Asked & Cited

Market map

The AI agent testing and evaluation market map, Q4 2026: 17 companies by approach

Which tools test AI agents before release, and which ones watch them after, sorted by how they actually work.

Pick by approach, not by brand. Eval and tracing platforms (Braintrust, LangSmith, Langfuse, Arize) fit teams that need one place to score and compare agent runs. Code-first frameworks (DeepEval, Promptfoo, LangWatch) fit engineers who want tests in CI. Simulation tools (Coval, Cekura, Hamming, Okareo, Roark, Bluejay, Maxim) fit agents that talk to people, especially by phone. Most funded teams end up using one from the first group plus one from the third.

How we chose. A company is included if it (1) sells or openly publishes a product whose main job is testing, evaluating or simulating AI agents or LLM apps, (2) has a pricing or product page we could fetch and read on 17 November 2026, and (3) is named by at least one of the buyer guides we read or appears in public sources as an active vendor. The map is grouped by approach, not ranked. We do not accept payment for inclusion or position, and sponsorship never changes the list. Read the editorial policy. Vendor facts come from the linked vendor pages; where we could not verify something, we say so.

What we measured: who the top buyer guides name

We ran four buyer phrasings through search, read the full text of the top four pages for each (16 pages), and counted which vendors each page names. The phrasings were "best AI agent testing and evaluation tools", "AI agent evaluation platforms compared", "LLM evaluation tools for startups" and "best voice AI agent testing tools". Counts are pages of 16 that mention the vendor at least once.

Comparison table

Last column of counts is the share of the 16 pages that name the vendor. Prices were read from each vendor page on 17 November 2026 and change often.

VendorBest forPricing modelNotable strengthNotable limitNamed onSource
BraintrustTeams that want every engineer, PM and reviewer in one eval workflowFree Starter; Pro $249/month; Enterprise customUnlimited users on every plan, including freeUsage meters (data and scores) and a $249 step to Pro15/16Source
LangSmithTeams building agents on LangChain or LangGraphFree Developer; Plus $39/seat/month; Enterprise customTracing, evals and agent deployment from the LangChain teamSelf-hosted and hybrid deployment need Enterprise13/16Source
LangfuseTeams that want open-source tracing and evals they can self-hostFree Hobby; Core $29/month; Pro $199/month; EnterpriseSelf-hostable, usage-based pricing with unlimited usersEvaluation is a feature of an observability-first product, so you build more of the test loop yourself14/16Source
Arize (AX and Phoenix)Teams that want OpenTelemetry-based tracing, online evals and a self-hosted optionAX Free; AX Pro $50/month; Enterprise custom; Phoenix free self-hostedOnline and offline evals, trajectory and session evals, unlimited users on all AX plansSelf-hosted AX is Enterprise only; Phoenix is source-available, not permissively open source11/16Source
Splunk Agent Observability (formerly Galileo)Enterprises already on Splunk or Cisco that want evals plus runtime guardrailsNot listed on pages we fetched; talk to salesEvaluation plus real-time guardrails across all trafficNow a module of a large observability suite, so it is a heavier purchase for a small team8/16Source
Opik (Comet)Cost-sensitive teams that want an open-source stack with a cheap hosted tierOpen source free; Cloud Free; Pro $19/month; Enterprise customOpen-source codebase that matches the hosted product, low entry priceCompliance and flexible deployment sit in Enterprise4/16Source
Confident AI and DeepEvalPython teams that want agent evals to run like unit tests in CIDeepEval open source; Confident AI Free; Starter $200/month; Team $2,000/monthPytest-style framework plus a hosted platform with chat simulationsFree tier is tight (5 test runs a week, 2 seats)11/16Source
PromptfooEngineers who want one config for quality checks and red teaming in CICommunity free (10k red-team probes/month); Enterprise customOpen-source CLI that does evals and red teaming togetherTeam sharing, continuous monitoring and SSO need Enterprise6/16Source
LangWatchTeams that want open-source simulated users in pytest or vitest, text and voiceFree Developer; Growth €29/core seat/month plus events; Enterprise customOpen-source Scenario SDK, self-hostable, voice testing includedEvent-based billing needs modelling; free plan caps at 3 scenarios6/16Source
Maxim AIProduct and engineering teams that want simulation, evals and observability in one placeSeat-based per third-party reports; check vendorText and voice simulation alongside evals and production tracingWe could not confirm current platform prices from Maxim's own pages8/16Source
CovalVoice-agent teams that need simulation in CI with compliance paperworkStarter $100/month; Growth $500/month; Enterprise from $4,500/monthSimulation, monitoring and human review queues; SOC 2 Type II, HIPAA, GDPR on every planNo free tier; the trial asks for a card6/16Source
CekuraVoice and chat agent teams that want pay-as-you-go testing$0.25/voice test minute; $0.05/monitored call; Startup $500/monthUsage pricing with 300 free credits (about 60 minutes) and no cardRed teaming, load testing and regression suites are Enterprise services5/16Source
HammingVoice teams in regulated settings that want red teaming, simulation and call monitoringNot published on the page we fetched; contact salesRed teaming, simulation and live-call monitoring in one productNo public pricing found5/16Source
BluejayTeams that want simulated users plus replay of real production callsFree start and demo on request; plan prices not on the page we fetchedDigital Humans across voice, chat and text; replay of production callsPricing details not published on the page we fetched5/16Source
OkareoTeams that need audio-native checks and synthetic production monitoringPro $199/month; Team $799/month; Enterprise custom; 14-day free trialEvaluates the audio itself, with audio, transcript and trace on one timelineIncluded simulation volumes are modest on Pro (125 simulations, 250 voice minutes)1/16Source
RoarkSmall voice teams that want pure usage billing and a free starter creditPay-as-you-go $0.15/sim minute; Team $500/month; Enterprise from $4,000/monthNo seat licences; $50 free credit for work emailsTeam commitment starts at $500; provider costs for placed calls are passed through3/16Source
PluraiTeams that want testing tied to a fix-and-retrain loopNot listed on the page we fetched; demo on requestFrames testing as a loop: simulate, fix, post-train, repeatEarly-stage positioning; little public pricing or documentation to compare0/16Source

Eval and tracing platforms

Record what the agent did, score it, and compare versions. Best when you need one system for development and production.

Braintrust

A hosted platform for tracing, offline experiments, scoring and human review. Its pricing page lists a $0 Starter (1 GB processed data, 10k scores, 14-day retention, unlimited users, projects and experiments), a $249 per month Pro plan, and custom Enterprise with on-prem or hosted deployment. It also advertises 6 to 12 months free for qualifying startups. Trade-off: scores and processed data are metered, so evaluation-heavy teams should model their own volume before comparing it with seat-priced tools. Source.

LangSmith

LangChain's commercial platform for tracing and evaluating agents. The Developer plan is free for one seat with up to 5k base traces a month. Plus is $39 per seat per month with 10k base traces, then pay as you go. Its pricing page also shows a $300M Series C at a $3B valuation for LangChain and credits of up to $10,000 for VC-backed startups. Trade-off: per-seat pricing grows with team size, and the self-hosted option sits behind an Enterprise contract. Source.

Langfuse

An open-source LLM engineering platform with tracing, datasets, experiments and evaluation, offered as Langfuse Cloud (US, EU or Japan data regions) or self-hosted. Hobby is free with 50k units a month. Core is $29 per month with 100k units and $8 per extra 100k. Pro is $199 per month. All listed tiers include unlimited users. Trade-off: it is strongest as the tracing backbone; simulated-user testing is not its centre of gravity. Source.

Arize (AX and Phoenix)

Two products: Phoenix, which you run yourself, and AX, the hosted platform. AX Free includes 25k spans a month and 15-day retention. AX Pro is $50 per month with 50k spans and 30 days. Its pricing grid lists trace-level evals for agent trajectories, session-level evals for multi-turn conversations, and agent-as-a-judge. Arize's own comparison page describes Phoenix's self-hosting as under the Elastic License 2.0. Trade-off: small span allowances mean busy agents move past the Pro tier quickly. Source.

Splunk Agent Observability (formerly Galileo)

Splunk's acquisition page says Cisco has completed its acquisition of Galileo, which is now Splunk Agent Observability. It describes the product as evaluating and improving agent behavior, observing performance, tracking usage and cost, and guardrailing actions in real time across 100% of traffic. Buyers who knew Galileo as an independent startup should evaluate it as part of the Splunk portfolio. Trade-off: we found no public price list on the Splunk pages we fetched. Source.

Opik (Comet)

Comet's open-source platform for agent tracing, test suites and assertions. The pricing page lists a free open-source option, a Free Cloud plan (up to 10 team members, 25k spans a month, 60-day retention), and Pro Cloud at $19 per month for 100k spans and up to 50 team members. Its product page cites 30+ evaluation metrics and guardrails. Trade-off: SOC 2, ISO and HIPAA compliance are listed only under Enterprise. Source.

Code-first test frameworks

Tests live in your repo and run in CI. Best when engineering owns quality and you want assertions, not dashboards.

Confident AI and DeepEval

DeepEval is the open-source framework; Confident AI is the hosted platform. The DeepEval quickstart runs evals with deepeval test run and can evaluate an agent's full trajectory with tracing. Confident AI's Free plan has 2 seats, 1 project and 5 test runs a week. Starter is $200 per month with unlimited seats and chat simulations. Team is $2,000 per month. It offers startup grants to companies that have raised under $5M. Trade-off: the jump from $200 to $2,000 is steep, and RBAC and SSO sit at the higher tier. Source.

Promptfoo

An open-source CLI and library for evaluating and red-teaming LLM apps, usable in CI/CD. Its Community plan is free forever and includes all evaluation features and 10k red-team probes a month; Enterprise adds team sharing, continuous monitoring and SSO, and an on-premise option exists. Trade-off: it is a test harness, not a simulated-caller product, so multi-turn voice testing needs another tool. Source.

LangWatch

An open-source platform for LLM evaluation, observability and agent testing. Its Scenario SDK (Python and TypeScript) runs a simulated user and a judge in your test runner. The Developer plan is free with 50k events a month, 2 users and 3 scenarios. Growth is €29 per core seat per month with 200k events included, then €5 per 100k. Its pricing page lists voice agent testing with latency metrics and noise and interruption injection. Trade-off: every LLM call, tool call and simulation step counts as an event, so costs track volume. Source.

Simulation and voice QA

Simulated callers or users hold full conversations with your agent. Best when the failures happen across many turns or on a phone line.

Maxim AI

Maxim's product page describes a simulation and evaluation platform for agents. Its pricing page, as fetched on 17 November 2026, lists pricing for Bifrost, its open-source AI gateway, rather than for the evaluation platform. Arize's November 2026 comparison reports platform plans of free (3 seats), $29 and $49 per seat per month, with 3, 7 and 30 days of retention; confirm with the vendor. Trade-off: short retention on lower tiers was the main limit named in that comparison. Source.

Coval

A voice and chat agent testing platform. Starter is $100 per month for 100 simulation minutes, Growth is $500 for 1,000, with overage at $0.40 and $0.25 per minute. Enterprise starts at $4,500 per month. It offers a CLI, API and MCP server and a 7-day trial. Its homepage lists Chime, Perplexity, ServiceNow, StubHub, Zoom, Toast and Upstart as teams it works with. Trade-off: the entry point is a paid plan, and concurrency starts at 5 simulations. Source.

Cekura

Automated QA for voice and chat agents. Pay as you go is $0.25 per voice testing minute and $0.05 per monitored call, with one free seat. The Startup plan is $500 per month for about 2,000 testing minutes and 10 seats. Every plan includes unlimited Python metrics and an API and MCP server, and the site lists HIPAA, GDPR and SOC 2. Trade-off: per-minute costs scale linearly with test volume. Source.

Hamming

Hamming's homepage positions it as a security layer for voice and chat agents: RedSwarm for red teaming, GreenLoop for simulation, and BlueTrace for monitoring production calls. It says scenarios are generated from your prompt and real calls, scored on 50+ built-in metrics, with 50K+ concurrent calls in CI/CD. Trade-off: without a public price, small teams cannot self-qualify. Source.

Bluejay

Bluejay says it tests, monitors and improves voice and chat agents. Its homepage describes lifelike Digital Humans used to validate workflows, replay production calls and load test, with a Start for Free option. Trade-off: we could verify the feature set but not plan prices from its own pages. Source.

Okareo

Okareo's pricing page lists Pro at $199 per month (125 simulations, 250 voice minutes, 1,500 check evaluations) and Team at $799 per month (300 simulations, 750 voice minutes). Both include synthetic production monitoring, where synthetic users probe live agents. Its voice page says it evaluates audio, not only transcripts, covering latency, prosody and dead air. Trade-off: usage beyond included limits is billed per unit. Source.

Roark

Roark's pricing page bills simulation at $0.15 per minute pay-as-you-go, $0.10 on the $500 Team plan, and $0.05 on Enterprise (from $4,000 per month). Sign-up with a work email adds $50 of credit. Its simulation page lists red teaming, load tests, regression runs in CI and 45 languages. Trade-off: telephony and speech provider costs are passed through at cost on calls Roark places. Source.

Emerging approach

Newer entrants with a distinct angle.

Plurai

Plurai calls itself the recursive self-improvement layer for AI agents. Its homepage describes a loop of simulating, fixing and post-training agents, offers a Claude Code quickstart, and links to a 2026 Gartner market guide for AI evaluation and observability platforms. Trade-off: we could verify the stated approach but not pricing or customer results, so run a pilot before relying on it. Source.

How to choose

  1. If the agent answers phone calls, start with a voice simulator. Text-only tests cannot reproduce interruptions, accents, noise or turn-taking. Shortlist two of Coval, Cekura, Okareo, Roark, Hamming and LangWatch, and run the same ten scenarios through each.
  2. If your agent is built on LangChain or LangGraph, trial LangSmith first. Otherwise start with a framework-neutral option: Braintrust, Langfuse, Arize or Opik.
  3. If engineering owns quality and wants tests in the repo, start with DeepEval, Promptfoo or the LangWatch Scenario SDK. All three have free open-source entry points.
  4. If data cannot leave your network, filter for self-hosting first: Langfuse, Opik, Phoenix and LangWatch publish self-hosted options; LangSmith and AX self-hosting are Enterprise.
  5. If you are pre-seed, compare startup programs: Braintrust (6 to 12 months free for qualifying startups), LangSmith (up to $10,000 credits for VC-backed startups) and Confident AI (grants under $5M raised).
  6. Model your volume before comparing prices. The units differ: seats (LangSmith), scores and data (Braintrust), spans (Arize, Opik), events (LangWatch), minutes (Coval, Cekura, Roark). Price the same 100,000 agent runs in each.
  7. Gate a release on it. Whatever you pick, confirm it can fail a pull request in CI. A tool that only shows a dashboard is monitoring, not testing.

FAQ

What is the difference between agent testing and agent observability?

Observability records what the agent did: traces, tool calls, latency and cost. Evaluation scores whether that behavior was correct. Testing runs the agent against scenarios before release. Most platforms here cover more than one, but their starting point differs, as the groups above show.

Do I need a simulation tool, or are evals enough?

Evals on fixed examples catch regressions on known cases. Simulation matters when quality depends on a multi-turn conversation or an audio channel, because a simulated caller reaches paths you did not script.

What is the cheapest way to start?

Open-source frameworks (DeepEval, Promptfoo, LangWatch Scenario) and free tiers at Braintrust, Langfuse, Arize and Opik cost nothing to begin. You will still pay for the LLM judge calls your tests make.

Which tools work for voice agents?

Coval, Cekura, Hamming, Bluejay, Okareo and Roark are built around voice, and LangWatch lists voice agent testing on its pricing page. Maxim says its platform simulates voice too; confirm channel support in a trial.

How was this list built?

By the criteria in the box above, using vendor pages fetched on 17 November 2026. No company paid for inclusion or placement.

Is your company missing from pages like this?

Funded vendors in this category often appear on none of the pages buyers read. That is measurable and fixable. See our GEO services, or start with an AI visibility audit. For the wider data, read who AI names first across 16 categories and how we measure AI visibility.

Sources (fetched 17 November 2026)
  1. Braintrust: https://www.braintrust.dev/pricing
  2. LangSmith: https://www.langchain.com/pricing
  3. Langfuse: https://langfuse.com/pricing
  4. Arize (AX and Phoenix): https://arize.com/pricing/
  5. Splunk Agent Observability (formerly Galileo): https://www.splunk.com/en_us/about-splunk/acquisitions/galileo.html
  6. Opik (Comet): https://www.comet.com/site/pricing/
  7. Confident AI and DeepEval: https://www.confident-ai.com/pricing
  8. Promptfoo: https://www.promptfoo.dev/pricing/
  9. LangWatch: https://langwatch.ai/pricing
  10. Maxim AI: https://www.getmaxim.ai/evals/products/agent-simulation-evaluation
  11. Coval: https://www.coval.ai/pricing
  12. Cekura: https://www.cekura.ai/pricing
  13. Hamming: https://hamming.ai
  14. Bluejay: https://getbluejay.ai
  15. Okareo: https://okareo.com/pricing
  16. Roark: https://roark.ai/pricing
  17. Plurai: https://www.plurai.ai/
  18. DeepEval quickstart: https://deepeval.com/docs/getting-started
  19. Promptfoo docs: https://www.promptfoo.dev/docs/intro/
  20. Splunk Agent Observability: https://www.splunk.com/en_us/products/agent-observability.html
  21. Arize, 7 best AI agent evaluation platforms (Phoenix license; Maxim plans): https://arize.com/resources/llm-and-agent-evaluation-platforms
  22. Arize, best AI observability tools for agents: https://arize.com/blog/best-ai-observability-tools-for-autonomous-agents-in-2026
  23. Buyer guide read for counts: CIO: https://www.cio.com/article/4207550/tools-for-evaluating-and-benchmarking-agentic-ai-systems.html
  24. Buyer guide read for counts: AgentOps Guide: https://agentopsguide.com/ai-agent-testing-tools/
  25. Buyer guide read for counts: Toolradar: https://toolradar.com/guides/best-ai-agent-testing-tools
  26. Buyer guide read for counts: LangWatch voice: https://langwatch.ai/blog/best-voice-agent-testing-platforms
  27. Buyer guide read for counts: Micdrop voice: https://micdrop.dev/blog/top-voice-agent-testing-tools

Where do you stand?

Send your website and the one buyer question you most want to be the answer to. We will send back a free snapshot within two business days.

Get a free visibility snapshot Book a call