Market map
The AI agent testing and evaluation market map, Q4 2026: 17 companies by approach
Which tools test AI agents before release, and which ones watch them after, sorted by how they actually work.
Pick by approach, not by brand. Eval and tracing platforms (Braintrust, LangSmith, Langfuse, Arize) fit teams that need one place to score and compare agent runs. Code-first frameworks (DeepEval, Promptfoo, LangWatch) fit engineers who want tests in CI. Simulation tools (Coval, Cekura, Hamming, Okareo, Roark, Bluejay, Maxim) fit agents that talk to people, especially by phone. Most funded teams end up using one from the first group plus one from the third.
What we measured: who the top buyer guides name
We ran four buyer phrasings through search, read the full text of the top four pages for each (16 pages), and counted which vendors each page names. The phrasings were "best AI agent testing and evaluation tools", "AI agent evaluation platforms compared", "LLM evaluation tools for startups" and "best voice AI agent testing tools". Counts are pages of 16 that mention the vendor at least once.
- The generalists dominate. Braintrust was named on 15 of 16 pages, Langfuse on 14, LangSmith on 13, and Arize and DeepEval on 11 each.
- Voice specialists are named only on voice queries. Coval appeared on 6 of 16, Cekura, Hamming and Bluejay on 5 each, Roark on 3, Okareo on 1. Most of those mentions came from the voice-testing phrasing.
- Plurai, a funded newcomer, appeared on none of the 16 pages.
- A bias to know about: many of the pages we read are published by vendors in the category (Arize, Braintrust, Confident AI, Pydantic, LangWatch, Prefactor, Feather, Regal, Micdrop). Vendor-written lists tend to name the author first. That is why we checked every entry against the vendor's own pages rather than the roundups.
Comparison table
Last column of counts is the share of the 16 pages that name the vendor. Prices were read from each vendor page on 17 November 2026 and change often.
| Vendor | Best for | Pricing model | Notable strength | Notable limit | Named on | Source |
|---|---|---|---|---|---|---|
| Braintrust | Teams that want every engineer, PM and reviewer in one eval workflow | Free Starter; Pro $249/month; Enterprise custom | Unlimited users on every plan, including free | Usage meters (data and scores) and a $249 step to Pro | 15/16 | Source |
| LangSmith | Teams building agents on LangChain or LangGraph | Free Developer; Plus $39/seat/month; Enterprise custom | Tracing, evals and agent deployment from the LangChain team | Self-hosted and hybrid deployment need Enterprise | 13/16 | Source |
| Langfuse | Teams that want open-source tracing and evals they can self-host | Free Hobby; Core $29/month; Pro $199/month; Enterprise | Self-hostable, usage-based pricing with unlimited users | Evaluation is a feature of an observability-first product, so you build more of the test loop yourself | 14/16 | Source |
| Arize (AX and Phoenix) | Teams that want OpenTelemetry-based tracing, online evals and a self-hosted option | AX Free; AX Pro $50/month; Enterprise custom; Phoenix free self-hosted | Online and offline evals, trajectory and session evals, unlimited users on all AX plans | Self-hosted AX is Enterprise only; Phoenix is source-available, not permissively open source | 11/16 | Source |
| Splunk Agent Observability (formerly Galileo) | Enterprises already on Splunk or Cisco that want evals plus runtime guardrails | Not listed on pages we fetched; talk to sales | Evaluation plus real-time guardrails across all traffic | Now a module of a large observability suite, so it is a heavier purchase for a small team | 8/16 | Source |
| Opik (Comet) | Cost-sensitive teams that want an open-source stack with a cheap hosted tier | Open source free; Cloud Free; Pro $19/month; Enterprise custom | Open-source codebase that matches the hosted product, low entry price | Compliance and flexible deployment sit in Enterprise | 4/16 | Source |
| Confident AI and DeepEval | Python teams that want agent evals to run like unit tests in CI | DeepEval open source; Confident AI Free; Starter $200/month; Team $2,000/month | Pytest-style framework plus a hosted platform with chat simulations | Free tier is tight (5 test runs a week, 2 seats) | 11/16 | Source |
| Promptfoo | Engineers who want one config for quality checks and red teaming in CI | Community free (10k red-team probes/month); Enterprise custom | Open-source CLI that does evals and red teaming together | Team sharing, continuous monitoring and SSO need Enterprise | 6/16 | Source |
| LangWatch | Teams that want open-source simulated users in pytest or vitest, text and voice | Free Developer; Growth €29/core seat/month plus events; Enterprise custom | Open-source Scenario SDK, self-hostable, voice testing included | Event-based billing needs modelling; free plan caps at 3 scenarios | 6/16 | Source |
| Maxim AI | Product and engineering teams that want simulation, evals and observability in one place | Seat-based per third-party reports; check vendor | Text and voice simulation alongside evals and production tracing | We could not confirm current platform prices from Maxim's own pages | 8/16 | Source |
| Coval | Voice-agent teams that need simulation in CI with compliance paperwork | Starter $100/month; Growth $500/month; Enterprise from $4,500/month | Simulation, monitoring and human review queues; SOC 2 Type II, HIPAA, GDPR on every plan | No free tier; the trial asks for a card | 6/16 | Source |
| Cekura | Voice and chat agent teams that want pay-as-you-go testing | $0.25/voice test minute; $0.05/monitored call; Startup $500/month | Usage pricing with 300 free credits (about 60 minutes) and no card | Red teaming, load testing and regression suites are Enterprise services | 5/16 | Source |
| Hamming | Voice teams in regulated settings that want red teaming, simulation and call monitoring | Not published on the page we fetched; contact sales | Red teaming, simulation and live-call monitoring in one product | No public pricing found | 5/16 | Source |
| Bluejay | Teams that want simulated users plus replay of real production calls | Free start and demo on request; plan prices not on the page we fetched | Digital Humans across voice, chat and text; replay of production calls | Pricing details not published on the page we fetched | 5/16 | Source |
| Okareo | Teams that need audio-native checks and synthetic production monitoring | Pro $199/month; Team $799/month; Enterprise custom; 14-day free trial | Evaluates the audio itself, with audio, transcript and trace on one timeline | Included simulation volumes are modest on Pro (125 simulations, 250 voice minutes) | 1/16 | Source |
| Roark | Small voice teams that want pure usage billing and a free starter credit | Pay-as-you-go $0.15/sim minute; Team $500/month; Enterprise from $4,000/month | No seat licences; $50 free credit for work emails | Team commitment starts at $500; provider costs for placed calls are passed through | 3/16 | Source |
| Plurai | Teams that want testing tied to a fix-and-retrain loop | Not listed on the page we fetched; demo on request | Frames testing as a loop: simulate, fix, post-train, repeat | Early-stage positioning; little public pricing or documentation to compare | 0/16 | Source |
Eval and tracing platforms
Record what the agent did, score it, and compare versions. Best when you need one system for development and production.
Braintrust
A hosted platform for tracing, offline experiments, scoring and human review. Its pricing page lists a $0 Starter (1 GB processed data, 10k scores, 14-day retention, unlimited users, projects and experiments), a $249 per month Pro plan, and custom Enterprise with on-prem or hosted deployment. It also advertises 6 to 12 months free for qualifying startups. Trade-off: scores and processed data are metered, so evaluation-heavy teams should model their own volume before comparing it with seat-priced tools. Source.
LangSmith
LangChain's commercial platform for tracing and evaluating agents. The Developer plan is free for one seat with up to 5k base traces a month. Plus is $39 per seat per month with 10k base traces, then pay as you go. Its pricing page also shows a $300M Series C at a $3B valuation for LangChain and credits of up to $10,000 for VC-backed startups. Trade-off: per-seat pricing grows with team size, and the self-hosted option sits behind an Enterprise contract. Source.
Langfuse
An open-source LLM engineering platform with tracing, datasets, experiments and evaluation, offered as Langfuse Cloud (US, EU or Japan data regions) or self-hosted. Hobby is free with 50k units a month. Core is $29 per month with 100k units and $8 per extra 100k. Pro is $199 per month. All listed tiers include unlimited users. Trade-off: it is strongest as the tracing backbone; simulated-user testing is not its centre of gravity. Source.
Arize (AX and Phoenix)
Two products: Phoenix, which you run yourself, and AX, the hosted platform. AX Free includes 25k spans a month and 15-day retention. AX Pro is $50 per month with 50k spans and 30 days. Its pricing grid lists trace-level evals for agent trajectories, session-level evals for multi-turn conversations, and agent-as-a-judge. Arize's own comparison page describes Phoenix's self-hosting as under the Elastic License 2.0. Trade-off: small span allowances mean busy agents move past the Pro tier quickly. Source.
Splunk Agent Observability (formerly Galileo)
Splunk's acquisition page says Cisco has completed its acquisition of Galileo, which is now Splunk Agent Observability. It describes the product as evaluating and improving agent behavior, observing performance, tracking usage and cost, and guardrailing actions in real time across 100% of traffic. Buyers who knew Galileo as an independent startup should evaluate it as part of the Splunk portfolio. Trade-off: we found no public price list on the Splunk pages we fetched. Source.
Opik (Comet)
Comet's open-source platform for agent tracing, test suites and assertions. The pricing page lists a free open-source option, a Free Cloud plan (up to 10 team members, 25k spans a month, 60-day retention), and Pro Cloud at $19 per month for 100k spans and up to 50 team members. Its product page cites 30+ evaluation metrics and guardrails. Trade-off: SOC 2, ISO and HIPAA compliance are listed only under Enterprise. Source.
Code-first test frameworks
Tests live in your repo and run in CI. Best when engineering owns quality and you want assertions, not dashboards.
Confident AI and DeepEval
DeepEval is the open-source framework; Confident AI is the hosted platform. The DeepEval quickstart runs evals with deepeval test run and can evaluate an agent's full trajectory with tracing. Confident AI's Free plan has 2 seats, 1 project and 5 test runs a week. Starter is $200 per month with unlimited seats and chat simulations. Team is $2,000 per month. It offers startup grants to companies that have raised under $5M. Trade-off: the jump from $200 to $2,000 is steep, and RBAC and SSO sit at the higher tier. Source.
Promptfoo
An open-source CLI and library for evaluating and red-teaming LLM apps, usable in CI/CD. Its Community plan is free forever and includes all evaluation features and 10k red-team probes a month; Enterprise adds team sharing, continuous monitoring and SSO, and an on-premise option exists. Trade-off: it is a test harness, not a simulated-caller product, so multi-turn voice testing needs another tool. Source.
LangWatch
An open-source platform for LLM evaluation, observability and agent testing. Its Scenario SDK (Python and TypeScript) runs a simulated user and a judge in your test runner. The Developer plan is free with 50k events a month, 2 users and 3 scenarios. Growth is €29 per core seat per month with 200k events included, then €5 per 100k. Its pricing page lists voice agent testing with latency metrics and noise and interruption injection. Trade-off: every LLM call, tool call and simulation step counts as an event, so costs track volume. Source.
Simulation and voice QA
Simulated callers or users hold full conversations with your agent. Best when the failures happen across many turns or on a phone line.
Maxim AI
Maxim's product page describes a simulation and evaluation platform for agents. Its pricing page, as fetched on 17 November 2026, lists pricing for Bifrost, its open-source AI gateway, rather than for the evaluation platform. Arize's November 2026 comparison reports platform plans of free (3 seats), $29 and $49 per seat per month, with 3, 7 and 30 days of retention; confirm with the vendor. Trade-off: short retention on lower tiers was the main limit named in that comparison. Source.
Coval
A voice and chat agent testing platform. Starter is $100 per month for 100 simulation minutes, Growth is $500 for 1,000, with overage at $0.40 and $0.25 per minute. Enterprise starts at $4,500 per month. It offers a CLI, API and MCP server and a 7-day trial. Its homepage lists Chime, Perplexity, ServiceNow, StubHub, Zoom, Toast and Upstart as teams it works with. Trade-off: the entry point is a paid plan, and concurrency starts at 5 simulations. Source.
Cekura
Automated QA for voice and chat agents. Pay as you go is $0.25 per voice testing minute and $0.05 per monitored call, with one free seat. The Startup plan is $500 per month for about 2,000 testing minutes and 10 seats. Every plan includes unlimited Python metrics and an API and MCP server, and the site lists HIPAA, GDPR and SOC 2. Trade-off: per-minute costs scale linearly with test volume. Source.
Hamming
Hamming's homepage positions it as a security layer for voice and chat agents: RedSwarm for red teaming, GreenLoop for simulation, and BlueTrace for monitoring production calls. It says scenarios are generated from your prompt and real calls, scored on 50+ built-in metrics, with 50K+ concurrent calls in CI/CD. Trade-off: without a public price, small teams cannot self-qualify. Source.
Bluejay
Bluejay says it tests, monitors and improves voice and chat agents. Its homepage describes lifelike Digital Humans used to validate workflows, replay production calls and load test, with a Start for Free option. Trade-off: we could verify the feature set but not plan prices from its own pages. Source.
Okareo
Okareo's pricing page lists Pro at $199 per month (125 simulations, 250 voice minutes, 1,500 check evaluations) and Team at $799 per month (300 simulations, 750 voice minutes). Both include synthetic production monitoring, where synthetic users probe live agents. Its voice page says it evaluates audio, not only transcripts, covering latency, prosody and dead air. Trade-off: usage beyond included limits is billed per unit. Source.
Roark
Roark's pricing page bills simulation at $0.15 per minute pay-as-you-go, $0.10 on the $500 Team plan, and $0.05 on Enterprise (from $4,000 per month). Sign-up with a work email adds $50 of credit. Its simulation page lists red teaming, load tests, regression runs in CI and 45 languages. Trade-off: telephony and speech provider costs are passed through at cost on calls Roark places. Source.
Emerging approach
Newer entrants with a distinct angle.
Plurai
Plurai calls itself the recursive self-improvement layer for AI agents. Its homepage describes a loop of simulating, fixing and post-training agents, offers a Claude Code quickstart, and links to a 2026 Gartner market guide for AI evaluation and observability platforms. Trade-off: we could verify the stated approach but not pricing or customer results, so run a pilot before relying on it. Source.
How to choose
- If the agent answers phone calls, start with a voice simulator. Text-only tests cannot reproduce interruptions, accents, noise or turn-taking. Shortlist two of Coval, Cekura, Okareo, Roark, Hamming and LangWatch, and run the same ten scenarios through each.
- If your agent is built on LangChain or LangGraph, trial LangSmith first. Otherwise start with a framework-neutral option: Braintrust, Langfuse, Arize or Opik.
- If engineering owns quality and wants tests in the repo, start with DeepEval, Promptfoo or the LangWatch Scenario SDK. All three have free open-source entry points.
- If data cannot leave your network, filter for self-hosting first: Langfuse, Opik, Phoenix and LangWatch publish self-hosted options; LangSmith and AX self-hosting are Enterprise.
- If you are pre-seed, compare startup programs: Braintrust (6 to 12 months free for qualifying startups), LangSmith (up to $10,000 credits for VC-backed startups) and Confident AI (grants under $5M raised).
- Model your volume before comparing prices. The units differ: seats (LangSmith), scores and data (Braintrust), spans (Arize, Opik), events (LangWatch), minutes (Coval, Cekura, Roark). Price the same 100,000 agent runs in each.
- Gate a release on it. Whatever you pick, confirm it can fail a pull request in CI. A tool that only shows a dashboard is monitoring, not testing.
FAQ
What is the difference between agent testing and agent observability?
Observability records what the agent did: traces, tool calls, latency and cost. Evaluation scores whether that behavior was correct. Testing runs the agent against scenarios before release. Most platforms here cover more than one, but their starting point differs, as the groups above show.
Do I need a simulation tool, or are evals enough?
Evals on fixed examples catch regressions on known cases. Simulation matters when quality depends on a multi-turn conversation or an audio channel, because a simulated caller reaches paths you did not script.
What is the cheapest way to start?
Open-source frameworks (DeepEval, Promptfoo, LangWatch Scenario) and free tiers at Braintrust, Langfuse, Arize and Opik cost nothing to begin. You will still pay for the LLM judge calls your tests make.
Which tools work for voice agents?
Coval, Cekura, Hamming, Bluejay, Okareo and Roark are built around voice, and LangWatch lists voice agent testing on its pricing page. Maxim says its platform simulates voice too; confirm channel support in a trial.
How was this list built?
By the criteria in the box above, using vendor pages fetched on 17 November 2026. No company paid for inclusion or placement.
Is your company missing from pages like this?
Funded vendors in this category often appear on none of the pages buyers read. That is measurable and fixable. See our GEO services, or start with an AI visibility audit. For the wider data, read who AI names first across 16 categories and how we measure AI visibility.
- Braintrust: https://www.braintrust.dev/pricing
- LangSmith: https://www.langchain.com/pricing
- Langfuse: https://langfuse.com/pricing
- Arize (AX and Phoenix): https://arize.com/pricing/
- Splunk Agent Observability (formerly Galileo): https://www.splunk.com/en_us/about-splunk/acquisitions/galileo.html
- Opik (Comet): https://www.comet.com/site/pricing/
- Confident AI and DeepEval: https://www.confident-ai.com/pricing
- Promptfoo: https://www.promptfoo.dev/pricing/
- LangWatch: https://langwatch.ai/pricing
- Maxim AI: https://www.getmaxim.ai/evals/products/agent-simulation-evaluation
- Coval: https://www.coval.ai/pricing
- Cekura: https://www.cekura.ai/pricing
- Hamming: https://hamming.ai
- Bluejay: https://getbluejay.ai
- Okareo: https://okareo.com/pricing
- Roark: https://roark.ai/pricing
- Plurai: https://www.plurai.ai/
- DeepEval quickstart: https://deepeval.com/docs/getting-started
- Promptfoo docs: https://www.promptfoo.dev/docs/intro/
- Splunk Agent Observability: https://www.splunk.com/en_us/products/agent-observability.html
- Arize, 7 best AI agent evaluation platforms (Phoenix license; Maxim plans): https://arize.com/resources/llm-and-agent-evaluation-platforms
- Arize, best AI observability tools for agents: https://arize.com/blog/best-ai-observability-tools-for-autonomous-agents-in-2026
- Buyer guide read for counts: CIO: https://www.cio.com/article/4207550/tools-for-evaluating-and-benchmarking-agentic-ai-systems.html
- Buyer guide read for counts: AgentOps Guide: https://agentopsguide.com/ai-agent-testing-tools/
- Buyer guide read for counts: Toolradar: https://toolradar.com/guides/best-ai-agent-testing-tools
- Buyer guide read for counts: LangWatch voice: https://langwatch.ai/blog/best-voice-agent-testing-platforms
- Buyer guide read for counts: Micdrop voice: https://micdrop.dev/blog/top-voice-agent-testing-tools
Where do you stand?
Send your website and the one buyer question you most want to be the answer to. We will send back a free snapshot within two business days.
Get a free visibility snapshot Book a call