FinMCP: why financial AI agents need their own benchmark
Your agent scores 87% on GAIA and 73% on WebArena. You deploy it to handle insurance underwriting queries. It fails at 40% of real tasks. The benchmarks told you something true. They just didn’t tell you the right thing.
This is the core problem with evaluating production agents on general benchmarks. They measure whether a model has broad capability. They don’t measure whether it will work in your specific environment, under your specific constraints, with your specific tools. The gap between those two things is where deployments go wrong.
FinMCP-Bench, published March 26, 2026 (arXiv:2603.24943), makes this concrete for financial agents. It’s the first benchmark built specifically for LLM agents using Model Context Protocol tools in real financial workflows. The numbers aren’t surprising so much as clarifying: they name the failure modes precisely, which is exactly what general benchmarks don’t do.
TL;DR: FinMCP-Bench tests LLM agents on 613 real financial tasks using 65 actual MCP tools. The best model scores 64% overall but collapses to a 7.4% exact match rate on multi-tool sequences. Generic benchmarks don’t surface this. For financial, legal, and medical agents, domain-specific evals aren’t optional — they’re the only signal that matters.
What generic benchmarks can’t measure
The best-scoring model on FinMCP-Bench achieves 64.27% overall. On the hardest task type, multi-tool sequences, exact match rates fall to 7.4% across all tested models. No general benchmark tracks this specific collapse. GAIA, WebArena, and most agent evals measure whether a model can navigate open-ended tasks, use web tools, and reason across multiple steps. What they don’t test is what happens when those steps must occur in a specific order mandated by regulation, not by logic.
In financial workflows, the order of tool calls is often legally prescribed. You fetch compliance eligibility before executing an allocation recommendation. You verify regulatory status before generating a prospectus summary. You resolve jurisdiction before applying tax treatment. A model that gets the right final answer but in the wrong sequence has produced a compliance violation, not a success.
Generic benchmarks also don’t test arithmetic precision under latency. Financial computations (NAV calculations, margin requirements, yield spreads) require exact numeric results. A model that rounds, approximates, or uses cached values from an earlier tool call introduces errors that compound across a multi-step workflow. Standard evals reward “approximately correct.” Financial production cannot.
The consequence is predictable: 88% of AI agents never reach production. For finance specifically, a 2026 survey found that incidents involving inaccurate or harmful AI outputs resulted in financial losses in 77% of cases. Agents that look capable on general benchmarks often aren’t capable at the thing you actually need them to do.
Related: Agent evaluation frameworks covers the broader evaluation landscape.
What FinMCP-Bench actually tests
The benchmark exposes exactly the failure modes that matter for financial deployment. FinMCP-Bench contains 613 samples across 10 main financial scenarios and 33 subcategories, built using 65 real MCP tools from production financial systems. The scenarios cover market analysis and research (141 samples), investment planning and allocation (101 samples), and a range of compliance, regulatory, and multi-source synthesis tasks.
Three task types capture different levels of complexity. Single-tool tasks (145 samples) require one tool call to answer a query. Multi-tool tasks (249 samples) require an average of 7.32 tool calls across 5.72 sequential steps, coordinating across data sources, compliance systems, and execution tools in a defined order. Multi-turn tasks (219 samples) test conversational coherence across 5.95 dialogue turns, where the agent must maintain context, track regulatory state, and update its reasoning as new information arrives.
The multi-tool category is where financial realism lives. Real investment workflows don’t call one API. They chain market data retrieval, portfolio constraint checks, compliance validation, and allocation execution, each step dependent on the last. A model that can pick the right tool for a single query but can’t sequence them correctly under constraint isn’t useful in production.
The benchmark also incorporates both real user queries from financial practitioners and synthetic queries designed to stress-test edge cases. That combination matters: synthetic queries alone don’t capture the ambiguity and domain vocabulary of real user requests, but real queries alone don’t systematically cover the failure modes you need to test.
How current agents score on FinMCP
Six models were evaluated, and the spread is wider than you’d expect from their general benchmark rankings. The top model, Qwen3-235B-A22B-Thinking, scores 64.27% overall Tool F1. Qwen3-30B-A3B-Thinking comes in at 55.58%, Qwen3-4B-Thinking at 50.08%, DeepSeek-R1 at 49.88%, Seed-OSS-36B at 39.34%, and GPT-OSS-20B at 32.62%.
That’s a 32-percentage-point spread between the best and worst models. On general benchmarks, that gap typically collapses. Models that differ by 30 points on domain-specific tasks often sit within 5-10 points of each other on MMLU or GAIA. Domain benchmarks reveal differentiation that general evals smooth over.
The multi-tool collapse is the most important finding. Across all six models, exact match rates on multi-tool sequences average 7.4%. Fewer than 8 in 100 attempts produce the correct tool sequence. The models can identify relevant tools. They just can’t reliably orchestrate them in the order that financial compliance demands.
Performance is also asymmetric across scenario types. Market analysis queries, which tend toward single-source lookups, see relatively higher scores. Multi-turn compliance conversations, where the agent must track evolving regulatory state across a dialogue, produce the lowest exact match rates of any task type. The models are better at retrieval than at stateful reasoning under constraint.
This doesn’t mean current models are useless for financial workflows. It means the failure modes are specific and testable. That’s exactly what a good benchmark gives you.
Extending the pattern: domain benchmarks for legal, medical, and infrastructure
FinMCP’s methodology is a template, not a one-off. The question it asks is worth asking in every domain that has real stakes: what does failure actually look like here, and can our benchmark catch it?
In legal, the answer is different from finance but the gap is similar. LegalBench, the crowd-sourced legal reasoning benchmark, puts the top model (Gemini 3 Pro) at 87% accuracy. That measures general legal reasoning. It doesn’t tell you whether an agent can draft a compliant NDA under Delaware law using a law firm’s specific document management tools, in the right format, with correct citations. LegalBenchmarks.ai is building toward that, but the field is early. The failure modes in legal are jurisdiction specificity and output format compliance, things that look fine in a generic eval and surface immediately in a real client engagement.
Medical is closer to where finance is now. Stanford’s MedAgentBench evaluates agents on clinical tasks inside a real EHR: ordering medications, retrieving patient data, coordinating care decisions. The insight is the same one FinMCP demonstrates: performing well on USMLE-style knowledge questions doesn’t predict whether a model can execute clinical workflows correctly inside a real system with real data and real consequence for error. Knowledge and workflow fidelity are different things.
Infrastructure agents face a problem that’s less about compliance and more about idempotency. An agent that calls a deployment API twice because it didn’t correctly track whether the first call succeeded creates a real incident. Generic benchmarks don’t test for double-execution. Neither do most internal eval suites, until something breaks in production.
Here’s the template FinMCP implicitly demonstrates for building a domain benchmark:
flowchart TD
A[Identify domain failure modes] --> B[Collect real tools from production]
B --> C[Build samples at 3 complexity levels]
C --> D1[Single-step: tool selection accuracy]
C --> D2[Multi-step: ordering under constraint]
C --> D3[Multi-turn: stateful coherence]
D1 --> E[Measure both Tool F1 and Exact Match Rate]
D2 --> E
D3 --> E
E --> F[Find the models that don't collapse on your hard cases]
The critical step is identifying the failure modes first. What does your agent do that, if wrong, produces a compliance violation, a safety incident, or a regulatory penalty? Start there. Everything else flows from it.
Score gap visualization: The following illustrates the kind of divergence between generic and domain-specific scores that FinMCP makes visible.
Model | Generic eval (est.) | FinMCP overall | Multi-tool EMR
Qwen3-235B | ~80% | 64.27% | ~7-8%
DeepSeek-R1 | ~78% | 49.88% | ~5-6%
GPT-OSS-20B | ~75% | 32.62% | ~3-4%
Generic eval scores are illustrative; FinMCP scores are from arXiv:2603.24943.
The point isn’t which specific model performs best. It’s that a model appearing near-competitive on general evals can be half as capable in a domain-specific context. The multi-tool collapse makes the gap even wider.
Building your domain benchmark: practical starting point
If you’re deploying agents in any regulated domain, here’s a concrete starting point based on FinMCP’s structure:
Step 1. List the five things your agent must never get wrong. These are your hard case seeds, the failure modes that matter in production.
Step 2. Collect the real tools your agent uses. Don’t simulate them. FinMCP uses 65 real financial MCPs because synthetic tool behavior doesn’t capture the error states and response formats that production tools produce.
Step 3. Build samples at three levels: single-tool (test tool selection), multi-tool (test sequencing under constraint), and multi-turn (test stateful coherence). Aim for a roughly 25/40/35 split, similar to FinMCP’s distribution.
Step 4. Measure two things separately: Tool F1 (does the model call the right tools?) and Exact Match Rate (does it call them in the right order?). The gap between those two numbers tells you whether you have a selection problem or a sequencing problem.
Step 5. Run your benchmark before selecting a model for production, not after. The variation across models on domain-specific tasks is large enough to change your deployment decision.
Related: Benchmarking AI agents in depth and testing AI agents in production.
FAQ
What is FinMCP-Bench? FinMCP-Bench (arXiv:2603.24943) is the first benchmark for evaluating LLM agents on real-world financial tasks via the Model Context Protocol. It has 613 samples across 10 financial scenarios using 65 real MCP tools, covering single-tool, multi-tool, and multi-turn task types. Published March 26, 2026, by researchers from Alibaba Cloud’s Qwen DianJin Team, YINGMI Wealth Management, and Soochow University.
How do models perform on FinMCP vs general benchmarks? The best model (Qwen3-235B) scores 64.27% on FinMCP overall. On multi-tool sequences, exact match rates drop to 7.4% across all tested models. General benchmarks don’t expose this failure mode because they don’t test tool sequencing under regulatory constraint.
What financial tasks does FinMCP-Bench cover? Ten main categories including market analysis and research (141 samples), investment planning and allocation (101 samples), and 31 other subcategories spanning compliance, multi-source synthesis, and regulatory handling.
Why do agents collapse on multi-tool financial sequences? Financial workflows require tool calls in a specific order prescribed by compliance requirements. Agents that get the right answer but in the wrong sequence have produced a compliance violation, not a success. Generic benchmarks don’t test ordering under constraint — they reward approximate correctness.
How do you build a domain-specific benchmark for other fields? FinMCP’s method: identify the irreducible failure modes in your domain first, collect real tools from production environments, build samples at three complexity levels (single-step, multi-step, multi-turn), measure tool selection and execution order separately. Legal and medical domains need the same approach — the pattern is transferable.
Want to work together?
I take on projects, advisory roles, and fractional CTO engagements in AI/ML. I also help businesses go AI-native with agentic workflows and agent orchestration.
Get in touch