Tool Overload Evaluations
6 min read
This page is the central hub for Unique's research on tool overload: what it is, when it shows up in client deployments, and which architecture choices reduce it.
What is tool overload?
The intuitive version is "too many tools confuse the model." We use a more precise definition:
Tool overload is a degradation in an agent's tool-selection accuracy or, in a multi-step task, in its ability to recover from a wrong tool selection and complete the task with a grounded answer, attributable to properties of the tools exposed to it rather than to task difficulty.
Four properties of the tools can cause it: how many tools there are, how much their definitions overlap, how complex those definitions are, and how much the tools return.
Our main finding: semantic overlap is what breaks the agent, not the number of tools. Growing a distinct tool set from 22 to 110 tools changed nothing we could measure. Adding just four tools that overlapped with ones already there cost up to 27 points of accuracy on a single tool choice and more than halved the correct answers on whole tasks. The number of tools still matters, as a cost and as the place where overlap creeps in, but the direct risk is two tools that plausibly claim the same job.
On dynamic tool loading: loading tool groups on demand solved the same 7 of 7 tasks as the full-tool single agent at about a fifth of the cost ($0.44 vs $1.96 per task), with 79,000 instead of 385,000 input tokens, fewer model calls (3.6 vs 5.3) and no latency penalty (1.02×). Fixing overlapping tools comes first.
Microsoft Research calls the same pattern tool-space interference, and reports that tools with overlapping names and scopes are common across independently published MCP servers.
Research program
We study tool overload as an evaluation ladder, where each page asks a slightly harder question than the one before it. This folder contains the detailed documentation of each experiment:
Defining Tool Overload with First-Tool Selection Accuracy
First results available When a user asks a question, does the agent choose the right first tool? This page isolates routing by measuring whether the correct tool appears in the agent's first batch of tool calls, before longer workflows introduce other ways to fail.
Single-Agent Tool Overload in Sequential Workflows
Results available After the first tool call, can one agent continue through a realistic multi-step task? It extends the question from routing to end-to-end completion, where state tracking, dependency order, argument carryover, and stopping behavior also matter.
Mitigating Tool Overload with Orchestrators and Dynamic Skills
Results available Do specialist agents or dynamic tool loading reduce interference, cost, or unnecessary calls? It compares mitigation architectures against the full-tool single agent.
Unique's Research Direction
Unique is building a practical research program around tool overload: what it is, when it appears in client deployments, and which architecture choices reduce it. The results come from four named studies, all run in one banking advisory workspace on the Unique platform with three OpenAI models.
Public study name | Public artifact | What it tested | Main result |
|---|---|---|---|
Tool Overload Baseline | GPT-5.1 with production-like tools from 2 to 105 tools, using real but mostly dissimilar connectors | 95.1% first-tool accuracy across the 1,680 main-sweep trials (2,076 including the position and ordering rounds); remaining misses concentrated around overlapping tool descriptions | |
First-Tool Selection Accuracy | GPT-4o, GPT-5.1, and GPT-5.5 across realistic tool sets, then the same setup with four deliberately overlapping tools | Realistic growth from 22 to 110 tools changed accuracy by less than one percentage point per model; four overlapping tools caused the large drops | |
Multi-Tool Task Completion | GPT-4o, GPT-5.1, and GPT-5.5 on 34 multi-step advisory tasks, on the same tool sets from 22 to 114 tools | Still flat from 22 to 110 tools; the four overlapping tools cut grounded answers from 50% to 23% | |
Architecture Evaluation | Single agent, orchestrator with specialist agents, and dynamic skills on the same tasks, at 110 and 114 tools | Dynamic skills matched the single agent on seven shared tasks at about a fifth of the cost (GPT-5.5); no architecture removed the overlap failure |
The operational lesson is not "keep the tool catalogue tiny." It is:
Keep useful, distinct tools available when the workflow needs them.
Remove or rewrite tools that claim overlapping responsibilities.
Treat tool descriptions as part of the product surface, not as internal metadata.
Measure cost because tool definitions are sent to the model on every call.
Key Research Takeaways
Realistic tool growth barely moved accuracy. Going from 22 to 110 real, mostly distinct tools changed first-tool accuracy by less than one percentage point for GPT-4o, GPT-5.1, and GPT-5.5.
Semantic collisions, not tool count, caused the failures. Adding just four broad, overlapping tools to the 110-tool set dropped GPT-5.1 from 92.6% to 65.7% (-26.9 pp) — while 41 real GitHub tools had barely moved it. The four were broad, long, and some asked to be preferred; the study shows the effect, not which of those properties drives it.
The misses were systematic. When a model failed on the collision set, it routed 73-100% of its misses straight into the four overlapping tools. It was not lost in a long list; it was following a description that made the wrong tool sound right.
Robustness varied by model, but every model failed the same way. With the four overlapping tools present, GPT-5.5 held 93.6%, GPT-4o 86.0% and GPT-5.1 65.7%. This is not a trend across model generations: GPT-5.1 dropped furthest, and its gap to GPT-5.5 also reflects a different reasoning setting. Every model still sent most or all of its misses to the decoys.
Position only helps once tools are confusable. In an exploratory round, tool order made no difference on clear prompts, but on prompts already known to be confusable the top two positions ran about 30 pp ahead of the middle and back.
Always-on tools are a cost decision. The fully equipped baseline ran roughly $40 per 1,000 GPT-5.1 calls, and rich FactSet definitions pushed requests to 35,000-50,000 input tokens.
What the follow-up studies found
First-tool selection is only the first rung. Modern agent benchmarks such as the Berkeley Function Calling Leaderboard, TPS-Bench, and TRAJECT-Bench increasingly separate single-call accuracy from longer trajectories. They measure whether agents can use tools in the right order, pass arguments forward, schedule independent calls, and stop when enough evidence has been collected.
We ran both follow-up studies on the same tool sets:
Multi-Tool Task Completion: 34 multi-step advisory tasks on Outlook, FactSet, InternalSearch and WebSearch. Grounded answers showed no detectable change from 22 to 110 tools, and the four overlapping tools cut them from 50% to 23%, pooled across models.
Architecture Evaluation: a single agent, an orchestrator with specialist agents and dynamic skills, run on the same tasks at 110 and 114 tools.
Advisor study: 14 practitioners worked with the single agent and the orchestrator. When the decoys sent the agent the wrong way, 10 of them noticed something was off, but only 5 worked out why.
What each architecture bought, and what it cost:
Architecture | Measured gain | Measured cost |
|---|---|---|
Single agent with all tools | Runs on all three models; no router can hide a needed tool | Highest cost per task; document-heavy tasks can outgrow the context window because tool results pile up |
Orchestrator with specialist agents | Completed tasks the single agent lost to context overflow (four single-trial cases) | About twice the model calls, 3.5 times the output and 1.5 times the model latency |
Dynamic skills | The same 7 of 7 tasks as the single agent on the clean tool set, at about a fifth of the cost (GPT-5.5) | Needs the vendor's tool-search feature (GPT-5.4 and later); with a plausible decoy in view it often stopped searching; its protection depends on a well-maintained group description |
No architecture removed the failure caused by the overlapping tools. The three are priced against each other, not ranked.
Research details and collaboration
This research is conducted at Unique AI by under the supervision of , with academic supervision by Prof. Dr. Gerhard Schwabe and Prof. Dr. Mateusz Dolata. The broader research question is how agentic systems should be architected so they remain reliable as enterprise tool catalogues grow.
The public artifacts are deliberately limited and curated:
System prompt used as the baseline production-style prefix.