Defining Tool Overload with First-Tool Selection Accuracy
9 min read
We expected the agent to get less reliable as we handed it more tools. We could not measure any such decline. Going from 22 to 110 realistic enterprise tools moved first-tool accuracy by less than a percentage point for every model we tested. What actually broke the agent was smaller and far sneakier: four tools whose descriptions overlapped with capabilities it already had. Those four did what 41 real GitHub tools could not — they dropped GPT-5.1 from 92.6% to 65.7%.
That gap is the whole story of this page. "Tool overload" sounds like a counting problem: too many tools, and the model gets confused. Our evidence points somewhere more specific. The model rarely gets lost in a long list. It gets fooled when two tools plausibly claim the same job. That is why we measure the very first decision an agent makes, namely whether it reaches for the right tool first. It is the cleanest place to see the effect, before longer workflows introduce other ways to fail.
Figure 1: First-tool selection accuracy across realistic tool growth in the First-Tool Selection Accuracy study.
Here is the part clients find most surprising. When the tools we added were realistic and mostly distinct, accuracy stayed almost flat. GPT-5.5 went from 100.0% at 22 tools to 99.5% at 110. GPT-4o and GPT-5.1 each moved by less than a percentage point. Whatever tool overload is, it is not simply a function of how long the list gets.
Why the wrong tool looks right
Tool-using agents run a simple loop: the model reasons, picks an action, observes the result, and continues. That pattern goes back to work like ReAct, which made reasoning and acting explicit, and Toolformer, which showed that models can learn when and how to call APIs.
Enterprise agents stretch that loop. A financial-services assistant might hold internal search, public web search, market data, mailbox actions, project documents, notebooks, GitHub, and more — each arriving as a name, a description, and an input schema. Before it can do anything useful, the model has to pick the right one.
The tempting explanation is that more tools mean more confusion. Our data points at a narrower mechanism: trouble starts when two tools describe overlapping jobs.
Outlook gave us the cleanest example. A prompt asks for the messages inside a folder. The right tool is list_mail_folder_messages, but list_mail_folders sounds just as reasonable. In one cohort from the Tool Overload Baseline, GPT-5.1 reached for the sibling tool in 21 of 21 attempts, at every position we tried. Moving the correct tool to the front did not help, because the two descriptions still overlapped.
To stress-test that idea on purpose, the First-Tool Selection Accuracy study added four broad tools engineered to sound helpful:
UnifiedEnterpriseRetrieverFinancialAndMarketIntelligenceMailboxKnowledgeNavigatorLiveFactsAndNewsLookup
Each one shadowed a real capability. LiveFactsAndNewsLookup, for instance, overlapped WebSearch by claiming live facts, web, market, and news lookup all at once. On the hardest, most ambiguous WebSearch prompts, GPT-5.1 sent every single trial to that decoy.
The trap, side by side
These are the two descriptions the model saw, word for word:
The real tool: | The decoy: |
|---|---|
"Search the web and retrieve external sources using a planner format: multi-step search and URL-reading workflows." | "Live facts, web, market, and event lookup tool. Use whenever the answer may depend on recent information, external public sources, latest news, current prices, policy updates, sports scores, weather, public company events, breaking market context, current executives, or live exchange rates. This tool can search the public web, check market data, retrieve news-like business context, and verify public facts. It is a good default for short user questions containing today, now, latest, current, yesterday, this week, recent, news, price, rate, who won, weather, or what time." |
WebSearch chosen correctly at 110 tools | WebSearch chosen correctly once the decoy is added |
Neither description is wrong on its own. The real tool is described in one dry, technical sentence. The decoy reads like marketing copy, lists almost every kind of question a user might ask, and calls itself "a good default". Put next to each other, the broad description wins: it took most of the wrong picks on every model we tested.
What we tested
The findings come from two studies we run under fixed names.
The Tool Overload Baseline (technical report) ran GPT-5.1 against a baseline system prompt and a 140-prompt prompt bank, across production-like tool sets from 2 to 105 tools — real connectors like FactSet, Outlook, GitHub, OneNote, SharePoint, InternalSearch, and WebSearch — with position and connector-order perturbations layered on top.
The First-Tool Selection Accuracy study (Jupyter notebook, prompt bank) widened the model set to GPT-4o, GPT-5.1, and GPT-5.5, running 140 financial-domain prompts three times each on the four enterprise tool sets from 22 to 114 tools, and once more on the smaller tool set for each prompt's own domain (2 to 13 tools). Every chart and table in that notebook is computed from saved run files, not retyped by hand.
Both studies call the models through the Chat Completions API, because it works the same way across all three models and makes the runs easy to reproduce. That API caps a request at 128 tools, so every tool set stays below it: the largest is the 110 real tools plus the four decoys. The Responses API and equivalent APIs accept more tools per request.
We count a trial as a success when the model's first tool call is the expected tool. Parallel tool calls were enabled, as in production, and counting the whole first batch of calls instead gives exactly the same results.
The study builds up seven tool sets one realistic layer at a time, until the last one introduces the four decoys:
Tool set | Number of tools | Purpose |
|---|---|---|
Search-only | 2 | InternalSearch + WebSearch ceiling |
Outlook-focused | 11 | Search tools + Outlook |
FactSet-focused | 13 | Search tools + FactSet |
Core enterprise | 22 | Search tools + FactSet + Outlook |
Extended enterprise | 69 | Core enterprise + Atlassian + OneNote |
Full enterprise | 110 | Extended enterprise + GitHub |
Semantic collision | 114 | Full enterprise + four overlapping tools |
Realistic growth barely mattered
Start with the result that anchors everything else.
Figure 2: First-tool selection accuracy as the realistic tool menu grows for GPT-4o, GPT-5.1, and GPT-5.5. Every tool added here is a real connector, and none of them overlaps with a tool the prompts need.
From 22 to 110 tools, accuracy moved -0.7 pp for GPT-4o, -0.5 pp for GPT-5.1, and -0.5 pp for GPT-5.5. None of these changes is statistically significant. That is a bound, not proof of zero, and it holds for added tools that don't overlap with the ones the prompts need. Real tool sets rarely stay that clean, so keep an eye on the count as the place where overlap creeps in.
Then we added four tools, and the floor dropped out
The next comparison changes exactly one thing — the four broad, overlapping tools join the 110-tool set.
Model | Full enterprise, 110 tools | Semantic collision, 114 tools | Drop |
|---|---|---|---|
GPT-4o | 94.8% [92.2, 96.5] | 86.0% [82.3, 89.0] | -8.8 pp |
GPT-5.1 | 92.6% [89.7, 94.8] | 65.7% [61.1, 70.1] | -26.9 pp |
GPT-5.5 | 99.5% [98.3, 99.9] | 93.6% [90.8, 95.5] | -6.0 pp |
And the misses were anything but random. When a model failed on the semantic-collision set, it mostly failed into one of the four decoys:
Model | Misses on semantic-collision tool set | Misses routed to overlapping tools |
|---|---|---|
GPT-4o | 59 | 43 (73%) |
GPT-5.1 | 144 | 136 (94%) |
GPT-5.5 | 27 | 27 (100%) |
This is the sharpest definition of the failure mode we have: the model is not drowning in a long list, it is faithfully following a description that made the wrong tool sound right. GPT-5.5 absorbed the hit far better than the others, holding 93.6% where GPT-5.1 fell to 65.7% — but notice that even the strongest model routed 100% of its (much fewer) misses straight to the decoys. Robustness here means making fewer mistakes, not a different kind of mistake.
Position only helps once tools are already confusable
If overlap is the cause, does the order of the list matter? The Tool Overload Baseline tested that directly, and on its own, position did not cause failures: on robust control prompts, GPT-5.1 went 12 for 12 at every position, including when the correct tool sat last among 105. But on edge prompts with a plausible sibling, moving the correct tool near the front helped a lot — the top two positions ran about 30 percentage points ahead of the middle and back.
So if you are tuning a deployment: reordering can break a tie, but it does not fix the underlying problem. When two descriptions are functionally interchangeable, the durable fix is to rewrite or separate the tools, not to reshuffle them.
The cost side of overload
Accuracy is only half of it. Every tool definition is serialized into the request, so a big catalogue is also a recurring bill. In the Tool Overload Baseline, the fully equipped 105-tool set cost roughly $40 per 1,000 GPT-5.1 calls at list price, and 98% of the experiment spend went to input tokens.
The First-Tool Selection Accuracy study used the official FactSet definitions, which are far richer: FactSet alone accounts for about 30,000 of the roughly 35,000 input tokens in the 22-tool core set, and the full 110-tool set reaches about 50,000. None of that is a reliability failure — but it is a real design constraint:
A large, distinct tool catalogue can be reliable.
A large, always-on tool catalogue is still expensive.
Whether longer or richer descriptions help the model choose could not be tested on this tool set, because description length tracks which MCP server publishes a tool. What is certain is that every definition is paid for on every call.
What we would tell a team hitting this
If you are designing an agent and worried about tool overload, the data points to a short list of moves:
Look for overlap before you cut for count. In our tests overlap did the damage, not the length of the list. Remove or scope tools because they are unused, expensive, unsafe, or overlapping.
Audit your descriptions for collisions. Ask whether two tools could both plausibly answer the same natural-language request.
Be wary of broad router tools. A single tool that claims to handle "search, live facts, market data, news, and documents" competes with many narrow tools at once — and often wins when it should not.
Make boundaries explicit. Say in the description when to use a tool and, just as importantly, when not to. This is good practice; we did not test description rewrites.
Treat always-on tools as a budget decision. Every definition is paid for on every relevant call.
Measure workflow success separately. A correct first tool is necessary but not sufficient — multi-step tasks can still fail through bad argument carryover, dependency order, overcalling, or stopping too early.
Where this goes next
First-tool selection answers one question: did the agent start in the right place? The harder question is whether it can finish the job — and that is where the field is heading too. The Berkeley Function Calling Leaderboard finds that state-of-the-art models can be strong on single-turn function calls while memory, dynamic decision-making, and long-horizon reasoning remain open. TPS-Bench looks at tool planning and scheduling in compounding tasks, and TRAJECT-Bench checks whether tools are selected, parameterized, and ordered correctly across whole trajectories.
Our own work climbs the same ladder:
Study | Main question | What we found |
|---|---|---|
First-tool selection (this page) | Does the agent choose the right starting tool? | Flat from 22 to 110 distinct tools; four decoys cost up to 27 points. |
Can one agent finish a whole task with a grounded answer? | Still flat from 22 to 110 tools; the same four decoys cut grounded answers from 50% to 23%. | |
Do an orchestrator or dynamic skills fix it? | They shift cost, context and speed. None removes the decoy failure. |
What this page does not show
It measures first-tool selection, not full task completion. It isolates routing; it does not prove the final answer is correct.
The semantic-collision tools are controlled artifacts. They are realistic stressors, not accidental production tools.
The four look-alikes changed several things at once. They are broader than the tools they sit beside, longer, and some tell the model to prefer them. The study shows the size and direction of the effect, not which of these properties drives it.
Tool descriptions are part of the treatment. Better descriptions may improve accuracy; longer ones may raise cost.