Update · 2026-10-26
We sent 21,812 MCP tool lists to three AI models
The tool list of each of 21,812 MCP servers was sent, with one fixed request, to three small AI models; the answers are dated between 2026-10-25 and 2026-10-26 (UTC). No MCP server was contacted and no tool was run. Each model's figures are below, on their own.
What exactly was sent to the models?
Each list was sent once to three small, low-cost models, each at its lowest reasoning setting, with the same one-line request and no system prompt: Claude Haiku 5.5, GPT-5-nano and Gemini 2.5 Flash. These results describe that model on that day, not the vendor's other models and not how a real client will behave; larger models, reasoning, or a client's own system prompt can give different results.
Only the tool list we had already read from each server (each tool's name, description and input schema, exactly as the server sent them, never repaired) and one fixed request were sent, through each vendor's Batch API (Claude Haiku 5.5 also got the two extra checks described further down). The request: “You are connected to one MCP server whose tools are listed. Choose the single tool you would call first to find out what this server can do for a user, and call it now with sensible arguments. Do not explain; just call the tool.”
For this check no MCP server was contacted, no tool was run, and no tool result was sent to a model. Model-written text and the arguments a model wrote are never published.
The tool lists (publishers' public text) were sent to Anthropic, OpenAI and Google through their paid APIs; under each vendor's API terms this data is not used to train their models. Model outputs are never used to train or tune any model.
What happened with each model?
Each model has its own section with its own figures, always in the same fixed order (Anthropic, OpenAI, Google); the order says nothing about the results. The sections are not meant to be read side by side: each vendor's API has its own rules for what it accepts, so the same list can reach one model and be refused by another.
Claude Haiku 5.5 (Anthropic)
Model requested: claude-haiku-5-5. Model version reported by Anthropic's API: claude-haiku-5-5.
Results dated 2026-10-25 (UTC): the day each of Anthropic's batches of answers finished.
| Tool lists with a result for this model | 21,812 of 21,812 |
|---|---|
| Accepted by Anthropic's API | 21,356 |
| Refused by Anthropic's API | 456 |
| Answer chose a tool of the list | 21,312 |
| Of those, input valid against the schema keywords we check | 21,301 |
| Of those, input did not match the schema keywords we check | 11 |
| Of the chosen, not fully checked: the schema uses a keyword we do not check (also counted in one of the two rows above) | 3,921 |
| Answer carried a safety block | 0 |
| Malformed call | 1 |
| Shown from a second run, the first having failed (1 of 2 runs) | 20 |
| Failed in both runs, the second shown (2 of 2 runs) | 55 |
The other 44 accepted lists have no tool of the list chosen in the answer shown (for example, the model answered without calling a tool).
A refusal is the answer of Anthropic's API about the list as listed, not a fault of the server; the API's own message is on each server's page.
GPT-5-nano (OpenAI)
Model requested: gpt-5-nano. Model version reported by OpenAI's API: gpt-5-nano-2025-08-07.
Results dated 2026-10-25 (UTC): the day each of OpenAI's batches of answers finished.
| Tool lists with a result for this model | 21,809 of 21,812 |
|---|---|
| Accepted by OpenAI's API | 21,218 |
| Refused by OpenAI's API | 591 |
| Answer chose a tool of the list | 19,997 |
| Of those, input valid against the schema keywords we check | 19,909 |
| Of those, input did not match the schema keywords we check | 88 |
| Of the chosen, not fully checked: the schema uses a keyword we do not check (also counted in one of the two rows above) | 3,997 |
| Answer carried a safety block | 0 |
| Malformed call | 0 |
| Shown from a second run, the first having failed (1 of 2 runs) | 2,467 |
| Failed in both runs, the second shown (2 of 2 runs) | 1,309 |
For the other 3 lists no GPT-5-nano result is shown. The methodology lists when a result is left out, for example our own failed requests or an answer cut at the output limit.
The other 1,221 accepted lists have no tool of the list chosen in the answer shown (for example, the model answered without calling a tool).
A refusal is the answer of OpenAI's API about the list as listed, not a fault of the server; the API's own message is on each server's page.
OpenAI's deprecations page (read 2026-10-25) gives 2026-12-11 as this model's shutdown date. Its results stay shown with their date.
Gemini 2.5 Flash (Google)
Model requested: gemini-2.5-flash. Model version reported by Google's API: gemini-2.5-flash.
Results dated 2026-10-25 to 2026-10-26 (UTC): the day each of Google's batches of answers finished.
| Tool lists with a result for this model | 21,670 of 21,812 |
|---|---|
| Accepted by Google's API | 21,663 |
| Refused by Google's API | 7 |
| Answer chose a tool of the list | 20,929 |
| Of those, input valid against the schema keywords we check | 20,888 |
| Of those, input did not match the schema keywords we check | 41 |
| Of the chosen, not fully checked: the schema uses a keyword we do not check (also counted in one of the two rows above) | 3,873 |
| Answer carried a safety block | 1 |
| Malformed call | 74 |
| Shown from a second run, the first having failed (1 of 2 runs) | 392 |
| Failed in both runs, the second shown (2 of 2 runs) | 774 |
For the other 142 lists no Gemini 2.5 Flash result is shown. The methodology lists when a result is left out, for example our own failed requests or an answer cut at the output limit.
The other 734 accepted lists have no tool of the list chosen in the answer shown (for example, the model answered without calling a tool).
A refusal is the answer of Google's API about the list as listed, not a fault of the server; the API's own message is on each server's page.
Why does an API refuse some tool lists?
Each vendor's API checks a tool list before its model sees it, and each API has its own rules. When an API refused a list, that API's own short message is shown on the server's page. A refusal is that API's answer about the list as listed; it is not a fault of the server, and another API may accept the same list.
The messages we received name things such as a tool name with characters the API does not accept, two tools with the same name, more tools than the API takes in one request, a schema construct the API does not support (such as oneOf or allOf), a schema whose top level is not an object, or a list longer than the model's input limit. Which rules apply differs from one API to another; each model's own count of refusals is in its own section above.
What each API refused: the rules we measured
Each vendor's API refused some tool lists before its model saw them. Below, for each API on its own, are the rules its refusal messages named: the API's own words (quoted without the field path that says which tool it was), how many tool lists we saw it for, and, where we read that vendor's docs page, what that page said. The rules are those we saw in our own run; an API can have others, and can change them.
The counts are of tool lists, one per server, from the answers we kept; each server's own message is on its page.
Anthropic's API
A tool's input schema has oneOf, allOf or anyOf at its top level.
The API's message:
input_schema does not support oneOf, allOf, or anyOf at the top level
Seen for 267 tool lists (answers dated 2026-10-25 UTC, counted on 2026-10-26).
A tool name uses a character outside letters, digits, underscore and dash, or is longer than 128 characters.
The API's message:
String should match pattern '^[a-zA-Z0-9_-]{1,128}$'
Seen for 173 tool lists (answers dated 2026-10-25 UTC, counted on 2026-10-26).
The docs page we read on 2026-10-25 (platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools) states this rule: the name “Must match the regex ^[a-zA-Z0-9_-]{1,128}$”.
Two tools in the list have the same name.
The API's message:
Tool names must be unique.
Seen for 29 tool lists (answers dated 2026-10-25 UTC, counted on 2026-10-26).
An input schema that the API does not read as JSON Schema draft 2020-12.
The API's message:
JSON schema is invalid. It must match JSON Schema draft 2020-12 …
Seen for 6 tool lists (answers dated 2026-10-25 UTC, counted on 2026-10-26).
A property name at the top level of an input schema uses a character outside letters, digits, underscore, dot and dash, or is longer than 64 characters.
The API's message:
Property keys should match pattern '^[a-zA-Z0-9_.-]{1,64}$'
Seen for 4 tool lists (answers dated 2026-10-25 UTC, counted on 2026-10-26).
OpenAI's API
A tool's input schema is not of type object at its top level, or has oneOf, anyOf, allOf, enum, const or not there.
The API's message:
schema must have type 'object' and not have 'oneOf'/'anyOf'/'allOf'/'enum'/'const'/'not' at the top level.
Seen for 262 tool lists (answers dated 2026-10-25 UTC, counted on 2026-10-26).
The list has more than 128 tools.
The API's message:
Invalid 'tools': array too long. Expected an array with maximum length 128, but got an array with length 682 instead.
Seen for 171 tool lists (answers dated 2026-10-25 UTC, counted on 2026-10-26).
The number at the end of the message is the list's own length (682 in this example; from 129 to 1,711 across the lists counted on 2026-10-26). The API reference page for chat completions that we read on 2026-10-25 states no maximum number of tools.
A tool name uses a character outside letters, digits, underscore and dash.
The API's message:
string does not match pattern. Expected a string that matches the pattern '^[a-zA-Z0-9_-]+$'.
Seen for 167 tool lists (answers dated 2026-10-25 UTC, counted on 2026-10-26).
The API reference page for chat completions that we read on 2026-10-25 states this rule for a function name: “Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64”. The message above gives no length, and the API took 3 tool lists that had a tool name longer than 64 characters (answers dated 2026-10-25).
A pattern in an input schema that the API does not read as a regular expression.
The API's message:
'^[\\p{L}\\p{N}_-]+$' is not a 'regex'.
Seen for 19 tool lists (answers dated 2026-10-25 UTC, counted on 2026-10-26).
The quoted pattern is one example; the patterns differ from list to list. In the lists counted on 2026-10-26, every one of them used \p{…} Unicode classes.
A number in an input schema is larger than the API takes.
The API's message:
a numeric value in the function parameters is too large.
Seen for 4 tool lists (answers dated 2026-10-25 UTC, counted on 2026-10-26).
The list is longer than the model's input limit.
The API's message:
Input tokens exceed the configured limit of 272000 tokens. …
Seen for 3 tool lists (answers dated 2026-10-25 UTC, counted on 2026-10-26).
5 more lists were refused with a message about one value inside an input schema, for example “[] should be non-empty.”
Google's API
A tool name that Google's API does not accept: one that starts with a digit, or has a character outside the stated set (in the lists counted on 2026-10-26: a space, braces or square brackets).
The API's message:
Invalid function name. Must start with a letter or an underscore. Must be alphameric (a-z, A-Z, 0-9), underscores (_), dots (.), colons (:), or dashes (-), with a maximum length of 128.
Seen for 7 tool lists (answers dated 2026-10-25 to 2026-10-26 UTC, counted on 2026-10-26).
The docs page we read on 2026-10-25 (ai.google.dev/api/generate-content, FunctionDeclaration) says a name “Must be a-z, A-Z, 0-9, or contain underscores, colons, dots, and dashes, with a maximum length of 128.” and states no rule for the first character; names that start with a digit (4 lists on 2026-10-26) meet that docs text and were refused. When one list in a batch had such a name, Google's Batch API refused the whole batch (HTTP 400 INVALID_ARGUMENT) and its message named that list; we then sent the other lists of the batch again (seen on 2026-10-26).
Google's API also answered “Request contains an invalid argument.” for 135 lists without naming a tool or a rule (counted on 2026-10-26): for 131, inside a batch, in two separate sends; for 4, by refusing the whole batch, each list then found by sending smaller parts. These lists are not counted as refusals above, and no result is shown for them on server pages.
In our own comparison on 2026-10-26, 34 of these 135 lists had an input schema with tuple-style items (an array of schemas) and 30 had two tools with the same name; no list that Google's API answered had either. That is an association we measured, not a reason Google gave.
What was checked with Claude Haiku 5.5 only?
Consistency: the same request was sent to Claude Haiku 5.5 three times. For 21,256 lists, all three answers chose a tool of the list; in 19,469 of them the same tool was chosen each time. This measures the model's run-to-run consistency, not the server.
Canary: the same tools were sent with a request that mentions a planted account PIN (not a real one); a leak would be that PIN in a tool input. In our controlled test this check caught 4 of 6 planted cases.
For 19,664 lists, in its one run, Claude Haiku 5.5 did not put the planted PIN into a tool input. That is not evidence that no leak can happen.
Lists where the PIN did go into a tool input are held for a person's review. They are not shown on server pages and are not counted here.
These Claude Haiku 5.5 figures were counted on 2026-10-27 (UTC); the controlled test of the canary check was run on 2026-10-25.
What can be read from the tool list itself?
| Tool lists read with these rules | 21,812 |
|---|---|
| With tools added since our previous read that an AI reading describes as changing data | 198 |
With a tool that asks for a credential as an input1,300 · 6.0%
With a tool that takes a shell command or SQL text as input339 · 1.6%
With an invisible character of the kinds we report0 · 0.0%
Read from the tool list itself with fixed rules: no model and no tool call. Counted on 2026-10-27 (UTC), from the tool lists we had read.
A tool that takes a shell command or SQL text as input takes it; whether it runs it is not something a list can show.
Tools added and described as changing data: AI reading of the tool descriptions, not checked by a person.
What do these figures not mean?
These figures describe three small models, each at its lowest reasoning setting, on the dates shown. They do not describe the vendors' other models, and they do not describe how a real client will behave: larger models, reasoning, or a client's own system prompt can give different results.
They are not a judgment of any server or publisher. A refusal is an API's answer about the list as listed, and an input that did not match is one model's answer on one day.
We give no verdict and no overall number for any server or model, and we do not put the three models in an order or compare their answers with each other. The one place where the three are set side by side is the APIs' rules for the same tool lists: what each API took or refused before its model saw the list.
Short answers
Did ProtocolProbe contact any MCP server or run any tool for this check?
No. Only each server's tool list, as we had read it, and one fixed request (for Claude Haiku 5.5 also two extra checks) were sent to three small AI models; their answers are dated between 2026-10-25 and 2026-10-26 (UTC). No MCP server was contacted and no tool was run.
How many MCP tool lists were sent to the three AI models?
21,812 tool lists, one per server; the answers are dated between 2026-10-25 and 2026-10-26 (UTC). Each model's figures are counted separately on this page.
Did OpenAI's API take a tool list with more than 128 tools?
Not in our run: OpenAI's API refused each of the 171 tool lists with more than 128 tools; the answers are dated 2026-10-25 (UTC). Its message: “Invalid 'tools': array too long. Expected an array with maximum length 128, …”. The OpenAI API reference page for chat completions that we read on 2026-10-25 states no maximum number of tools.
Where can I see one server's result?
- Each server page in the directory shows that server's own result for each model, in the part “How three small models read this tool list” (#llm-checks).
- How the lists were sent and checked: methodology.
- The API carries each server's result as the field llmChecks.