io.github.JcJamet/ia-qa-eval

None of its 38 tools are described as changing anything. For 38 the reading was uncertain, so no mark is shown.

• get_testing_guidelineson 1 other server

  “Query the IA-QA methodology knowledge base.”

  Publisher declares: read-only yes · destructive no · idempotent yes · open-world no

  1 input

  • topicstring“The testing topic to retrieve guidelines for.”
• find_toolon 17 other servers

  “Search available MCP tools by keyword or category before calling them.”

  Publisher declares: read-only yes · destructive no · idempotent yes · open-world no

  4 inputs, 1 required

  • querystringrequired“Keyword(s) to search in tool name and description (e.g.”
  • categorystring“Optional: filter by category — data | encoding | text | llm | qa | rag | dev | security | web”
  • with_schemaboolean“Set true to include inputSchema in results (default: false)”
  • max_resultsnumber“Maximum tools to return (default 10, max 50).”
• hallucination_checkon 3 other servers

  “Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context.”

  Publisher declares: read-only yes · destructive no · idempotent yes · open-world no

  3 inputs, 2 required

  • answerstringrequired“The LLM-generated answer to verify”
  • contextstringrequired“The source/reference text that should ground the answer”
  • strictboolean“If true, every sentence in the answer must be supported (default: false)”
• run_eval_contracton 1 other server

  “Parse a .ia-eval.yaml LLM test suite, call the specified LLM model for each scenario, run all configured scorers, and return a structured JSON report with per-scenario Pass/Fail verdicts and a Markdo…”

  Publisher declares: read-only yes · destructive no · idempotent no · open-world yes

  5 inputs

  • contract_pathstring“Absolute or relative path to a .ia-eval.yaml file (required unless inline_contract is provided)”
  • inline_contractobject“Raw contract object (alternative to contract_path).”
  • api_keysobject“API keys to use for LLM generation (all optional — falls back to server env vars)”
  • overridesobject“Override contract defaults”
  • validate_onlyboolean“Check the contract against the IA-EVAL schema and stop: no model call, no key, deterministic.”
• run_semantic_testson 1 other server

  “Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L.”

  Publisher declares: read-only yes · destructive no · idempotent yes · open-world no

  5 inputs, 1 required

  • casesarrayrequired“Array of (actual, expected) pairs to evaluate.”
  • thresholdsobject“Pass/fail thresholds (defaults: cosine 0.75, rouge_l 0.5).”
  • require_allboolean“If true (default), all cases must pass for overall PASS.”
  • modestring“tfidf (default): fast, free, lexical.”
  • api_keystring“OpenAI API key — required only when mode is embeddings.”
• generate_ci_workflowon 2 other servers

  “Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA.”

  Publisher declares: read-only yes · destructive no · idempotent yes · open-world no

  12 inputs

  • gatestring“Which gate to emit.”
  • contract_pathstring“Path to the .ia-eval.yaml contract, relative to the repo root (default: evals/smoke.ia-eval.yaml).”
  • providerstring“LLM provider the contract runs against — decides which repository secret the workflow wires (default: groq).”
  • cli_toolsarray“IA-QA tool names to run as deterministic gates, e.g.”
  • triggersarray“Workflow triggers (default: push + pull_request).”
  • cronstring“Cron expression when triggers include 'schedule' (default: '0 6 * * 1' — Mondays 06:00 UTC).”
  • min_scorenumber“Override the contract min_score (0-100).”
  • fail_on_failboolean“Fail the build on a FAIL/PARTIAL verdict (default: true).”
  • workflow_namestring“Workflow display name (default: "IA-QA Quality Gate").”
  • node_versionstring“Node version for the CLI steps (default: "20").”
  • app_start_commandstring“Command that boots the app for the selector_drift gate, e.g.”
  • app_base_urlstring“URL the drift gate waits for before capturing (default: http://127.0.0.1:3000).”
• needle_haystack_generateon 1 other server

  “Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position.”

  Publisher declares: read-only yes · destructive no · idempotent no · open-world no

  4 inputs, 2 required

  • needlestringrequired“The fact to hide (e.g.”
  • questionstringrequired“The question to ask the LLM (e.g.”
  • tokensinteger“Target haystack size in tokens (default: 5000, max: 100000)”
  • positionstring“Where to insert the needle: "start", "middle", "end", "random" (default: "middle")”
• generate_eval_yamlon 1 other server

  “Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do.”

  Publisher declares: read-only yes · destructive no · idempotent no · open-world yes

  4 inputs, 1 required

  • descriptionstringrequired“Plain-language description of what the LLM under test should do.”
  • system_promptstring“Optional system prompt of the LLM under test.”
  • scenario_countnumber“Number of scenarios to generate (default: 5).”
  • task_typestring“Optional task type hint to guide evaluator selection.”
• metamorphic_checkon 1 other server

  “Reference-free stability primitive: instead of comparing an answer to a ground truth, it checks that an assistant's answer stays INVARIANT when the QUESTION is transformed (typo, casing, paraphrase, …”

  Publisher declares: read-only yes · destructive no · idempotent yes · open-world no

  7 inputs, 2 required

  • baseobjectrequired“The reference run: the original question and the answer your system produced for it.”
  • variantsarrayrequired“Answers produced for transformed versions of the same question, each tagged with the relation that was applied.”
  • modestring“tfidf (default): free, lexical, deterministic — but a genuine paraphrase rarely reaches 0.80, so gate on case/typo and treat paraphrase as a trend.”
  • api_keystring“OpenAI API key — required only when mode is embeddings.”
  • thresholdsobject“Per-relation threshold overrides.”
  • baseline_guardobject“Correctness floor applied to the BASE answer before anything is scored.”
  • require_allboolean“If true (default), every gated variant must pass.”
• test_skillon 1 other server

  “Test a SKILL.md in the open Agent Skills format (https://agentskills.io/specification — read by Claude Code, Codex, GitHub Copilot, Cursor, Gemini CLI, OpenCode and others) in two stages.”

  Publisher declares: read-only yes · destructive no · idempotent no · open-world yes

  8 inputs, 1 required

  • skill_mdstringrequired“Full content of the SKILL.md file to test, YAML frontmatter included.”
  • dir_namestring“Name of the folder holding the SKILL.md.”
  • scenariosarray“Fixed user requests for a reproducible gate: [{"prompt": "...", "should_trigger": true}].”
  • other_skillsarray“The other skills installed next to this one, [{"name", "description"}], so triggering is tested against the real neighbours.”
  • scenario_countnumber“Number of requests to generate when scenarios is omitted: half should-load, half near misses.”
  • lint_onlyboolean“Run the format check only: no model call, no key, deterministic.”
  • modelstring“LLM model ID to use for both scenario generation and testing (e.g.”
  • api_keystring“API key for the chosen model provider.”
• llm_output_validatoron 1 other server

  “Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language detection, and safety patterns.”

  Publisher declares: read-only yes · destructive no · idempotent yes · open-world no

  9 inputs, 1 required

  • outputstringrequired“The LLM output text to validate”
  • expected_formatstring“Expected output format”
  • must_includestring“Comma-separated strings that MUST appear in the output”
  • must_not_includestring“Comma-separated strings that must NOT appear (e.g.”
  • max_lengthnumber“Maximum character length for the output”
  • min_lengthnumber“Minimum character length for the output”
  • expected_languagestring“Expected language of the output (en, fr, es, de…).”
  • check_safetyboolean“Check for PII patterns (emails, phones, SSN), profanity signals, and prompt leakage”
  • check_json_schemastring“If expected_format is JSON, provide required keys as comma-separated list to validate the structure”
• compare_responseson 2 other servers

  “Compare two ALREADY-PRODUCED outputs (e.g.”

  Publisher declares: read-only yes · destructive no · idempotent yes · open-world no

  9 inputs, 2 required

  • response_astringrequired“First output (e.g.”
  • response_bstringrequired“Second output (e.g.”
  • label_astring“Label for output A (e.g.”
  • label_bstring“Label for output B (e.g.”
  • taskstring“The task/prompt both outputs were answering — used by the LLM judge for context”
  • referencestring“Optional ground-truth / expected answer.”
  • check_jsonboolean“Try to parse as JSON and compare structurally (keys, types, values)”
  • modelstring“Optional judge model id (BYOK).”
  • api_keystring“Optional API key for the judge model (BYOK).”

Each quoted sentence is the first sentence of the publisher’s own description. The kind of tool and the “changes data” mark are our AI model’s reading of that description, not something we tested: two differently worded runs had to agree, otherwise no label is shown, and nobody has checked them by hand. A tool without the “changes data” mark is described as only reading, searching or calculating, or the two runs did not agree. No tool was called. Read the publisher’s full description before relying on it. “Publisher declares” lists the hints the server itself sends with the tool (MCP annotations: read-only, destructive, idempotent, open-world); they are the publisher's claims, not checked by us. Inputs are the parameters in the tool's own input schema: name, declared type, whether it is required, and the first sentence of its description. Read 2026-10-27.

The registry checks an entry when it is published (each new version), and nothing re-checks it on a schedule. A difference below is about the listing, not a judgement of the publisher.