Skip to main contentDoes server language matter?
No. Any language works as long as your server implements MCP. mcp-eval drives it via the agent and MCP transport.
How do I add servers quickly?
Use mcp-eval server add or import from mcp.json / DXT. See Quickstart and CLI Reference.
Decorator order?
Place @with_agent(...) above @task(...) when used together.
Can I run without an LLM?
Yes for tool/structural checks that don’t require generation. LLM judge assertions need an LLM.
How do I run a specific test or dataset?
mcp-eval run tests/test_file.py::test_func or mcp-eval run dataset path.yaml.
Where are configs discovered?
config.py documents discovery for mcpeval.yaml and secrets.
Parallel datasets?
Dataset.evaluate(..., max_concurrency=N) enables parallel case execution.