​
CLI reference

Complete reference for MCP-Eval command-line interface, including commands and flags.

​
CLI commands

CommandDescriptionExample
mcp-eval initInitialize a new MCP-Eval projectmcp-eval init
mcp-eval generateGenerate test scenarios for MCP serversmcp-eval generate --style pytest
mcp-eval runExecute test files and datasetsmcp-eval run tests/
mcp-eval datasetRun dataset evaluationmcp-eval dataset datasets/basic.yaml
mcp-eval validateValidate configurationmcp-eval validate --quick
mcp-eval doctorDiagnose setup issuesmcp-eval doctor --full
mcp-eval issueCreate GitHub issue with diagnosticsmcp-eval issue --title "Bug report"
mcp-eval server addAdd MCP server to configmcp-eval server add
mcp-eval server listList configured serversmcp-eval server list -v
mcp-eval agent addAdd test agent to configmcp-eval agent add
mcp-eval agent listList configured agentsmcp-eval agent list --name default
mcp-eval versionShow version informationmcp-eval version

​
Setup & Configuration

​
init

Initialize a new MCP-Eval project with interactive setup. 
mcp-eval init [OPTIONS]
FlagDescriptionDefaultExample
--out-dirProject directory for configs.--out-dir ./my-project
--templateBootstrap template: empty, basic, samplebasic--template sample
What it does:
  • Creates mcpeval.yaml and mcpeval.secrets.yaml
  • Prompts for LLM provider and API key
  • Auto-detects and imports servers from .cursor/mcp.json or .vscode/mcp.json
  • Configures default agent with instructions
  • Sets up judge configuration for test evaluation
Source: generator.py:841-1015

​
server add

Add MCP server to configuration. 
mcp-eval server add [OPTIONS]
FlagDescriptionDefaultExample
--out-dirProject directory.--out-dir ./project
--from-mcp-jsonImport from mcp.json file---from-mcp-json .cursor/mcp.json
--from-dxtImport from DXT file---from-dxt manifest.dxt
Source: generator.py:1529-1633

​
agent add

Add test agent configuration. 
mcp-eval agent add [OPTIONS]
FlagDescriptionDefaultExample
--out-dirProject directory.--out-dir ./project
Source: generator.py:1635-1701

​
Test Generation

​
generate

Generate test scenarios and write test files for MCP servers. 
mcp-eval generate [OPTIONS]
FlagDescriptionDefaultExample
--out-dirProject directory.--out-dir ./tests
--styleTest format: pytest, decorators, dataset---style pytest
--n-examplesNumber of scenarios to generate6--n-examples 10
--providerLLM provider: anthropic, openai---provider anthropic
--modelSpecific model to use---model claude-3-opus-20240229
--verboseShow detailed error messagesFalse--verbose
--outputExplicit output file path---output tests/custom.py
--updateAppend tests to existing file instead of creating new---update tests/test.py
What it does:
  • Discovers server tools via MCP protocol
  • Generates test scenarios with AI
  • Refines assertions for each scenario
  • Validates generated Python code
  • Outputs test files or datasets
Source: generator.py:1017-1346 Update mode: When using --update, the command appends new tests to an existing file rather than creating a new one. The file path provided to --update becomes the target file. Example: 
# Append 5 new tests to existing file
mcp-eval generate --update tests/test_server.py --n-examples 5

​
Test Execution

​
run

Execute test files and generate reports. 
mcp-eval run <path|file|file::function> [OPTIONS]
FlagDescriptionDefaultExample
-v, --verboseDetailed outputFalse-v
--jsonOutput JSON report---json results.json
--markdownOutput Markdown report---markdown results.md
--htmlOutput HTML report---html results.html
--max-concurrencyParallel execution limit---max-concurrency 4
Accepts all standard pytest options Source: runner.py

​
dataset

Run dataset evaluation. 
mcp-eval dataset <dataset_file> [OPTIONS]
Same options as run command. Source: runner.py

​
Inspection & Validation

​
server list

List configured MCP servers. 
mcp-eval server list [OPTIONS]
FlagDescriptionDefaultExample
--project-dirProject directory.--project-dir ./project
-v, --verboseShow full detailsFalse-v
Source: list_command.py:20-102

​
agent list

List configured agents. 
mcp-eval agent list [OPTIONS]
FlagDescriptionDefaultExample
--project-dirProject directory.--project-dir ./project
-v, --verboseShow full instructionsFalse-v
--nameShow specific agent---name default
Source: list_command.py:104-185

​
validate

Validate MCP-Eval configuration and connections. 
mcp-eval validate [OPTIONS]
FlagDescriptionDefaultExample
--project-dirProject directory.--project-dir ./project
--servers/--no-serversValidate serversTrue--no-servers
--agents/--no-agentsValidate agentsTrue--no-agents
--quickSkip connection testsFalse--quick
What it checks:
  • API keys are configured
  • Judge model is set
  • Servers can be connected to
  • Agents reference valid servers
  • LLM connections work
Source: validate.py:342-514

​
Debugging & Diagnostics

​
doctor

Comprehensive diagnostics for troubleshooting. 
mcp-eval doctor [OPTIONS]
FlagDescriptionDefaultExample
--project-dirProject directory.--project-dir ./project
--fullInclude connection testsFalse--full
What it checks:
  • Python version and packages
  • Configuration files
  • Environment variables
  • System information
  • Recent test errors
  • Provides fix suggestions
Source: doctor.py

​
issue

Create GitHub issues with diagnostic information. 
mcp-eval issue [OPTIONS]
FlagDescriptionDefaultExample
--project-dirProject directory.--project-dir ./project
--titleIssue title---title "Connection timeout"
--no-include-outputsSkip test outputsFalse--no-include-outputs
--no-open-browserDon’t open browserFalse--no-open-browser
Source: issue.py

​
version

Show version information. 
mcp-eval version
Source: init.py:34-42

​
Configuration Files

MCP-Eval uses two primary configuration files:

​
mcpeval.yaml

Main configuration containing:
  • Server definitions (transport, command, args, env)
  • Agent definitions (name, instruction, server_names)
  • Judge configuration (provider, model, min_score)
  • Default agent setting
  • Reporting configuration

​
mcpeval.secrets.yaml

Sensitive configuration containing:
  • API keys for LLM providers
  • Authentication tokens
  • Other secrets
Both files are created by mcp-eval init and can be edited manually.

​
Environment Variables

MCP-Eval respects these environment variables:
VariableDescriptionExample
ANTHROPIC_API_KEYAnthropic Claude API keysk-ant-...
OPENAI_API_KEYOpenAI API keysk-...
GOOGLE_API_KEYGoogle API key...
COHERE_API_KEYCohere API key...
AZURE_API_KEYAzure OpenAI API key...
MCPEVAL_CONFIGPath to config file./config/mcpeval.yaml
MCPEVAL_SECRETSPath to secrets file./config/secrets.yaml

​
Typical Workflow

​
1. Initialize Project

# Create new project
mcp-eval init

# Or with sample files
mcp-eval init --template sample

​
2. Configure Servers & Agents

# Add server interactively
mcp-eval server add

# Import from existing config
mcp-eval server add --from-mcp-json .cursor/mcp.json

# Add test agent
mcp-eval agent add

# List what's configured
mcp-eval server list -v
mcp-eval agent list

​
3. Validate Setup

# Full validation with connections
mcp-eval validate

# Quick config check only
mcp-eval validate --quick

# Diagnose issues
mcp-eval doctor --full

​
4. Generate Tests

# Generate pytest tests
mcp-eval generate --style pytest --n-examples 10

# Add more tests to existing file
mcp-eval generate --update tests/test_server.py --n-examples 5

# Generate dataset for batch testing
mcp-eval generate --style dataset

​
5. Execute Tests

# Run all tests
mcp-eval run tests/

# Run specific test with reports
mcp-eval run tests/test_fetch.py -v --json report.json --markdown report.md

# Run dataset
mcp-eval dataset datasets/basic.yaml

​
6. Debug Issues

# If tests fail
mcp-eval doctor

# Create GitHub issue
mcp-eval issue --title "Tests failing with timeout"

​
Test Styles

MCP-Eval supports three test formats:

​
pytest

Standard pytest format with test functions and assertions. Best for integration with existing Python test suites.

​
decorators

MCP-Eval’s decorator-based format using @task and @setup. Provides rich async support and session management.

​
dataset

YAML-based test cases for batch evaluation. Ideal for non-programmers and test data management.

​
See also