🚀 Welcome to mcp-eval
! You’re about to supercharge your MCP development with powerful testing capabilities. This guide will have you testing MCP servers and agents in just 5 minutes!
What you’ll learn
By the end of this quickstart, you’ll be able to:- ✅ Install and configure
mcp-eval
for your project - ✅ Connect your MCP servers for testing
- ✅ Write and run your first test
- ✅ Understand test reports and iterate on failures
- ✅ Choose the right testing style for your needs
Before you begin
Let’s make sure you have everything ready:System requirements
Python 3.10+
Required for running
mcp-eval
Download Python →MCP Server
Any MCP-compatible server to testBrowse MCP servers →
API Key
Claude or OpenAI key for LLM featuresGet Claude API →
New to MCP? No worries! Check out the MCP documentation to understand the basics of Model Context Protocol servers. You’ll be testing them like a pro in no time!
Your 5-minute journey to testing mastery
1
Install `mcp-eval` and configure API keys
First, let’s get This makes the Next, add mcp-eval as a dependency for your project:Alternatively:Now set up your API key for the best experience:
mcp-eval
installed for your project.We recommend using uv to install mcp-eval
as a global tool:mcp-eval
CLI available globally on your system.Language agnostic testing:
mcp-eval
can test MCP servers written in any language - Python, TypeScript, Go, Rust, Java, etc. As long as your server implements the MCP protocol, mcp-eval can test it!Using uv in a project
Pro tip: Claude Sonnet or Opus models provide the best results for test generation and LLM judge evaluations!
2
Initialize your test project
Let’s set up your testing environment with our interactive wizard:This friendly wizard will:
- 🎯 Ask for your preferred LLM provider and model
- 📝 Create
mcpeval.yaml
with your configuration - 🔐 Set up
mcpeval.secrets.yaml
for secure API key storage - 🤖 Help you define your first test agent
- 🔧 Import any existing MCP servers
3
Configure MCP servers
Before we can test an MCP server, you need to tell mcp-eval how to connect to it.
Adding your MCP server
You have several ways to add a server to your configuration:- Interactive Add
- From mcp.json
- From DXT
- Manual Edit
The easiest way - let mcp-eval guide you:This will prompt you for:
- How to add (interactive, from-mcp-json, or from-dxt)
- Server name (e.g., “fetch”)
- Command to run (e.g., “uvx mcp-server-fetch”)
- Any arguments or environment variables
Common server examples
Here are some popular MCP servers you might want to test:Verify your server configuration: After adding a server, you can verify it’s working:
4
Run your first test
Time for the exciting part - running your first test! We’ll use the included fetch server example to demonstrate.First, let’s make sure we have an example test. If you used What’s happening:
Example structure: The examples assume you have the fetch server configured. If you’re testing a different server, you’ll need to adjust the test code accordingly.
mcp-eval init
, you might already have one. Otherwise, let’s run:- 🏃 Running decorator-style tests from the example file
- 📊 Verbose output (
-v
) shows test progress - 📝 Markdown report for documentation
- 🌐 HTML report for interactive exploration
5
Explore your test results
Open your shiny new test report to see the details:Understanding the HTML report:The interactive report shows:
- 📊 Overview dashboard - Pass/fail rates, performance metrics
- 🔍 Test details - Each test with all assertions
- 🛠️ Tool usage - What tools were called and when
- 💭 LLM reasoning - The agent’s thought process
- ⚡ Performance - Response times and efficiency metrics
- 🎯 Failed assertions - Detailed diffs and explanations
- Did the right tools get called?
- Was the output accurate?
- How efficient was the agent’s approach?
- What was the LLM judge’s assessment?
Test failed? Don’t worry! Check the assertion details to understand why. Common issues:
- Tool not found (check server configuration)
- Content mismatch (adjust your assertions)
- Timeout (increase timeout in config)
What’s next? Write your own test!
Now that you’ve run the example, let’s write your very first custom test:Choose your testing style
- Task Decorators
- Pytest
Best for: Quick, readable tests
Your test file structure
Create a new test filetests/test_my_server.py
:
Troubleshooting common issues
My server isn't being found
My server isn't being found
Solution: Check your Also verify the server name matches what you’re using in your agent’s
mcpeval.yaml
to ensure the server is properly configured:server_names
.Tests are timing out
Tests are timing out
Solution: Increase the timeout in your configuration:
API key errors
API key errors
Solution: Ensure your API key is set correctly:
Resources to level up
Ready to become amcp-eval
expert? Here’s your learning path:
Complete Examples
Full test suites showing all testing patterns
Common Workflows
Step-by-step guides for typical testing scenarios
Configuration Guide
Deep dive into all configuration options
Best Practices
Pro tips for writing maintainable tests
Get help
- 💬 Questions? Check our FAQ or troubleshooting guide
- 🐛 Found a bug? Report it on GitHub
- 💡 Have ideas? We’d love to hear them in discussions
Congratulations! 🎉 You’ve successfully set up
mcp-eval
and run your first tests. You’re now ready to ensure your MCP servers and agents work flawlessly. Happy testing!