> ## Documentation Index
> Fetch the complete documentation index at: https://mcp-eval.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# mcp-eval Documentation

> The comprehensive testing framework for MCP servers and tool-using agents.

> [!NOTE]
>
> **Your flight simulator for MCP servers and agents** — Connect agents to real MCP servers, run realistic scenarios, and calculate metrics for tool calls and more.

> [!NOTE]
>
> [Model Context Protocol](https://modelcontextprotocol.io/docs/getting-started/intro)  standardizes how applications provide context to large language models (LLMs). Think of MCP like a USB-C port for AI applications.
>
> **`mcp-eval`** ensures your MCP servers, and agents built with them, work reliably in production.

## What `mcp-eval` Does for You

> [card] Test MCP Servers
>
> Ensure your MCP servers respond correctly to agent requests and handle edge cases gracefully

> [card] Evaluate Agents
>
> Measure how effectively agents use tools, follow instructions, and recover from errors

> [card] Track Performance
>
> Monitor latency, token usage, cost, and success rates with OpenTelemetry-backed metrics

> [card] Assert Quality
>
> Use structural checks, LLM judges, and path efficiency validators to ensure high quality

## Get Started in 30 Seconds

We recommend using [uv](https://docs.astral.sh/uv/):

```bash uv (recommended) theme={null}
# Install mcp-eval globally (for CLI)
uv tool install mcpevals

# Add mcp-eval dependency to your project
uv add mcpevals

# Initialize your project (interactive setup)
mcp-eval init

# Add your MCP server to test
mcp-eval server add

# Auto-generate tests with an LLM
mcp-eval generate

# Run decorator/dataset tests
mcp-eval run tests/

# Run pytest tests (use pytest)
uv run pytest -q tests
```

```bash pip theme={null}
# Install mcp-eval
pip install mcpevals

# Initialize your project
mcp-eval init

# Add your MCP server
mcp-eval server add

# Run decorator/dataset tests
mcp-eval run tests/

# Run pytest tests (use pytest)
pytest -q tests
```

> [!NOTE]
>
> **Test any MCP server:** It doesn't matter what language your MCP server is written in - Python, TypeScript, Go, Rust, Java, or any other. As long as it implements the MCP protocol, `mcp-eval` can test it!

> [!TIP]
>
> You're ready to start testing! [Continue with the Quickstart →](https://mcp-eval.ai/quickstart.md)

## 🎮 Choose Your Testing Adventure

What are you evaluating today?

> [tab] I'm testing an MCP Server
>
> **You built an MCP server** (in any language!) and want to ensure it handles agent requests correctly.
>
> > [card] Perfect! Let's test your server
> >
> > mcp-eval will spin up an AI agent to test your server with realistic requests, edge cases, and error scenarios.
> >
> > **Your server could be:**
> >
> > * A streamable HTTP database connector
> > * An SSE API wrapper
> > * A stdio file system server
> > * Any server that speaks MCP!
>
> > [card] [Start Here](https://mcp-eval.ai/server-evaluation.md)
> >
> > MCP Server Testing Guide
>
> > [card] [See Examples](https://mcp-eval.ai/examples-mcp-server-fetch.md)
> >
> > Testing the Fetch Server

> [tab] I'm testing an Agent
>
> **You built an AI agent** that uses MCP servers and want to ensure it uses tools effectively.
>
> > [card] Perfect! Let's evaluate your agent
> >
> > mcp-eval will connect your agent to MCP servers and verify it uses tools correctly, handles errors, and meets performance targets.
> >
> > **Your agent could be:**
> >
> > * A customer service bot
> > * A coding assistant
> > * A deep research agent
> > * Any MCP agent!
>
> > [card] [Start Here](https://mcp-eval.ai/agent-evaluation.md)
> >
> > Agent Evaluation Guide
>
> > [card] [Learn Patterns](https://mcp-eval.ai/common-workflows.md)
> >
> > Common Testing Patterns

> [tab] I want both!
>
> **You're building a complete system** with both MCP servers and agents.
>
> > [card] Awesome! Test the whole stack
> >
> > mcp-eval can test your entire integration - ensuring servers handle requests correctly AND agents use tools effectively.
>
> > [card] [Quick Start](https://mcp-eval.ai/quickstart.md)
> >
> > 5-minute setup
>
> > [card] [Concepts](https://mcp-eval.ai/concepts.md)
> >
> > Core concepts
>
> > [card] [Examples](https://mcp-eval.ai/examples.md)
> >
> > Browse all examples

## Why Teams Choose `mcp-eval`

* **Production-readiness**: Built on OpenTelemetry for enterprise-grade observability
* **Multiple test styles**: Choose between decorators, pytest, or dataset-driven testing
* **Rich assertions**: Content checks, tool verification, performance gates, and LLM judges
* **CI/CD friendly**: GitHub Actions support, JSON/HTML reports, and regression detection
* **Language agnostic**: Test MCP servers written in any language

## Quick Navigation

> [card] [Quickstart](https://mcp-eval.ai/quickstart.md)
>
> Get up and running in 5 minutes

> [card] [Common Workflows](https://mcp-eval.ai/common-workflows.md)
>
> Step-by-step guides for typical tasks

> [card] [API Reference](https://mcp-eval.ai/api-catalog.md)
>
> Complete assertion catalog and APIs

## Learning Path

> [tab] Getting Started
>
> > [card] [Overview](https://mcp-eval.ai/overview.md)
> >
> > Understand `mcp-eval`'s architecture and philosophy
>
> > [card] [Quickstart](https://mcp-eval.ai/quickstart.md)
> >
> > Your first test in 5 minutes
>
> > [card] [Concepts](https://mcp-eval.ai/concepts.md)
> >
> > Core concepts and terminology

> [tab] Writing Tests
>
> > [card] [Assertions](https://mcp-eval.ai/assertions.md)
> >
> > The unified Expect API for all assertions
>
> > [card] [Common Workflows](https://mcp-eval.ai/common-workflows.md)
> >
> > Practical testing patterns
>
> > [card] [Test Generation](https://mcp-eval.ai/test-generation.md)
> >
> > AI-powered test creation

> [tab] Evaluation Types
>
> > [card] [Server Evaluation](https://mcp-eval.ai/server-evaluation.md)
> >
> > Testing MCP server implementations
>
> > [card] [Agent Evaluation](https://mcp-eval.ai/agent-evaluation.md)
> >
> > Measuring agent effectiveness
>
> > [card] [Datasets](https://mcp-eval.ai/datasets.md)
> >
> > Systematic evaluation suites

> [tab] Configuration
>
> > [card] [Configuration](https://mcp-eval.ai/configuration.md)
> >
> > Settings and customization
>
> > [card] [CI/CD](https://mcp-eval.ai/ci-cd.md)
> >
> > GitHub Actions and automation
>
> > [card] [Reports](https://mcp-eval.ai/reports.md)
> >
> > Understanding test outputs

> [tab] Reference
>
> > [card] [CLI Reference](https://mcp-eval.ai/cli-reference.md)
> >
> > Complete command documentation
>
> > [card] [API Reference](https://mcp-eval.ai/api-catalog.md)
> >
> > Detailed API documentation
>
> > [card] [Troubleshooting](https://mcp-eval.ai/troubleshooting.md)
> >
> > Common issues and solutions
>
> > [card] [FAQ](https://mcp-eval.ai/faq.md)
> >
> > Frequently asked questions

## Example: Your First Test

```python test_fetch.py theme={null}
from mcp_eval import task, Expect

@task("Verify fetch server works correctly")
async def test_fetch(agent, session):
    # Ask the agent to fetch a webpage
    response = await agent.generate_str("Fetch https://example.com and summarize it")
    
    # Assert the right tool was called
    await session.assert_that(Expect.tools.was_called("fetch"))
    
    # Verify the content is correct
    await session.assert_that(Expect.content.contains("Example Domain"), response=response)
    
    # Check performance
    await session.assert_that(Expect.performance.response_time_under(5000))
```

```python pytest_style.py theme={null}
import pytest
from mcp_eval import create_agent, Expect

@pytest.mark.asyncio
async def test_fetch_with_pytest():
    agent = await create_agent("claude-3-5-sonnet")
    response = await agent.generate_str("Fetch https://example.com")
    
    assert "Example Domain" in response
    assert agent.tools_called == ["fetch"]
```

> [!TIP]
>
> [See more examples →](https://mcp-eval.ai/examples.md)

## Join the Community

> [card] [GitHub](https://github.com/lastmile-ai/mcp-eval)
>
> Report issues and contribute

> [card] [Discord](https://lmai.link/discord/mcp-eval)
>
> Get help and share experiences


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.