> ## Documentation Index
> Fetch the complete documentation index at: https://mcp-eval.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Quickstart Guide

> Get `mcp-eval` up and running in 5 minutes. Learn to install, configure, and run your first tests for MCP servers and agents.

> 🚀 **Welcome to `mcp-eval`!** You're about to supercharge your MCP development with powerful testing capabilities. This guide will have you testing MCP servers and agents in just 5 minutes!

## What you'll learn

By the end of this quickstart, you'll be able to:

* ✅ Install and configure `mcp-eval` for your project
* ✅ Connect your MCP servers for testing
* ✅ Write and run your first test
* ✅ Understand test reports and iterate on failures
* ✅ Choose the right testing style for your needs

**Time to complete:** \~5 minutes

## Before you begin

Let's make sure you have everything ready:

### System requirements

<CardGroup cols={3}>
  <Card title="Python 3.10+" icon="python">
    Required for running `mcp-eval`

    [Download Python →](https://www.python.org/downloads/)
  </Card>

  <Card title="MCP Server" icon="server">
    Any MCP-compatible server to test

    [Browse MCP servers →](https://github.com/modelcontextprotocol/servers)
  </Card>

  <Card title="API Key" icon="key">
    Claude or OpenAI key for LLM features

    [Get Claude API →](https://console.anthropic.com/)
  </Card>
</CardGroup>

<Tip>
  **New to MCP?** No worries! Check out the [MCP documentation](https://modelcontextprotocol.io) to understand the basics of Model Context Protocol servers. You'll be testing them like a pro in no time!
</Tip>

## Your 5-minute journey to testing mastery

<Steps>
  <Step title="Install `mcp-eval` and configure API keys">
    First, let's get `mcp-eval` installed for your project.

    We recommend using [uv](https://docs.astral.sh/uv/) to install `mcp-eval` as a global tool:

    ```bash theme={null}
    uv tool install mcpevals
    ```

    This makes the `mcp-eval` CLI available globally on your system.

    <Info>
      **Language agnostic testing:** `mcp-eval` can test MCP servers written in **any language** - Python, TypeScript, Go, Rust, Java, etc. As long as your server implements the MCP protocol, mcp-eval can test it!
    </Info>

    Next, add mcp-eval as a dependency for your project:

    # Using uv in a project

    ```bash theme={null}
    uv add mcpevals
    ```

    Alternatively:

    ```bash theme={null}
    pip install mcpevals
    ```

    Now set up your API key for the best experience:

    ```bash theme={null}
    # We recommend Claude for superior test generation and judging
    export ANTHROPIC_API_KEY="sk-ant-..."

    # Alternative: OpenAI
    export OPENAI_API_KEY="sk-..."
    ```

    <Note>
      **Pro tip:** Claude Sonnet or Opus models provide the best results for test generation and LLM judge evaluations!
    </Note>

    {/* TODO: Screenshot showing successful installation output */}
  </Step>

  <Step title="Initialize your test project">
    Let's set up your testing environment with our interactive wizard:

    ```bash theme={null}
    mcp-eval init
    ```

    This friendly wizard will:

    * 🎯 Ask for your preferred LLM provider and model
    * 📝 Create `mcpeval.yaml` with your configuration
    * 🔐 Set up `mcpeval.secrets.yaml` for secure API key storage
    * 🤖 Help you define your first test agent
    * 🔧 Import any existing MCP servers

    **What happens during init:**

    ```
    ? Select your LLM provider: Anthropic
    ? Select model: claude-3-5-sonnet-20241022
    ? Import servers from mcp.json? Yes
    ? Path to mcp.json: .cursor/mcp.json
    ✓ Found 2 servers: fetch, filesystem
    ? Create a default agent? Yes
    ? Agent name: TestBot
    ? Agent instruction: You test MCP servers thoroughly
    ✓ Configuration saved to mcpeval.yaml
    ✓ Secrets saved to mcpeval.secrets.yaml
    ```

    {/* TODO: Screenshot of the interactive init prompts and successful completion */}

    {/* TODO: Screenshot showing the created files in the project directory */}
  </Step>

  <Step title="Configure MCP servers">
    Before we can test an MCP server, you need to tell mcp-eval how to connect to it.

    <Info>
      Connection works over any supported [transport](https://modelcontextprotocol.io/specification/2025-06-18/basic/transports) (stdio, websocket, sse, streamable\_http).
      You can import server configurations from [`mcp.json`](https://code.visualstudio.com/docs/copilot/customization/mcp-servers) or [`dxt`](https://github.com/anthropics/dxt/blob/main/README.md) files,
      or specify them interactively using the `mcp-eval server add` command.
    </Info>

    ### Adding your MCP server

    You have several ways to add a server to your configuration:

    <Tabs>
      <Tab title="Interactive Add">
        The easiest way - let mcp-eval guide you:

        ```bash theme={null}
        mcp-eval server add
        ```

        This will prompt you for:

        * How to add (interactive, from-mcp-json, or from-dxt)
        * Server name (e.g., "fetch")
        * Command to run (e.g., "uvx mcp-server-fetch")
        * Any arguments or environment variables

        Example interaction:

        ```
        ? How would you like to add the server? interactive
        ? Server name: fetch  
        ? Command: uvx mcp-server-fetch
        ? Add environment variables? No
        ✓ Added server 'fetch'
        ```
      </Tab>

      <Tab title="From mcp.json">
        If you're using Cursor or VS Code with MCP:

        ```bash theme={null}
        mcp-eval server add --from-mcp-json .cursor/mcp.json
        ```

        This imports servers from your IDE's MCP configuration. The tool will:

        1. Read the mcp.json file
        2. Show you available servers
        3. Let you choose which ones to add
      </Tab>

      <Tab title="From DXT">
        ```bash theme={null}
        mcp-eval server add --from-dxt ~/Desktop/my-server.dxt
        ```

        This imports servers from DXT manifest files.
      </Tab>

      <Tab title="Manual Edit">
        Edit `mcpeval.yaml` directly:

        ```yaml theme={null}
        mcp:
          servers:
            # Example: fetch server
            fetch:
              command: "uvx"
              args: ["mcp-server-fetch"]
              env: 
                UV_NO_PROGRESS: "1"
            
            # Example: filesystem server
            filesystem:
              command: "npx"
              args: ["-y", "@modelcontextprotocol/server-filesystem", "/path/to/dir"]
            
            # Example: custom streamable http server
            local_http:
              description: "A streamable HTTP server."
              transport: streamable_http
              url: http://0.0.0.0:3156/mcp
              headers:
                my-header: "some_value"
        ```

        See the [Configuration Guide](./configuration#server-configuration) for all options.
      </Tab>
    </Tabs>

    ### Common server examples

    Here are some popular MCP servers you might want to test:

    <CodeGroup>
      ```bash fetch theme={null}
      # Fetch server (web content)
      uvx mcp-server-fetch
      ```

      ```bash filesystem theme={null}
      # Filesystem server
      npx -y @modelcontextprotocol/server-filesystem /path/to/directory
      ```

      ```bash github   theme={null}
      # GitHub server
      npx -y @modelcontextprotocol/server-github
      ```
    </CodeGroup>

    <Note>
      **Verify your server configuration:** After adding a server, you can verify it's working:

      ```bash theme={null}
      # List all configured servers
      mcp-eval server list

      # Validate server connectivity
      mcp-eval validate
      ```
    </Note>

    {/* TODO: Screenshot showing successful server import with list of discovered tools */}
  </Step>

  <Step title="Run your first test">
    Time for the exciting part - running your first test! We'll use the included fetch server example to demonstrate.

    <Warning>
      **Example structure:** The examples assume you have the fetch server configured. If you're testing a different server, you'll need to adjust the test code accordingly.
    </Warning>

    First, let's make sure we have an example test. If you used `mcp-eval init`, you might already have one. Otherwise, let's run:

    ```bash theme={null}
    mcp-eval run examples/mcp_server_fetch/tests/test_decorator_style.py \
      -v \
      --markdown test-reports/results.md \
      --html test-reports/index.html
    ```

    **What's happening:**

    * 🏃 Running decorator-style tests from the example file
    * 📊 Verbose output (`-v`) shows test progress
    * 📝 Markdown report for documentation
    * 🌐 HTML report for interactive exploration

    **Expected output:**

    ```
    Running tests...
    ✓ test_basic_fetch_decorator - Test basic URL fetching [2.3s]
      ✓ fetch_tool_called: Tool 'fetch' was called
      ✓ contains_domain_text: Content contains "Example Domain"
      ✓ fetch_success_rate: Tool success rate 100%

    ✓ test_content_extraction_decorator - Test extraction quality [3.1s]
      ✓ fetch_called_for_extraction: Tool 'fetch' was called
      ✓ extraction_quality_assessment: LLM judge score 0.92

    Results: 2 passed, 0 failed
    Reports saved to test-reports/
    ```

    {/* TODO: Screenshot of terminal showing colorized test run output with progress */}

    {/* TODO: Screenshot of the test summary showing passed/failed counts */}
  </Step>

  <Step title="Explore your test results">
    Open your shiny new test report to see the details:

    ```bash theme={null}
    # Open the HTML report in your browser
    open test-reports/index.html

    # Or view the markdown report
    cat test-reports/results.md
    ```

    **Understanding the HTML report:**

    The interactive report shows:

    * 📊 **Overview dashboard** - Pass/fail rates, performance metrics
    * 🔍 **Test details** - Each test with all assertions
    * 🛠️ **Tool usage** - What tools were called and when
    * 💭 **LLM reasoning** - The agent's thought process
    * ⚡ **Performance** - Response times and efficiency metrics
    * 🎯 **Failed assertions** - Detailed diffs and explanations

    **Common things to check:**

    * Did the right tools get called?
    * Was the output accurate?
    * How efficient was the agent's approach?
    * What was the LLM judge's assessment?

    {/* TODO: Screenshot of HTML report overview page showing test results summary */}

    {/* TODO: Screenshot of a failed assertion with detailed diff view */}

    {/* TODO: Screenshot of tool usage timeline visualization */}

    <Warning>
      **Test failed?** Don't worry! Check the assertion details to understand why. Common issues:

      * Tool not found (check server configuration)
      * Content mismatch (adjust your assertions)
      * Timeout (increase timeout in config)
    </Warning>
  </Step>
</Steps>

## What's next? Write your own test!

Now that you've run the example, let's write your very first custom test:

### Choose your testing style

<Tabs>
  <Tab title="Task Decorators" icon="at">
    **Best for:** Quick, readable tests

    ```python theme={null}
    from mcp_eval import task, Expect

    @task("My first test")
    async def test_my_server(agent, session):
        response = await agent.generate_str(
            "Use my tool to do something"
        )
        
        await session.assert_that(
            Expect.tools.was_called("my_tool"),
            response=response
        )
    ```
  </Tab>

  <Tab title="Pytest" icon="flask">
    **Best for:** Integration with existing pytest suites

    ```python theme={null}
    import pytest
    from mcp_eval import Expect

    @pytest.mark.asyncio
    async def test_my_server(mcp_agent):
        response = await mcp_agent.generate_str(
            "Use my tool to do something"
        )
        
        await mcp_agent.session.assert_that(
            Expect.tools.was_called("my_tool"),
            response=response
        )
    ```

    Run with uv:

    ```bash theme={null}
    uv run pytest -q tests
    # or a single file/function
    uv run pytest tests/test_my_server.py -v
    uv run pytest tests/test_my_server.py::test_my_server -v
    ```
  </Tab>
</Tabs>

### Your test file structure

Create a new test file `tests/test_my_server.py`:

```python theme={null}
"""Tests for my awesome MCP server."""

from mcp_eval import task, setup, Expect

@setup
def configure_tests():
    """Any setup needed before tests run."""
    print("🚀 Starting my server tests!")

@task("Test basic functionality")
async def test_basic_operation(agent, session):
    """Verify the server responds correctly to basic requests."""
    
    # 1. Send a prompt to the agent
    response = await agent.generate_str(
        "Please use the calculator to add 2 + 2"
    )
    
    # 2. Check that the right tool was called
    await session.assert_that(
        Expect.tools.was_called("calculate"),
        name="calculator_used"
    )
    
    # 3. Verify the response content
    await session.assert_that(
        Expect.content.contains("4"),
        name="correct_answer",
        response=response
    )
    
    # 4. Check efficiency (optional)
    await session.assert_that(
        Expect.performance.max_iterations(3),
        name="completed_efficiently"
    )

@task("Test error handling")
async def test_error_recovery(agent, session):
    """Verify graceful error handling."""
    
    response = await agent.generate_str(
        "Try to divide by zero, then recover"
    )
    
    # Use LLM judge for complex behavior
    await session.assert_that(
        Expect.judge.llm(
            rubric="Agent should handle error gracefully and provide helpful response",
            min_score=0.8
        ),
        name="error_handling_quality",
        response=response
    )
```

Run your new test:

```bash theme={null}
mcp-eval run tests/test_my_server.py -v --html reports/my_test.html
```

## Troubleshooting common issues

<Accordion title="My server isn't being found">
  **Solution:** Check your `mcpeval.yaml` to ensure the server is properly configured:

  ```yaml theme={null}
  mcp:
    servers:
      my_server:
        command: "python"
        args: ["path/to/server.py"]
  ```

  Also verify the server name matches what you're using in your agent's `server_names`.
</Accordion>

<Accordion title="Tests are timing out">
  **Solution:** Increase the timeout in your configuration:

  ```yaml theme={null}
  execution:
    timeout_seconds: 600  # 10 minutes
  ```
</Accordion>

<Accordion title="API key errors">
  **Solution:** Ensure your API key is set correctly:

  ```bash theme={null}
  # Check if it's set
  echo $ANTHROPIC_API_KEY

  # Or add to mcpeval.secrets.yaml
  anthropic:
    api_key: "sk-ant-..."
  ```
</Accordion>

## Resources to level up

Ready to become a `mcp-eval` expert? Here's your learning path:

<CardGroup cols={2}>
  <Card title="Complete Examples" href="./examples" icon="code">
    Full test suites showing all testing patterns
  </Card>

  <Card title="Common Workflows" href="./common-workflows" icon="arrows-turn-to-dots">
    Step-by-step guides for typical testing scenarios
  </Card>

  <Card title="Configuration Guide" href="./configuration" icon="gear">
    Deep dive into all configuration options
  </Card>

  <Card title="Best Practices" href="./best-practices" icon="star">
    Pro tips for writing maintainable tests
  </Card>
</CardGroup>

## Get help

* 💬 **Questions?** Check our [FAQ](./faq) or [troubleshooting guide](./troubleshooting)
* 🐛 **Found a bug?** [Report it on GitHub](https://github.com/lastmile-ai/mcp-eval/issues)
* 💡 **Have ideas?** We'd love to hear them in [discussions](https://github.com/lastmile-ai/mcp-eval/discussions)

***

**Congratulations!** 🎉 You've successfully set up `mcp-eval` and run your first tests. You're now ready to ensure your MCP servers and agents work flawlessly. Happy testing!