> ## Documentation Index
> Fetch the complete documentation index at: https://mcp-eval.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets

> Define Cases and Datasets for systematic evaluation; run programmatically or from files.

### API

* `Case[Input, Output, Metadata]`
* `Dataset[Input, Output, Metadata]`

Source: [datasets.py](https://github.com/lastmile-ai/mcp-eval/blob/main/src/mcp_eval/datasets.py)

### Programmatic

```python theme={null}
from mcp_eval import Case, Dataset, ToolWasCalled, ResponseContains

cases = [
  Case(
    name="fetch_example",
    inputs="Fetch https://example.com",
    evaluators=[ToolWasCalled("fetch"), ResponseContains("Example Domain")],
  )
]

dataset = Dataset(name="Fetch Suite", cases=cases)
report = await dataset.evaluate(lambda inputs, agent, session: agent.generate_str(inputs))
report.print(include_input=True, include_output=True)
```

Parallel evaluation:

```python theme={null}
report = await dataset.evaluate(
  lambda inputs, agent, session: agent.generate_str(inputs),
  max_concurrency=4,
)
```

### YAML/JSON

Save/load via `Dataset.to_file` and `Dataset.from_file`. Schema: [mcpeval.config.schema.json](https://github.com/lastmile-ai/mcp-eval/blob/main/schema/mcpeval.config.schema.json).

YAML example (from [basic\_fetch\_dataset.yaml](https://github.com/lastmile-ai/mcp-eval/blob/main/examples/mcp_server_fetch/datasets/basic_fetch_dataset.yaml)):

```yaml theme={null}
name: "Basic Fetch Dataset"
server_name: "fetch"
cases:
  - name: "simple_fetch"
    inputs: "Fetch https://example.com"
    expected_output: "Example Domain"
    evaluators:
      - ToolWasCalled:
          tool_name: "fetch"
      - ResponseContains:
          text: "Example Domain"
```

### Concurrency

`Dataset.evaluate(..., max_concurrency=N)` runs cases in parallel.

### Examples

* [test\_dataset\_style.py](https://github.com/lastmile-ai/mcp-eval/blob/main/examples/mcp_server_fetch/tests/test_dataset_style.py)
* [datasets/](https://github.com/lastmile-ai/mcp-eval/tree/main/examples/mcp_server_fetch/datasets/)