CI/CD

Run action

The recommended approach is to use the reusable workflow which handles all the setup, testing, and deployment. For the complete workflow configuration, visit mcpeval.yml.

name: MCP-Eval CI

on:
  push:
    branches: [main, master, trunk]
  workflow_dispatch:

jobs:
  call-mcpeval:
    uses: lastmile-ai/mcp-eval/.github/workflows/mcpeval-reusable.yml
    with:
      deploy-pages: true
    permissions:
      contents: read
      pages: write
      id-token: write
      pull-requests: write
    secrets: inherit

Alternatively, you can directly use the action in your workflow:

steps:
  - name: Checkout
    uses: actions/checkout@v4

  - name: Run MCP-Eval (uv)
    id: mcpeval
    uses: lastmile-ai/mcp-eval/.github/actions/mcp-eval/run
    with:
      python-version: "3.11"
      working-directory: .
      run-args: "-v"
      tests: tests/
      reports-dir: mcpeval-reports
      json-report: mcpeval-results.json
      markdown-report: mcpeval-results.md
      html-report: mcpeval-results.html
      artifact-name: mcpeval-artifacts
      pr-comment: "true"
      set-summary: "true"
      upload-artifacts: "true"
    env:
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

  - name: Generate and upload badges
    uses: lastmile-ai/mcp-eval/.github/actions/mcp-eval/badges
    with:
      report-path: ${{ steps.mcpeval.outputs.results-json-path }}
      output-dir: badges
      format: "both"
      upload-artifacts: "true"
      artifact-name: mcpeval-badges

Sources:

Action: action.yaml
README: run/README.md
Workflows: mcpeval.yml, mcpeval-reusable.yml

Publish HTML and Badges via Pages

The workflow automatically deploys both the HTML report and badges to GitHub Pages when pushing to main/master branches. After deployment, badges are accessible at https://<username>.github.io/<repo>/badges/. To enable GitHub Pages deployment:

Enable GitHub Pages in your repository settings
The workflow will automatically deploy on pushes to main/master
Badges and reports will be available at your Pages URL

Badges

After deployment to GitHub Pages, reference badges using your Pages URL:

[![mcp-tests](https://img.shields.io/endpoint?url=https://YOUR_USERNAME.github.io/YOUR_REPO/badges/mcp-tests.json&cacheSeconds=300)](https://YOUR_USERNAME.github.io/YOUR_REPO/)
[![mcp-cov](https://img.shields.io/endpoint?url=https://YOUR_USERNAME.github.io/YOUR_REPO/badges/mcp-cov.json&cacheSeconds=300)](https://YOUR_USERNAME.github.io/YOUR_REPO/)

For example,

Getting Started

Core Concepts

Writing Tests

Building with LLMs

Evaluation Guides

Configuration

CI/CD & Deployment

Test Reporting

API Reference

CLI Reference

Resources

Run action

Publish HTML and Badges via Pages

Badges

​Run action

​Publish HTML and Badges via Pages

​Badges

Run action

Publish HTML and Badges via Pages

Badges