Security Guide - mcp-eval

🔒 Security first! This guide covers essential security practices for mcp-eval, from protecting API keys to ensuring compliance with security standards. Keep your tests and data safe.

Security quick reference

Jump to what you need:

API Keys

Secure credential handling

Data Protection

Encrypt and protect test data

Network Security

Secure communications

Access Control

Authentication and authorization

Compliance

Meet regulatory requirements

Incident Response

Handle security incidents

API key management

Never commit secrets

🚨 Critical: Never commit API keys to version control!

# .gitignore - ALWAYS include these
mcpeval.secrets.yaml
mcp-agent.secrets.yaml
.env
.env.local
*.key
*.pem
secrets/
.anthropic/
.openai/

Secure storage options

Environment variables (basic)

# Set in shell profile (~/.bashrc, ~/.zshrc)
export ANTHROPIC_API_KEY="sk-ant-..."
export OPENAI_API_KEY="sk-..."

# Never log or print these!
echo $ANTHROPIC_API_KEY  # DON'T DO THIS

Secrets file (better)

# mcpeval.secrets.yaml
anthropic:
  api_key: "sk-ant-..."  # Encrypted at rest
  
# Set file permissions (Unix/Linux)
chmod 600 mcpeval.secrets.yaml  # Owner read/write only

Secret management systems (best)

AWS Secrets Manager
HashiCorp Vault
Azure Key Vault

# integrations/aws_secrets.py
import boto3
import json

class AWSSecretManager:
    """Fetch secrets from AWS Secrets Manager."""
    
    def __init__(self, region='us-east-1'):
        self.client = boto3.client('secretsmanager', region_name=region)
    
    def get_api_key(self, secret_name):
        """Retrieve API key from AWS Secrets Manager."""
        try:
            response = self.client.get_secret_value(SecretId=secret_name)
            secret = json.loads(response['SecretString'])
            return secret['api_key']
        except Exception as e:
            # Log error without exposing secret
            print(f"Failed to retrieve secret: {type(e).__name__}")
            raise

# Use in configuration
from mcp_eval.config import set_settings

secret_mgr = AWSSecretManager()
anthropic_key = secret_mgr.get_api_key('mcp-eval/anthropic')

set_settings({
    'anthropic': {'api_key': anthropic_key}
})

# integrations/vault_secrets.py
import hvac

class VaultSecretManager:
    """Fetch secrets from HashiCorp Vault."""
    
    def __init__(self, vault_url, token):
        self.client = hvac.Client(url=vault_url, token=token)
    
    def get_api_key(self, path):
        """Retrieve API key from Vault."""
        response = self.client.secrets.kv.v2.read_secret_version(
            path=path
        )
        return response['data']['data']['api_key']

# Use in configuration
vault = VaultSecretManager(
    vault_url='https://vault.company.com',
    token=os.environ['VAULT_TOKEN']
)

anthropic_key = vault.get_api_key('secret/mcp-eval/anthropic')

# integrations/azure_secrets.py
from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential

class AzureSecretManager:
    """Fetch secrets from Azure Key Vault."""
    
    def __init__(self, vault_url):
        credential = DefaultAzureCredential()
        self.client = SecretClient(
            vault_url=vault_url,
            credential=credential
        )
    
    def get_api_key(self, secret_name):
        """Retrieve API key from Azure Key Vault."""
        secret = self.client.get_secret(secret_name)
        return secret.value

# Use in configuration
azure_vault = AzureSecretManager(
    vault_url='https://myvault.vault.azure.net/'
)

anthropic_key = azure_vault.get_api_key('anthropic-api-key')

API key rotation

Implement regular key rotation:

# security/key_rotation.py
import datetime
import secrets

class APIKeyRotation:
    """Manage API key rotation."""
    
    def __init__(self, secret_manager):
        self.secret_manager = secret_manager
    
    def should_rotate(self, key_name, max_age_days=90):
        """Check if key needs rotation."""
        metadata = self.secret_manager.get_metadata(key_name)
        created_date = metadata['created_date']
        age = (datetime.now() - created_date).days
        return age > max_age_days
    
    def rotate_key(self, key_name, provider):
        """Rotate an API key."""
        # Generate new key from provider
        new_key = provider.generate_new_key()
        
        # Store new key
        self.secret_manager.update_secret(key_name, new_key)
        
        # Deactivate old key (after grace period)
        provider.schedule_deactivation(old_key, grace_hours=24)
        
        # Log rotation (without exposing keys)
        self.log_rotation(key_name)
        
        return new_key

Data protection

Encrypt sensitive test data

# security/encryption.py
from cryptography.fernet import Fernet
import json
import base64

class TestDataEncryption:
    """Encrypt sensitive test data."""
    
    def __init__(self, key=None):
        if key:
            self.cipher = Fernet(key)
        else:
            # Generate new key
            self.cipher = Fernet(Fernet.generate_key())
    
    def encrypt_test_data(self, data):
        """Encrypt test data."""
        json_str = json.dumps(data)
        encrypted = self.cipher.encrypt(json_str.encode())
        return base64.b64encode(encrypted).decode()
    
    def decrypt_test_data(self, encrypted_data):
        """Decrypt test data."""
        decoded = base64.b64decode(encrypted_data)
        decrypted = self.cipher.decrypt(decoded)
        return json.loads(decrypted.decode())

# Usage in tests
encryptor = TestDataEncryption()

# Encrypt sensitive test inputs
sensitive_data = {
    "user_id": "12345",
    "ssn": "123-45-6789",
    "credit_card": "4111-1111-1111-1111"
}

encrypted = encryptor.encrypt_test_data(sensitive_data)

# Use encrypted data in test
@task("Test with encrypted data")
async def test_sensitive_operation(agent, session):
    # Decrypt only when needed
    data = encryptor.decrypt_test_data(encrypted)
    
    # Use data in test
    response = await agent.generate_str(
        f"Process user {data['user_id']}"
    )
    
    # Clear sensitive data from memory
    del data

Sanitize test outputs

# security/sanitization.py
import re

class OutputSanitizer:
    """Sanitize sensitive information from test outputs."""
    
    # Patterns for sensitive data
    PATTERNS = {
        'api_key': r'(sk-[a-zA-Z0-9]{48})',
        'email': r'([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,})',
        'ssn': r'\b(\d{3}-\d{2}-\d{4})\b',
        'credit_card': r'\b(\d{4}[-\s]?\d{4}[-\s]?\d{4}[-\s]?\d{4})\b',
        'ip_address': r'\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b',
    }
    
    def sanitize(self, text):
        """Remove sensitive information from text."""
        sanitized = text
        
        for name, pattern in self.PATTERNS.items():
            sanitized = re.sub(
                pattern,
                f'[REDACTED_{name.upper()}]',
                sanitized
            )
        
        return sanitized

# Use in test reporting
sanitizer = OutputSanitizer()

@task("Test with sanitized output")
async def test_with_sanitization(agent, session):
    response = await agent.generate_str("Get user data")
    
    # Sanitize before logging or reporting
    safe_response = sanitizer.sanitize(response)
    print(f"Response: {safe_response}")
    
    # Original response for assertions
    await session.assert_that(
        Expect.content.contains("user_id"),
        response=response
    )

Secure file handling

# security/secure_files.py
import tempfile
import shutil
from pathlib import Path

class SecureFileHandler:
    """Handle test files securely."""
    
    def create_secure_temp_file(self, content, prefix="test_"):
        """Create temporary file with secure permissions."""
        # Create file with restricted permissions
        fd, path = tempfile.mkstemp(prefix=prefix)
        
        try:
            # Write content
            with os.fdopen(fd, 'w') as f:
                f.write(content)
            
            # Set restrictive permissions (owner only)
            os.chmod(path, 0o600)
            
            return path
        except Exception as e:
            # Clean up on error
            os.unlink(path)
            raise
    
    def secure_cleanup(self, path):
        """Securely delete file."""
        if Path(path).exists():
            # Overwrite with random data before deletion
            with open(path, 'ba+', buffering=0) as f:
                length = f.tell()
                f.seek(0)
                f.write(os.urandom(length))
            
            # Remove file
            os.unlink(path)

Network security

TLS/SSL configuration

# mcpeval.yaml - Secure network settings
network:
  # Enforce TLS
  require_tls: true
  min_tls_version: "1.2"
  
  # Certificate verification
  verify_certificates: true
  ca_bundle: "/path/to/ca-certificates.crt"
  
  # Client certificates
  client_cert: "/path/to/client.crt"
  client_key: "/path/to/client.key"
  
  # Timeouts
  connect_timeout: 30
  read_timeout: 60

Secure proxy configuration

# security/proxy.py
import os

class SecureProxy:
    """Configure secure proxy settings."""
    
    @staticmethod
    def configure():
        """Set up secure proxy configuration."""
        proxy_url = os.environ.get('HTTPS_PROXY')
        
        if proxy_url:
            # Parse and validate proxy URL
            from urllib.parse import urlparse
            parsed = urlparse(proxy_url)
            
            # Ensure HTTPS proxy
            if parsed.scheme != 'https':
                raise ValueError("Only HTTPS proxies are allowed")
            
            # Set proxy with authentication if needed
            if parsed.username and parsed.password:
                # Don't log credentials!
                proxy_auth = f"{parsed.username}:***@"
            else:
                proxy_auth = ""
            
            safe_proxy = f"https://{proxy_auth}{parsed.hostname}:{parsed.port}"
            
            return {
                'http': safe_proxy,
                'https': safe_proxy,
                'no_proxy': 'localhost,127.0.0.1'
            }
        
        return None

Network isolation

# docker-compose.security.yml
version: '3.8'

services:
  mcp-eval:
    image: mcp-eval:latest
    networks:
      - test-network
    security_opt:
      - no-new-privileges:true
    cap_drop:
      - ALL
    cap_add:
      - NET_BIND_SERVICE
    read_only: true
    tmpfs:
      - /tmp
      - /var/run

networks:
  test-network:
    driver: bridge
    internal: true  # No external access
    ipam:
      config:
        - subnet: 172.28.0.0/24

Access control

Role-based access

# security/rbac.py
from enum import Enum
from functools import wraps

class Role(Enum):
    ADMIN = "admin"
    DEVELOPER = "developer"
    TESTER = "tester"
    VIEWER = "viewer"

class Permissions(Enum):
    RUN_TESTS = "run_tests"
    VIEW_RESULTS = "view_results"
    MODIFY_CONFIG = "modify_config"
    ACCESS_SECRETS = "access_secrets"

# Role permissions mapping
ROLE_PERMISSIONS = {
    Role.ADMIN: [
        Permissions.RUN_TESTS,
        Permissions.VIEW_RESULTS,
        Permissions.MODIFY_CONFIG,
        Permissions.ACCESS_SECRETS
    ],
    Role.DEVELOPER: [
        Permissions.RUN_TESTS,
        Permissions.VIEW_RESULTS,
        Permissions.MODIFY_CONFIG
    ],
    Role.TESTER: [
        Permissions.RUN_TESTS,
        Permissions.VIEW_RESULTS
    ],
    Role.VIEWER: [
        Permissions.VIEW_RESULTS
    ]
}

def require_permission(permission):
    """Decorator to check permissions."""
    def decorator(func):
        @wraps(func)
        def wrapper(user, *args, **kwargs):
            if not has_permission(user, permission):
                raise PermissionError(
                    f"User {user.name} lacks {permission.value} permission"
                )
            return func(user, *args, **kwargs)
        return wrapper
    return decorator

def has_permission(user, permission):
    """Check if user has permission."""
    user_permissions = ROLE_PERMISSIONS.get(user.role, [])
    return permission in user_permissions

# Usage
class User:
    def __init__(self, name, role):
        self.name = name
        self.role = role

@require_permission(Permissions.RUN_TESTS)
def run_tests(user, test_suite):
    """Run tests - requires RUN_TESTS permission."""
    print(f"{user.name} running {test_suite}")

Authentication

# security/auth.py
import jwt
import datetime
from passlib.hash import bcrypt

class Authentication:
    """Handle user authentication."""
    
    def __init__(self, secret_key):
        self.secret_key = secret_key
    
    def hash_password(self, password):
        """Hash password securely."""
        return bcrypt.hash(password)
    
    def verify_password(self, password, hashed):
        """Verify password against hash."""
        return bcrypt.verify(password, hashed)
    
    def generate_token(self, user_id, role, expires_hours=24):
        """Generate JWT token."""
        payload = {
            'user_id': user_id,
            'role': role,
            'exp': datetime.datetime.utcnow() + 
                   datetime.timedelta(hours=expires_hours)
        }
        return jwt.encode(payload, self.secret_key, algorithm='HS256')
    
    def verify_token(self, token):
        """Verify and decode JWT token."""
        try:
            payload = jwt.decode(
                token,
                self.secret_key,
                algorithms=['HS256']
            )
            return payload
        except jwt.ExpiredSignatureError:
            raise ValueError("Token has expired")
        except jwt.InvalidTokenError:
            raise ValueError("Invalid token")

Compliance and auditing

Audit logging

# security/audit.py
import json
import datetime
from pathlib import Path

class AuditLogger:
    """Log security-relevant events."""
    
    def __init__(self, log_file="audit.log"):
        self.log_file = Path(log_file)
        # Ensure log file has restrictive permissions
        self.log_file.touch(mode=0o600)
    
    def log_event(self, event_type, user, details, status="success"):
        """Log an audit event."""
        event = {
            "timestamp": datetime.datetime.utcnow().isoformat(),
            "event_type": event_type,
            "user": user,
            "status": status,
            "details": self._sanitize_details(details)
        }
        
        # Append to audit log
        with open(self.log_file, 'a') as f:
            f.write(json.dumps(event) + '\n')
    
    def _sanitize_details(self, details):
        """Remove sensitive data from details."""
        # Remove any API keys or secrets
        sanitized = {}
        for key, value in details.items():
            if 'key' in key.lower() or 'secret' in key.lower():
                sanitized[key] = "[REDACTED]"
            else:
                sanitized[key] = value
        return sanitized
    
    def log_test_run(self, user, test_suite, results):
        """Log test execution."""
        self.log_event(
            "test_run",
            user,
            {
                "test_suite": test_suite,
                "passed": results['passed'],
                "failed": results['failed'],
                "duration_ms": results['duration_ms']
            }
        )
    
    def log_config_change(self, user, config_path, changes):
        """Log configuration changes."""
        self.log_event(
            "config_change",
            user,
            {
                "config_file": config_path,
                "changes": changes
            }
        )
    
    def log_secret_access(self, user, secret_name, purpose):
        """Log access to secrets."""
        self.log_event(
            "secret_access",
            user,
            {
                "secret_name": secret_name,
                "purpose": purpose
            }
        )

Compliance checks

# security/compliance.py

class ComplianceChecker:
    """Check compliance with security standards."""
    
    def check_gdpr_compliance(self, test_data):
        """Verify GDPR compliance."""
        issues = []
        
        # Check for PII
        if self._contains_pii(test_data):
            issues.append("Test data contains PII - ensure consent")
        
        # Check data retention
        if not self._has_retention_policy(test_data):
            issues.append("No data retention policy defined")
        
        # Check encryption
        if not self._is_encrypted(test_data):
            issues.append("PII data not encrypted")
        
        return issues
    
    def check_sox_compliance(self, audit_logs):
        """Verify SOX compliance for financial data."""
        issues = []
        
        # Check audit trail completeness
        if not self._complete_audit_trail(audit_logs):
            issues.append("Incomplete audit trail")
        
        # Check access controls
        if not self._proper_access_controls():
            issues.append("Insufficient access controls")
        
        return issues
    
    def check_hipaa_compliance(self, test_data):
        """Verify HIPAA compliance for health data."""
        issues = []
        
        # Check for PHI
        if self._contains_phi(test_data):
            # Verify encryption
            if not self._is_encrypted_hipaa(test_data):
                issues.append("PHI not properly encrypted")
            
            # Check access logs
            if not self._has_access_logs():
                issues.append("PHI access not logged")
        
        return issues

Security testing

Security-focused test cases

# tests/test_security.py
from mcp_eval import task, Expect

@task("Test SQL injection prevention")
async def test_sql_injection(agent, session):
    """Verify protection against SQL injection."""
    
    # Attempt SQL injection
    malicious_input = "'; DROP TABLE users; --"
    response = await agent.generate_str(
        f"Search for user: {malicious_input}"
    )
    
    # Should handle safely
    await session.assert_that(
        Expect.judge.llm(
            "Response shows input was sanitized and no SQL was executed",
            min_score=0.9
        ),
        response=response
    )
    
    # Verify no actual damage
    await session.assert_that(
        Expect.tools.failed("database"),  # DB operation should fail safely
        name="injection_prevented"
    )

@task("Test XSS prevention")
async def test_xss_prevention(agent, session):
    """Verify protection against XSS attacks."""
    
    xss_payload = "<script>alert('XSS')</script>"
    response = await agent.generate_str(
        f"Display message: {xss_payload}"
    )
    
    # Should escape or sanitize
    await session.assert_that(
        Expect.content.not_contains("<script>"),
        name="script_tag_escaped",
        response=response
    )

@task("Test authentication bypass attempts")
async def test_auth_bypass(agent, session):
    """Test resistance to authentication bypass."""
    
    bypass_attempts = [
        "admin' OR '1'='1",
        "admin'--",
        "../../../etc/passwd"
    ]
    
    for attempt in bypass_attempts:
        response = await agent.generate_str(
            f"Login as: {attempt}"
        )
        
        await session.assert_that(
            Expect.judge.llm(
                "Login attempt was properly rejected",
                min_score=0.9
            ),
            name=f"blocked_{attempt[:10]}",
            response=response
        )

Incident response

Security incident handling

# security/incident_response.py
import datetime
from enum import Enum

class IncidentSeverity(Enum):
    CRITICAL = 1
    HIGH = 2
    MEDIUM = 3
    LOW = 4

class SecurityIncidentHandler:
    """Handle security incidents."""
    
    def __init__(self, notifier, logger):
        self.notifier = notifier
        self.logger = logger
    
    async def handle_incident(self, incident_type, severity, details):
        """Respond to security incident."""
        
        incident_id = self._generate_incident_id()
        
        # Log incident
        self.logger.log_incident(
            incident_id,
            incident_type,
            severity,
            details
        )
        
        # Take immediate action based on severity
        if severity == IncidentSeverity.CRITICAL:
            await self._critical_response(incident_id, details)
        elif severity == IncidentSeverity.HIGH:
            await self._high_response(incident_id, details)
        
        # Notify stakeholders
        await self.notifier.send_incident_alert(
            incident_id,
            incident_type,
            severity,
            details
        )
        
        return incident_id
    
    async def _critical_response(self, incident_id, details):
        """Handle critical incidents."""
        # Immediate actions for critical incidents
        actions = []
        
        # Rotate potentially compromised keys
        if 'api_key' in details.get('affected_resources', []):
            actions.append("Rotate all API keys")
            await self._rotate_all_keys()
        
        # Disable affected accounts
        if 'user_account' in details.get('affected_resources', []):
            actions.append("Disable affected accounts")
            await self._disable_accounts(details['affected_users'])
        
        # Stop all test executions
        actions.append("Halt all test executions")
        await self._stop_all_tests()
        
        return actions
    
    async def _rotate_all_keys(self):
        """Emergency key rotation."""
        # Implementation for key rotation
        pass

Security checklist

Use this checklist for security reviews:

Configuration security

API keys stored securely (not in code)
Secrets files excluded from version control
File permissions set correctly (600 for secrets)
TLS/SSL enabled and configured
Network isolation implemented

Data security

Test data encrypted at rest
Sensitive data sanitized in logs
PII/PHI handled according to regulations
Secure file deletion implemented
Data retention policies defined

Access control

Authentication required for sensitive operations
Role-based access control implemented
Audit logging enabled
Session timeout configured
Multi-factor authentication available

Compliance

GDPR requirements met (if applicable)
HIPAA requirements met (if applicable)
SOX requirements met (if applicable)
Security testing included in CI/CD
Regular security audits scheduled

Incident response

Incident response plan documented
Alert notifications configured
Backup and recovery procedures tested
Security patches applied regularly
Vulnerability scanning automated

Security resources

OWASP Top 10

Essential security risks to address

CIS Controls

Security best practices framework

NIST Framework

Cybersecurity framework guidelines

Security Tools

Open source security tools

Stay secure! Following these security practices ensures your mcp-eval testing environment remains protected while maintaining compliance with security standards. Remember: security is everyone’s responsibility! 🔒

Getting Started

Core Concepts

Writing Tests

Building with LLMs

Evaluation Guides

Configuration

CI/CD & Deployment

Test Reporting

API Reference

CLI Reference

Resources

​Security quick reference

API Keys

Data Protection

Network Security

Access Control

Compliance

Incident Response

​API key management

​Never commit secrets

​Secure storage options

​Environment variables (basic)

​Secrets file (better)

​Secret management systems (best)

​API key rotation

​Data protection

​Encrypt sensitive test data

​Sanitize test outputs

​Secure file handling

​Network security

​TLS/SSL configuration

​Secure proxy configuration

​Network isolation

​Access control

​Role-based access

​Authentication

​Compliance and auditing

​Audit logging

​Compliance checks

​Security testing

​Security-focused test cases

​Incident response

​Security incident handling

​Security checklist

​Configuration security

​Data security

​Access control

​Compliance

​Incident response

​Security resources

OWASP Top 10

CIS Controls

NIST Framework

Security Tools

Security quick reference

API key management

Never commit secrets

Secure storage options

Environment variables (basic)

Secrets file (better)

Secret management systems (best)

API key rotation

Data protection

Encrypt sensitive test data

Sanitize test outputs

Secure file handling

Network security

TLS/SSL configuration

Secure proxy configuration

Network isolation

Access control

Role-based access

Authentication

Compliance and auditing

Audit logging

Compliance checks

Security testing

Security-focused test cases

Incident response

Security incident handling

Security checklist

Configuration security

Data security

Access control

Compliance

Incident response

Security resources