🛠️ Installation

The framework requires Python version >= 3.11, nodejs and jq.

conda create -n mcpbench python=3.11 -y
conda activate mcpbench
pip install -r requirements.txt

🚀 Quick Start

Please first determine the type of MCP server you want to use:

• If it is a remote host (accessed via SSE, such as ModelScope, Smithery, or localhost), you can directly conduct the evaluation.
• If it is started locally (accessed via npx using STDIO), you need to launch it.

Launch MCP Server (optional for stdio)

First, you need to write the following configuration:

{
    "mcp_pool": [
        {
            "name": "firecrawl",
            "run_config": [
                {
                    "command": "npx -y firecrawl-mcp",
                    "args": "FIRECRAWL_API_KEY=xxx",
                    "port": 8005
                }
            ]
        }  
    ]
}

Save this config file in the configs folder and launch it using:

sh launch_mcps_as_sse.sh YOUR_CONFIG_FILE

For example, save the above configuration in the configs/firecrawl.json file and launch it using:

sh launch_mcps_as_sse.sh firecrawl.json

Launch Evaluatio...