GitTaskBench

👋 To address this gap, we introduce GitTaskBench. Our benchmark focuses on tasks whose complexity and practical value demand leveraging repository-level code, mirroring how developers solve real problems using existing GitHub projects.

🔍 We carefully selected 54 representative tasks with real-world economic value, and for each task, searched and identified a corresponding GitHub repository that meets strict selection criteria (the repository for each task is fixed to ensure benchmark completeness, as some agent frameworks do not support searching for appropriate repositories). This setup allows us to systematically evaluate LLM agents' ability to utilize open-source repositories to solve complex, realistic problems.

👉 By doing so, GitTaskBench offers a more authentic and comprehensive assessment of agent performance in practical, repository-driven environments.

🚀 How to Run

⚡ If you only want to know how to use GitTaskBench, start here.

0. Directory structure

└── QuantaAlpha/GitTaskBench/

├── README.md
├── setup.py
├── requirements.txt
├── Task_Success_Criteria.xlsx   # listed clearly
├── code_base/                   # all used repositories
│   ├── AnimeGANv3/
│   └── ...
├── queries/                     # all task definitions
│   ├── AnimeGANv3_01/
│   │   └── query.json
│   ├── AnimeGANv3_02/
│   │   └── query.json
│   └── ...
├── run_auto_prompt/             # generate all prompts
│   ├── new_run_setup.py
│   └── get_new_run_prompt.sh
├── Aider/                       # agent framework
│   └── ... 
├── SWE_agent/                   # agent framework
│   └── ... 
├── OpenHands/                   # agent framework
│   └── ...
├── config/                      # task evaluation configs
│   ├── AnimeGANv3_01/
│   │   └── task_info.yaml
│   ├── AnimeGANv3_02/
│   │   └── task_info.yaml
│   ├── AnimeGANv3_03/
│   └── ...
├── groundtruth/                 # ground truth
│   ├── Trafilatura_02/
│   │   └── gt.md
│   └── Trafilatura_03/...
├── output_for_show/             #  agent's outputs
│   ├── AnimeGANv3_01/
│   │   └── output.png
│   └── AnimeGANv3_02/...
├── gittaskbench/                # evaluation settings
│   ├── __init__.py
│   └── ...
├── test_scripts/                # test scripts
│   ├── AnimeGANv3_01/
│   │   └── test_script.py
│   ├── AnimeGANv3_02/
│   │   └── test_script.py
│   └──...
├── test_results_for_show/       # analysis results
│   ├── AnimeGANv3_02/
│   │   └── results.jsonl
│   └──...
└── test_reports/                # summary report
    ├── evaluation_report_openhands_gpt4o_100iters.txt
    ├── evaluation_report_openhands_gpt4o_70iters.txt
    ├── evaluation_report_openhands_gpt4o_30iters.txt
    └── ...

1. Set Up ⚙️

GitTaskBench offers easy-to-use shell commands to ensure reproducible evaluations. To build GitTaskBench from source, follow bellow steps.

First, create a new conda environment:

conda create -n gittaskbench python=3.10 -y
conda activate gittaskbench

pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 \
  --extra-index-url https://download.pytorch.org/whl/cu113

Then, you can install gittaskbench with pip:

git clone https://github.com/your-org/GitTaskBench.git
cd GitTaskBench
# config
pip install -e .

also you can

# config
pip install -r requirements.txt

2. Quick Start 💡

• Single Task Evaluation:

If you need to evaluate a single, specific task, you can use the following command. The example below shows how to evaluate the Trafilatura_01 task:

cd GitTaskBench
# The outputs are saved in the DEFAULT "./output" directory, for example: "./output/Trafilatura_01/output.txt"

gittaskbench grade --taskid Trafilatura_01

Running the command will produce an analysis report (.jsonl) at the DEFAULT path (./test_results/Trafilatura_01). See test_results_for_show/ for a sample.

The complete commands can be found in the 🤖 Automation Evaluation section.

• All Tasks Evaluation

When you need to evaluate all tasks, you can use the --all parameter. This command will automatically iterate through and execute the evaluation of all tasks:

gittaskbench grade --all

• Test Results Analysis

After completing the evaluation, if you want to analyze & summary the test results, you can use the statistics command. This command will analyze & summary the evaluation results in the specified directory and output an analysis report (.txt):

gittaskbench eval

See test_reports/ for a sample.

👉 That’s it. With the above commands you can run, and analyze the agent performance on GitTaskBench.

📊 Benchmark Overview

GitTaskBench is a comprehensive benchmark designed to evaluate the capabilities of intelligent agents across multiple modalities and task complexities. It encompasses 54 tasks spanning 7 key domains.

Each domain features a curated set of tasks that reflect real-world applications and research challenges. These tasks assess an agent's autonomous ability to interpret complex instructions, process multi-modal inputs, perform reasoning, understand and explore the GitHub repositories, and deliver accurate, meaningful outputs.

The GitTaskBench data curation and processing pipeline is illustrated below.

✅ Task Distribution

| Domain | Task List | |----------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Image Processing | Style Transfer, Image Coloring, Image Restoration, Scratch Detection, Image Enhancement, Background Processing, Watermark Embedding | | Video Processing | Video Action Analysis, Style Transfer, Video Coloring | | Speech Processing | Speech Recognition, Speech Separation, Speech Enhancement, Noise Reduction, Speech Analysis | | Physiological Signals Processing | EDA (Electrodermal Activity) Data Analysis, ECG (Electrocardiogram) Data Analysis, EOG (Electrooculogram) Data Analysis | | Security and