OpenVLHarness

✨ Highlights

• Training-free. Works with the backbone's existing weights through native tool calling; nothing is fine-tuned.
• Nine capabilities. Visual grounding, OCR, depth, camera trajectory, zoom, text search, image search, webpage visits and a Python coding agent, each behind a task-level interface that returns evidence the model can read.
• Multimodal memory. Images, boxes, masks, depth maps and web references stay addressable for the whole question, so later tools and code can build on them.
• Consistent gains. Across 22 benchmarks in four domains, seven backbones gain 9.7 to 12.6 points on average over the same models without tools.
• Any registered model. Presets for seven of the paper's eight backbones, plus every Qwen3-VL Instruct size and OpenAI's GPT-5 to GPT-6.1 models in the model registry.

🗂️ Table of Contents

• How it works
• Results
• Installation
• Quick start
• Evaluation
• Configure and extend
• Repository layout
• Documentation and tests
• Misc
• Citation

🔍 How it works

The model chooses tools, inspects their evidence, and continues until it can answer. Three components let it combine evidence across steps:

• Capability layer: nine task-level interfaces turn tool outputs into readable observations, including annotated images, summaries, and compact numerical data.
• Multimodal memory: images, structured data, and web references remain addressable throughout a question. Each new question starts with fresh memory.
• Python Coding Agent: the backbone writes code in a separate context to calculate over stored boxes, masks, depth maps, and other tool results.

For example, an agent can locate objects, estimate depth, and compute their spatial relationship; or crop an unfamiliar object, search for visual matches, and verify the answer on a webpage.

Every preset uses Qwen3-VL-8B-Instruct for multimodal search summarization and webpage extraction, independently of the main model. The coding agent uses the same backbone as the main model.

📈 Results

Average score over the paper's 22 benchmarks (counting and grounding, multimodal search, general VQA and hallucination, and spatial understanding):

Qwen3-VL results are the mean of three runs. Per-benchmark results, comparisons with other harnesses, and recorded trajectories are on the project page.

🛠️ Installation

Use Python 3.10 or newer. From a source checkout:

git clone https://github.com/uclanlp/OpenVLHarness.git
cd OpenVLHarness
python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[qwen,server]'

This installs the agent, Qwen tokenizer support, and the tool router. The model servers have separate environments and dependencies. For a client connecting to an existing router, pip install -e . is sufficient; add the qwen extra when using Qwen for local context counting or raw trace export. Add the eval extra to run benchmarks (see Evaluation).

🚀 Quick start

1. Pick a preset

The release includes agent presets for seven of the paper's eight backbones:

The paper's Qwen2.5-VL-7B-Instruct results, compared with training-based harnesses in its appendix, used a separate configuration that this release does not include.

The registry also covers every Qwen3-VL Instruct size (2B to 235B-A22B, with FP8 builds) and OpenAI's GPT-5 to GPT-6.1 models; to use one, copy its family's preset and change one line (other registered models).

2. Prepare models and tools

To run all nine capabilities, prepare these services before starting the agent:

• Models: serve Qwen locally or connect to a GPT or Kimi endpoint, following the model setup guide. Every preset also needs a Qwen3-VL-8B-Instruct endpoint for search tools; the 8B preset can share it with the main model, while the 32B preset needs a second server for it. Sharing one server is the simplest deployment and scores within run-to-run noise of the paper setup; its cost is throughput, since search summarization and webpage extraction queue behind agent turns on the same GPU. To reproduce the paper exactly, or to keep agent throughput at higher concurrency, serve a second Qwen3-VL-8B instance and point AUXILIARY_MODEL_BASE_URL at it. The guide includes the tested model revision, serving versions, and GPU configuration.
• Perception: launch SAM 3, PaddleOCR-VL, and Depth Anything 3 using the tool service guide. These run as separate GPU services behind the router.
• Search: the three search tools call Serper, a paid Google Search API with a free starter quota: web/image/news search, Google Lens reverse image search, and page scraping for webpage visits. Configure its key using SERPER_API_KEY or SERPER_API_KEY_FILE. Reverse image search additionally needs a URL Serper can fetch for the query image: either the built-in image host behind a tunnel (no cloud account) or an Amazon S3 or Google Cloud Storage bucket, where the image is uploaded and handed to Serper as a signed URL. Follow the search and storage setup.

For the local Qwen setup, set these variables in both the router and agent terminals. Adjust the addresses to match your deployment:

export MODEL_BASE_URL=http://127.0.0.1:8222/v1
export MODEL_API_KEY=empty
export AUXILIARY_MODEL_BASE_URL=http://127.0.0.1:8222/v1
export AUXILIARY_MODEL_API_