codex-vision-proxy

What it thinks is what it sees — a way to make a text-only model "see" images with its mind's eye, a vision toolkit, and drop-in setups for Codex, Claude Code, Pi, Oh My Pi, and OpenCode.

🌐 中文 ｜ English

If your Codex is already connected to a text-only model like DeepSeek V4, but you're frustrated that it can't see images — every attempt to look at one is blocked by the system — this repository offers a way to let a text-only model call Codex's built-in view_image without errors. Instead of failing, it returns a task-aware description shaped by the agent's original viewing intent, keeping the text-only model's experience as close as possible to a multimodal one, without introducing extra MCPs, skills, or CLIs, and without the risk of repeated configuration. It also provides an optional vision toolkit that leverages multimodal models for image Q&A, OCR, visual grounding, and more.

All code has been verified in real Codex + DeepSeek sessions, and the same pipeline has been live-verified end-to-end in Claude Code, Pi, Oh My Pi, and OpenCode. Use cases include but are not limited to: image Q&A, screenshot analysis, Computer Use GUI operation, and multi-step image reasoning.

Supported Agents

All entry points share the same describe layer — the focus hint, the verbatim-transcription contract, the re-query channel note, and the per-(image, prompt) cache — and the same three VISION_* env vars.

Most vision wrappers simply turn an image into a generic description and leave the text model to recover the original task afterward.

codex-vision-proxy preserves why the agent is looking. It extracts the viewing intent from the user message or the assistant's stated reason for calling view_image, then passes that intent to the vision model as a focus hint. The result is a task-aware description that emphasizes what matters for the current step—not a generic "detailed description." Lower cost, higher accuracy, and faster response times.

If the agent you're using isn't Codex: Claude Code can point ANTHROPIC_BASE_URL at the same proxy, Pi / Oh My Pi / OpenCode get the same describe layer as single-file native extensions, and any agent with a shell can install the visual toolkit — CLIs that let agents interact with images.

If this project helps you, feel free to star🌟 & follow～ I'll keep sharing more practical tools and tips.

Real-world Effects

Left: DeepSeek V4 answers a UI style question with similar-style comparisons. Right: DeepSeek V4 debugs a field-name mismatch from a screenshot.

Left: multi-round image Q&A after installing the optional glance CLI. Right: after installing ground, DeepSeek V4 locates screen elements to play chess autonomously.

Highlights

• Descriptions target the current question: every image gets a focus hint — a pasted image carries its own message's text, an image fetched via view_image carries the assistant's stated reason for looking — so the description covers the details this turn actually needs instead of being a generic caption.
• Pasted images and view_image both work: images pasted directly (message.content) and images passed when the model calls view_image (function_call_output.output) are both understood.
• Parallel multi-image understanding: multiple images in one request hit the vision model concurrently — N images cost roughly the latency of 1, no waiting image by image.
• Optional glance: a concise standalone CLI for image Q&A and OCR — the follow-up channel when a description misses a detail you need.
• Optional ground: locate a target in an image with natural language and get a bounding box in original pixel coordinates — for GUI-automation clicks and zoom-in crops.
• Optional detect: inventory the elements of a screen or region in one call — the scaffold for rebuilding a UI from a screenshot.
• Optional trace: local, deterministic image-to-SVG tracing, no vision API involved — for reproducing icons/graphics as vectors and measuring exact shape geometry.
• More vision tools may be added later

Usage

This repository doesn't provide a universal one-click installer. The recommended way is to hand the repository link to your Codex agent:

I've already got a text-only model working in Codex. Please read this repository's README first, then follow AGENT_INSTALL.md to deploy and verify view_image on the current system.

Detailed steps are in the Codex Agent Installation Guide. After installation and a Codex restart, just paste an image or let the model call the built-in view_image.

Prerequisites

• Codex already working with a text-only model (e.g. DeepSeek V4)
• Python 3.11+
• An OpenAI-compatible vision API that supports /chat/completions and image_url

Configuration

Only these env vars are required:

Upstream authentication is still sent by Codex and passed through by the proxy, so there's no need to store it again in the env.

Optional Tool: glance (Recommended)

glance is a standalone CLI for asking questions about an image directly, to fill in specific details.

For a global command, let Codex create a wrapper following the install guide. The call then becomes:

glance screenshot.png -q "What is the dominant color of this image?"
glance screenshot.png --ocr

Answer:

The dominant colors of this image are **white and light gray, with light blue accents.**

Username
Password
Login

Optional Tool: ground

ground is a standalone CLI for locating objects or regions in an image:

ground screenshot.png "Send button"

x1: 1067, y1: 841, x2: 1108, y2: 881

It analyzes one full image per call and outputs the target's pixel coordinates in the original image. With --region X1,Y1,X2,Y2 it searches only that box and still reports original-image coordinates.

Optional Tool: detect

detect is a standalone CLI that inventories the elements of an image (or a region) — a numbered list with exact visible text and pixel boxes:

detect page.png
detect page.png "buttons"
detect page.png --region 238,600,953,671

1. bottom-left Do anything x1: 253, y1: 601, x2: 328, y2: 609
2. bottom-left + x1: 254, y1: 650, x2: 268, y2: 665
3. bottom-right stop button x1: 924, y1: 645, x2: 952, y2: 670

A full-screen pass is a fast first draft; for completeness on dense screens, inventory region by region.

Optional Tool: trace

trace vectorizes an image (or a cropped region) into SVG locally and deterministically — coordinates come from the actual pixels, not from a vision model's estimates. Use it for exact shape geometry: reproducing icons/logos as SVG, reading a diagram's layout, or measuring elements. Requires the optional vtracer (and pillow for --region).

trace diagram.png --polygon
trace screenshot.png --region 1563,514,1668,621 -o icon.svg

Install the vision-tools Skill (optional)

One way to install the extra vision tools into Codex is the bundled vision-tools skill, which tells Codex what glance/ground are and how to use them. Install it with the official skills CLI:

npx skills add Anionex/codex-vision-proxy --skill vision-tools -a codex -g --copy -y

Or copy the folder manually:

cp -r skills/vision-tools ~/.codex/skills/

Restart Codex afterwards.

How It Works

Codex -> 127.0.0.1:19100 -> your existing text-only upstream
             |
             +-- when the request contains images:
                 focus hint (the user's request, or the assistant's
                 stated reason for calling view_image)
                   -> vision prompt -> text description -> image replaced

The vision prompt is not a fixed "describe this image". The proxy attaches a focus hint so the description covers what actually matters right now: a pasted image carries the user's request, while an image fetched via view_image carries the assistant's own stated reason for looking (falling back to the user text when the tool was called silently). Descriptions are cached per (image, prompt); both hint sources sit in the immutable conve