DSH Vision Toolkit

DSH Vision Toolkit brings agent-vision-toolkit into DeepSeek Harness as a native Profile Bundle.

Give text-only DSH agents eyes—and keep vision in the harness—with intent-aware image Q&A, OCR, original-pixel grounding, UI restoration, pixel verification, managed Artifacts, and Web Settings. Ten independent tools replace shell glue with structured schemas and Agent-scoped progressive exposure.

Upstream toolkit: Anionex/agent-vision-toolkit · Project website: agent-vision.anionex.me

English | 中文

Why this exists

agent-vision-toolkit treats vision as an Agent-callable capability rather than a property of the base model. Its method carries the reason for looking into the visual request, moves from the whole image to targeted regions, and verifies coordinates, colors, geometry, and differences with focused tools instead of accepting a generic description as evidence.

DSH Vision Toolkit preserves that method while replacing CLI installation and Bash argument construction with native schemas, DSH Credentials, lifecycle-managed runtime preparation, structured Session-log results, previewable Artifacts, dedicated Web cards, and Settings. The Agent loads one versioned Skill and receives the ten visual schemas only when the current task needs them.

The package delivers the committed P0 and P1 product scope. P2's stable ctx.visionToolkit service remains deliberately unpublished until an independent plugin becomes a real consumer; the internal runtime does not pretend that an unvalidated ecosystem API is stable.

Proven use cases from agent-vision-toolkit

The first two panels are official upstream reference runs from the same pinned agent-vision-toolkit lineage packaged by this bundle. The image Q&A and screenshot-guided debugging panel is a live DeepSeek Harness Web session, showing the same workflows through DSH. See the asset provenance record for the upstream source images.

Infographic restoration: screenshot to editable HTML/CSS

Left: source screenshot. Right: the editable HTML/CSS result from the upstream infographic-restoration reference.

UI restoration: sketch to working interface

Left: hand-drawn input. Right: the upstream reconstructed interface; the complete method lives in the UI restoration playbook.

Image Q&A and screenshot-guided debugging

Left: intent-aware image Q&A in DSH Web. Right: a DSH Web screenshot-debugging turn that lists the concrete UI differences and continues toward vision_pixel_diff. The upstream workflow source is the same agent-vision-toolkit reference.

DSH Vision Toolkit adds native tool schemas, versioned lifecycle, Credentials, structured Session results, Artifacts, Web presentation, Settings, and progressive exposure around these upstream capabilities. The next section is the reproducible proof executed and checked into this DSH repository.

DSH-native proof: reference-to-pixel verification

The checked-in UI-restoration workflow renders an intentionally inaccurate HTML implementation, measures a 6.04% pixel difference across six non-zero regions, iterates, and reaches an exact 0% difference against the reference at 1200 × 720.

Highlights

• See images without bloating every prompt: only vision_toolkit_activate is initially visible; loading vision-tools mounts ten independent schemas for that Agent and keeps version/health administration out of model context.
• Act on coordinates instead of parsing prose: grounding and detection return original-image pixel boxes, while every model-visible result remains structured text or JSON.
• Deliver files, not temporary output: crop, trace, OCR, pixel diff, foreground extraction, and HTML rendering produce described Artifacts that the Web client can preview, download, or open locally.
• Keep runtime and credentials controlled: DSH Credentials hold API keys, managed mode prepares an exact isolated Python environment, and a failed Settings candidate cannot replace the serving generation.
• Close the visual verification loop: local HTML rendering and pixel-diff ranking support reference → implementation → screenshot → measured iteration without a model-native image channel.
• Use the same bundle in Web and Headless profiles: Web adds cards, previews, Settings, and health actions; Headless receives the same tool semantics and complete structured results.

Quick start

Prerequisites: access to this private repository, DeepSeek Harness, Python 3.11+, and pnpm available to dsh plugin. Clone the release checkout with your authenticated GitHub credentials, add it to the profiles you use, and confirm the bundle row:

git clone https://github.com/dsh-external/dsh-vision-toolkit.git
PLUGIN="$PWD/dsh-vision-toolkit"
dsh plugin --profile web add "$PLUGIN"
dsh plugin --profile headless add "$PLUGIN"
dsh --profile web --dump-config | grep vision-toolkit
dsh --profile headless --dump-config | grep vision-toolkit

Restart a running Web profile, open Settings → Vision Toolkit, select a DSH Credential for remote tools, and explicitly run Test connection. In a conversation, make an image available as a workspace path, invoke /vision-tools, and ask the Agent to call a specific vision_* tool. Local crop, trace, pixel, color, foreground, and HTML operations do not require a visual API credential.

How it works

flowchart LR
    User["Workspace image or local HTML"] --> Skill["vision-tools Skill"]
    Skill --> Activate["Agent-scoped activation"]
    Activate --> Tools["10 independent vision_* tools"]
    Tools --> Runtime["Shared VisionToolkitRuntime"]
    Credentials["DSH Credentials"] --> Runtime
    Settings["Web Settings and health"] --> Runtime
    Runtime --> Upstream["Pinned agent-vision-toolkit"]
    Runtime --> Remote["Configured vision API"]
    Upstream --> Result["Text, coordinates, JSON"]
    Remote --> Result
    Runtime --> Artifacts["Workspace Artifacts"]
    Result --> Session["Reconstructable Session log"]
    Artifacts --> Web["Preview, download, or open file"]

Tool definitions call one runtime; the runtime validates paths, limits, credentials, cancellation, and deadlines before dispatching to the pinned upstream snapshot or configured OpenAI-compatible vision endpoint. Web presentation consumes the same structured results and Artifact descriptors, so it does not change Headless behavior. Health, connection testing, and version inspection stay in Settings rather than model tool schemas.

Tool