nnInteractive Agent

Say what to segment. The agent does the rest.

nnInteractive as a tool for AI agents. Tell your agent in plain text what to segment ("segment every rib individually and save the result"). The agent prompts nnInteractive with scribbles (boxes where a box fits the object, points for tiny regions), checks the 3D result and corrects it, as a human annotator would. It works on CT, MRI, microscopy, electron microscopy and photographs.

What you get:

• An MCP server (nninteractive) with the tools listed under Tools. Any agent that speaks MCP can use it: Claude Code, Codex, open models behind Claude Code, and others.
• A skill (nninteractive-segmentation) that teaches the agent the workflow: prompt rules, a correction playbook and how to verify. The MCP server also offers it as an MCP prompt for agents that do not load skills.
• A web demo (optional): a browser page with 2D/3D views and a chat, to show the agent to people who have no agent set up.

How well it works: on our benchmark (14 public CT, MRI, PET and micro-CT images, 51 named targets, nnInteractive v1.0), Claude Code and Codex reach a mean Dice of 0.81 to 0.85 (averaged per image) in about four minutes per image. nnInteractive alone, with one click per target, reaches 0.71. When the agents are told where each target is, their prompts are about as good as simulated prompts placed with the reference mask, at the same number of prompts. VoxTell, a 3D model that segments from text alone, reaches 0.54: it does well on standard radiology targets and fails on the leg muscles, the foot bones and the animals.

Technical report: nnInteractive Agent: General-Purpose AI Agents as 3D Image Annotators (arXiv, 2026). It describes the tools and the skill and evaluates them on the benchmark.

nnInteractive itself runs on your machine (a CUDA GPU is strongly recommended) or on an nninteractive-server on a GPU machine elsewhere.

Installation

Requires Python 3.10 or newer.

pip install "nninteractive-agent[local] @ git+https://github.com/MIC-DKFZ/nninteractive-agent"

Extras combine: [local,web]. Local inference needs an NVIDIA GPU with about 10 GB of memory (small objects work with less); on a CPU it works, but every prompt takes long. If pip installs a PyTorch that does not match your CUDA driver, install PyTorch first (pytorch.org), then this package.

Use it from Claude Code or Codex

Set up

Install Claude Code or Codex and log in. Then:

nninteractive-agent setup claude        # or: nninteractive-agent setup codex

setup does three things:

• nnInteractive: for local inference it checks PyTorch and your GPU and downloads the nnInteractive weights (about 400 MB, into ~/.nninteractive). With --server it connects to the server once instead.
• The MCP server: it registers the server for your user, with the Python you installed into, so it works without activating that environment.
• The skill: Claude Code gets it in ~/.claude/skills, plus a verifier subagent in ~/.claude/agents. Codex gets it in ~/.codex/skills.

From then on, every new Claude Code or Codex session has the tools and the skill, in any directory. A session that never segments anything only starts a small Python process: PyTorch and the model load with the first image.

The choices are stored in the agent's MCP configuration. To change them, for example to switch from local inference to a server, run setup again with the new options.

Use

Start the agent in the directory where results should go, and ask in plain text:

claude "Segment the liver and the spleen in /data/ct.nii.gz, each as its own object, and save the result."
codex "Segment every nucleus in /data/cells.tif and save the result."

Or start claude or codex without a prompt and ask there. The agent loads the skill on its own when you ask it to segment, delineate, outline, label or count structures in an image file. The image and its objects stay loaded for the whole session, so you can follow up: "the liver leaks into the heart, fix that", "now also both kidneys, each as its own object".

• Inputs: NIfTI, NRRD, MHA, TIFF (stacks), PNG/JPG and NPY.
• Results: an instance map with one id per object (<image>_instances in the input's format) and objects.json with names and notes. They go wherever you tell the agent, or to ./runs/<time>/output/.
• Good requests say what counts as one object ("each rib", "each nucleus") and name the structures the way a radiologist or biologist would.
• Tool approvals:
  • Claude Code asks before each nnInteractive tool it has not been allowed to use yet. Answer "don't ask again", or add "mcp__nninteractive" to permissions.allow in ~/.claude/settings.json to allow all of them.
  • codex exec (non-interactive) needs the tools approved up front: codex exec -c mcp_servers.nninteractive.default_tools_approval_mode="approve" "...".
  • Before you allow the python tool for good, read Security.

Tested with Claude Code 2.1 (Opus 5.5), Codex 0.162.1 (gpt-6.1-sol) and Claude Code on vLLM (Qwen3.8-27B, see Open models).

Other agents

nninteractive-agent setup print

This prints an mcpServers entry for any MCP client (Cursor, opencode, Goose, ...) and the path of the skill folder. Copy the folder to wherever your agent looks for skills. If the agent has no skills, it can read the MCP prompt nninteractive-segmentation (also available as resources under nninteractive://skill/); the server's instructions point the agent to it.

The agent needs a model that sees images: the tools return rendered views.

Claude Code can also load the tools and the skill as a plugin, for one session only, from an environment where nninteractive-agent-mcp is on the PATH:

claude --plugin-dir "$(python -c 'import nninteractive_agent.skill as s; print(s.PLUGIN_DIR)')"

Open models

Claude Code can drive any model behind an Anthropic-compatible API, for example a vLLM server (it serves /v1/messages). The model must:

• see images, including images inside tool results;
• call tools. In vLLM this needs --enable-auto-tool-choice and the model's --tool-call-parser; otherwise tool calls come back as plain text.

We tested Qwen3.8-27B:

nninteractive-agent setup claude
export ANTHROPIC_BASE_URL=http://localhost:8000 ANTHROPIC_API_KEY=dummy ENABLE_TOOL_SEARCH=false
export ANTHROPIC_DEFAULT_OPUS_MODEL=Qwen/Qwen3.8-27B ANTHROPIC_DEFAULT_SONNET_MODEL=Qwen/Qwen3.8-27B \
       ANTHROPIC_DEFAULT_HAIKU_MODEL=Qwen/Qwen3.8-27B
claude --model Qwen/Qwen3.8-27B --effort medium

Pick an effort the server accepts: vLLM with Qwen takes low, medium and xhigh, not high. Open models work, but they are slower and less precise. On a spleen CT (spleen_10 of the Medical Segmentation Decathlon, nnInteractive v1.0 on the same machine), Qwen3.8-27B at medium effort reached Dice 0.93 with five prompts, but took 34 minutes; Claude Code and Codex need about four minutes per image in the benchmark, for 3.6 targets on average. Its first boxes are often too large, so do not tell it to skip verification.

Web demo

A demonstrator, separate from the tools above: the same tools and skill, driven by an agent behind a browser page. The people using it need no agent of their own, and the web demo does not need setup.

pip install "nninteractive-agent[local,web] @ git+https://github.com/MIC-DKFZ/nninteractive-agent"
nninteractive-agent-web --examples ~/images

It prints a link, http://127.0.0.1:8443/?key=...: open it in a browser on the same machine. The start-up lines also show where nnInteractive runs, its model license, the agent's LLM and the python mode. If the demo runs on