Gauge CLI

An open-source CLI for hillclimbing Agent Experience.

Make your Skills, CLIs, SDKs, MCPs and docs easier for coding agents to discover and use. Gauge runs real agents against your product, captures where they get stuck, and lets your own coding agent test fixes before you publish them.

The full loop runs locally with Docker and your existing agent subscriptions or API keys. No Gauge account required.

Website · Documentation · Hosted Gauge · npm · Changelog

What does Gauge find?

Gauge runs and analyzes coding sessions to find the issues that coding agents run into.

Common findings include:

• An interactive command blocks a coding agent
• An agent reaches for deprecated commands or outdated API patterns
• A Skill goes unused even when it should help with the task
• An agent confidently claims a limitation that does not exist
• An agent cannot figure out the intended workflow across multiple commands

The right fix might change error messages, Skill content, help text, code, or documentation. The Gauge CLI lets you validate proposed fixes before release.

Quickstart

Gauge is designed to be used by your coding agent. Point it at this repository and ask it to get started, or follow the steps below in the project you want to improve.

You need Node.js 22.13+, npm, Docker running Linux containers, and credentials for the agent you want to test. Gauge supports Claude Code, Codex, Pi, and OpenCode. It can use native Claude/Codex logins or API keys; agent usage is billed through your provider or subscription.

1. Install Gauge and its runtime

npm install -g @withgauge/cli
gauge local runtime pull

Gauge downloads the published Docker image pinned to your CLI release. The image contains the agent harnesses and standard development tools.

2. Set up your project

From your project directory (Git is optional):

gauge init --local

This creates a private .gauge/ workspace, including a SQLite database and local backend preference. Plain init changes no tracked files. Add --cases when you want shared Markdown cases: it also writes a gauge.json manifest, gauge-cases/, and guidance in AGENTS.md. Commit those shared files. Keep .gauge/ private: it contains durable measurements and analyses.

3. Install the skill

The Gauge skill teaches your coding agent where to find the CLI's instructions:

gauge install-skill

How the improvement loop works

Hillclimbing means making a change, measuring whether it helped, and repeating.

1. Define realistic tasks. Your agent saves evals or preference prompts with stable identities, editable defaults, and observable outcomes. Use optional Markdown cases for fully specified measurements intended for sharing or CI.
2. Test your current product. In local mode, Gauge starts fresh Docker sessions and injects the working-tree version of your configured inputs. Each test agent sees the task and configured inputs within its sandbox. Local mode supports Skills, SDKs, CLIs, and files. Hosted compute adds support for docs, authenticated MCPs, and authenticated APIs.
3. Find friction. Gauge normalizes events from Claude Code, Codex, Pi, and OpenCode into consistent transcripts. In local mode, your agent reads the evidence, checks the outcome against the criteria, and identifies what slowed or blocked the task. It publishes structured Findings and judgments through gauge run analysis save. In hosted mode, this analysis happens automatically after each session. gauge run insight shows the evidence.
4. Change and retest. Your agent edits the product or its guidance and reruns the affected cases against the new artifacts.
5. Analyze the evidence. Check whether the original problem disappeared, whether the task succeeded, and whether the change introduced new friction.

Skills and file inputs are read from your working tree. For an npm CLI or SDK, build and pack a new tarball before rerunning. Gauge installs these pre-release artifacts in the test sessions, so you can validate a fix before publishing it.

The CLI handles execution and evidence collection. In local mode, your coding agent authors the changes and analyzes the local results.

A closer look at an eval

Saved evals and preference prompts live in .gauge/gauge.sqlite locally or in your hosted organization. Shared file-based cases optionally live in gauge-cases/; gauge.json declares project inputs and automation. Both measurement sources support local and hosted execution, with the input differences described below. Start with a saved local eval; use a shared case when the task is ready for teammates or CI.

Create skills/release-notes/SKILL.md:

---
name: release-notes
description: Write release notes for a new version in the team's format.
---

Write RELEASE-NOTES.md with the version as its title, an Added section for new
features, and a Fixed section for bug fixes. Use one bullet per change.

For a fresh project, use this gauge.json:

{
  "version": 2,
  "inputs": {
    "release-notes": {
      "type": "skill",
      "path": "skills/release-notes"
    }
  }
}

Save the task and criteria:

gauge eval create --local --name "Release notes" \
  --agent codex --samples 1 \
  --text "Write the release notes for version 1.2.0. We added dark mode and fixed a login crash." \
  --criterion "Team format: RELEASE-NOTES.md has a version 1.2.0 title, dark mode under Added, and the login crash fix under Fixed."

Use the returned eval ID to preview, then launch when ready:

gauge run --eval <eval-id> --local --input release-notes --dry-run
gauge run --eval <eval-id> --local --input release-notes
gauge instructions session-analysis --local

The preview checks local readiness without launching an agent. Your coding agent then reads the session and publishes its analysis. After changing the Skill, rerun the same eval ID: the new batch snapshots the current input files while the baseline remains intact. Keep the task, criteria, and model pin fixed, analyze both batches, then compare:

gauge run --eval <eval-id> --local --input release-notes
gauge run compare --local --baseline batch=<before-id> --candidate batch=<after-id>

For a choice study, use gauge preference create --kind open-ended with a task that lets the agent choose its approach, then gauge run --preference <id>. Supplying the Skill measures choice after that exposure, not discovery on the web.

For a shared or automated version of the eval, use the same input configuration and create gauge-cases/release-notes/basic.md:

---
criteria:
  - name: Release notes follow the team format
    rubric: RELEASE-NOTES.md has a version 1.2.0 title, dark mode under Added, and the login crash fix under Fixed.
config:
  agents:
    - agent: CODEX_CLI
  sampleCount: 1
  inputs: [release-notes]
---

Write the release notes for version 1.2.0. We added dark mode and fixed a login crash.

This case uses Codex; use CLAUDE_CODE, PI, or OPENCODE to test another harness with its configured credentials. The agent starts in an empty workspace with the Skill installed. The task does not explicitly tell it to use the Skill, so the session also reveals whether it discovers the guidance.

gauge run --verify --local
gauge run --dry-run --local
gauge run --local
gauge run list --local -o json

Verification checks the config and cases. The dry run previews the sessions and checks local runtime and credential readiness. Neither launches an agent.

Results, transcripts, and available diffs stay under .gauge/. Have your coding agent follow gauge instructions session-analysis --local to assess each criterion and report friction with supporting evidence.

After editing the Skill, rerun the case:

gauge run --local --files "gauge-cases/release-notes/*.md"
gauge run compare --local --baseline batch=<before-id> --candidate batch=<after-id>

Use the batch IDs printed by the runs. The comparison summarizes completion, available turns/duration, and published criterion judgments. Read the findings and evidence to determine whether the fix worked.

What is Gauge?

Gauge helps teams improve how AI agents discover, choose, and use their products. Agent Experience measures whether agents can successfully use your product. Agent Preference examines which tools agents choose for a task and why.

This CLI puts that work in the terminal where you and your coding agent already work. Local execution supports the complete evaluate, analyze, fix, and retest loop. The hosted platform adds managed infrastructure and shared measurement over time.

Local or hosted

Sign in with gauge auth login to use hosted Chat and Agents. On a remote terminal, use gauge auth login --device. See signing in for help with login and API keys.