AgentMeasure
Find repeated failures and retries in your Codex sessions, with local evidence.
Healthcheck reads existing Codex Desktop rollout logs and produces a terminal summary and a local HTML report. It checks duplicate records, retry chains, and consecutive tool failures. Missing evidence is UNPROVABLE, never silently zero.
# Requires Python 3.9+, Git, and pipx. Installation uses the network.
pipx install "git+https://github.com/roy-tong/AgentMeasure#subdirectory=healthcheck"
agentmeasure demo # synthetic example; no personal logs needed
agentmeasure check # your local Codex sessions, last 7 days
Analysis runs locally with no runtime network calls. Engineering preview: Codex CLI is not yet independently verified; Claude Code is not supported yet. The Git install above is available now; PyPI publishing is being prepared.
Quick start and supported formats · Try it and share feedback safely · What our contributions changed · 中文
Found a useful result? Keep a snapshot and compare your next run. Feedback is optional: you can use the tool without opening an issue or uploading logs.
Measurement infrastructure
The open yardstick for agent usage and AI outcomes. Test whether your agent metrics mean what their labels claim.
Conformance checks for AI-agent telemetry — a retry is one logical operation, not two requests; a reasoning-token subset must not be added into totals; a cache hit is not a new measurement. Every check reports PASS / FAIL / UNPROVABLE, and UNPROVABLE is a first-class result: when the evidence to decide is absent, it is disclosed, never zeroed.
# .github/workflows/conformance.yml — turn measurement assumptions into CI checks
- uses: roy-tong/AgentMeasure@2cf476d6f7d0fc45401db5a822e1f12de009ac74
with:
fixture: fixtures/telemetry.jsonl # your FMT-002 event fixture
Run conformance locally · See real failures · Read the spec · 中文
AgentMeasure separates execution facts from logical operations, evidence from inference, and economic settlement from value — an open measurement layer for the Agent Capability Economy. One number for your dashboards (logical operations), the evidence for what each number counts, and explicit disclosure for what it cannot prove.
Today: measure agent-facing capability usage — attempts, operations, retry inflation, success rates with numerators you can audit. Next: make capabilities comparable and meterable — including the outcome units now being billed (what counts as one resolution, one completed task). Long term: provide the measurement foundation for Capability as a Service (CaaS).
Reach → Choice → Use → Utility → Value
AgentMeasure is not a payment protocol, marketplace, or universal ranking system. It standardizes the facts and measurement semantics those systems can build on.
Why now
AI services have started charging by the outcome: Zendesk at $1.50–2.00 per automated resolution since Aug 2024, Intercom Fin at $0.99 per resolution with a money-back guarantee, Sierra's outcome-based enterprise contracts. But what counts as one outcome has no standard — retries, reopens, and silent "assumed resolutions" all change the number, and the bill. When money rides on a measured unit, someone has to define the unit. That is what we build. → Read the note
Principles (written down before anyone asked)
- Free, forever. Spec, engine, SDK, conformance, local dashboard — the free list is part of governance, not revocable later. Adoption is the product.
- Local-first, opt-in. Raw data never leaves your machine by default; only aggregated, anonymized evidence returns, and only if you opt in.
- The referee doesn't open a store. No paid rankings, no recommendation slots, no custody of funds. Neutrality is the product.
Website · Send a trace → get a measurement check · Free 7-day audit — apply · 30 Projects / 30 Days campaign · Whitepaper · Core Specification
Start with the story: When the Software Consumer Becomes an Agent (EN) · 当软件的消费者变成 Agent (ZH)
Try it in 2 minutes
./examples/demo-e2e.sh
Mock MCP server → canonical observations → local metrics, all on your machine, no
cloud. The demo is reproducible: it runs in an isolated workspace (never touches
~/.agentmeasure) — same fixture + same policy = same result (42 calls → 84
canonical observations, caller claims claude:14 · codex:14 · unknown:14).
Then read why we audit the ecosystem's usage claims: Benchmark Run #001 — six real claims profiled with the Evidence Profile (multi-axis, no composite scores), and Pipeline Validation #001 — our own fixture, kept out of the ranking as a reference baseline.
Run a preregistered experiment — AgentMeasure Lab
python3 lab/am lab selftest # planted uplift recovered + honest null
python3 lab/am lab init # workspace + example experiment
python3 lab/am lab preregister am-lab/experiments/example-manifest.json
python3 lab/am lab run am-lab/experiments/example-manifest.prereg.json
The open experiment engine (lab/): task set × harness matrix × factor
variants → Reach → Choice → Success → Consumption funnel → effect sizes with
confidence intervals, guardrails, honest nulls, and an offline HTML report that opens
with a bilingual decision-maker one-pager. Preregistration is enforced (hypothesis /
primary metric / guardrails / analysis plan hashed before the run — with a scale /
power / budget preview), seeds replay deterministically, and a budget circuit breaker
stops safely with the data it has. Selection uplift that loses consumption is
rejected at the decision exit (unverified_growth — do not ship), and candidates
that make no more money at higher cost are flagged as dominated.
The shipped demo runs on a synthetic harness (planted ground truth at realistic, literature-scale amplitudes, disclosed in every report) — it validates the engine, not real-agent claims. Real harness adapters (Claude Code / Codex) are runner plugins against the same interface; that is the highest-value contribution right now. Docs: lab/README.md · formats: lab/schemas/ (experiment manifest / funnel events / report).
Why the split matters, in one line
1 user intent
2 provider attempts
1 final success
That is 1 operation, 2 attempts, operation success 100%, attempt success 50%, attempts per operation 2.
It is not "2 operations with 50% success".
Attempts are execution facts. Operations are logical intents. Mixing the two is how agent reliability reports get distorted.
External provider trials
Start smaller than a trial: send a trace, get a measurement check — 20–100 anonymized trace/log rows (or a public export), mapped locally, with a short report of what safely counts as attempts vs operations, where retries may inflate usage, and what the telemetry cannot prove. No SDK, no integration; raw data stays local unless you explicitly authorize sharing a sanitized sample. A sample check is not a full audit and claims no causal effect.
If the check pays for itself, AgentMeasure is looking for 3 MCP/API providers for short external measurement runs.
The goal is not to prove the model works. It is to find where it breaks on real provider traffic.
A trial can stay fully local, requires no agent-side install, and can start with one capability.
Best feedback: a concrete trace, counterexample, or field that cannot be measured safely.
→ The formal offer, schedule, and FAQ live in product/AUDIT.md · apply i