Bug Hunt Bench — which AI coding model fixes the most real bugs?
105 real bugs, hidden in two production codebases. Frontier coding models — GPT-6, Claude, Grok, Gemini, DeepSeek, Kimi, GLM and more — get one round per repo in their own agentic CLI (Codex CLI, Claude Code, Grok CLI, Antigravity CLI) to find and fix what they can. Every diff is graded blind against a withheld answer key. The score counts planted bugs only.
Live board: bughunt.productcompass.pm · Method, caveats and definitions · Raw data · Findings by wave
If the numbers save you a benchmark run of your own, star this repo — that is what keeps the bench findable, and new models are added as they ship.
Updated Oct 1, 2026 · 164 rows from 195 scored runs · 40 models · 29 of 105 bugs have never been fixed by any model.
Current leader: Sonnet 5.5 at max effort — 51.3 / 105 (22.7/45 on repo 1, 28.7/60 on repo 2).
Best per lab: Anthropic: Sonnet 5.5 (max) 51.3 · OpenAI: GPT-6 Astra (max) 45 · Meta: Muse Spark 1.3 (max) 32.2 · Unbiased: Pareto (ex-Union Alpha) (default) 30.7 · xAI: Grok 4.7 (xhigh) 28.8 · Alibaba: Qwen3.8-Flash (max) 26 · Xiaomi: MiMo-V2.6-Flash (default) 23.3 · Google: Gemini 3.7 Flash (high) 22 · DeepSeek: DeepSeek V4.1 Flash (max) 21.7 · Moonshot AI: Kimi K3 (default) 21 · Z.ai: GLM-5.3 (max) 19 · Tencent: Hy4 Preview (default) 18
| # | Model | Harness | Effort | Runs | Fixed /105 | Repo 1 /45 | Repo 2 /60 | Extras | Wall | Cost | Date |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | Sonnet 5.5 | Claude Code | max | 3 | 51.3 | 22.7 | 28.7 | 26.3 | 287 min | $154.03 | 2026-09-29 |
| 2 | GPT-6 Astra | Codex CLI | max | 3 | 45 | 22 | 23 | 55 | 90 min | $33.03 | 2026-09-14 |
| 3 | GPT-6.1 Sol | Codex CLI | max | 3 | 44.3 | 22.3 | 22 | 65.3 | 139 min | $5.88 | 2026-09-30 |
| 4 | GPT-5.6 Sol | Codex CLI | max | 2 | 43.5 | 19 | 24.5 | 48.5 | 253 min | $95.35 | 2026-09-16 |
| 5 | GPT-6 Astra | Codex CLI | xhigh | 1 | 43 | 23 | 20 | 53 | 59 min | $24.22 | 2026-09-04 |
| 6 | Fable 5.1 | Claude Code | max | 1 | 43 | 19 | 24 | 11 | 73 min | $87.18 | 2026-09-01 |
| 7 | GPT-6.1 Sol | Codex CLI | xhigh | 3 | 42.7 | 22.3 | 20.3 | 56.7 | 99 min | $4.35 | 2026-09-30 |
| 8 | Opus 5.5 | Claude Code | max | 3 | 41.7 | 19 | 22.7 | 9 | 67 min | $58.53 | 2026-09-23 |
| 9 | GPT-5.6 Sol | Codex CLI | xhigh | 1 | 39 | 18 | 21 | 59 | 126 min | $52.75 | 2026-08-28 |
| 10 | GPT-6.1 Sol | Codex CLI | high | 2 | 36.5 | 17 | 19.5 | 40 | 70 min | $3.26 | 2026-09-30 |
| 11 | Opus 5.5 | Claude Code | xhigh | 3 | 36 | 17 | 19 | 13.3 | 42 min | $34.98 | 2026-09-23 |
| 12 | Sonnet 5.5 | Claude Code | xhigh | 3 | 36 | 16 | 20 | 14 | 73 min | $42.86 | 2026-09-29 |
| 13 | GPT-6 Astra | Codex CLI | high | 1 | 35 | 19 | 16 | 40 | 40 min | $20.60 | 2026-09-04 |
| 14 | Sonnet 5.5 | Claude Code | high | 3 | 34.3 | 15.7 | 18.7 | 10.7 | 28 min | $19.85 | 2026-09-29 |
| 15 | GPT-6 Astra | Codex CLI | medium | 1 | 34 | 19 | 15 | 33 | 28 min | $15.78 | 2026-09-05 |
| 16 | GPT-5.6 Sol | Codex CLI | high | 1 | 34 | 13 | 21 | 28 | 67 min | $33.92 | 2026-07-31 |
| 17 | Fable 5.1 | Claude Code | high | 1 | 33 | 15 | 18 | 7 | 36 min | $48.58 | 2026-09-01 |
| 18 | Muse Spark 1.3 | Muse Code / Meta API | max | 5 | 32.2 | 14.2 | 18 | 24 | 86 min | $18.11 | 2026-09-17 |
| 19 | GPT-5.6 Terra | Codex CLI | max | 1 | 32 | 16 | 16 | 45 | 160 min | $27.98 | 2026-08-27 |
| 20 | Opus 5.5 | Claude Code | high | 3 | 31.7 | 13 | 18.7 | 10.7 | 24 min | $22.25 | 2026-09-23 |
| 21 | GPT-5.6 Luna | Codex CLI | max | 3 | 31.3 | 16 | 15.3 | 54.7 | 187 min | $3.15 | 2026-09-22 |
| 22 | Pareto (ex-Union Alpha) | Claude Code / OpenRouter | default | 3 | 30.7 | 16 | 14.7 | 10.3 | 38 min | $4.81 | 2026-09-17 |
| 23 | Opus 5.5 | Claude Code | medium | 3 | 30.3 | 12.7 | 17.7 | 13.3 | 17 min | $15.68 | 2026-09-23 |
| 24 | GPT-6 Sol | Codex CLI | max | 3 | 29.3 | 15.3 | 14 | 41.3 | 63 min | $9.33 | 2026-09-23 |
| 25 | GPT-6.1 Sol | Codex CLI | medium | 2 | 29 | 16 | 13 | 30.5 | 30 min | $1.76 | 2026-09-30 |
| 26 | GPT-5.6 Sol | Codex CLI | medium | 1 | 29 | 13 | 16 | 24 | 48 min | $15.77 | 2026-09-06 |
| 27 | Fable 5.1 | Claude Code | low | 1 | 29 | 13 | 16 | 6 | 33 min | $33.00 | 2026-09-02 |
| 28 | Fable 5.1 | Claude Code | xhigh | 1 | 29 | 13 | 16 | 4 | 60 min | $53.61 | 2026-09-10 |
| 29 | Fable 5 | Claude Code | max | 1 | 29 | 12 | 17 | 5 | 57 min | $112.40 | 2026-08-01 |
| 30 | Grok 4.7 | Grok Build CLI (ACP) | xhigh | 4 | 28.8 | 13 | 15.8 | 31.8 | 46 min | $22.89 floor | 2026-09-21 |
| 31 | Grok 4.6 | Grok Build CLI (ACP) | xhigh | 3 | 28.7 | 11.7 | 17 | 18 | 43 min | $18.60 floor | 2026-09-14 |
| 32 | GPT-6 Astra | Codex CLI | low | 1 | 27 | 18 | 9 | 25 | 32 min | $11.69 | 2026-09-05 |
| 33 | Opus 5 | Claude Code | max | 1 | 27 | 13 | 14 | 2 | 60 min | $54.94 | 2026-08-01 |
| 34 | Grok 4.7 | Grok Build CLI (ACP) | medium | 3 | 26.7 | 12 | 14.7 | 22.7 | 31 min | $16.76 floor | 2026-09-21 |
| 35 | Qwen3.8-Flash | Claude Code / Alibaba API | max | 1 | 26 | 13 | 13 | 7 | 97 min | $1.81 | 2026-09-11 |
| 36 | Opus 5 | Claude Code | xhigh | 1 | 26 | 14 | 12 | 3 | 50 min | $63.25 | 2026-08-28 |
| 37 | Qwen3.8-Max | Claude Code / Alibaba API | max | 3 | 25.7 | 12.3 | 13.3 | 7 | 117 min | $26.76 | 2026-09-15 |
| 38 | GPT-6 Sol | Codex CLI | xhigh | 1 | 25 | 10 | 15 | 24 | 51 min | $7.67 | 2026-09-23 |
| 39 | Opus 5 | Claude Code | medium | 1 | 24 | 11 | 13 | 4 | 30 min | $37.40 | 2026-08-27 |
| 40 | Fable 5 | Claude Code | high | 1 | 24 | 9 | 15 | 3 | 31 min | $74.52 | 2026-07-26 |
| 41 | MiMo-V2.6-Flash | Claude Code / OpenRouter | default | 3 | 23.3 | 8 | 15.3 | 7.3 | 71 min | $0.49 bill | 2026-09-22 |
| 42 | Sonnet 5.5 | Claude Code | medium | 3 | 23.3 | 8.7 | 14.7 | 8.7 | 17 min | $11.11 | 2026-09-30 |
| 43 | Qwen3.8-Flash | Claude Code / Alibaba API | low | 1 | 23 | 11 | 12 | 7 | 98 min | $1.37 | 2026-09-11 |
| 44 | GPT-5.6 Luna | Codex CLI | xhigh | 1 | 23 | 10 | 13 | 53 | 136 min | $2.50 | 2026-08-28 |
| 45 | Grok 4.6 | Grok Build CLI (ACP) | high | 1 | 23 | 7 | 16 | 16 | 33 min | $15.73 floor | 2026-08-28 |
| 46 | MiMo-V2.6-Pro | Claude Code / OpenRouter | default | 3 | 22.7 | 11.3 | 11.3 | 5.3 | 91 min | $0.86 bill | 2026-09-22 |
| 47 | GPT-6.1 Sol | Codex CLI | low | 2 | 22.5 | 11.5 | 11 | 24 | 25 min | $1.34 | 2026-10-01 |
| 48 | Opus 5.5 | Claude Code | low | 3 | 22.3 | 11.3 | 11 | 9.7 | 12 min | $8.34 | 2026-09-23 |
| 49 | Grok 4.6 | Grok Build CLI (ACP) | medium | 1 | 22 | 9 | 13 | 11 | 26 min | $5.80 floor | 2026-08-28 |
| 50 | Gemini 3.7 Flash | Gemini CLI (retired) + model-pinning gateway | high | 1 | 22 | 8 | 14 | 4 | 97 min | $8.43 | 2026-08-24 |
| 51 | DeepSeek V4.1 Flash | Claude Code / DeepSeek API | max | 3 | 21.7 | 11.3 | 10.3 | 6.7 | 56 min | $0.78 bill | 2026-09-15 |
| 52 | Fable 5.1 | Claude Code | medium | 1 | 21 | 8 | 13 | 3 | 23 min | $23.73 | 2026-09-10 |
| 53 | Kimi K3 | Claude Code / OpenRouter | default | 1 | 21 | 4 | 17 | 6 | 108 min | $25.27 | 2026-07-26 |
| 54 | Opus 5 | Claude Code | high | 1 | 21 | 11 | 10 | 6 | 37 min | $41.51 | 2026-07-26 |
| 55 | Muse Spark 1.3 | Muse Code / Meta API | xhigh | 3 | 20.3 | 8 | 12.3 | 13.3 | 64 min | $13.71 | 2026-09-14 |
| 56 | GPT-6 Sol | Codex CLI | high | 1 | 20 | 10 | 10 | 15 | 34 min | $4.16 | 2026-09-23 |
| 57 | GPT-5.6 Terra | Codex CLI | xhigh | 1 | 20 | 9 | 11 | 29 | 57 min | $8.98 | 2026-08-28 |
| 58 | Grok 4.7 | Grok Build CLI (ACP) | high | 3 | 19.3 | 8.7 | 10.7 | 15.7 | 38 min | $12.29 floor | 2026-09-21 |
| 59 | DeepSeek V4.1 Flash | Claude Code / DeepSeek API | high | 1 | 19 | 9 | 10 | 4 | 26 min | $0.31 bill | 2026-09-10 |
| 60 | Sonnet 5.5 | Claude Code | low | 3 | 19 | 6.7 | 12.3 | 8 | 8 min | $5.73 | 2026-09-30 |
| 61 | GLM-5.3 | Claude Code / Z.ai API | max | 1 | 19 | 9 | 10 | 2 | 40 min | $15.93 | 2026-09-10 |
| 62 | Muse Spark 1.3 | Muse Code / Meta API | high | 3 | 18.7 | 8.7 | 10 | 19.3 | 76 min | $12.94 | 2026-09-14 |
| 63 | GPT-6 Luna | Codex CLI | max | 3 | 18.3 | 6.3 | 12 | 22.7 | 99 min | $0.52 | 2026-09-23 |
| 64 | MiMo-V2.6-Pro | Claude Code / Xiaomi first-party | enabled | 1 | 18 | 9 | 9 | 4 | 75 min | $0.59 | 2026-09-22 |
| 65 | Hy4 Preview | Claude Code / OpenRouter | default | 1 | 18 | 9 | 9 | 2 | 68 min | $3.13 bill | 2026-08-28 |
| 66 | GPT-5.6 Terra | Codex CLI | high | 1 | 18 | 7 | 11 | 17 | 28 min | $5.89 | 2026-08-27 |
| 67 | Gemini 3.8 Flash | Antigravity CLI | high | 3 | 18 | 6.7 | 11.3 | 4 | 39 min | $11.03 | 2026-09-15 |
| 68 | GLM-5.3 Flash | Claude Code / Z.ai API | max | 3 | 17.7 | 9.3 | 8.3 | 3.3 | 63 min | $1.03 | 2026-09-15 |
| 69 | Grok 4.5 | Grok Build CLI (ACP) | high | 1 | 17 | 5 | 12 | 10 | 28 min | $8.50 floor | 2026-08-06 |
| 70 | Muse Spark 1.2 | Claude Code / Meta API | xhigh | 1 | 17 | 6 | 11 | 12 | 36 min | $13.99 | 2026-08-06 |
| 71 | Ox Alpha (stealth) | Claude Code / OpenRouter | default | 1 | 16 | 8 | 8 | 3 | 59 min | free | 2026-08-25 |
| 72 | GLM-5.3 Flash | Claude Code / Z.ai API | high | 1 | 16 | 9 | 7 | 4 | 62 min | $0.94 | 2026-09-13 |
| 73 | DeepSeek V4-Pro | Claude Code / DeepSeek API | max | 1 | 16 | 8 | 8 | 4 | 37 min | $1.89 bill | 2026-09-10 |
| 74 | Gemini 3.7 Flash | Antigravity CLI | high | 1 | 16 | 4 | 12 | 2 | 23 min | $6.36 | 2026-08-28 |
| 75 | Grok 4.7 | Grok Build CLI (ACP) | low | 3 | 15.7 | 10.3 | 5.3 | 18.3 | 22 min | $9.36 floor | 2026-09-21 |
| 76 | GPT-5.6 Terra | Codex CLI | medium | 1 | 15 | 4 | 11 | 6 | 20 min | $3.87 | 2026-08-27 |