name: falsify-the-problem description: Challenge the current problem formulation before solving it. Use when explicitly invoked, or when meaningful framing uncertainty and non-trivial intervention cost or risk coexist; do not auto-activate for verified diagnoses or mechanical edits.

falsify-the-problem

Falsify the problem before solving it.

Answer Are we solving the right problem? Challenge the current formulation, seek disconfirming evidence, and hand the result back to the host agent. Do not design or implement the solution inside this skill. Use the same protocol across domains; examples are not domain-specific workflows.

1. Decide whether to enter

• Explicit invocation of falsify-the-problem: always enter. Strong existing evidence can justify a brief KEEP without alternatives or a new test.
• Automatic activation: require BOTH meaningful formulation uncertainty AND non-trivial intervention cost or risk. Neither alone is enough.
• Useful signals include solution-shaped requests, a hypothesis treated as fact, repeated failed attempts, and costly or hard-to-reverse proposed changes. Signals are reasons to check the gate, not unconditional triggers.
• Usually do not auto-activate for a verified diagnosis, obvious syntax or compilation error, typo, translation, formatting, mechanical transformation, deterministic local edit, or very cheap reversible intervention.
• Mere logical possibility of another explanation is insufficient.

BYPASS is an activation decision, never an evidence verdict. When bypassing, leave the original task with the host; do not manufacture an adversarial report. If explicitly invoked on a supported diagnosis, use KEEP, not BYPASS.

2. Separate the claims and preserve their sources

Distinguish these five objects, even when the user combines them in one sentence:

Observation != Interpretation != Causal Hypothesis != Problem Formulation != Solution.

For each consequential observation, retain the source and relevant scope:

• tool-observed: name the file, command, trace, or other inspected artifact.
• user-reported: retain attribution, including measurements supplied by a user.
• document-reported: identify the document and its claim.
• experimentally observed: identify the experiment and conditions.
• measurement-derived: identify inputs, transformation, and sampling limits.

These labels can combine. Reading a report with a tool verifies what the report says, not that its underlying claim is true. A local simulated fixture is evidence about that fixture, not independent verification of a real incident. Never upgrade user reports into independently verified facts. Record uncertainty about coverage, freshness, comparability, or missing data where it could change the conclusion. Do not fabricate observations or execution.

State the Current Problem Formulation explicitly. If inferred from a proposed solution, mark it as inferred; do not attribute it to the user as a stated belief. Keep this original formulation as the reference for the overall verdict through this pass. A replacement is REFORMULATE, not a silently relabeled KEEP. Preserve claim strength: a possible contributor, a primary explanation, and an exclusive explanation are different claims. Do not infer exclusivity from a proposed solution or refute a stronger claim than the request requires.

3. Build materially different live formulations

Ask what would change the kind of problem being investigated. Competing causes within one formulation are not automatically competing problem formulations.

Two formulations should materially differ in at least one consequential dimension: failing variable, mechanism class, abstraction level, system boundary, expected evidence, investigation path, or intervention target.

If two formulations predict nearly the same evidence and lead to nearly the same investigation path, they are probably superficial variants. Merge them. For example, weak embeddings, small embeddings, and old embeddings usually belong to one representation-quality formulation. Retrieval construction, ranking, generation grounding, evaluation validity, and query distribution can change the framing when they predict meaningfully different evidence.

Keep only plausible, decision-relevant alternatives. Do not mechanically fill a quota. Let evidence and the pending decision determine the live set; more candidates are not evidence of better reasoning. Presentation limits in section 9 do not justify dropping a materially different, decision-relevant candidate.

Candidates need not be mutually exclusive. If several mechanisms coexist, test which framing explains the material failure at the relevant scope; do not force a winner from evidence that supports a mixed or narrower formulation.

4. Find load-bearing assumptions and distinct predictions

Prioritize assumptions whose failure would materially weaken or collapse a live formulation. Avoid exhaustive lists or generic questions about everything. Include measurement validity only when it bears on the decision.

For each important live formulation, connect:

Formulation -> load-bearing assumption -> observable prediction -> contrary evidence

A prediction must distinguish live formulations, not restate their names. Example: a service-time framing predicts elevated service spans in slow requests; a capacity framing predicts rising pre-service wait with concurrency; a measurement framing predicts disagreement between raw samples and reported aggregation. State the population, comparison, and meaningful difference when available. Do not invent numerical thresholds without a basis; explain qualitative criteria.

Before choosing the test, make one lightweight Shared-Frame / Coverage Check: do the live formulations all rely on one consequential, untested premise about the relevant boundary, measurement path, causal layer, population, time scope, or optimization target? Select what matters here; do not audit a category checklist. If that premise failing would undermine the whole set, and a plausible alternative would change the decision and predict different evidence, add at most ONE Outside-Frame Challenger. Name the shared premise and its contrasting prediction briefly when decision-relevant. Generic "measurement could be wrong" is insufficient; ground the challenger in this task. If none is warranted, proceed without one. Do not chain coverage audits or expand the set merely to satisfy a quota.

5. Select ONE Primary Discriminating Test for this round

When beginning an investigation round under unresolved uncertainty, choose exactly ONE Primary Discriminating Test. This is one per round, not one forever. Choose the cheapest currently available test with substantial discriminatory information relative to cost, risk, and execution time.

Ask: would plausible different outcomes materially change the relative support of the live formulations? If not, choose a better test. An outcome predicted by several live formulations may localize the failure without distinguishing them. Prefer a matched comparison that separates the relevant factor; otherwise retain those formulations rather than assigning the outcome to one by default.

Specify:

• Test: one bounded comparison or observation, its source, and its scope.
• Why this first: which live formulations it separates and why its value exceeds the cost of the next plausible option.
• Possible outcomes: what would strengthen, weaken, or kill which candidates. Include inconclusive or mixed outcomes where plausible.
• Remaining uncertainty: what this test cannot establish.

One test can require several retrieval or computation steps if they serve one predeclared comparison and outcome map. It cannot hide independent investigations inside a bundle such as "check logs, metrics, configuration, and tests." Include an unaccounted or outside-boundary outcome when the observation method could omit relevant stages; an artifact's label does not define its boundaries. Handle such a result through the evidence update below, not a forced winner. Do not call it a "Decisive Test" or imply that every result must settle the issue.

6. Gather evidence instead of deflecting it

STOP solutioning != STOP investigating.

Inspect evidence directly when it is available, permitted, low-risk, and reasonably cheap. Use the host's tools as appropriate: local artifacts, code, history, logs, traces, metrics, configurations, documentation, existing tests, safe diagnostics, low-risk experiments, or benchmarks. No specific tool or runtime is required. Choose the investigation for its discrimination, not because a tool is available.

Read-only exploration needed to locate the selected evidence is allowed. Bound it to the comparison; do not turn discovery into a generic diagnostic checklist. Record what was actually inspected or run and the result, including failures.

If the selected source is inaccessible, prefer an available lower-cost proxy only if it can discriminate; record the limitation. If no useful proxy exists, state the one pending Primary Discriminating Test and request only the minimum missing discriminating evidence or access. Do not ask users to do work the host can safely perform. Do not claim the pending test ran or loop through identical requests.

Investigation does not authorize implementation. Before formulation validation, do not implement the requested solution, migrate architecture, introduce infrastructure, make expensive production changes, take irreversible action, or optimize around an unvalidated premise. Calling an intervention a "test" does not make it low-risk. Respect the host's existing permissions and user task scope.

7. Update from evidence, then decide whether to continue

Evidence beats rhetoric and the agent's previous conclusion. Accept contradictory evidence; do not reinterpret it merely to preserve an earlier position.

After the selected test:

1. Attribute the result and compare it with the predicted outcomes.
2. If a consequential result fits no live formulation, mark it OUTSIDE CURRENT FRAME: the set may be incomplete. This is an evidence diagnostic, not a sixth verdict. Preserve the unexplained residual; do not award it to the closest candidate. Repeat the lightweight coverage check once for this new evidence, adding at most one grounded challenger if warranted. Otherwise leave it unresolved.
3. Update affected candidates using the labels below with a short evidence reason, and re-check load-bearing assumptions invalidated by the test.
4. Merge redundant candidates; retain uncertainty and scope limits.
5. Apply the STOP conditions. If a further discriminating observation is worth its cost, start the next round with the updated live set and one new Primary Test.

Missing evidence is not contrary evidence. An absent signal kills a candidate only when coverage and detection were sufficient for that signal to be expected. If new evidence undermines a prior kill, explicitly reopen the candidate and give the reason; labels are revisable conclusions, not permanent states. Candidate updates and the overall verdict have different meanings and scopes.

8. STOP, give an overall verdict, and hand back control

End the adversarial pass when any of these holds:

• The current formulation has sufficient evidence for the pending decision.
• No meaningful materially different alternative remains, or remaining alternatives are too weak to justify delaying the decision.
• Further investigation has poor information gain relative to cost, risk, or time.
• Another formulation is sufficiently supported to replace the original.
• The original formulation is dead and no useful cheap investigation remains.
• Essential discriminating evidence is unavailable and no useful proxy exists; hand off the minimal evidence request rather than stall indefinitely.

Sufficiency is proportional to the pending intervention and evidence quality. Running out of tests, access, time, or alternatives is not itself evidence for KEEP. Do not become contrarian for its own sake. KEEP is a normal successful outcome.

Use exactly one overall verdict about the original Current Problem Formulation:

For REFORMULATE, always output Reformulated Problem: with the replacement and its supporting evidence. KILL does not assert a replacement; REFORMULATE does. If several descriptions fit, choose the most informative supported verdict: an established replacement takes REFORMULATE over KILL or WEAKEN; absent one, contradicted necessary assumptions justify KILL, reduced support justifies WEAKEN, and non-discrimination justifies INSUFFICIENT EVIDENCE.

Skill STOP != Host Agent STOP. KEEP ends this skill's responsibility. When the original request already authorizes implementation, the host can continue in the same turn without asking again. Validation is not new permission for extra work. KEEP validates the problem framing, not the effectiveness of the proposed solution. Mark any subsequent diagnosis or implementation as host/downstream continuation so a completed framing pass does not silently turn into a root-cause workflow.

After REFORMULATE, the host may run a brief validation pass with the replacement as the new Current Problem Formulation, reusing the evidence already gathered. If it warrants KEEP, release it without redundant testing; then the host can solve within the user's authorized scope. Do not implement the original solution merely because an alternative explanation was found. The skill itself never designs it.

9. Default to compact output; disclose detail as needed

Complete the relevant checks, but show only decision-relevant evidence and rationale, not an exhaustive reasoning transcript or hidden chain of thought. Choose the shortest path that preserves the decision, discrimination, and scope:

These are soft presentation guides, not word quotas or correctness gates; use equivalent brevity in other languages. Normal output usually shows 2-4 live formulations including the current one, at most 3-4 by default. Do not conceal a consequential challenger to fit that range: merge genuine redundancies, or expand the output when needed. A supported single formulation needs no padding.

Keep provenance compact (for example, "report.txt, supplied summary"), never omit it or upgrade it. State whether the one primary comparison ran or is pending; preliminary localization is not a discriminating test when all candidates predict it. Give one clear verdict and release status, plus Reformulated Problem: when required. Headings are optional; do not mechanically reproduce all workflow steps.

On later turns, use delta output: new evidence/source, what changed, affected candidate updates and scope, a new Primary Test only if useful, then verdict/handoff. Do not repeat unchanged observations, formulations, assumptions, or predictions. Reference prior candidates briefly so the update remains intelligible.

Do not repeat the user prompt, narrate internal exploration, stage a debate, explain the methodology unless asked, or list discarded/merged candidates unless their change matters now. Stop once the framing decision is supported; mark any authorized downstream work separately and let the host continue.

falsify-the-problem — An Agent Skill for Problem Framing

English | 简体中文

Project website

Stop solving the wrong problem faster.

An open-source Agent Skill for problem framing and assumption testing across coding, debugging, architecture, scientific research, and product decisions. It asks Are we solving the right problem? before proposing solutions. Start with installation and a first invocation.

Illustrative scenarios, not production incidents or measured results:

One Markdown file. No dependency, API key, server, or database. The distinctive focus is a falsifiable framing, materially different alternatives, one primary discriminating test per investigation round, evidence updates, and an explicit handoff before solution design. KEEP is a successful result.

Installation

The complete runtime artifact is SKILL.md. Evaluation files are development materials, not dependencies. The skill itself needs no package, API key, or service; your host agent has its own setup requirements.

Codex: choose an installation scope

Codex is one way to use this cross-domain skill. According to the official skills documentation, local skills can live in either of these locations:

Choose one scope. Run the commands below in a terminal, not in the Codex prompt. Git must be installed. Project commands start in the root of the project where you want to use the skill; user commands can start in any directory. These commands clone into a new folder and will fail if the destination is already populated. Keep an existing installation instead of overwriting it.

Windows PowerShell, project scope:

New-Item -ItemType Directory -Force -Path .agents/skills | Out-Null
git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problem

Windows PowerShell, user scope:

$skillRoot = Join-Path $env:USERPROFILE '.agents/skills'
New-Item -ItemType Directory -Force -Path $skillRoot | Out-Null
git clone https://github.com/ordinary-s/falsify-the-problem.git (Join-Path $skillRoot 'falsify-the-problem')

macOS / Linux, Bash or Zsh, project scope:

mkdir -p .agents/skills
git clone https://github.com/ordinary-s/falsify-the-problem.git .agents/skills/falsify-the-problem

macOS / Linux, Bash or Zsh, user scope:

mkdir -p "$HOME/.agents/skills"
git clone https://github.com/ordinary-s/falsify-the-problem.git "$HOME/.agents/skills/falsify-the-problem"

Alternatively, download or clone this repository elsewhere and copy only SKILL.md into the chosen falsify-the-problem folder. Codex detects skill changes automatically; restart it if the skill does not appear. In Codex CLI or the IDE extension, use /skills or type $ to select the skill.

First invocation in Codex CLI or the IDE extension

Paste this into the Codex prompt, not PowerShell or a shell:

$falsify-the-problem
Our API has become slower and we are considering a cache.
Inspect the available evidence, distinguish observations from assumptions,
and test the problem framing before proposing a solution.

The local setup commands do not establish skill activation or model behavior. macOS and Linux installation and cross-host behavior have not been verified in this documentation change. See Usage for ordinary-language examples.

Other hosts

1. Download or clone this repository into a local falsify-the-problem folder.
2. For a host with skill-folder support, place the folder containing SKILL.md in that host's documented skill location and use its normal reload/discovery procedure. You can copy just SKILL.md into a folder of the same name.
3. For a host that accepts reusable Markdown instructions, attach or load the complete SKILL.md and explicitly request its use before solving the task.

Paths, discovery, invocation syntax, and tool permissions are host-specific. The skill is designed to be portable across agents that support reusable Markdown instructions or skill-like workflows; identical behavior across hosts is not guaranteed.

Why

A requested solution often contains an untested definition of the problem. Optimizing that solution can make the wrong investigation expensive. This skill exposes the definition and seeks cheap evidence that could overturn it. It is useful across coding, debugging, architecture, research, and product work. The protocol stays the same across domains.

How it works

1. Preserve observations and their provenance; separate interpretation, causal hypothesis, problem formulation, and proposed solution.
2. Identify the current formulation and plausible, materially different alternatives.
3. Find load-bearing assumptions and derive distinct observable predictions.
4. Select one Primary Discriminating Test per round, balancing information against cost, risk, and time. Inspect available, permitted, cheap evidence directly.
5. Strengthen, weaken, kill, or merge candidates from the result. Investigate again when another discriminating observation is worth its cost.
6. Stop with an overall verdict and return control to the host agent.

Before choosing the Primary Discriminating Test, the skill checks whether all live formulations share the same consequential, untested frame. It may add one grounded outside-frame challenger; unexplained evidence can reopen that check. Output defaults to compact Fast / Normal / Deep paths, with later turns showing only changed evidence and decisions.

BYPASS is an activation decision, not a verdict. KILL does not imply a known replacement. After REFORMULATE, the host can validate the replacement in a brief new pass using existing evidence, obtain KEEP, and continue within task scope.

STOP solutioning does not mean STOP investigating. Skill STOP does not mean Host Agent STOP. After KEEP, a host already authorized to implement can proceed in the same turn without asking for permission again. No new authority is granted.

Quick example

User: "API P99 increased from 300 ms to 2.8 s. Add Redis."

• Observation: user-reported latency increase; not independently verified.
• Interpretation: the database is slow. Causal hypothesis: repeated reads delay requests. Solution: Redis. These are inferred, not measured facts.
• Current formulation: database service time materially dominates the slow tail.
• Alternatives: queueing/capacity, downstream dependency delay, or a measurement artifact. Runtime pauses and workload shifts may also matter if evidence suggests them.
• Load-bearing assumption: slow requests actually spend material time in DB work.
• Primary Discriminating Test: inspect an available, representative sample of slow and typical requests from the same window; reconcile end-to-end latency with queue, DB, and downstream time in one comparison.
• Outcomes: elevated DB time strengthens the current framing; rising queue wait with stable DB time weakens it and supports capacity; inconsistent totals support measurement trouble. Sparse or overlapping spans leave uncertainty.

If the evidence has not been inspected: INSUFFICIENT EVIDENCE, solutioning NOT YET. The host should inspect it when accessible. If reliable observations instead establish queue wait as the material delay: REFORMULATE, with that revised problem. Neither result automatically authorizes a Redis implementation. This is an illustrative example, not a reported production incident.

When to use

Invoke explicitly when you want the current framing challenged. Automatic use requires both meaningful formulation uncertainty and non-trivial intervention cost or risk. Typical signals include:

• A proposed solution arriving before the problem has been tested.
• A causal hypothesis being treated as an observation.
• Repeated failed debugging or research attempts that preserve the same framing.
• Technology selection or an expensive product change preceding diagnosis.

When NOT to use

Usually skip automatic activation for verified diagnoses, obvious compiler errors, typos, formatting, transl