Bug Diagnosis with Discriminating Experiments
Isolate bug causes with explicit hypotheses, discriminating tests, evidence logs, and verified fixes.
Before & after
Same prompt, same agent — the only difference is the skill.
Flaky tests are frustrating. Here are common causes: - **Timing/race conditions** — async operations completing in different orders - **Shared state** — tests polluting each other when run in sequence - **Test order dependency** — passing alone but failing in a suite - **External dependencies** — network calls, real time, filesystem - **Random data** — faker or random seeds producing edge cases occasionally Try running the test in isolation many times to see if it fails alone. If it only fails in the suite, it's likely shared state or ordering. Adding retries can hide the problem, so I'd avoid that until you understand the cause. What does the test do?
One in twenty means you can't verify a fix by running it once — that's
the actual problem, and it's the first thing to solve.
BUILD THE LOOP BEFORE TOUCHING THE CODE
1. Reproduce on demand. Run it 200× in a loop, capture failures.
If it fails ~10 times, you now have a signal.
If it never fails alone, it's suite interaction — go to step 3.
2. Establish a baseline failure rate. Write the number down. Every
"fix" gets measured against it, not against a single green run.
3. Bisect the environment, not the code:
alone, 200× → isolates the test itself
full suite, random order → shared state or ordering
full suite, fixed seed → random data
full suite, --workers=1 → concurrency
Whichever column changes the rate is your class of bug. This is
four experiments and it eliminates three of four possibilities.
INSTRUMENT THE FAILURE, DON'T READ IT
On failure only, dump: wall-clock timestamps per step, the random
seed, test execution order, and any shared fixture state. A flake you
can't reproduce is a flake you didn't record enough about.
THEN FORM ONE HYPOTHESIS
State it so it can be wrong: "the assertion runs before the debounce
fires." Change exactly one thing. Re-run 200×. Compare to baseline.
10/200 → 0/200 is a fix.
10/200 → 3/200 is a different bug underneath, not partial success.
WHAT NOT TO DO
Don't add a retry, don't add a sleep, don't mark it skip. Each one
converts a reproducible signal into a silent one, and you'll pay for
it when the same race shows up in production where there's no retry.
What does the loop give you at 200 runs — does it fail alone?About this skill
The bug disappeared after a change, but you still cannot explain why. Or three attempted fixes later, it keeps coming back. This skill turns debugging into a short evidence loop: state competing causes, choose a test that separates them, record what happened, and fix the mechanism the evidence supports.
What it does
- Converts vague bug reports into observable failures with explicit trigger conditions.
- Separates established facts, plausible causes, and missing evidence.
- Designs small experiments with different predicted outcomes for competing hypotheses.
- Narrows failures across inputs, environments, state transitions, and system boundaries.
- Investigates intermittent bugs using controlled timing, recorded attempts, and comparable conditions.
- Verifies that a fix addresses the cause and checks nearby paths for the same defect.
What you get
A compact diagnosis with an evidence trail, the best-supported cause, and the next decisive experiment or verified fix. When execution access is unavailable, you get precise probes and expected outcomes with unverified claims clearly labeled.
Best for
Developers using Claude or Claude Code to investigate recurring bugs, flaky behavior, unexplained regressions, and fixes that seem to work without establishing the cause.
SKILL.md preview
---
name: bug-diagnosis-with-discriminating-experiments
description: Diagnoses bugs through an evidence loop that compares competing hypotheses, runs small discriminating experiments, and verifies the causal mechanism before fixing it. Use when the user says "help me diagnose this bug", "my fix keeps breaking again", "this only fails sometimes", "why does this work locally", "stop guessing at fixes", "find the root cause", or "the error disappeared but I do not know why".
---
# Bug Diagnosis with Discriminating Experiments
… (sign up to view the full skill)More development skills
View all Development skills →React and Next.js Regression Test Coverage
Generate React and Next.js tests with Jest, Testing Library, MSW, and Playwright, prioritized by regression risk.
Playwright E2E Tests and Flake Repair
Build reliable Playwright tests, repair flakes from evidence, migrate suites, and configure faster, diagnosable CI.
Pre-Ship Verification for Code Changes
Before calling a code change done, Claude grades every claim by evidence, tries to break it, and reports what's verified vs assumed.
TestRail Case and CI Result Sync
Sync automated tests to TestRail cases and scoped runs with stable mappings, retry handling, and verified result counts.