Chaos Experiment Design with Blast Radius and Abort Criteria
Chaos experiment plans with SLI hypotheses, blast-radius and error-budget math, auto-abort criteria, and a readout.
Before & after
Same prompt, same agent — the only difference is the skill.
Chaos engineering is a discipline for testing system resilience by intentionally injecting failures. For a payment service, testing database disconnection would be valuable. You'd want to define what success looks like—how quickly the system recovers, whether it gracefully degrades, etc. What's your current infrastructure? Are you using Kubernetes? Do you have monitoring in place? Understanding your setup would help me suggest specific tools and experiment design.
CHAOS EXPERIMENT: Payment Service Database Failover HYPOTHESIS Payment service maintains transaction integrity and recovers to healthy state within 30 seconds when primary database connection is severed. STEADY STATE BASELINE → Payment API processing 1,000 requests per minute → 99.5% success rate on transaction completion → P95 latency at 200ms BLAST RADIUS → Scope: Production payment API pods only (not admin dashboard) → Blast duration: 120 seconds maximum → Automatic rollback: kill switch if success rate drops below 95% EXPERIMENT SEQUENCE Step 1: Kill 30% of database connections Step 2: Observe error handling, request routing to replica Step 3: Measure recovery time and transaction retry behavior Step 4: Inject gradual connection delay (500ms to 5sec) SUCCESS CRITERIA → Zero corrupted transactions during outage → Automatic failover to replica within 10 seconds → No customer-facing errors (client receives retry directive) → Recovery to baseline throughput within 45 seconds ABORT TRIGGERS (stop experiment immediately) → Any transaction data corruption detected → Success rate below 94% for 15 consecutive seconds → P99 latency exceeds 5 seconds POSTMORTEM CHECKLIST → Did failover happen automatically? → Were alerts fired correctly? → Time to detect → time to mitigate → Any manual intervention needed?
About this skill
Most chaos experiments fail in one of three ways. They run against "the system looks healthy" instead of a real metric. Nobody works out how much of the error budget a bad run could burn. Or the abort plan is a person watching a dashboard. This skill turns "let's break the queue and see" into an experiment you can defend in a change review, run safely, and learn from.
What it does
- Writes a falsifiable hypothesis tied to a user-facing SLI, with a measured baseline and noise band.
- Calculates blast radius as traffic share x duration x worst-case failure rate, then checks it against your remaining error budget.
- Sets two-layer abort criteria (guardrail and hard stop) wired to alarms, and sizes the halt time so it is shorter than your tolerance.
- Picks the fault that tests the right mechanism. Latency tests timeouts and breakers. Kills test restarts and failover. It then drafts the spec for Chaos Mesh, Litmus, AWS FIS, Gremlin, Chaos Toolkit, or Toxiproxy.
- Catches broken premises before you run, such as single-replica targets, faults you can't observe, and weaknesses you already know about.
- Turns results into a readout: held, degraded, refuted, or inconclusive, with owned follow-ups and a re-run trigger.
What you get
A one-page experiment plan covering hypothesis, scope, blast-radius math, abort conditions, escalation ladder, and fault config. After the run, a readout with findings and tickets.
Best for
SRE, platform, and backend teams running their first production fault injections or tightening an existing gameday practice on Kubernetes or AWS.
SKILL.md preview
---
name: chaos-experiment-design
… (sign up to view the full skill)More development skills
View all Development skills →Chaos Experiment and Game Day Planner
Plans chaos experiments and game days: hypothesis, blast radius, abort conditions, tool manifests, rollback, and post-run review.
Claude Code Cost and Latency Routing
Configure Claude Code Router with model tiers, explicit fallbacks, compatibility checks, and a cost-aware validation plan.
CI/CD Pipeline Builder with Gated Deploys
Generates CI/CD workflows from your repo's real stack: cached fast-fail checks, pinned actions, and staging-to-prod deploys with gates.
Terraform Modules and Safe Deployments
Get reusable Terraform modules, safe state migrations, explicit provider wiring, and CI gates tied to reviewed plans.