Chaos Experiment and Game Day Planner
Plans chaos experiments and game days: hypothesis, blast radius, abort conditions, tool manifests, rollback, and post-run review.
Before & after
Same prompt, same agent — the only difference is the skill.
Designing chaos experiments is a good way to test how resilient your system is to failures. The basic idea is to introduce failures in a controlled way and see how your system responds. You might experiment with things like taking down a service, increasing latency, or simulating a database failure. You'd want to define what you're trying to learn, set up the experiment carefully so you can control the failure, observe what happens, and then have a rollback plan in case things go wrong. You should probably start with smaller, lower-risk experiments and work your way up to more severe scenarios. If you can tell me about your system architecture and what failure scenarios you're most concerned about, I can help you design a specific experiment.
CHAOS EXPERIMENT DESIGN: API LATENCY INJECTION EXPERIMENT OBJECTIVE Test: How does the system handle when payment-processing API responds in 5 seconds instead of 200ms? Success criteria: Orders queue instead of failing; no 500 errors; recovery within 30 seconds of latency removal EXPERIMENT MANIFEST Title: Payment API Degradation Timing: Tuesday 2pm UTC (low-traffic window, not Friday evening) Duration: 10 minutes (bounded blast radius) Target service: Payment processor (East region first) FAILURE INJECTION MECHANISM Tool: Chaos Monkey (Netflix; inject latency via proxy) Configuration: Service: paymentgateway-api Failure type: Network latency Latency added: 4.8 seconds (5s total response time) Percentage: 100% (all payment calls) Region: us-east-1 (primary only, not backup) METRICS TO MONITOR (Real-time dashboard) • Payment success rate (target: stays > 90%) • API response time p99 (expect: 5+ seconds during experiment) • Order queue depth (should increase, then drain after recovery) • Error rate (target: stays < 1%) • Customer-facing latency (web checkout time) BLAST RADIUS ASSESSMENT • Blast radius: Payment flow only; checkout page slows but doesn't error • Customers impacted: Checkout users during 10-min window (est. 1-2K transactions) • Critical risk: If queue overflows -> orders lost (MITIGATED by queue monitoring) ROLLBACK PROCEDURE Manual kill-switch: Kill chaos experiment pod (1 minute to recover) Automatic rollback: If order queue > 5K, abort experiment (prevents cascade) Verification: Confirm payment success rate > 95% within 2 min of rollback POST-MORTEM TEMPLATE • What we learned: System handles graceful degradation; queue strategy works • Gaps identified: Alerts triggered late (should alert at queue 1K) • Follow-up: Add health-check endpoint for payment queue depth • Next experiment: Test database failover (higher risk, requires pre-approval)
About this skill
"Let's test resilience" usually ends one of two ways. You break staging and learn nothing, or you break production and learn too much. This skill plans chaos work the way an experienced SRE does. You get a falsifiable hypothesis on a business metric and a fault sized against your real timeouts and retries. You also get an abort that has been tested before the experiment starts, and a ladder for growing blast radius in safe steps.
What it does
- Turns "what if X fails" into a steady-state metric, a numeric hypothesis, and a held / refuted / inconclusive verdict rule
- Checks your timeout and retry config before sizing the fault, so a latency test doesn't quietly turn into a timeout test or a retry storm
- Writes ready-to-adapt manifests for Chaos Mesh, LitmusChaos, AWS FIS, Toxiproxy, or tc netem, with stop conditions tied to real alarms
- Builds an abort and rollback plan, including manual restore steps for faults that don't revert on their own
- Designs game days with separate roles (conductor, responders, safety officer, scribe), injects, a timeline, and detection and mitigation timings
- Turns results into owned fix tickets and a rerun plan
What you get
An experiment plan you can run: hypothesis, blast radius, fault manifest, abort and rollback steps, a go/no-go checklist, and a results template. Larger requests also get a full game day run sheet.
Best for
SREs, platform engineers, and on-call leads running their first chaos experiments or getting a game day program going on Kubernetes or AWS.
SKILL.md preview
---
name: chaos-experiment-game-day-planner
… (sign up to view the full skill)More development skills
View all Development skills →Claude Code Cost and Latency Routing
Configure Claude Code Router with model tiers, explicit fallbacks, compatibility checks, and a cost-aware validation plan.
CI/CD Pipeline Builder with Gated Deploys
Generates CI/CD workflows from your repo's real stack: cached fast-fail checks, pinned actions, and staging-to-prod deploys with gates.
Terraform Modules and Safe Deployments
Get reusable Terraform modules, safe state migrations, explicit provider wiring, and CI gates tied to reviewed plans.
Dockerfile and Compose Review: Size, Cache, Security
Rewrites Dockerfiles and compose files for smaller images, cache-friendly builds, and non-root, secret-free containers, with reasons.