A/B Test Power and Validity Designer
Exact A/B sample sizes and MDE with math shown, plus the traps: unit mismatch, multiple variants, peeking, guardrail power, and SRM.
Before & after
Same prompt, same agent — the only difference is the skill.
Designing a good A/B test requires thinking through what you actually want to measure and what change you think will improve things. You'll want to state a clear hypothesis, decide on your success metric, figure out how many people you need in each group to detect a meaningful difference, and think about how long the test should run. You'll also want to make sure you're testing only one variable at a time so you know what's causing the change. It's helpful to think about external factors that might skew results—like day of week effects or seasonal patterns. If you can tell me what aspect of your signup flow you want to improve and what your current conversion rate is, I can help you design a more specific test.
A/B TEST DESIGN: SIGNUP CTA BUTTON COLOR HYPOTHESIS "Changing the primary CTA button from blue to orange will increase signup conversion by 8% due to higher visual contrast and reduced decision fatigue." VARIANT SPECIFICATIONS Control: Blue button (#0071E3), standard text "Sign Up" Treatment: Orange button (#FF6B35), text unchanged -> Only variable changed (follows testing discipline) PRIMARY METRIC Signup completion rate (users who filled all fields + confirmed email) Current baseline: 18.2% | Target improvement: 8% → 19.7% Secondary metrics: Time to complete signup, email confirmation click rate STATISTICAL REQUIREMENTS • Sample size: 10,500 users per variant (87% power, 5% significance level) • Daily traffic: Assuming 1,200 users/day on signup • Runtime: 18 days (9 days per variant if split 50/50) • Minimum detectable effect: 1.5% (0.27 percentage points) TEST EXECUTION • Randomization: By session ID (consistent experience per user) • Exclusions: Admins, internal testers, bot traffic • Stratification: Segment by traffic source (organic vs paid) to account for intent differences ANALYSIS & DECISION RULES -> If orange > blue AND p < 0.05: Deploy orange (statistically significant win) -> If orange = blue (difference < 0.5%): Deploy control (no meaningful difference, revert) -> If blue > orange: Investigate qualitative feedback (UX issue with orange variant?) RISK MITIGATION • Kill switch: Stop test if any variant drops below 15% (alert on anomaly) • Peeking rule: No analysis until reaching 80% planned sample size • Post-test: Run holdout (10% users keep control) to measure longer-term impact
About this skill
Get the numbers for a product experiment right before you spend weeks of traffic on it. Every sample size and MDE comes with the formula, inputs, and arithmetic shown, and every design is checked for the errors that quietly invalidate results.
What it does
- Computes exact per-arm sample sizes for conversion and continuous metrics, or the smallest effect your traffic can detect
- Catches relative vs absolute MDE mix-ups, unequal-split costs, and multi-variant alpha corrections
- Handles account-level randomization (design effect), ratio metrics (delta method), triggering, heavy tails, and CUPED
- Sets guardrails as non-inferiority tests and states the drop they can actually detect
- Quantifies the peeking problem, provides sequential stopping boundaries, and diagnoses sample ratio mismatch
What you get
A quick calculation with the math shown, a findings list for an existing test plan, or a full design spec: unit, metrics, sample, duration, stopping rule, validity checks, and decision rule.
Best for
Product managers, analysts, and growth engineers running A/B tests where a wrong call is expensive.
SKILL.md preview
---
name: experiment-designer
… (sign up to view the full skill)More research & data skills
View all Research & Data skills →Source-Reconciled Research Brief
Turn supplied links into a concise brief with claim-level citations, reconciled source conflicts, and explicit evidence gaps.
Source-Traced Research Bundle
Compile many sources into one bundle where every claim is atomic, sourced, dated, tiered, and conflict-checked, in Markdown or JSON.
Research Routing and Cited Briefings
Route research to the right method, then produce a cited briefing with evidence gaps, disagreements, and a source audit.
X/Twitter Research with Cited Post Evidence
Research X topics, threads, and accounts with cited posts, deliberate sampling, and clear limits on sentiment claims.