Statistical Tests, Power & A/B Readouts
Get hypothesis tests, effect sizes, confidence intervals, and sample-size calculations with experiment validity checks.
Before & after
Same prompt, same agent — the only difference is the skill.
An 8% lift sounds promising, but whether it's real depends on a few things like sample size, how long the test ran, and your baseline numbers. Larger sample sizes and longer test durations generally give you more confidence in the result. Without seeing your actual data, I can't say for certain if the lift is statistically significant. If you share the conversion rates and sample sizes for both variations, I could look at the numbers more carefully.
HYPOTHESIS TEST RESULTS TEST SETUP Control conversion rate: 12.4% (n=8452 users) Treatment conversion rate: 13.4% (n=8396 users) Observed lift: 8.1% (relative) Absolute difference: +1.0pp STATISTICAL ANALYSIS Two-proportion Z-test Z-statistic: 2.14 P-value: 0.032 (p < 0.05) Conclusion: STATISTICALLY SIGNIFICANT at 95% confidence level CONFIDENCE INTERVAL 95% CI on lift: 0.1-1.9pp Interpretation: We're 95% confident true lift is between 0.1 and 1.9 percentage points Note: Lower bound (0.1pp) is practically small EFFECT SIZE Cohen's h: 0.04 (small effect) Practical significance: Modest improvement, not transformational FALSE POSITIVE RISK P-value 0.032 vs typical threshold 0.05 Risk of false positive: 3.2% (acceptable) But: Running multiple tests or peeking at data increases false positive risk SAMPLE SIZE ADEQUACY Powered for: 80% detection of 1.2pp lift Actual detected: 1.0pp lift Duration: 14 days at current traffic (sufficient) RECOMMENDATION Deploy treatment. The lift is real and statistically significant. Caution: Effect is small; monitor post-launch for sustainability. Next test: Run longer (28 days) to confirm lift persists over time. IMPORTANT: This analysis is statistical illustration, not investment or trading advice
About this skill
“B is up 8%, but is that enough to ship?” Raw lift hides uncertainty, and a small p-value can hide a broken experiment. This skill turns counts, summary statistics, or supplied data into a statistical readout that separates effect size, uncertainty, validity, and the decision the evidence supports.
What it does
- Selects tests for independent groups, paired measurements, sparse conversion counts, and clustered experiments.
- Checks assignment balance, denominators, repeated observations, missing outcomes, and early stopping before interpreting results.
- Reports absolute effects, relative lift, confidence intervals, and p-values together.
- Calculates sample size from a baseline, minimum detectable effect, significance level, power, and allocation.
- Handles multiple comparisons and distinguishes inconclusive results from evidence of equivalence.
- Reviews existing analyses for incorrect tests, misleading significance claims, and unsupported launch recommendations.
What you get
A concise decision readout or experiment-sizing calculation, with assumptions, reproducible formulas or inline code when needed, and the next action supported by the evidence. Quick questions get quick answers; fuller analyses include validity checks and calculation details.
Best for
Analysts, product teams, engineers, and researchers who need defensible hypothesis tests and experiment interpretations without confusing statistical significance with business value.
SKILL.md preview
---
name: statistical-tests-power-ab-readouts
description: Analyze hypothesis tests and A/B results by validating the experimental unit, selecting an appropriate test, and reporting effect sizes with confidence intervals; calculate sample sizes from explicit design assumptions. Use when the user says "is this A/B result significant?", "how many users do we need?", "my test keeps flipping winners", "check this statistical analysis", "compare these before-and-after measurements", "does no significance mean no effect?", or "we tested several metrics and found a winner".
---
# Statistical Tests, Power & A/B Readouts
… (sign up to view the full skill)More development skills
View all Development skills →Pandas Data Wrangling Without Silent Errors
Correct, fast pandas code for merging, cleaning, groupby, reshaping, and time series, with checks that catch silent row loss.
Apache Spark Job Tuning and Debugging
Finds why a Spark job is slow or failing from the plan and UI metrics, then gives the code and config fix plus how to verify it.
Reliable Data Pipelines and Analytics Models
Build and repair ETL/ELT pipelines with replay-safe writes, explicit data contracts, and actionable quality checks.
Answer Business Questions with Validated SQL and Charts
Turn a business question into a pinned metric definition, checked SQL, the right chart, and a plain finding with confidence and caveats.