Experiments, Causal Effects & Predictive Models
Design defensible experiments, estimate causal effects, and build leakage-safe tabular models with decision-ready uncertainty.
Before & after
Same prompt, same agent — the only difference is the skill.
A/B testing is a great way to validate ideas before rolling them out. You'll want to define a clear hypothesis about what change might improve conversion. You should think about how to randomly split your users into test and control groups. It's important to have a large enough sample size so your results are statistically significant. You might want to look at multiple metrics, not just conversion, to make sure you're not hurting other parts of the experience. You could run the test for a week or two to gather enough data. Then you'll analyze the results using statistical methods to see if there's a real difference. Would you like help defining your hypothesis or choosing metrics?
A/B TEST DESIGN: Subscription Conversion HYPOTHESIS Changing pricing display from annual to monthly (with annual option) will increase conversion without reducing LTV. SAMPLE SIZE CALCULATION Baseline conversion: 2.8% Minimum detectable effect: 0.4 percentage points (15% relative lift) Statistical power: 90% Significance: 0.05 (two-tailed) Sample per arm: 18,400 users Total required: 36,800 users Expected duration: 12 days TEST SETUP Control: Annual pricing, 49,000/year, savings emphasis Treatment: Monthly option first (4,099/month), annual in fine print Randomization: User ID hash to treatment, stratified by cohort ANALYSIS PLAN Primary: Two-proportion z-test (alpha 0.05) Secondary: Kaplan-Meier curves for 90-day retention (Logrank test) Monitored: Revenue per convert, CAC, support tickets Sensitivity: Bonferroni-corrected for secondaries (alpha 0.025) FAILURE CRITERIA Treatment LTV-12m lower than control: Do not roll out Churn spike month 2: Revert and investigate Lift below 10%: Extend to 21 days (underpowered) INTERPRETATION GUIDE 0-5% lift: Implement if change cost is low 5-15% lift: Implement with monitoring 15%+ lift: Fast-track to full rollout
About this skill
Your A/B test shows a lift, but should you ship it? Your model scores well, but will it work on next month's customers? This skill checks assignment, causal assumptions, and validation splits before turning results into a decision.
What it does
- Sizes experiments around baseline rates, worthwhile effects, eligible traffic, and the actual randomization unit.
- Analyzes treatment effects with confidence intervals, practical thresholds, and appropriate handling of repeated looks and multiple comparisons.
- Evaluates whether observational data supports causal inference, including difference-in-differences and its identification limits.
- Builds reproducible tabular classification and regression workflows with preprocessing inside validation folds.
- Detects leakage from future information, repeated customers, overlapping label windows, and features created after treatment.
- Selects evaluation metrics, classification thresholds, and deployment checks that reflect the decision's costs.
What you get
A scoped experiment plan, causal analysis, or predictive modeling workflow with assumptions, uncertainty, and a recommended next action. When implementation is requested, you get self-contained Python, R, or SQL matched to the supplied schema, with execution status stated explicitly.
Best for
Product analysts, applied data scientists, and technical teams making decisions from experiments, observational records, or structured tabular models.
SKILL.md preview
---
name: experiments-causal-effects-predictive-models
description: >-
Design and analyze experiments, estimate observational causal effects, and build tabular predictive models by checking identification, data leakage, validation design, and decision uncertainty. Use when the user says "is this A/B test a winner?", "how many users do we need?", "did this rollout cause the increase?", "my model performs well offline but fails in production", "build a churn prediction model", "review this difference-in-differences analysis", or "turn these results into a business decision".
---
# Experiments, Causal Effects & Predictive Models
… (sign up to view the full skill)More development skills
View all Development skills →Pandas Data Wrangling Without Silent Errors
Correct, fast pandas code for merging, cleaning, groupby, reshaping, and time series, with checks that catch silent row loss.
Apache Spark Job Tuning and Debugging
Finds why a Spark job is slow or failing from the plan and UI metrics, then gives the code and config fix plus how to verify it.
Reliable Data Pipelines and Analytics Models
Build and repair ETL/ELT pipelines with replay-safe writes, explicit data contracts, and actionable quality checks.
Statistical Tests, Power & A/B Readouts
Get hypothesis tests, effect sizes, confidence intervals, and sample-size calculations with experiment validity checks.