A/B Test Setup
When the user wants to plan, design, or implement an A/B test or experiment.
Before & after
Same prompt, same agent — the only difference is the skill.
A/B testing checkout sounds like a good idea. You'd want to run different versions of the page and see which converts better. The most common things to test are like the button color, copy, or number of form fields. You'd need to make sure you have enough traffic to get statistical significance. It depends on your current conversion rate and how many visitors you get. What version are you thinking of testing, and how long do you want to run it?
A/B TEST DESIGN: Mobile Checkout Conversion HYPOTHESIS "Reducing form fields from 8 to 5 on mobile checkout will increase completion rate by 8% (from 62% to 67%)" Logic: Drop optional upsell questions + address verification (auto-fill via ZIP + country) to reduce cognitive load Risk: Might lose shipping accuracy on first-time users if address validation too permissive TEST PARAMETERS Metric: Completion rate (orders placed / checkouts started) on mobile only (≤768px) Current baseline: 62% (measured last 30 days) Effect size target: 8% relative lift (62% → 67%) Statistical power: 80% (standard) Significance threshold: p<0.05 SAMPLE SIZE CALCULATOR • Current CVR: 62%, Baseline traffic: 8,000 mobile checkouts/month • Daily mobile traffic: 267 checkouts • Minimum sample per variant: 2,900 (need 43 days at 50/50 split) • Safe duration: 50 days to reach 95% confidence CONTROL VARIANT (A) 8 fields: email, shipping address, city, state, ZIP, country, phone, promo code Button label: "Place Order" Estimated form completion time: 90 seconds (from interaction analytics) TREATMENT VARIANT (B) 5 fields: email, ZIP (triggers address lookup), country, phone, promo code Auto-fill: Address pulled from ZIP + country (using postal API) Button label: "Complete Order" (minor copy test) Estimated form completion time: 45 seconds SUCCESS CRITERIA (All must pass to declare winner) ✓ Treatment CVR > 67% (statistically significant vs 62% baseline) ✓ Error rate (invalid addresses) <3% (ensure address lookup didn't break deliveries) ✓ Customer support impact: chargeback/address disputes flat or declining ✓ Checkout abandonment at address step <5% (users don't get stuck on auto-fill) FAILURE MODE: Declare no winner if ✗ Treatment CVR = 64% (lifted, but not 8% target, might be random noise) ✗ Error rate jumps to 8% (address accuracy trade-off not worth 5% CVR gain) ✗ Support tickets for "wrong address shipped" spike 40% (hidden cost) TIMELINE Days 1–2: QA both variants, code review Day 3: Deploy at 50/50 split to 5% of mobile traffic (test 1,350 users), monitor dashboards Days 4–8: Ramp to 50% mobile traffic if no critical issues (13,500 users in test) Days 9–50: Run to completion (2,900 per variant) Day 51: Analysis, decision, rollout if winner declared SAFETY RAILS • Kill switch: If error rate >5% immediately revert to A (automated alert at 2%) • Monitoring: Watch revenue per session, support tickets, payment declined rate daily • Holdout: Keep 5% of traffic on control to detect drift over 50 days IMPORTANT: Results are illustrative sample numbers. Actual effect depends on your traffic, existing CVR, and user base. This is not financial or conversion advice.
About this skill
name: ab-test-setup description: Use when When the user wants to plan, design, or implement an A/B test or experiment.
Ab Test Setup
When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "conversion experiment," "statistical significance," or "test this." For tracking implementation, see analytics-tracking.
What you get
- Public GitHub repo (alirezarezvani/claude-skills)
- the ab-test-setup skill folder with SKILL.md. Part of a 337-skill / 30-agent / 70-command install.
Customize your output
- Fork the repo and adapt the skill's instructions and references to your workflow.
Example output
Activates automatically when your request matches Ab Test Setup; chains with the other skills, agents, and commands in the collection.
Best for
Creators, builders, and teams using Claude Code.
SKILL.md preview
---
name: ab-test-setup
description: Use this skill when the user wants to plan, design, or implement an A/B test or experiment, including split tests, multivariate tests, or hypothesis-driven conversion experiments.
version: 1.0.0
category: Marketing
author: AgentVolt
license: proprietary
tags:
- marketing
---
# Ab Test Setup
Designs a rigorous A/B or multivariate test — hypothesis, variants, sample size, and success criteria — before any implementation begins.
## When to use
- Planning a split test, A/B test, or multivariate experiment
… (sign up to view the full skill)Featured in
More marketing & growth skills
View all Marketing & Growth skills →Paid Ads
When the user wants help with paid advertising campaigns on Google Ads, Meta (Facebook/Instagram), LinkedIn, Twitter/X, or other ad platforms.
Onboarding CRO
When the user wants to optimize post-signup onboarding, user activation, first-run experience, or time-to-value.
Social Media Manager
When the user wants to develop social media strategy, plan content calendars, manage community engagement, or grow their social presence across platforms.
Analytics Tracking
Set up, audit, and debug analytics tracking implementation — GA4, Google Tag Manager, event taxonomy, conversion tracking, and data quality.