Experiment Designer
Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.
Before & after
Same prompt, same agent — the only difference is the skill.
Designing a good A/B test requires thinking through what you actually want to measure and what change you think will improve things. You'll want to state a clear hypothesis, decide on your success metric, figure out how many people you need in each group to detect a meaningful difference, and think about how long the test should run. You'll also want to make sure you're testing only one variable at a time so you know what's causing the change. It's helpful to think about external factors that might skew results—like day of week effects or seasonal patterns. If you can tell me what aspect of your signup flow you want to improve and what your current conversion rate is, I can help you design a more specific test.
A/B TEST DESIGN: SIGNUP CTA BUTTON COLOR HYPOTHESIS "Changing the primary CTA button from blue to orange will increase signup conversion by 8% due to higher visual contrast and reduced decision fatigue." VARIANT SPECIFICATIONS Control: Blue button (#0071E3), standard text "Sign Up" Treatment: Orange button (#FF6B35), text unchanged -> Only variable changed (follows testing discipline) PRIMARY METRIC Signup completion rate (users who filled all fields + confirmed email) Current baseline: 18.2% | Target improvement: 8% → 19.7% Secondary metrics: Time to complete signup, email confirmation click rate STATISTICAL REQUIREMENTS • Sample size: 10,500 users per variant (87% power, 5% significance level) • Daily traffic: Assuming 1,200 users/day on signup • Runtime: 18 days (9 days per variant if split 50/50) • Minimum detectable effect: 1.5% (0.27 percentage points) TEST EXECUTION • Randomization: By session ID (consistent experience per user) • Exclusions: Admins, internal testers, bot traffic • Stratification: Segment by traffic source (organic vs paid) to account for intent differences ANALYSIS & DECISION RULES -> If orange > blue AND p < 0.05: Deploy orange (statistically significant win) -> If orange = blue (difference < 0.5%): Deploy control (no meaningful difference, revert) -> If blue > orange: Investigate qualitative feedback (UX issue with orange variant?) RISK MITIGATION • Kill switch: Stop test if any variant drops below 15% (alert on anomaly) • Peeking rule: No analysis until reaching 80% planned sample size • Post-test: Run holdout (10% users keep control) to measure longer-term impact
About this skill
name: experiment-designer description: Use when Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.
Experiment Designer
Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.
What you get
- Public GitHub repo (alirezarezvani/claude-skills)
- the experiment-designer skill folder with SKILL.md. Part of a 337-skill / 30-agent / 70-command install.
Customize your output
- Fork the repo and adapt the skill's instructions and references to your workflow.
Example output
Activates automatically when your request matches Experiment Designer; chains with the other skills, agents, and commands in the collection.
Best for
Creators, builders, and teams using Claude Code.
SKILL.md preview
---
name: experiment-designer
description: Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.
version: 1.0.0
category: Business & Ops / Product
author: AgentVolt
license: proprietary
tags:
- business-ops
- product
---
# Experiment Designer
Turns a product idea into a testable experiment: a falsifiable hypothesis, the right sample size, a prioritization score, and a disciplined read of the result.
## When to use
… (sign up to view the full skill)More business & ops skills
View all Business & Ops skills →Product Manager Toolkit
Comprehensive toolkit for product managers including RICE prioritization, customer interview analysis, PRD templates, discovery frameworks, and go-to-market strategies.
Product Skills
Coordinates a set of product-management sub-skills — prioritization, discovery, UX research, competitive teardown, analytics, roadmapping.
UX Researcher & Designer
UX research and design toolkit for Senior UX Designer/Researcher including data-driven persona generation, journey mapping, usability testing frameworks, and research synthesis.
Product Discovery
Use when validating product opportunities, mapping assumptions, planning discovery sprints, or testing problem-solution fit before committing delivery resources.