← All skills · Automation · A/B Test Planner

A/B Test Planner — finished output
Runs inClaude Code Codex Cursor Muse Dot Grok Bot Cue OpenClaw Hermes
Needs:The test brief: ab-test.txt (a Hypothesis line, a Page line, a Metric line with the metric name and | Baseline: 5%, an MDE line as a relative lift, a Control line, one Variant line per variant, a Traffic line with visitors per day — plus optional Split, Secondary, Guardrail, Confidence, Power, and Note lines).

Similar Skills

View all skills →

What you get

  • A pre-registered test plan: hypothesis, arms, sample size, duration, and the decision rule, written before the test starts
  • A per-variant sample size computed from your baseline and MDE — 8,160 per variant for a 5% baseline and a 20% lift, cross-checked against the published reference table
  • An expected duration in days at your daily traffic and split — 9 days at 2,000 visitors a day, 50/50
  • A decision rule that names the call point and the win threshold — no peeking, no early stop for a win
  • Every gap, input error, and warning named in the report

Good fit

  • Planning a test before spending traffic on it — and wanting the sample size decided up front
  • Deciding whether a change is even worth testing — a small MDE on a low baseline needs more visitors than you have
  • Judging a finished test against a rule you wrote before seeing the results

Not for

  • Running the test — the script plans it, your testing tool runs it
  • Analysing results or calling a winner from data — the script never reads your analytics
  • Guaranteeing the test will detect the effect — the sample size is a planning estimate, not a promise

Sample output

An example of what this Skill produces. Sample content, shown for illustration.

A/B Test Plan — sample

Metric: Signup start rate | Baseline: 5% | MDE: +20% relative (+1.00 pp) | Arms: 2 (Control + 1 variant) | Confidence: 95% | Power: 80%

Sample size: 8,160 per variant | 16,320 total | About 9 days at 2,000 visitors/day (50/50 split)

Decision rule (pre-registered): call the test only when the smallest arm reaches 8,160 visitors; a variant wins only if it beats control by at least 1.00 pp and the difference is significant at 95% confidence. A significant result smaller than the MDE is real but too small to ship on this hypothesis. An inconclusive result is not a reason to extend the test — plan a new one.

Sample sizes are planning estimates from the normal approximation at the stated confidence and power — not a guarantee, and this script does not analyse results.

About this Skill

You say
We want to test moving the signup CTA above the fold on the BeanLoop pricing page — plan the test before we run it.

How to install

  1. Copy the install prompt into your agent: Read https://skillu.dev/install/agent.md, then install the A/B Test Planner skill (ab-test-planner).
  2. Or download the Skill package from this page after launch and place its folder in your agent’s skills directory.
  3. Call it with a slash command, e.g. /ab-test-planner, and describe what you want in plain words.

FAQ

Does Ready to run mean the change will win?

No. It means every part of the plan is present and the input had no errors — the plan is complete, not the result. Most tests lose or tie; a plan that says so plainly is doing its job.

Why is the sample size so large for a small lift?

Because a small relative lift is a small absolute difference — 20% on a 5% baseline is only 1 percentage point, and telling 5% from 6% reliably takes about 8,160 visitors per variant. Halve the MDE and the sample roughly quadruples.

Get Premium access Browse more skills