A/B testing
Definition
A/B testing is a controlled experiment in which two versions of something — a page, a message, a feature, a price — are shown to randomly split groups of users simultaneously, to determine which version performs better on a defined metric.
A/B testing (also called split testing) is the most rigorous form of growth experimentation because it controls for timing, audience mix, and external factors. Both variants run at the same time on a randomly split audience, which eliminates the confounds that plague before/after comparisons.
The mechanics: define the metric you're trying to move, create the two variants, split your audience randomly (usually 50/50, though uneven splits are valid for higher-risk tests), run until you reach statistical significance, then read the result.
Statistical significance matters. If you stop the test as soon as you see the number you wanted, you're likely reading noise. Standard practice is to pre-calculate the required sample size and run until that threshold is reached.
A/B testing requires traffic volume. A product with 200 new signups per week cannot run a meaningful A/B test on a metric that moves by a few percentage points — the confidence intervals will be too wide. Low-traffic products should use qualitative testing (user interviews, prototype tests) instead.
Variants to commonly test: onboarding flows, paywall designs, email subject lines, call-to-action copy, pricing page layouts, notification timing. The best A/B tests are driven by a specific hypothesis, not by "let's just see which one users prefer."
Examples
- Testing two versions of an onboarding welcome screen to see which increases activation rate
- Running a 50/50 split on two pricing page layouts to measure conversion rate difference
- Testing a different call-to-action label on the primary button to see which drives more taps
- Sending two subject lines to equal halves of an email list to optimize open rate
Snap a a/b testing. Ship its actions.
BoardSnap turns any whiteboard — including this one — into a summary and action plan.