What is an A/B test?
An A/B test splits an audience randomly between two or more variants of an experience and measures which performs better on a chosen goal. The randomness is the whole technology: because the groups differ only in the variant they saw, a performance gap can be attributed to the change itself rather than to who happened to see it.
The mechanics
Users are assigned to variants by weight (commonly 50/50) and stickily — the same user always sees the same variant, so nobody experiences both. A control (the current version) anchors the comparison; the goal metric (completion is the usual choice for onboarding flows, since starts only measure reach) is tallied per variant as conversion.
The statistics that protect you
Early results wobble — day-two "trends" reverse routinely, which is why serious tools gate verdicts behind statistical confidence (typically a significance test at the 95% threshold). "Not yet significant" means keep collecting, not lean in and squint. Low-traffic surfaces may take weeks to reach significance; for those, shipping judgment and watching the funnel beats a test that never concludes.
Hygiene rules
- One variable per test. Variants differing in five ways make a win unattributable.
- No mid-test edits. Changing a variant's content mid-run muddies what was measured — pause, conclude, iterate in the next test.
- Test the big levers. Length (three steps vs. five) and first-step framing move onboarding numbers far more than word-tweaks.
In onboarding practice, A/B testing settles the arguments intuition can't — built no-code over any guide, with confidence surfaced honestly.
See the concepts running live.
Wakeline puts the whole vocabulary to work — tours, checklists, targeting, funnels — no-code on your app, free for 1,000 monthly active users.