A/B Testing
A/B testing (split testing) means showing two versions of something to different users and measuring which performs better. Version A vs. Version B. Data wins, opinions lose.
What is A/B Testing? How to Run Split Tests
How it works: 50% of users see the blue button, 50% see the green button. After a week, you check which color had more clicks. If green wins by a statistically significant margin, you ship green to everyone.
What to A/B test: (1) Pricing pages (price points, messaging, CTAs), (2) Onboarding flows (steps, copy, visuals), (3) Email subject lines (open rates), (4) Landing page headlines (conversion rates), (5) Signup forms (number of fields, button text).
What NOT to A/B test when you're small: Don't test 20 things at once. Don't test on 50 visitors (not statistically significant). Don't test things that don't matter (button shadow style won't move revenue). Test big swings on high-traffic pages first.
Statistical significance: You need enough sample size to trust results. If version A converts 10/100 users and version B converts 12/100, that's not significant—could be random noise. Tools like Optimizely or VWO calculate significance for you. General rule: need 100+ conversions per variant minimum.
A/B testing mistakes: (1) Stopping tests too early (impatience kills accuracy), (2) Testing too many variables at once (can't isolate what worked), (3) Not having a hypothesis (random testing wastes time), (4) Ignoring segment differences (B might win overall but lose for mobile users).
Compounding improvements: A 10% improvement to your landing page conversion, plus 10% to email open rates, plus 10% to trial-to-paid conversion = 33% total revenue increase. Small wins compound.
Examples
Obama's 2008 campaign A/B tested email subject lines and raised $60M more than they would have without testing. Booking.com runs thousands of A/B tests simultaneously—data-driven culture.