A/B testing

Version A is the page as it stands, called the control. Version B carries the change, called the variant. Each visitor is assigned at random and keeps seeing the same version. Because both groups arrive over the same period from the same sources, a gap between them can be put down to the change rather than to a campaign or a slow week.

Sample size decides whether the result means anything. You estimate it before starting from three inputs: the current conversion rate, the smallest lift worth detecting and how much certainty you want. Small lifts on low conversion rates need far more visitors than most teams expect, and the math is unkind to a page that gets a few thousand visits a week.

Why it matters for a marketing site

A homepage change made on taste can cost pipeline, and nobody finds out which change did it. A test limits that risk to part of the traffic and leaves a record of what was tried. The catch is volume. Plenty of B2B pages will never collect enough visitors for a clean answer, and on those pages a few customer calls will teach more.

How I handle it

Experiments are not part of a migration band. I do not run them across a cutover, because the move itself changes the numbers you would be reading. Once the site is live, a test is a scoped request: one hypothesis, one metric, and the sample size worked out before it starts. With PostHog flags I test the first render and the identity change, since a flag that resolves late flashes the default version and counts against the page.

Example

Say a demo page converts 2% of visitors and the team wants to detect a lift to 2.5%. At 95% confidence and 80% power, the standard two-proportion formula asks for about 13,800 visitors per version. At 3,000 visits a week, that is a nine-week test. Most teams should test a bigger change or a busier page.

Common mistakes

• Stopping when one version pulls ahead. Early leads come and go. Set the sample size first and wait for it.
• Testing on thin traffic. A test that cannot reach its sample size produces a number that looks like an answer and is not one.
• Changing ten things at once. You learn that the bundle won, not which part of it did.
• Using a 301 to send visitors to the variant. Google asks for a 302 on redirect tests, because a 301 says the original page is gone.