A/B Test vs. Before-After

Why controlled experiments catch effects that time-series comparisons miss — or invent.

Lesson 1 of 2

You Pick the Launch Point

Drag the launch line. The before-after compares the after window to the before window — drift between them swings its verdict around. The A/B test compares control vs. treatment in the same periods, so drift cancels and its answer stays near the truth wherever you launch.

Simulation

Drift shape

Total sample users in the test

Window ± periods (0 = all)

Baseline rate % conversion

True effect % real lift

MDE % lift to detect

Sampling noise % of expected

Drift % per period

Lesson 1 of 2

Takeaways Over Simulations

How Reliable Is Each Method?

Now we run a whole program of 1,000 experiments. Each one has its own random true effect (mostly small, some real wins or losses — set the typical size with Effect spread) and launches at a random point in time. For each, we ask: did the method call it significant and in the right direction? Each verdict is correct (green), significant-but-wrong / a false positive (red), or not significant (the grey remainder).

With no drift the two methods agree. Turn on drift (or pick Upward/Downward): the before-after starts calling experiments significant in the wrong direction — confidently misreading which tests won — while the A/B test keeps getting the direction right wherever it launches.

Simulation

Drift shape

Total sample users in the test

Baseline rate % conversion

Effect spread ± typical real lift
(varies per test)

Sampling noise % of binomial
(100 = realistic min)

Drift % per period

A/B test

Before-after

Lesson 2 of 2