Experiment Stopping Rules

When should you call a test, and how much speed can you trade for honesty?

Lesson 1 of 1

How Long Should You Run Each Test?

The same stream of ideas runs through six stopping rules at once; they differ only in when they call each test. Long, well-powered horizons are accurate but slow. Short fixed durations bank sooner but miss real wins. Sequential rules (mSPRT, AGILE) stop early with error control; naive peeking stops early without it. The chart shows the true lift each rule ships, and the tick at its end shows how much it over-claimed.

Simulation

%
conversion
%
fixed horizons
%
typical size
visitors / test
tests / month

All rules use α = 5%, but naive peeking re-checks it at every peek, so its real error rate climbs far higher.

Takeaways Over Simulations

References