Experiment Stopping Rules
When should you call a test, and how much speed can you trade for honesty?
Lesson 1 of 1
How Long Should You Run Each Test?
The same stream of ideas runs through six stopping rules at once; they differ only in when they call each test. Long, well-powered horizons are accurate but slow. Short fixed durations bank sooner but miss real wins. Sequential rules (mSPRT, AGILE) stop early with error control; naive peeking stops early without it. The chart shows the true lift each rule ships, and the tick at its end shows how much it over-claimed.
Simulation
%
conversion
%
fixed horizons
%
typical size
visitors / test
tests / month
All rules use α = 5%, but naive peeking re-checks it at every peek, so its real error rate climbs far higher.
Takeaways Over Simulations
References
- Georgi Z. Georgiev. (2017). Efficient A/B Testing in Conversion Rate Optimization: The AGILE Statistical Method.
- Georgi Z. Georgiev. (2022, updated 2023). Comparison of the Statistical Power of Sequential Tests: SPRT, AGILE, and Always Valid Inference.