50 Tests vs. 100 Tests

Both lines use the same win, flat, and loss mix. The only difference is experiment volume: one program runs 50 tests, while the other runs 100 tests. Stat-sig positives ship, stat-sig negatives do not, and flat results add no lift. The chart shows how extra test volume compounds when more winning results have a chance to appear.

Simulation

Test count I

Test count II

Win / Flat / Loss %

100 tests: true lift values

Lesson 1 of 2

Takeaways Over 1,000 Simulations

Monthly Test Velocity

Each line uses the same test quality mix over 12 months, but runs at a different velocity. Higher effective velocity lets the team work through more ideas, so more winners have a chance to compound.

Simulation

Test velocity I tests/month

Test velocity II tests/month

Win / Flat / Loss %

Simulated tests: true lift values

Lesson 2 of 2

Takeaways Over 1,000 Simulations

Growth, Decay, and Compounding

Teams rarely run at constant velocity -- they scale up over time. And idea pools rarely stay fresh -- win rates tend to decay as the obvious wins are exhausted. Each line has its own starting velocity, monthly velocity growth, and win rate decay. Watch how growth and diminishing returns interact over up to 36 months.

Simulation

Months

Win / Flat / Loss %

Starting velocity I tests/month

Ending velocity I tests/month

Win rate floor I % by end

Starting velocity II tests/month

Ending velocity II tests/month

Win rate floor II % by end

Lesson 3 of 3

Experiment Velocity

50 Tests vs. 100 Tests

Simulation

Takeaways Over 1,000 Simulations

Monthly Test Velocity

Simulation

Takeaways Over 1,000 Simulations

Growth, Decay, and Compounding

Simulation

Takeaways Over 1,000 Simulations