Prioritized Experiments

Does it matter which experiments you run first?

Imagine you have 100 A/B tests to run. Some will be winners, some losers, and many will be flat. You reject stat-sig negatives (don't ship them) and ship stat-sig positives.

If a human can correctly guess the direction of a test even 60% of the time, should you let them prioritize the order? The simulation below shows what happens.

Simulation

Human accuracy

60%

Correct directional guesses

Tests

100 tests: true lift values

Over 1,000 Simulations

Takeaway

The final cumulative value is the same regardless of ordering — you ship the same set of winning tests either way. But the shape matters: human-prioritized ordering reaches gains earlier, meaning your product improves faster.

Even 60% directional accuracy — barely better than a coin flip — produces a visible advantage. This is because front-loading likely winners means you compound gains sooner, while random ordering spreads winners and losers evenly across the timeline.