Experiment Prioritization

Does it matter which experiments you run first?

Lesson 1 of 7

Perceived Winners First?

Imagine you have a finite set of 100 A/B tests to run. Some will be winners, some losers, and many will be flat. You reject stat-sig negatives (don't ship them) and ship stat-sig positives. If a human can correctly guess the direction of a test even 59% of the time, should you let them prioritize the order by winners first? The simulation below shows what happens.

Simulation

Human directional accuracy

59%

For sorting positives first

Win / Flat / Loss %

100 tests: true lift values

Lesson 1 of 7

Takeaways Over 1,000 Simulations

Ordering does not magically create more winners in a finite set. What it does change is when you reach them. Front-loading likely winners compounds gains sooner, so the product gets better faster.

Only Test Positive Estimates From A Finite Set?

Let's see what happens when we apply our intuition to gate experiments believed to be negative. Imagine having a finite set of 100 ideas while comparing two policies: only running tests with positive estimates versus running all test ideas (with negative estimates in the second half).

Simulation

Human directional accuracy

59%

For sorting positives first

Win / Flat / Loss %

Temporary negative-test loss 50% exposure to the losing variant.
Loss lasts two test slots, then returns to zero.

Candidate tests: true lift values

Lesson 1 of 7

Takeaways Over 1,000 Simulations

If you use intuition to decide whether a test gets run at all, you may avoid waste. But at 59% accuracy you will still filter out some real winners, and you may also remove the failures that would have generated valuable follow-up ideas.

What If The Idea Backlog Is Infinite?

When new ideas keep coming in, skipping a negative-estimated test doesn't waste a slot — a fresh candidate fills it immediately. So the question shifts: is there still a case for running negative-estimated ideas at all? Here we compare test positives only and random ordering against two strategies that also continue into negative-estimated ideas: adding them within the main prioritized queue, or running them in parallel alongside positives.

Simulation

Human directional accuracy

59%

For sorting positives first

Win / Flat / Loss %

One incoming batch: true lift values

Lesson 1 of 7

Takeaways Over 1,000 Simulations

With an infinite backlog, skipping a negative-estimated idea costs nothing — a fresh positive candidate takes its place. Slotting negatives into the main queue after positives (line 2) means the queue gets slower, and that drag shows up in the results — positive-only testing (line 1) stays ahead.

Direction vs. Impact Magnitude Sorting?

Filter & sort by direction uses predicted direction to both filter and order: only positive-estimated tests run, sorted positives-first. Filter & sort by impact magnitude ignores direction entirely and filters + sorts by estimated effect size, running the biggest estimated effects first. This lesson isolates the two signals to show which carries more value on its own.

Simulation

Human directional accuracy

59%

For sorting positives first

Win / Flat / Loss %

Impact estimate error

Median absolute error when estimating effect size

100 tests: true lift values

Lesson 1 of 7

Takeaways Over 1,000 Simulations

Filter & sort by direction uses a binary signal — predicted win or loss — and ignores magnitude entirely. Filter & sort by impact magnitude also filters out negatives, but ranks what remains by estimated effect size rather than a directional call. The impact estimate error slider shows how quickly the magnitude signal degrades as estimates get noisier.

Does Iterating On Results Help?

We compare three strategies: test positives only, iterate on all results, or iterate only on results above an effect size threshold. Follow-up tests keep a configurable relative gain from the source test. A negative follow-up that lands is treated as flipping the original loss into a smaller positive gain.

Simulation

Human directional accuracy

59%

For sorting positives first

Win / Flat / Loss %

Effect threshold

To iterate on

Effect decay

Effect retained

Winner success

35%

Positive follow-up win chance

Loser success

42%

Loss-to-win follow-up chance

One incoming batch: true lift values

Lesson 1 of 7

Takeaways Over 1,000 Simulations

Only Iterating On Large Effects?

Lesson 5 showed that iterating on large effects can beat iterate all. Here the threshold strategy becomes iterate >=X% and <=-X% — large winners and large losers, since a strong loss is worth a follow-up that can flip it into a smaller win.

Simulation

Human directional accuracy

59%

For sorting positives first

Win / Flat / Loss %

Effect threshold

To iterate on

Iteration effort

Slot cost per iteration

Effect decay

100%

Follow-up effect retained

Winner success

35%

Positive follow-up win chance

Loser success

42%

Loss-to-win follow-up chance

One incoming batch: true lift values

Lesson 1 of 7

Takeaways Over 1,000 Simulations

Sequential iteration has to earn its slot. Larger source effects can survive decay better, while smaller source effects often get diluted enough that the next fresh positive-estimated idea is a stronger bet.

References

GoodUI. (2022). Beyond Opinions About Opinions: What 70,149 Guesses Tell Us About Predicting A/B Tests.
GoodUI. (2026). Experiment Estimation Study: How Well Can 1,391 Guesses Predict A/B Test Effects?.
Frazer Mawson. LinkedIn post on iteration win rates for winners and losers.