Hello Power

Learn how power affects your experiments.

Lesson 1 of 5

Power 101

Statistical power is the probability of detecting a true effect as a Stat-sig result if it exists. It's a critical concept on which experimentation and A/B testing are built. Let's start with the simplest visual exercise of this.

Question Imagine you would like to detect a 2% effect with 80% probability - what should you set your power to?

Takeaways

Power is the probability of detecting a true effect when one exists.
If you want an 80% chance of detecting an effect, set power to 80%.
Most experiments aim for 80% power as a standard threshold.

What If You Knew The True Effect?

Imagine you know the true relative effect of what you're measuring. You run 10 independent A/B tests.

Question If you set your statistical power to 80%, how many of your A/B tests would end up stat-sig?

Takeaways

With 80% power, roughly 80% of experiments will detect a true effect as stat-sig.
Even with a real effect, ~20% of tests may miss it (false negatives).
Higher power means more consistent detection of real effects.

Sample Size and MDE

Sample size and MDE (Minimum Detectable Effect) are directly linked.

Scenario: 10% baseline, 80% power, 5% alpha

Question What is the relationship between sample size and MDE?

Drag sliders to explore

Takeaways

Sample size and MDE have an inverse-square relationship.
4× sample size = ½ MDE (you can detect half the effect).
This tradeoff is fundamental to experiment planning.

From 8% MDE to 2%

You've seen the quadrupling rule: halve your MDE, pay 4× in samples. This inverse-square relationship means detecting smaller effects gets expensive fast.

Scenario: 10% baseline, 80% power, 5% alpha

Question If 8% MDE requires N samples, how many would a 2% MDE need?

Takeaways

Halving MDE = 4× sample size (inverse-square relationship).
Going from 8% → 2% MDE (4× reduction) requires 16× the sample size.
This is why choosing the right MDE is critical for experiment feasibility.

Find the Sweet Spot

Higher power requires more sample size - but how much more? Explore the curve below to find a good balance between detection rate and sample cost.

Scenario: 10% baseline, 2% MDE (relative lift, i.e. 10% → 10.2%)

Question What power level gives you good detection without excessive cost?

Power: 20%

Takeaways

The sample size curve gets steeper above ~80-85% power.
Around 75-85% power offers the best efficiency (power gained per sample).
This is why 80% power is the industry standard - good detection without excessive cost.