Your test came back and you know what you gotta do.
ooh la la
p-value1.000
Nope, still not significant!
In statistics, significance functions as a strict yes-or-no. In a frequentist fixed-horizon experiment, you pre-register an alpha (say 0.05). If the p-value doesn't reach it, the result is not significant, and you treat it as such.
And that's a useful outcome too. It suggests the effect is either null or smaller than your test was sized to detect.
The key thing is to choose your design before you run it, and stick to it.
Luckily, there are more ways to design experiments than many people realize.
Rerun + meta-analyze
Run it again and pool both results. See Rerun, an open-source project built for this, as well as GoodUI Meta-Analysis.
Sequential test
Always-valid p-values, so peeking and stopping early or running longer are allowed. Try the Stat Sig Calculator.
Non-inferiority test
Ship if it's not meaningfully worse, within a set margin. Also in the Stat Sig Calculator.
Bayesian test
Report the chance to beat control and the expected loss.
Higher alpha
Accept more false positives (say α = 0.10) when shipping a dud is cheap. See the Power Visualizer.
Higher MDE
Only chase effects big enough to matter, and size the test for them. See the Power Planner.
Plan enough power
Size the test for 80-90% power at your MDE, so a real effect is likely to clear α rather than land near it. See the Power Planner.
Compound treatments
Combine several changes that are each likely to help, for a bigger effect that's easier to detect. See A/B Tests Big and Small.
Variance reduction
Pre-declare CUPED or a covariate to cut the noise.
Tighter triggers
Only count users who actually reach the change, so unaffected visitors don't dilute the effect.
One-sided test
If you'd only ever ship a win, test in that one direction.
Exploratory
Collect data to learn and shape future tests, without needing a ship decision.
Above all, stick to your original design but maintain openness to rerun.
Inspired by Matthew Hankins's collection of 500+ ways to describe a near-miss p‑value.