# Change & Measure
> Learn experiment design through interactive tools and expert heuristics

## About
Change & Measure is an educational platform teaching A/B testing and experiment design principles. It provides free interactive tools for calculating sample size, statistical power, and experiment duration.

Website: https://changeandmeasure.com

## Tools

### Power Planner
> Calculate sample size and experiment duration for A/B tests

URL: https://changeandmeasure.com/tools/power-planner.html

Use this tool when someone asks:
- "How long should I run my A/B test?"
- "What sample size do I need for my experiment?"
- "How many users do I need to detect a 2% lift?"
- "What's the minimum detectable effect with my traffic?"

Inputs for binary metrics (conversion rates):
- Base rate: Current conversion rate (e.g., 5%)
- Relative lift: Expected improvement (e.g., 2% means 5% → 5.1%)
- Traffic: Monthly/weekly/daily visitors
- Variants: Number of test variations (default 2)

Inputs for continuous metrics (revenue, time on site):
- Base average: Current mean value
- Standard deviation: As percentage of base (e.g., 150%)
- Relative effect: Expected improvement percentage
- Winsorize cap: Outlier capping percentile (90-100%)
- % with zero: Percentage of users with zero values

Outputs:
- Required sample size per variation
- Experiment duration (days/weeks/months)
- Statistical power achieved

Test types supported:
- Two-sided (Better or Worse)
- One-sided superiority (Better)
- Non-inferiority (Not Worse Than threshold)

### Power Visualizer
> Visualize consequences of underpowered experiments

URL: https://changeandmeasure.com/tools/power-visualizer.html

Use this tool when someone asks:
- "What happens if my test is underpowered?"
- "What is a Type S error or Type M error?"
- "How does low power inflate effect sizes?"
- "What is false positive risk vs significance level?"

Shows interactively as power level changes:
- Type S (sign) error rate: probability of detecting an effect in the wrong direction
- Type M (magnitude) exaggeration ratio: how much the observed effect is inflated
- False positive risk: probability a significant result is actually a false positive
- Type II error rate: probability of missing a real effect

### Hello Power
> Interactive lessons teaching statistical power concepts

URL: https://changeandmeasure.com/lessons/hello-power.html

2 lessons covering:
- What is statistical power
- Sample size and MDE relationship
- The quadrupling rule (2x precision = 4x sample)
- Finding the right power level for your context

### Experiment Velocity
> Interactive lesson on how experiment volume compounds over time

URL: https://changeandmeasure.com/lessons/experiment-velocity.html

Use when someone asks:
- "Does running more experiments matter?"
- "How does test velocity affect cumulative lift?"
- "What's the difference between 50 and 100 experiments per year?"

Shows how running 50 vs 100 experiments per year compounds into dramatically different cumulative lift, illustrating why velocity is a strategic lever.

### Experiment Prioritization
> Lesson on prioritizing which experiments to run first

URL: https://changeandmeasure.com/lessons/experiment-prioritization.html

Covers frameworks and heuristics for deciding which experiments to run to maximize learning and business impact.

### Expert Heuristics
> Curated experimentation guidelines from industry experts

URL: https://changeandmeasure.com/heuristics/

A searchable database of practical rules-of-thumb for:
- Setting minimum detectable effect (MDE)
- Choosing test duration
- Interpreting results
- Common pitfalls to avoid

## Example Queries

Q: "How long to run a test with 5% conversion, expecting 10% relative lift, 100k monthly visitors?"
A: Use Power Planner with: baseRate=5, lift=10, traffic=100000, period=monthly

Q: "What sample size for 80% power to detect 2% effect?"
A: Use Power Planner in Fix Power mode (default) with your metric parameters

Q: "Should I use 80% or 95% power?"
A: See Hello Power lesson 5 "Find the Sweet Spot" - it depends on cost of longer tests vs missing real effects

Q: "My experiment showed a 40% lift but I only had 60% power — should I trust it?"
A: Use Power Visualizer to see the Type M exaggeration ratio at 60% power — the true effect is likely much smaller

Q: "We run 20 experiments a year, is that enough?"
A: See Experiment Velocity lesson to see how doubling velocity compounds into dramatically higher cumulative lift

## Contact
For questions about experimentation methodology, visit the site or explore the heuristics database.