Effect Size

The P-value answers the question: "Is there an effect?" The Effect Size answers the question: "How big is it?"

These are not the same.

The Huge Study Problem

If you study 1,000,000 people, you can find a "statistically significant" difference in height between people who like cats and people who like dogs.

Maybe the difference is 0.001 mm. The p-value will be < 0.0001. But the effect size is zero. It is meaningless.

Measuring Magnitude

We use metrics like Cohen's d to measure effect size.

• 0.2 = Small (Hard to see with the naked eye).
• 0.5 = Medium (Visible).
• 0.8 = Large (Obvious).

The Diagnosis

Don't get dazzled by tiny p-values.

If a new diet pill has a p-value of 0.001, ask: "How much weight did they lose?" If the answer is "100 grams," the effect size is negligible. Keep your money.