Type I vs Type II Errors

Science is never 100% sure. It deals in probabilities. This means it makes mistakes.

There are two main types of mistakes.

Type I Error: The False Positive

This is the "Boy Who Cried Wolf."

You claim there is an effect when there is none. You say the drug works, but it is just sugar. You say the house is on fire, but it is just toast.

In statistics, this is rejecting the null hypothesis when it is true. We usually set the limit for this at 5% (p < 0.05). We accept a 5% chance of being a fool.

Type II Error: The False Negative

This is the "Wolf Who Ate the Boy."

You claim there is no effect, when there actually is one. You say the drug is useless, but it actually cures cancer. You say the house is safe, but you are standing in flames.

This often happens when sample sizes are too small. You didn't have enough power to see the truth.

The Trade-off

You cannot eliminate both.

If you make your test very strict to avoid False Positives (Type I), you will miss real effects (Type II). If you make your test very sensitive to catch everything (avoid Type II), you will generate many False Positives (Type I).

The Diagnosis

Airport security accepts many Type I errors (searching innocent people) to avoid a Type II error (missing a bomb).

The justice system accepts Type II errors (letting a criminal go) to avoid Type I errors (convicting an innocent).

Know which error is worse for your situation.