P Values and Confidence Intervals

Expansion

What a p value is: the probability of observing a result at least as extreme as this one if the null hypothesis were true.

What it is not

• Not the probability that the null hypothesis is true
• Not the probability that the result occurred by chance
• Not a measure of the size or importance of an effect
• A p of 0.049 and 0.051 are essentially the same evidence; the 0.05 threshold is arbitrary

Errors and power

• Type 1 error (alpha): rejecting a true null, so a false positive. Conventionally set at 0.05
• Type 2 error (beta): failing to reject a false null, so a false negative
• Power = 1 - beta, conventionally 80 or 90 per cent, and determined mainly by sample size and effect size
• Absence of evidence is not evidence of absence: a non-significant result in an underpowered study says nothing

Confidence intervals are more informative because they convey both the effect size and its precision. A 95 per cent interval means that if the study were repeated many times, 95 per cent of such intervals would contain the true value.

• If the interval for a difference crosses 0, or for a ratio crosses 1, the result is not statistically significant
• A wide interval indicates an imprecise estimate, usually from a small sample
• An interval can exclude the null yet include only clinically trivial values, which is statistical significance without clinical significance

Multiple comparisons: testing many outcomes or subgroups inflates the false positive rate, which is why primary outcomes are pre-specified, and why a positive subgroup finding in an otherwise negative trial should be treated as hypothesis generating.