P-value

Definition

A p-value is the probability, assuming the null hypothesis is true, of getting a result at least as extreme as the one observed. It measures how surprising the data are under that assumption; it is not the probability that the hypothesis is true or false.

In plain terms

Start with a default claim, the null hypothesis, such as “this coin is fair” or “the new rule makes no difference”. Ask: if that were true, how often would chance alone produce data at least as unusual as ours? That frequency is the p-value. A small one means the data would be rare under the null, which counts against it.

The American Statistical Association has listed what a p-value does not tell you: not the probability that the null is true, not the size or importance of an effect, and not a licence to conclude anything from a single threshold such as 0.05.

Example

A coin is flipped 20 times and lands heads 15 times. Null hypothesis: the coin is fair.

The result would be fairly rare for a fair coin, but it is not proof that the coin is biased: a fair coin does this about once in 25 runs, and if you flipped many coins one of them would do it.

Common misunderstandings

• “p = 0.04 means there is a 96% chance the hypothesis is true.” The p-value assumes the null and says nothing about its probability.
• “p above 0.05 means there is no effect.” It means the data did not rule out chance; there may be an effect too small for this sample to show.
• “A smaller p-value means a bigger effect.” With enough data, tiny effects produce tiny p-values (see Effect Size).