Multiple Comparisons Problem

Definition

The multiple comparisons problem is that the more tests, rules or variants you try on the same data, the more likely some will look significant by chance alone. Reported success then overstates real effects unless the number of attempts is counted and corrected for.

In plain terms

A single test at the 5% level will wrongly “find” an effect one time in twenty when there is none. Run twenty independent tests and the chance that at least one does is not 5% but about 64%. The same happens, less visibly, when an analyst tries several time windows, thresholds and subgroups and reports the one that worked: Andrew Gelman calls this the “garden of forking paths”.

Remedies: decide the analysis before looking (Pre-registration), keep data in reserve for a final test (Holdout Set), count every variant tried, and use corrections that control the error across the family of tests, such as Bonferroni’s (divide the threshold by the number of tests) or the false discovery rate procedure of Benjamini and Hochberg.

Example

m independent tests, every one of them on data with no real effect, each judged at the 5% level:

With 20 tests, a Bonferroni-corrected threshold would be 0.05 / 20 = 0.0025 for each, which restores an overall 5% error rate.

Common misunderstandings

• “A p-value below 0.05 means a 95% chance the effect is real.” It does not; the answer depends on how many things were tried and how plausible the effect was to begin with.
• “Only formal tests count.” Every variant, window, threshold or subgroup you looked at is another comparison.
• “Corrections are a statisticians’ fussiness.” Without them, a collection of random rules reliably produces some that look good.