Try Enough Rules and One Will Work: Multiple Comparisons and Backtest Overfitting

Imagine twenty people each flip a coin ten times. Somebody will probably get eight or nine heads, and a newspaper could call that person a “coin-flipping genius”. Nothing about their skill is special; with twenty tries, someone is bound to do well. The same thing happens when an analyst tries twenty rules on the same data and reports the best. The result is real, but it describes luck, not the rule. This is the multiple comparisons problem, and when it hits a historical test it is called backtest overfitting (Backtesting, Overfitting).

The arithmetic of many tries

A single test judged at the 5% level has a 5% chance of “finding” an effect that does not exist. With more independent tests, the chances of at least one false find add up:

Number of tests	Chance of at least one false “significant” result
1	5.0%
5	22.6%
10	40.1%
20	64.2%
100	99.4%

The formula is 1 − 0.95ᵐ for m independent tests. The tests do not have to be formal. Every time window, threshold, subgroup, filter or version of a rule that you look at is a comparison, including the ones you drop without writing them down. Andrew Gelman and Eric Loken call these hidden choices the “garden of forking paths”.

How big is the effect on a backtest?

A simulation makes it concrete. Twenty rule variants are given no skill at all: each earns the same true return, about 6% a year, plus random monthly noise. Each is tested on 8 years of data; the best is picked and then followed for two more years it has never seen. The experiment is repeated 3,000 times and averaged.

	Annualised return
An average variant, in the 8-year test	6.2%
The best of the 20, in the 8-year test	16.3%
That same best variant, in the next 2 years	6.0%

The winner earns 10 points more than any variant deserves, and the advantage vanishes the moment it meets new data. The more variants tried, the larger the gap. How much better the best of N random results looks, measured in standard deviations above average:

Variants tried (N)	Expected best result, in standard deviations above average
1	0.0
5	1.2
20	1.9
100	2.5
1,000	3.2

With a thousand variants, the best random one looks like a 3-sigma discovery. A researcher who reports only the winner, and not that a thousand were tried, is not necessarily dishonest, but the report misleads.

Counting your attempts

The first defence is bookkeeping. Keep a log of every variant you evaluate (the rule, the parameters, the data, the result), including the abandoned ones. The number of entries is the number that matters when you judge the survivor. It is easy to underestimate: ten parameters with five values each is already about ten million combinations.

Correcting for them

Two standard corrections let you keep a claim honest when you cannot avoid many tests.

Bonferroni: divide the significance threshold by the number of tests. With 20 tests, each must reach 0.05 / 20 = 0.0025. It is simple and strict, and it protects against any false find.

False discovery rate (Benjamini–Hochberg): rather than avoiding every false find, control the share of false finds among the discoveries. Sort the p-values and compare the i-th smallest with (i / m) × 0.05. Everything up to the largest p-value that passes is declared a discovery. It is less strict and finds more.

A worked example with ten p-values from ten tests:

Rank	p-value	Bonferroni threshold (0.05 / 10)	BH threshold (rank / 10 × 0.05)	Passes Bonferroni?	Passes BH?
1	0.001	0.005	0.005	Yes	Yes
2	0.008	0.005	0.010	No	Yes
3	0.012	0.005	0.015	No	Yes
4	0.030	0.005	0.020	No	No
5	0.040	0.005	0.025	No	No
6–10	0.20 to 0.90	0.005	0.030 to 0.050	No	No

Judged one at a time at 5%, five results look “significant”. After Bonferroni one remains; after Benjamini–Hochberg three. Which correction fits depends on the cost of a false find against the cost of a missed one.

Designing so that you need fewer corrections

Decide before you look. Write down the rule, the metric, the data and the success criterion before running the test (Pre-registration). A rule chosen in advance is one comparison, not twenty.
Keep a holdout. Reserve data that no variant has seen, and test only the finalist on it, once (Holdout Set).
Prefer simple rules with few knobs. Every free parameter multiplies the forks.
Demand a reason. A result with a plausible mechanism and consistent behaviour across periods is better evidence than a number that merely cleared a threshold.
Look at all the variants, not just the winner. If neighbouring parameter values do much worse, the winner is probably a fluke.

Questions to ask of any claim

How many things were tried before this one was reported?
Was the rule fixed before the data were examined?
Does it hold on data that were not used to find or tune it?
Is the effect large and consistent, or just past a threshold?

If the answers are “many”, “no”, “no” and “just past”, the claim is a lottery winner and should be treated as one.