Evidlake
Menu

Backtesting

Also known as: historical simulation, back-test, 回测.

Evaluating a rule or model by replaying it on historical data as if it had been used at the time. Informative only if the replay is honest.

Updated Last reviewed

Synthetic dataExamples on this page use synthetic data created for illustration.

Not adviceThis is educational material. It is not investment, business or other professional advice.

Definition

Backtesting evaluates a rule or model by replaying it on historical data as if it had been used at the time. It is informative only if the replay uses nothing that was unknowable then, and if the rule was not tuned until it fitted that very history.

In plain terms

A backtest is cheap evidence: the future has not happened, but the past has. Its weaknesses are well known. Information from later than the decision date leaks in (see Look-ahead Bias); only the things that survived are in the data (Survivorship Bias); costs and delays are ignored; and, most of all, every variant tried on the same history raises the chance that one of them looks good by luck (Multiple Comparisons Problem).

Good practice is to fix the rule and the evaluation before looking at results, to keep a stretch of history unseen until the end (Holdout Set), to step through time (Walk-forward Validation), and to report how many variants were tried.

Example

A simulation in which 20 rule variants have no skill: each earns the same true return of about 6% a year, plus random monthly noise. Each is tested on 8 years, and the best is then followed for 2 more years. The experiment is repeated 3,000 times and averaged.

Annualised return
An average variant, in the 8-year test6.2%
The best of the 20, in the 8-year test16.3%
That same “best” variant, in the next 2 years6.0%

Picking the winner from 20 tries manufactures a 10-point edge from pure noise, and it vanishes the day the rule is used for real.

Common misunderstandings

  • “A good backtest predicts good performance.” It predicts it only if the replay was honest and the rule was not selected from many tries on the same history.
  • “More history is always better.” Old periods may follow different conditions; check whether the result holds across them.
  • “Costs and delays are details.” Ignoring them can turn a loser into an apparent winner.
  • Look-ahead Bias

    A study error where information that was not yet available at the time of a decision is used to evaluate that decision.

  • Overfitting

    When a model learns the noise in its training data instead of the real pattern, and so fails on new data.

  • Holdout Set

    Data set aside before any building begins and used once, at the end, to estimate performance on unseen data.

  • Multiple Comparisons Problem

    The more tests, rules or variants you try on the same data, the more likely some look significant by chance. The number of tries has to be counted.

  • Walk-forward Validation

    Evaluating a forecasting method by training on data up to a date, testing on the period just after, then moving the date forward and repeating.

References

  1. David H. Bailey, Jonathan M. Borwein, Marcos López de Prado, Qiji Jim Zhu. Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance . Notices of the American Mathematical Society, 2014. DOI: 10.1090/noti1105
  2. Marcos López de Prado. Advances in Financial Machine Learning . Wiley, 2018. ISBN 978-1119482086.