Evidlake
Menu

Honest Validation for Time-Ordered Data: Holdout, Cross-Validation and Embargo

Random splits quietly leak the future when data are ordered in time. How to split by time, step forward through history, and purge and embargo the overlap that still leaks.

Evidlake Editorial Updated Last reviewed 4 min read

Synthetic dataExamples on this page use synthetic data created for illustration.

Not adviceThis is educational material. It is not investment, business or other professional advice.

On this page
  1. Step 1: split by time, not at random
  2. Step 2: step forward through history
  3. Step 3: look at the labels
  4. Step 4: purge and embargo
  5. Step 5: do the preparation inside the fold
  6. Step 6: protect the final test
  7. What each device guards against
  8. A procedure to follow

The purpose of validation is to find out how a method will do on data it has never seen. The standard tools (a holdout set, cross-validation) were designed for data in which each observation is independent of the others, like a pile of unrelated patients. Data ordered in time break that assumption: Wednesday’s value is related to Tuesday’s, a label such as “growth over the next week” reaches into the future, and the real future arrives after the past. Apply the standard tools naively and the validation score is optimistic, sometimes wildly so. This article walks through the adjustments.

Step 1: split by time, not at random

The simplest fix is a time-ordered holdout: train on the earliest data, validate on the next stretch, and keep the latest stretch as a final test used once (Holdout Set). A random split is wrong here because it puts the future on both sides of the line: a model trained on Wednesday’s neighbours can interpolate Wednesday, which no real forecaster could.

Write the split down before you start, and do not move it after you have seen the results.

Step 2: step forward through history

One holdout gives one score, which may be lucky. Walk-forward validation repeats the process: train up to a cut-off, test on the following period, move the cut-off forward, repeat (Walk-forward Validation). Every test uses only the past, and the scores across folds show whether performance is steady.

FoldTrained onTested on
1Months 1–6Month 7
2Months 1–7Month 8
3Months 1–8Month 9

You can keep the training window expanding (everything so far) or sliding (only the latest stretch). Use sliding if you suspect old data no longer describe today (Concept Drift).

Step 3: look at the labels

Even with a time split, there is a quieter leak. Many labels are defined over a window: “did the price rise over the next 5 days?”, “was the item still listed 30 days later?”. A training example on day 18 with a 5-day label depends on days 19 to 23. If your test period begins on day 21, the example’s label contains information from inside the test. A model trained on it has partly seen the answer.

Timeline of 40 days. Training days 1 to 15, purged days 16 to 20, test days 21 to 30, embargoed days 31 to 35, training days 36 to 40. A bracket shows that the label of a training example on day 18 reaches into the test period.
One test window, with a purge before it and an embargo after it.

Step 4: purge and embargo

The remedy has two parts (Purging and Embargo):

  • Purge: remove from training every example whose label window overlaps the test period. With 5-day labels and a test beginning on day 21, that is the examples on days 16 to 20.
  • Embargo: when training data also come from after the test period (as in k-fold schemes), skip a further stretch right after the test, because nearby observations are correlated and would carry information about the test back into training. Days 31 to 35 in the figure.

How long should they be? The purge must be at least as long as the label window. The embargo is a judgment: about as long as the memory of the series (how many days until yesterday stops helping to predict today), often a small fraction of the data. Err towards longer; it costs a little training data and buys honesty.

Step 5: do the preparation inside the fold

Any step that learns from data (scaling, selecting features, filling gaps, choosing thresholds) must be fitted on the training part of each fold only, then applied to the test part. Scaling the whole dataset first lets the test period influence the training (Data Leakage).

Step 6: protect the final test

Cross-validation and walk-forward are used for choosing: which model, which parameters. Each time you use a score to choose, the score becomes a little optimistic. So keep the final stretch aside and use it once, on the finalist (Try Enough Rules and One Will Work: Multiple Comparisons and Backtest Overfitting). If it disappoints, the honest response is to accept the result, not to adjust and test again; a second look at the same data is no longer a test.

What each device guards against

DeviceGuards againstNot enough on its own for
Time-ordered holdoutThe future informing the pastOne lucky or unlucky period
Walk-forward foldsA single unrepresentative periodLabels that overlap folds
PurgeLabels that reach into the test periodCorrelation after the test
EmbargoInformation leaking back from nearby observationsTuning on the final test
Final untouched testOverfitting through repeated choosingChanges in the world after you finish

A procedure to follow

  1. Write down the split rule and the label window before looking at results.
  2. Order data by time; define the final test as the latest stretch and lock it.
  3. Set up walk-forward folds on the rest, with purge at least as long as the label window and an embargo.
  4. Fit every data-dependent preparation inside each fold.
  5. Choose models using the fold scores, and note how many variants you tried.
  6. Evaluate the finalist once on the locked test, report the score with its uncertainty, and compare it with a simple baseline (Baseline).
  7. After release, keep checking against fresh outcomes: the world will move (Concept Drift).

References

  1. Rob J. Hyndman, George Athanasopoulos. Forecasting: Principles and Practice, section 5.10: Time series cross-validation . OTexts, 2021.
  2. Marcos López de Prado. Advances in Financial Machine Learning, chapter 7: Cross-Validation in Finance . Wiley, 2018. ISBN 978-1119482086.
  3. Christoph Bergmeir, José M. Benítez. On the Use of Cross-validation for Time Series Predictor Evaluation . Information Sciences, 2012. DOI: 10.1016/j.ins.2011.12.028
  4. Jeff Racine. Consistent Cross-validatory Model-selection for Dependent Data: hv-block Cross-validation . Journal of Econometrics, 2000. DOI: 10.1016/S0304-4076(00)00030-0
  • Holdout Set

    Data set aside before any building begins and used once, at the end, to estimate performance on unseen data.

  • Cross-validation

    Estimating how well a method will do on new data by repeatedly leaving part of the data out, training on the rest, and rotating.

  • Purging and Embargo

    Removing training observations whose time span overlaps the test period, and skipping a short stretch after it, so no information leaks between the two.