Read
The Probability of Backtest Overfitting
Bailey et al.
This 2015 paper by Bailey, Borwein, Lopez de Prado, and Zhu introduces a general computational framework for quantifying the probability of backtest overfitting (PBO), defined as the probability that the strategy configuration delivering maximum in-sample performance systematically underperforms the median configuration out of sample. The framework is implemented via combinatorially symmetric cross-validation (CSCV), a nonparametric, model-free procedure that partitions historical performance series into subsets and exhaustively swaps in-sample and out-of-sample roles to produce a bootstrapped distribution of OOS outcomes without requiring a fixed hold-out set.
Key takeaways
- PBO is formally defined as the probability that an optimally selected in-sample strategy underperforms the median of all tested configurations out of sample, capturing the degree to which the strategy selection process has been conducive to overfitting rather than discovery of genuine predictive structure.
- CSCV avoids the credibility problem of the hold-out method by not requiring a pre-committed out-of-sample window; instead it exhaustively partitions performance time series into all combinatorially symmetric pairings of IS and OOS subsets, producing a bootstrapped distribution of OOS performance that reflects the full selection process.
- Standard hold-out cross-validation fails to control for backtest overfitting in investment research because it treats the validation as a single trial, ignores the number of configurations tested before the holdout was applied, has high estimation variance on small samples, and can use the most or least representative portion of the data depending on where the split is placed.
- The framework is generic and model-free, requiring only the time series of backtested performance for each strategy configuration and no knowledge of the underlying trading rule or forecasting equation, making it applicable across diverse strategy types and compatible with any user-chosen performance statistic beyond the Sharpe ratio.
- A backtest is characterized as overfit when the IS-optimal configuration systematically loses its relative rank out of sample, which occurs because parameters are selected to exploit noise patterns present in the training period that are either diluted or actively reversed in new data, particularly under mean-reverting financial processes.
- The paper demonstrates that minimum backtest length is a related but distinct concept from PBO: MinBTL is a necessary but not sufficient condition for avoiding overfitting, while PBO provides a direct probabilistic measure of whether the selection process itself has inflated the apparent quality of the chosen strategy.
Reflections
No notes on this one yet. I add reflections as I finish or revisit a book.