TradeHouse Fidelity Lab · Guides
Probability of backtest overfitting (PBO)
The probability of backtest overfitting is a measure introduced by David Bailey, Jonathan Borwein, Marcos López de Prado and Qiji Jim Zhu in The Probability of Backtest Overfitting (Journal of Computational Finance, 2017). It asks a direct question: if you pick the best configuration on one part of the data, how often does that pick turn out to be below median on the rest?
The method: combinatorially symmetric cross-validation
- Run every configuration you tried over the same period and build a matrix of returns: rows are time periods, columns are configurations.
- Split the rows into S equal blocks (S even, for example 16).
- For every way of choosing S/2 blocks as the in-sample set, treat the remaining blocks as out of sample.
- In each split, find the configuration with the best in-sample performance and record its rank among all configurations out of sample.
- Convert that rank to a relative position and take its logit. A negative logit means the in-sample winner landed below median out of sample.
- PBO is the share of splits where the logit is negative or zero.
How to read it
| PBO | Reading |
|---|---|
| Near 0 | The in-sample winner usually stays near the top out of sample. Selection is adding information. |
| Around 0.5 | Picking the in-sample winner is no better than picking at random. |
| Above 0.5 | The selection process is actively choosing configurations that do worse out of sample. |
Inputs and limits
- You need the return series of every configuration you tried, not just the winner. If you only kept the winner, PBO cannot be computed honestly.
- Blocks should be long enough to hold several independent observations each.
- PBO describes the selection process. It does not say whether the chosen strategy has positive expected return; pair it with a deflated Sharpe ratio and a frozen-rule out-of-sample test.
Related checks in Fidelity Lab
Fidelity Lab does not report a PBO number. It addresses the same problem from two other directions: a walk-forward with the threshold frozen on the training split, and Benjamini-Hochberg control at q=0.05 across every rule or cohort you submit, so the bar rises with the number of things you tried.
Try it on your own trades. The free Fidelity Lab tier runs one report a month on up to 5,000 legs: all 11 contamination detectors, cost re-netting and the walk-forward verdict. Run a free report or see pricing.
Frequently asked questions
Is PBO the same as the deflated Sharpe ratio?
No. The deflated Sharpe ratio adjusts one Sharpe ratio for the number of trials and its distribution. PBO measures how reliable the act of picking the best configuration was.
How many configurations do I need for PBO?
There is no fixed minimum, but with only a handful of configurations the ranks are coarse and the estimate is noisy.
Related guides
- Backtest overfitting: a practical guide
- Walk-forward validation vs cross-validation
- Backtest validation tool: what it should check
- Trading strategy audit service
- Overfitting in backtrader
- Trading strategy robustness tests
- Monte Carlo backtests
- Look-ahead bias in backtests
- Survivorship bias in backtests
- Data snooping in trading
- Out-of-sample testing in trading
- What is backtest overfitting?
- Deflated Sharpe ratio calculator
- Fidelity Lab
Educational content, not investment advice. Updated 2026-10-28.