TradeHouse Fidelity Lab · Guides
Backtest overfitting: a practical guide
A backtest is overfit when its result depends on the particular history it was tuned on rather than on something that will repeat. The danger is not that overfit backtests look bad. It is that they look good, often better than honest ones, because the search process selected them for looking good.
Where overfitting comes from
- Too many trials. Every parameter value, filter and universe you try is another draw. The best of 200 draws from pure noise can show a Sharpe ratio well above 1.
- Leakage. Information from the test period reaches the rule: a filter added after seeing a drawdown, a feature computed with future data, a close price used for a same-bar entry.
- Small effective samples. Thousands of trades clustered in a few days are a few observations, not thousands.
- Unrealistic costs. A thin edge measured before fees and slippage is often an edge that does not exist after them.
How to measure it
| Method | Question it answers | Guide |
|---|---|---|
| Walk-forward with a frozen rule | Does the rule chosen on past data still work on data it never saw? | Walk-forward schedule |
| Deflated Sharpe ratio | Is the best Sharpe still significant once you account for how many trials produced it? | Calculator |
| Probability of backtest overfitting (PBO) | How often does the in-sample winner rank below median out of sample? | PBO explained |
| Multiplicity control | After testing many rules, which survivors are not just false discoveries? | Data snooping |
| Robustness and Monte Carlo tests | Does the result survive small changes in parameters, costs and trade order? | Robustness tests |
A checklist before you trade a backtest
- Write down how many variants you tried, including the ones you discarded.
- Confirm every feature and every fill price was knowable at decision time (look-ahead bias).
- Confirm the universe includes things that died, delisted or were voided (survivorship bias).
- Re-net costs from the venue's real fee schedule, not a flat guess.
- Count independent days, not rows.
- Score a frozen rule on a held-out period you have not looked at (out-of-sample testing).
- Adjust the headline Sharpe for the number of trials.
What a clean result means
Passing these checks does not prove an edge. It removes the most common ways a backtest lies. The honest next step after a pass is a small live or paper allocation with the rule frozen, not full size.
Try it on your own trades. The free Fidelity Lab tier runs one report a month on up to 5,000 legs: all 11 contamination detectors, cost re-netting and the walk-forward verdict. Run a free report or see pricing.
Frequently asked questions
What is the difference between overfitting and curve fitting?
In trading the two terms are used for the same thing: tuning a rule to the noise of one sample.
Can I avoid overfitting by using fewer parameters?
It helps, but it does not remove the trials you ran before choosing the final rule. Those trials still count.
Related guides
- Walk-forward validation vs cross-validation
- Backtest validation tool: what it should check
- Probability of backtest overfitting (PBO)
- Trading strategy audit service
- Overfitting in backtrader
- Trading strategy robustness tests
- Monte Carlo backtests
- Look-ahead bias in backtests
- Survivorship bias in backtests
- Data snooping in trading
- Out-of-sample testing in trading
- What is backtest overfitting?
- Deflated Sharpe ratio calculator
- Fidelity Lab
Educational content, not investment advice. Updated 2026-10-28.