TradeHouse Fidelity Lab · Guides
Data snooping in trading
Data snooping happens when the same data set is used again and again to search for a strategy. Each test is reasonable on its own. Together they guarantee that something will look significant by chance, and that is the thing that gets reported.
A simple illustration
If you test 100 rules that have no edge at a 5% significance level, you should expect about five of them to pass. Report only those five and the research looks successful. Nothing in the five results reveals the 95 that failed.
Ways to control it
| Method | Idea | Trade-off |
|---|---|---|
| Bonferroni correction | Divide the significance level by the number of tests | Very conservative when tests are many or correlated |
| Benjamini-Hochberg (1995) | Control the expected share of false discoveries among the rules you accept | More power than Bonferroni; allows some false positives by design |
| White's Reality Check (2000) | Bootstrap test of whether the best rule beats a benchmark, given all rules tried | Needs the return series of every rule |
| Hansen's SPA test (2005) | Refines Reality Check to be less affected by poor rules in the set | Same data requirement |
| Deflated Sharpe ratio | Adjust the best Sharpe for the number of trials | Needs an honest trial count; see the calculator |
| Fresh hold-out data | Test the final rule once on data never used in the search | Data can only be used once; see out-of-sample testing |
The hidden trials problem
Every correction depends on knowing how many things were tried. Researchers routinely undercount: ideas dropped after a quick look, parameter changes made on a hunch, and earlier projects on the same data all count. When in doubt, assume the number is larger.
How Fidelity Lab handles it
Fidelity Lab applies Benjamini-Hochberg at q=0.05 across every train-frozen rule or cohort evaluated in a submission. Send twenty cohorts and the bar each one must clear rises accordingly.
Try it on your own trades. The free Fidelity Lab tier runs one report a month on up to 5,000 legs: all 11 contamination detectors, cost re-netting and the walk-forward verdict. Run a free report or see pricing.
Frequently asked questions
Is data snooping the same as p-hacking?
They describe the same behaviour. P-hacking is the term common in academic research; data snooping is the common term in finance.
Does machine learning make data snooping worse?
It can, because models search a very large space automatically. The same corrections and hold-out discipline apply.
Related guides
- Backtest overfitting: a practical guide
- Walk-forward validation vs cross-validation
- Backtest validation tool: what it should check
- Probability of backtest overfitting (PBO)
- Trading strategy audit service
- Overfitting in backtrader
- Trading strategy robustness tests
- Monte Carlo backtests
- Look-ahead bias in backtests
- Survivorship bias in backtests
- Out-of-sample testing in trading
- What is backtest overfitting?
- Deflated Sharpe ratio calculator
- Fidelity Lab
Educational content, not investment advice. Updated 2026-10-28.