TradeHouse Fidelity Lab · Guides
Out-of-sample testing in trading
An out-of-sample test scores a trading rule on data that played no part in designing it. It is the most direct evidence that a backtest result is not just a description of the past.
How to set one up
- Split by time. Keep the most recent part of the data aside. Random splits leak information between periods.
- Freeze the rule. Fix every parameter, filter and threshold on the in-sample data before the out-of-sample data is touched.
- Score it once. Run the frozen rule on the hold-out period and record the result.
- Apply real costs. Use the same fee and slippage model you would use live.
- Judge against a bar set in advance. Decide before looking what result would count as a pass.
Mistakes that spoil it
- Peeking. Running the hold-out, adjusting, and running again turns it into in-sample data. After a few rounds it offers no protection.
- Too short a hold-out. A few weeks of data is a handful of independent observations. The result is mostly noise.
- Regime mismatch. A hold-out that is unusually calm or volatile compared with the training period can mislead in either direction.
- Leakage through features. Scalers or normalisations fitted on all data carry hold-out information into training.
Single hold-out, walk-forward or live?
| Approach | Strength | Weakness |
|---|---|---|
| Single hold-out | Simple and clean if used once | One period, one result |
| Walk-forward | Many out-of-sample windows, mimics re-fitting over time | Each window is shorter |
| Paper or small live trading | Truly unseen data, real fills | Slow, and costs time or a little capital |
How Fidelity Lab does it
Fidelity Lab runs a point-in-time walk-forward with the threshold frozen on the training split, re-nets fees from a per-venue schedule and checks whether the out-of-sample window holds enough independent days. If the in-sample edge does not reproduce, the verdict is DISSOLVES and the report says whether out-of-sample decay or fees killed it.
Try it on your own trades. The free Fidelity Lab tier runs one report a month on up to 5,000 legs: all 11 contamination detectors, cost re-netting and the walk-forward verdict. Run a free report or see pricing.
Frequently asked questions
What share of data should be out of sample?
Common splits keep 20 to 30 percent aside, but the real requirement is enough independent days in the hold-out to tell an edge from noise.
Can I reuse a hold-out period after changing the strategy?
Not as a clean test. Once a period has influenced a decision it is in-sample. Use a later period or forward data instead.
Related guides
- Backtest overfitting: a practical guide
- Walk-forward validation vs cross-validation
- Backtest validation tool: what it should check
- Probability of backtest overfitting (PBO)
- Trading strategy audit service
- Overfitting in backtrader
- Trading strategy robustness tests
- Monte Carlo backtests
- Look-ahead bias in backtests
- Survivorship bias in backtests
- Data snooping in trading
- What is backtest overfitting?
- Deflated Sharpe ratio calculator
- Fidelity Lab
Educational content, not investment advice. Updated 2026-10-28.