Advanced
Backtesting Trading Strategies Without Fooling Yourself
Every backtest flatters the strategy that produced it. The useful question is by how much, and each of the five biases here has a procedure that puts a number on it.
Every backtest overstates live performance. That is a structural property of testing a rule on data you have already seen, and it reflects nothing whatever about the competence of the person running the test.
The useful question is by how much, and which particular biases are doing the inflating.
The five biases, and what each one costs
1. Overfitting, usually the largest
You test a moving-average crossover. The 20/50 is mediocre, so you try 18/44, which is better. An RSI filter helps. So does a volatility condition, and a rule that skips Mondays, and by this point the equity curve has turned beautiful.
What has been found is the parameter set that best fits the noise in this particular sample.
The tell is fragility. Change 18/44 to 19/45. If the result collapses you have fitted noise, because a real effect would not care about a one-period change, and a genuine edge tends to show a broad plateau of acceptable settings around it.
2. Survivorship bias
Backtesting “all S&P 500 stocks” against today’s constituent list quietly removes every company that went bankrupt, was delisted, or was dropped from the index along the way. Your test never experiences those losses. For a long-biased strategy the inflation can be dramatic.
The remedy is point-in-time constituent data. It is expensive. That expense is among the main reasons cheap backtests are unreliable.
3. Look-ahead bias
Information that was unavailable at the time, used as though somebody had it. It creeps in quietly:
- Using a day’s close to decide a trade executed at that same close
- Using restated earnings revised months after the original release
- Using index membership before the change was announced
- Filtering a universe on a characteristic measured later in the period
Each of those injects future knowledge into a decision taken in the past. One rule catches most of them: at every decision point, ask what you would truly have known at that moment.
4. Cost omission
The most quantifiable bias, and the one most often skipped. Test a strategy on closing prices with spread, slippage and commission all set to zero and you have tested it in a market that does not exist.
| Trades a year | Cost at 0.1% | Cost at 0.35% |
|---|---|---|
| 20 | 2% | 7% |
| 100 | 10% | 35% |
| 500 | 50% | 175% |
A strategy showing 40% annual returns at 500 trades a year is a losing strategy once costs are modelled honestly. Assume the high end. See trading costs for realistic figures.
5. Regime dependence
A strategy tested on 2010 to 2020 was tested almost entirely on a bull market with falling rates. Mean-reversion strategies looked exceptional over that stretch. Outside it they behaved very differently.
So test across regimes deliberately: a bear market, a high-volatility stretch, a sideways year. A strategy that works in one regime alone amounts to a bet on that regime continuing, which is a considerably larger position than most people realise they are taking when they describe themselves as trading a system.
Validation that catches these
Out-of-sample testing comes first. Hold back at least a third of the data, build the strategy entirely on the rest, and test once on the held-back set.
Then the discipline nobody keeps. Out-of-sample data is single use. Look at it, revise the strategy, and the held-back set has quietly become in-sample, its value as independent evidence gone permanently.
Walk-forward analysis. Optimise on window 1, test on window 2, roll both forward, repeat, and measure only the out-of-sample segments. It is harder to pass, because it asks whether the strategy keeps working as conditions shift, window after window.
Parameter sensitivity is cheaper. Plot performance across a grid of settings and look for a plateau, since a lone spike surrounded by poor results is overfitting made visible.
Trade-order randomisation is the last of them. Shuffle the sequence of your trades and examine the distribution of the resulting equity curves, which separates the part of the curve’s shape that is edge from the part that is the order in which the wins happened to arrive.
The honest reporting standard
State a result alongside all of the following:
- Trade count and the standard error of expectancy; see expectancy and sample size
- The number of variants tested, which is the multiple-comparisons cost
- Cost assumptions, explicitly
- Maximum drawdown and its duration, since the length is what breaks people more often than the depth
- Performance broken out by regime, alongside the aggregate figure
A strategy reported this way looks a good deal worse than the same strategy presented as a single equity curve. That gap is the amount by which you were previously fooling yourself.
The practical shortcut
Full walk-forward analysis is more machinery than many people want. Three checks catch most of the damage.
- Nudge every parameter by 10%. If results collapse, stop.
- Model costs at double what you think they will be. If the edge vanishes, it was never there.
- Test on one instrument you did not develop it on. Portability is the cheapest proxy for robustness available.
Surviving all three guarantees nothing. Failing any of them is strong evidence that there is no edge to find, and an hour of work will have told you so, at a price considerably below a year of trading capital.
Frequently asked questions
Why do backtested strategies fail in live trading?
Usually because several small optimistic biases compound. Parameters were chosen using the same data that then measured performance, costs were understated or left out entirely, the universe quietly excluded companies that went bankrupt, and the data contained information nobody had at the time. Each of those is modest on its own. Stacked together they routinely turn a losing strategy into a curve that looks profitable.
What is overfitting in trading?
Tuning a strategy until it fits the noise in your historical sample, at the expense of any pattern that will outlive it. The clearest symptom is a result that falls apart when you nudge the parameters slightly, or when you apply the same rules to a different period or instrument. A real edge survives small changes to its settings, and that is a test worth running before anything else.
What is survivorship bias?
Testing on a universe that contains only the companies still listed today. Firms that went bankrupt or were delisted are simply absent, so your test never experiences the losses they caused. It inflates returns for any long-biased strategy, and it is among the easiest biases to introduce entirely by accident, since most convenient data sets are built this way.
What is walk-forward analysis?
Optimising parameters on one window of data, testing them on the next unseen window, then rolling both windows forward and repeating the exercise. Performance is measured only on the out-of-sample segments. It asks more of a strategy than a single train-and-test split does, because it checks whether the thing keeps working as conditions shift across several successive windows.
How much out-of-sample data do I need?
As a rough guide, hold back at least a third of your data and leave it alone until the strategy is final. The stricter discipline, and the one people find hardest, is that out-of-sample data is single use. Every time you check it and then go back and revise, it has become in-sample, and whatever independent evidence it might have given you is gone.