Backtesting Trading Strategies Without Fooling Yourself

Every backtest overstates live performance. That is a structural property of testing a rule on data you have already seen, whatever your competence.

The useful question is by how much, and which specific biases are inflating it.

The five biases, and what each one costs

1. Overfitting, usually the largest

You test a moving-average crossover. The 20/50 is mediocre, so you try 18/44. Better. Then you add an RSI filter, a volatility condition, and skip Mondays. The equity curve becomes beautiful.

You have not found an edge. You have found the parameter set that best fits the noise in this particular sample.

The tell is fragility. Change 18/44 to 19/45. If the result collapses, you fitted noise, because a real effect would not care about a one-period change. A genuine edge shows a broad plateau of acceptable settings rather than a single sharp peak.

2. Survivorship bias

Backtesting “all S&P 500 stocks” using today’s constituent list silently excludes every company that went bankrupt, was delisted, or was removed from the index. Your test never experiences those losses.

For any long-biased strategy this inflates returns, sometimes dramatically. The fix needs point-in-time constituent data, which is expensive, and that expense is one of the main reasons cheap backtests are unreliable.

3. Look-ahead bias

Using information that was not available at the time. It creeps in subtly:

• Using a day’s close to decide a trade executed at that same close
• Using restated earnings that were revised months after the original release
• Using index membership before the change was announced
• Filtering a universe on a characteristic measured later in the period

Each injects future knowledge. The rule that catches most of them: at every decision point, ask what you would genuinely have known at that moment.

4. Cost omission

The most quantifiable bias, and the one most often skipped. A strategy tested on closing prices with no spread, no slippage and no commission is testing a market that does not exist.

Trades a year	Cost at 0.1%	Cost at 0.35%
20	2%	7%
100	10%	35%
500	50%	175%

A strategy showing 40% annual returns at 500 trades a year is a losing strategy once costs are modelled honestly. Always assume the high end; see trading costs for realistic figures.

5. Regime dependence

A strategy tested on 2010–2020 was tested almost entirely on a bull market with falling rates. Mean-reversion strategies looked exceptional in that period and behaved very differently outside it.

Test across regimes deliberately: a bear market, a high-volatility stretch, a sideways year. A strategy that only works in one regime is a bet on that regime continuing, which is a far larger position than you think you are taking.

Validation that catches these

Out-of-sample testing. Hold back at least a third of the data. Build entirely on the rest. Test once on the held-back set.

The discipline nobody keeps: out-of-sample data is single use. Look at it, revise the strategy, and it has become in-sample. Its value as an independent test is gone permanently.

Walk-forward analysis. Optimise on window 1, test on window 2, roll both forward, repeat. Measure only the out-of-sample segments. It is harder to pass because it asks whether the strategy keeps working as conditions shift, not merely whether it worked once.

Parameter sensitivity. Plot performance across a grid of settings. You want a plateau. A lone spike surrounded by poor results is overfitting made visible.

Trade-order randomisation. Shuffle the sequence of your trades and look at the distribution of resulting equity curves. This separates how much of your curve’s shape is edge from how much is the order the wins happened to arrive in.

The honest reporting standard

When you state a result, state it with:

• Trade count and the standard error of expectancy; see expectancy and sample size
• The number of variants tested, which is the multiple-comparisons cost
• Cost assumptions, explicitly
• Maximum drawdown and its duration, since the length is what breaks people, not the depth
• Performance by regime, not only in aggregate

A strategy reported this way looks considerably worse than the same strategy shown as a single equity curve. That gap is precisely the amount you were previously fooling yourself by.

The practical shortcut

If full walk-forward analysis is more machinery than you want, three checks catch most of the damage:

1. Nudge every parameter by 10%. If results collapse, stop.
2. Model costs at double what you think. If the edge vanishes, it was never there.
3. Test on one instrument you did not develop it on. Portability is the cheapest proxy for robustness there is.

A strategy surviving all three is not guaranteed to work. But one failing any of them is almost certainly not an edge, and you have learned that for an hour’s work instead of a year’s trading capital.