Backtest, Then Forward Test With Small Size

When a backtest and a live account disagree, two separate explanations are available, and a good deal depends on which one is operating. The backtest may have been optimistic about the market. It may also have assumed a trader who does not exist. Forward testing at small size is the cheapest method available for establishing how much of each you are dealing with.

The school science project that works perfectly on the kitchen table and produces nothing at all in front of the class is the same phenomenon with a spreadsheet in the leading role. What was controlled during the rehearsal is not controlled during the performance, and the gap between the two is the thing being measured.

What a backtest cannot see

Fills, to begin with. A historical test works from closing or mid-prices, whereas the order you actually send arrives at the ask, with slippage, into a market that may well have moved in the interval between the signal and the fill, and the difference between the two prices is seldom in your favour.

Hesitation is the second thing, and it is invisible by construction. Every signal in a backtest is taken. In the live account you skip the ones that look ugly on the screen, you add one or two that look attractive and were never in the rules at all, and neither adjustment is recorded anywhere that the test can reach.

Then there is the data you did not have at the time. A signal computed on today’s close and executed at today’s close is a small cheat. Small cheats accumulate. Restated earnings. Index changes announced after the fact. Delisted companies that quietly left the dataset. Each one leaks a little of the future into the past.

Regime is the fourth. A rule tuned on a bull market is, in the end, a rule about bull markets.

It follows that every backtest overstates live performance to some degree, this being a structural property of testing a rule against data you have already looked at. The question worth asking is by how much.

What forward testing adds

Run the strategy in real time, on real quotes, with real orders, at a size where the money does not matter, and three things become visible that no backtest is capable of showing you.

The first is execution cost as actually incurred: the spread you crossed, the slippage you took, the fills that never happened because a limit order sat one cent away from the print.

The second is your own behaviour. Whether you take every signal or quietly edit the list. Whether the stop stays where the rules put it, or gets moved in the minute before it is due to be hit. The distance between the result the rules produced and the result you produced is a number, and it is frequently the largest number in the whole exercise.

The third is a sample that is genuinely out of sample. None of the forward period existed when the rules were written, so whatever it returns stands as the first honest estimate of the edge you have.

Why small, and how small

Small enough that a loss produces no emotional response at all. That is the whole specification. If a stop-out stings even slightly, the experiment has quietly turned into a different experiment, one in which the subject is you under pressure and the answer that comes back will be an answer about you. Fifty shares where you would normally trade a thousand is about the right order of magnitude. Paper trading belongs one step earlier in the sequence, being useful for mechanics and useless for the hesitation problem, since taking every signal costs a person nothing when the money is imaginary.

An illustration

Take a method backtested with some care and then run forward at small size for a hundred trades. Suppose the forward expectancy arrives at roughly two-thirds of the historical figure. The instinct is to blame costs. Costs rarely explain it. On inspection, most of the gap sits in another column of the forward log, the one recording signals the rules produced and the trader declined, because those skipped trades, taken as a group, did better than the trades that were actually placed, a sentence that is a good deal easier to write down than it is to read back in your own log. The rules were sound. The filter laid on top of them was subtracting value, quietly, on every pass, and no backtest could have revealed it for the simple reason that the backtest contained no trader.

Running the two stages

1. Backtest with costs set at the pessimistic end, across at least one regime that does not suit the strategy, and keep a count of the variants tried before you settled on this one. The backtesting guide lists the biases in full.
2. Freeze the rules in writing. Forward testing a strategy that is still being adjusted amounts to another backtest, run more slowly.
3. Trade it live at a size that cannot hurt, for a number of trades fixed before you begin. Fifty is a minimum for anything. A hundred is better.
4. Log every signal, taken or declined, with the outcome of each. The difference between those two columns is your behavioural cost.
5. Compare forward expectancy against the backtest, and read thirty trades is not a track record before deciding what the comparison is entitled to mean.

Only then, and only where the lower bound of the forward result sits above zero, does size increase. The expectancy and sizing guide covers how far it should be allowed to go.