Backtesting without fooling yourself · Chapter 4 of 6

The costs everyone forgets

Advanced6 min read · part of Backtesting without fooling yourself

A backtest run at the mid price with no commission is not an optimistic estimate of a strategy. It is a test of a different strategy — one that trades in a market that does not exist. And the gap between the two grows with frequency, so the methods that look best in a frictionless test are exactly the ones most damaged by reality.

The spread is the first cost and the easiest to get wrong, because it is not constant. It widens at the open, around news, in the last hour, and in thin instruments overnight — which is to say, precisely when a lot of strategies want to trade. Using an average spread on a strategy that concentrates its entries in the widest minutes of the day understates the cost systematically rather than randomly.

Slippage is the second, and it is asymmetric in a way that is easy to miss. Your stop is more likely to fill worse than requested than better, because stops cluster and get taken out in fast moves. Limit orders have the opposite problem: in the backtest they always fill when price touches, and in reality the ones that touch and reverse are exactly the ones that fill, while the ones that touch and run often do not.

Then the boring ones: commission per side, financing on positions held overnight, and currency conversion when the instrument settles in something other than your account currency. Individually small, and on a method taking several trades a day they are frequently the whole margin between profitable and not.

The test to apply is a sensitivity check rather than a precise figure. Run the backtest again with costs at, say, one and a half times your estimate. A strategy that survives that has some room; one that turns negative was never really profitable, it was profitable to within the error of your cost assumption — which is not the same thing at all.

What to take away

A frictionless backtest tests a different strategy, and higher frequency widens the gap.
Spread widening correlates with when strategies want to trade — the error is systematic.
Test at 1.5× your cost estimate; surviving that is the meaningful result.

Where it goes wrong

Using a single average spread for a strategy that trades the open.
Assuming limit orders fill whenever price touches the level.
Ignoring overnight financing on a swing method.

This chapter, measured against your own trades

In the app the same chapter ends in your figures rather than an example: how often you did the thing it describes, over your last ninety days. You pick one change to make, and Evidence checks afterwards whether it actually changed — from your journal, arithmetic, no opinion involved. Questions you get wrong come back a week later and again a month after that.

Open the free plan →

Check that it stuck

Answers shown — in the app these are asked before you see them, and the ones you get wrong come back after a week.

Why is using an average spread systematically wrong for many strategies?

Spreads widen exactly when many strategies want to trade, so the error is one-directional — A strategy concentrating entries in the widest minutes pays more than average every time, so the bias does not cancel out.

What is the asymmetry in limit-order fills that backtests miss?

The ones that touch and reverse fill; the ones that touch and run often do not — The backtest fills every touch, which quietly hands the strategy the good outcomes it would have missed.

A strategy is profitable at your cost estimate and negative at 1.5× it. What should you conclude?

It is profitable only to within the error of your assumption, which is not a result — The margin is inside the uncertainty of an input. That is not an edge, it is a rounding artefact.

← In-sample, out-of-sample, and the one rule about it Monte-Carlo: the same edge, a different path →