Backtesting without fooling yourself · Chapter 1 of 6

The rear-view mirror knows everything

Advanced7 min read · part of Backtesting without fooling yourself

The first and largest problem with testing on history is that history is complete. You know what happened next, and you cannot un-know it. Every judgement made while looking at a finished chart is made with information the trader in that moment did not have, and almost none of that leakage feels like cheating while it is happening.

The obvious form is lookahead in the rules: using a day’s high in a signal evaluated intraday, using the close of the bar you enter on, sizing from a volatility figure computed over the whole test period. These are bugs, they are findable, and any result that depends on one is worthless rather than merely optimistic.

The subtle form has no bug in it at all. You scroll to an interesting stretch of chart, you see the move, and then you decide which rule would have caught it. The rule is then tested on the same data that suggested it. Nothing in the code is wrong; the whole procedure is circular, and it produces beautiful equity curves that never survive contact with a live market.

A third form is survivorship, and it applies to instruments as much as to strategies. Testing a method on the instruments you currently trade means testing on the ones that did not make you abandon them. The selection already happened, silently, before the test started.

The structural defence is a replay that hides the future — trading forward candle by candle, without the right-hand side of the chart visible, and recording the decision before revealing the outcome. It is slower than scrolling a finished chart, and that slowness is the point: it is the only way to find out what you would actually have done rather than what you can see should have been done.

What to take away

Lookahead in the rules is a bug; lookahead in how the rule was chosen is a procedure.
A rule found by looking at a stretch of chart cannot be tested on that stretch.
A replay that hides the future is the structural fix, not a matter of willpower.

Where it goes wrong

Using the entry bar’s close, or the day’s high, in a signal evaluated during the day.
Designing the rule on the same data used to validate it.
Testing only on instruments you already chose to keep trading.

This chapter, measured against your own trades

In the app the same chapter ends in your figures rather than an example: how often you did the thing it describes, over your last ninety days. You pick one change to make, and Evidence checks afterwards whether it actually changed — from your journal, arithmetic, no opinion involved. Questions you get wrong come back a week later and again a month after that.

Open the free plan →

Check that it stuck

Answers shown — in the app these are asked before you see them, and the ones you get wrong come back after a week.

You spot a great move on a chart and design a rule that would have caught it, then test that rule on the same period. What is wrong?

The procedure is circular — the data that suggested the rule cannot also validate it — No bug is required for this failure. The leakage is in how the rule was chosen, not in how it was computed.

Which of these is a lookahead bug?

Using the current bar’s close in a signal you act on within that bar — The close is not known until the bar ends. Acting on it inside the bar uses information that did not exist yet.

Why does a replay that hides the future help?

It records the decision before the outcome is visible, which is the only way to know what you would have done — Willpower does not stop hindsight. Removing the information does.

← All chapters How a perfect backtest gets built →