Evidence Trading
Backtesting without fooling yourself · Chapter 2 of 6

How a perfect backtest gets built

Advanced7 min read · part of Backtesting without fooling yourself

Overfitting is not a mistake people make through carelessness. It is what happens when a careful person works hard on a fixed set of data, and the effort itself is the mechanism. Every parameter you adjust to improve the result is a small amount of the test period’s specific noise being copied into the strategy.

The tell is not that the result is good. It is that the result is fragile. A robust parameter sits on a plateau: 18, 20 and 22 all work about as well, so the exact value does not matter much. An overfit parameter sits on a spike, where 20 is excellent and 19 and 21 are mediocre. That spike is a description of one particular history, and there is no reason for the next history to have its spike in the same place.

The number of choices matters more than any single one. Entry condition, exit condition, two thresholds, a session filter and a stop multiple is six knobs; a few dozen combinations tried across them will produce something impressive from noise alone, on any data, including data you generated randomly. This is worth doing once as an exercise, because seeing a beautiful equity curve emerge from pure noise permanently changes how you read your own.

The defences are simple and unpopular. Prefer fewer parameters. Prefer round values you did not tune. Check the neighbourhood of every parameter rather than its peak. Count how many variants you tried, and remember that the best of fifty is expected to look good even if none of them are.

And be specific about what you are asking the test for. A backtest that answers "does this idea have an edge" needs far less tuning than one that answers "what is the optimal setting", and the second question is much less useful than it appears — because the optimal setting is optimal for a period that will not repeat.

What to take away

  • Overfitting is the by-product of effort on fixed data, not of carelessness.
  • Robust parameters sit on plateaus; overfit ones sit on spikes.
  • The best of fifty variants looks good even when none of them work.

Where it goes wrong

  • Reporting the peak parameter without checking its neighbours.
  • Adding a filter to remove the losing trades you can see in the test.
  • Losing count of how many variants were tried.

This chapter, measured against your own trades

In the app the same chapter ends in your figures rather than an example: how often you did the thing it describes, over your last ninety days. You pick one change to make, and Evidence checks afterwards whether it actually changed — from your journal, arithmetic, no opinion involved. Questions you get wrong come back a week later and again a month after that.

Open the free plan →

Check that it stuck

Answers shown — in the app these are asked before you see them, and the ones you get wrong come back after a week.

A parameter of 20 performs excellently while 19 and 21 are mediocre. What does that suggest?
The result is fitted to this period’s noise and unlikely to hold — Real effects are broad. A spike that narrow is a description of one particular history.
Why does the number of variants tried matter?
The best of many variants looks good by chance even if none has an edge — Selecting the maximum from a large set is itself a way of fitting noise, whatever each individual test showed.
Which question needs less tuning and is more useful?
Does this idea have an edge at all — The optimum is optimal for a period that will not repeat. Whether the idea works at sensible settings is the durable question.
← The rear-view mirror knows everything In-sample, out-of-sample, and the one rule about it →