In-sample, out-of-sample, and the one rule about it
The standard defence against overfitting is to split the history: develop on one part, then test once on a part you never looked at. It works, and it is the most commonly broken rule in the field, because the way it fails is so easy to talk yourself into.
The rule is that out-of-sample data can be used once. Look at it, get a disappointing result, adjust the strategy and look again, and it has become development data. It cannot be reset. Everything after that first look is in-sample, and the test you thought you were running no longer exists.
That constraint is what makes the split expensive, and it is why it should be spent deliberately. Do the development properly first — settle the idea, the parameters and the costs on the development period — and only then spend the reserved period, on a strategy you consider finished, with the criterion for pass or fail written down before you look.
A useful discipline for the split itself: reserve the most recent stretch rather than a random slice. Markets change over time, so testing on the most recent period asks the only question that matters, which is whether the method works in conditions like the ones you are about to trade. A random slice from the middle answers a weaker question.
Walk-forward testing generalises this: develop on a window, test on the next, roll both forward, repeat, and evaluate only the concatenated out-of-sample results. It costs more history and more patience, and it is the closest thing there is to an honest estimate of what a method would have done. It also frequently reports that a strategy which looked excellent on a single split does not survive, which is precisely the information you were looking for.
What to take away
- Out-of-sample data can be used once. Looking twice turns it into development data.
- Reserve the most recent period, not a random slice.
- Walk-forward is the honest version, and it fails a lot of strategies for good reason.
Where it goes wrong
- Adjusting the strategy after a poor out-of-sample result and re-testing.
- Deciding the pass criterion after seeing the result.
- Reserving a slice from the middle of the history.
This chapter, measured against your own trades
In the app the same chapter ends in your figures rather than an example: how often you did the thing it describes, over your last ninety days. You pick one change to make, and Evidence checks afterwards whether it actually changed — from your journal, arithmetic, no opinion involved. Questions you get wrong come back a week later and again a month after that.
Open the free plan →Check that it stuck
Answers shown — in the app these are asked before you see them, and the ones you get wrong come back after a week.