A backtest is an argument, and most backtests argue for more than the data supports. The rules below are how we keep an argument honest. We apply them to our own research and to every strategy a client asks us to validate.
How a test is run
- Clean data first. Before any signal is tested, the data are checked for look-ahead, meaning information used before it was available, for survivorship, meaning instruments that later disappeared and were left out, and for how futures contracts were rolled and joined into one series.
- No flattering position sizing. A signal is first judged at a fixed size of one contract or one lot. Compounding, even at 1% fixed-fractional sizing, makes a weak signal look strong, so sizing is studied separately and later.
- Nothing that hides the signal. No take-profit or stop-loss orders, trailing stops or other exits that can be tuned until the history looks good. A signal is judged on its own exits and on time-based exits.
- Costs from the start. Spread, slippage and commission are in every test, set conservatively for the market and the size traded.
- Small search spaces. The fewer variations tried, the less the best one owes to luck. When many variations have been tried, the result is corrected for it.
- Data the strategy has never seen. A large share of the history is held back from all development and opened once, at the end.
How a result is judged
- A result has to hold across neighbouring parameter values, not only at the best one.
- A result has to hold on related instruments and in different periods, unless there is a clear reason why it should not.
- An edge can shrink and a market can change character. A result is examined period by period, to see whether the edge is decaying and whether it depends on one kind of market.
- The worst losing run and the longest time below a previous high are estimated before trading starts, so live results have a limit to be judged against.
- Live trading is compared with the research continuously. A strategy that drifts away from its research is flagged and, by rule, switched off.
What a client receives
Every validation ends in a written verdict. With it comes the list of every test that was run, including the ones that failed, and a statement of what evidence would change the verdict.
What we do not publish
This site shows no trading performance figures. Client research stays with the client, and a chart of our own results would prove less than the method behind it. The library lists the published papers behind these rules, as questions any client can ask.