What the verdict tests

A verdict is the answer to one question: did this strategy earn its result, or did the search find luck? A run passes through stages, cheap ones first. A strategy stops at the first stage it fails, and the verdict names that stage.

The stages#

1. Cheap checks. Do costs leave any profit, are there enough trades to say anything, and is the strategy's list of settings short enough to trust? A strategy that fails here never reaches the expensive stages.
2. Hold-out. The last part of the history is kept back while the search runs. The strategy must still work on it. See the hold-out test.
3. Rolling consistency. The earlier history is cut into consecutive windows. The strategy must work in most of them, not in one lucky stretch. See the rolling test.
4. The expensive tier. Four tests that ask how likely it is that the winner is an accident:
   • The probability of backtest overfitting: how often the best setting of one half of the history is below average on the other half.
   • The deflated Sharpe ratio: the Sharpe ratio, reduced for how many things were tried.
   • Plateau: whether the neighbours of the winning settings also work, or only that exact point.
   • Walk-forward: settings chosen on one window, then tried on the next, again and again.

The cost of the trades is tested again with wider spreads. A strategy that dies when the spread triples was living on the spread. See the cost stress.

What the words on a result mean#

A result says Validated when a candidate cleared every stage. Otherwise it says where the strategy was killed, for example at the gates, at the hold-out or at the expensive tier. Killed is a result, not an error: it tells you which question the strategy could not answer.