Can a strategy pass every backtest check and still be luck?
Yes. We built one by accident, nearly published it, and then found the check that catches it.
This page is about the one test that is missing from almost every backtest you will ever be shown, including — until September 2026 — ours.
The sweep this came from is #11 in the research log.
The strategy that passed everything
The volatility term structure compares spot fear to three-month fear: the VIX against the VIX3M. When the ratio goes above 1.0 the market is paying more for protection now than for protection in three months, which happens on about 7.6% of trading days and essentially only during stress. The hypothesis, registered before the run: buy the index when that ratio crosses back below 1.0, because the panic is subsiding. Exit on a trailing stop.
Here is what it produced on SPY, QQQ and IWM across two independent windows, 2010 to 2018 and 2018 to 2026:
| Cell | Trades | Profit factor | Return | Strip-best | Max drawdown |
|---|---|---|---|---|---|
| SPY 2010–18 | 38 | 2.96 | +68.3% | +56.6% | 12.0 |
| SPY 2018–26 | 41 | 1.71 | +26.9% | +14.1% | 9.4 |
| QQQ 2010–18 | 38 | 3.99 | +91.6% | +77.6% | 8.1 |
| QQQ 2018–26 | 39 | 1.87 | +44.0% | +22.6% | 12.2 |
| IWM 2010–18 | 38 | 3.10 | +83.7% | +71.3% | 10.7 |
| IWM 2018–26 | 41 | 1.44 | +23.7% | +10.1% | 14.5 |
Go down our vetting bar with that table:
- Profitable after deleting the best trade? Yes, in all six cells.
- Two independent multi-year windows? Yes, both positive everywhere.
- More than one instrument? Three.
- Net of realistic fees? Yes — and still passing at 3.3 times the fee assumption.
- Drawdown within reason? Single digits to 14.5, against a bar of 20.
It also survived a one-day signal lag and next-open fills, tested separately, because the VIX prints its official close fifteen minutes after the equity close and we did not want that fifteen minutes to be the edge. Roughly 240 trades across the passing cells — a larger sample than most of what is already in this library.
This was one commit from shipping as entry #7. The write-up was drafted.
The question that killed it
Then someone asked the obvious thing: is this the signal, or is it a good exit applied to a market that mostly goes up?
The test is embarrassingly cheap. Keep everything — the window, the instrument, the trailing exit, the fees. Throw away the entry signal. Enter on random days instead, the same number of them the real strategy took. Do it 400 times. See where the real result lands in that distribution.
| Cell | Real return | Null median | Null 95th pct | Where the real result lands |
|---|---|---|---|---|
| SPY 2010–18 | +68.3% | +9.5% | +33.0% | 100th percentile |
| SPY 2018–26 | +26.9% | +14.4% | +50.4% | 75th percentile |
| QQQ 2010–18 | +91.6% | +18.2% | +46.9% | 100th percentile |
| QQQ 2018–26 | +44.0% | +24.0% | +65.9% | 80th percentile |
| IWM 2010–18 | +83.7% | +11.7% | +47.1% | 100th percentile |
| IWM 2018–26 | +23.7% | +9.5% | +48.5% | 72nd percentile |
In the first window the signal is unambiguous: 400 random draws never once beat it, on any of the three instruments. In the second window it is inside the noise on all three. SPY's +26.9% sits in a distribution whose own 95th percentile is +50.4% — random timing with the same exit beat it a quarter of the time.
A strategy that cannot beat a coin flip in its most recent window is not a strategy, whatever five other checks say. It is not in the library.
Why the other five checks missed it
Look at what they actually ask. Strip-best-trade asks whether the result depends on one trade. Two windows asks whether it survives a regime change. Cross-asset asks whether it is one symbol's quirk. Fees ask whether it survives friction. Drawdown asks whether you could hold it.
Every one of those is a question about the returns. Not one of them is a question about the signal. They are all perfectly capable of being satisfied by "long equities, sometimes, with a sensible trailing exit" — which in a rising market is a profitable thing to do, robustly, across windows and instruments and fee levels, for reasons that have nothing to do with the rule you think you are trading.
That is the hole. The null test is the patch.
What it did to our own shelf
A new standard that only applies to future candidates is not a standard. So every published entry went through it, on the window its own page displays, using the exact committed configuration:
| Entry | Window | Real return | Null median | Percentile | |
|---|---|---|---|---|---|
| RSI(2) dip-buy | 2018–26 | +36.4% | +10.3% | 98.3rd | ✅ |
| BTC funding-squeeze 1h | 2022–26 | +12.9% | −4.8% | 95.5th | ✅ |
| BTC funding-squeeze 4h | 2024–26 | +13.6% | −4.0% | 96.3rd | ✅ |
| BTC funding-squeeze 4h | 2022–24 | +15.3% | −0.2% | 90th | ⚠️ |
| Double-7s | 2010–18 | +56.4% | +26.8% | 98.5th | ✅ |
| Double-7s | 2018–26 | +45.7% | +32.5% | 76.3rd | ❌ |
| SMA-200 timing | 2010–18 | +64.4% | +46.2% | 89th | ⚠️ |
| SMA-200 timing | 2018–26 | +76.5% | +59.9% | 77.3rd | ❌ |
The crypto rows are the most interesting in this library. Random entries on 4h and 1h Bitcoin with the same trailing exit have a negative median return — you lose money entering at random and trailing out. The funding signal converts that into +13% to +15%. There is no ambiguity about whether the signal is doing the work.
Two entries do not clear it, and they are still on the shelf with the result attached:
- Double-7s is 98.5th percentile in its first window and 76.3rd in the second. The mechanism was clearly real and the modern window cannot be told apart from random timing.
- SMA-200 trend timing does not clear it in either window — and this one deserves a fair reading rather than a headline. Its published claim was never excess return; its own write-up says holding beats it on raw return and that the point is the drawdown column, a third of holding's. A return-based null is the wrong instrument for that claim. What the test adds is that its return is what the exposure bought, not what the signal earned.
Nothing was delisted. The entries keep their evidence and now carry this too, which is the only version of "we publish our failures" that means anything once a failure lands on our own shelf.
The honest limitations of the test itself
The null usually realises fewer trades and less exposure than the real strategy, because a random signal that fires while a position is already open gets ignored. Less exposure means a lower null return, which pushes every percentile in these tables upward, in the strategy's favour. SMA-200's null ran at 58.5% exposure against the real strategy's 80.6%, which is the extreme case. So these are floors, and an exposure-matched null would be harsher, not kinder.
A uniform-random null also ignores the fact that real signals cluster in specific conditions. A block bootstrap would be a stricter test. We used the simpler one, and it was already enough to stop a strategy from shipping.
What to do with this
If someone shows you a backtest, the two cheapest questions in finance are:
- What happens if you delete the best trade?
- What would random entries with the same exit have made?
The second one takes about twenty lines of code and it is almost never run, because it has an uncomfortable habit of answering. Ours runs against the committed strategy configurations themselves, so the numbers above are reproducible rather than asserted — and they are what it said about our own shelf.
A backtest that has not answered both questions has not told you whether the rule works. It has told you that the market went up.