Over five million backtests, more than three thousand assets, seven timeframes, eighty-plus validated strategies across thirteen edge families.
That scale is the product and the problem.
Test enough combinations and some will look extraordinary by chance. No edge, just probability working at high volume. The bad results are easy to dismiss. The good ones are the failure mode: the strategies that look real because the sample was large enough for luck to impersonate a pattern. This is the multiple-testing trap. The more combinations you test, the higher the bar any apparent winner has to clear before you take it seriously.
Most backtesting advice ignores this. It gives you a fixed hurdle, profit factor above some number, drawdown below some number, and tells you that if the backtest clears it, you have an edge. The bar was set for one test, not five million.
Luck is not the villain. Probability is behaving exactly as it should. The search is what manufactures the false positives. Look in enough places and your own process guarantees something will look like an edge.
What a random strategy scores
A randomly-entered strategy on a daily equity market scores higher than most traders expect. Markets trend. A long-only strategy with no skill can post a profit factor well above 1.5 on daily equities, clearing the same bar a validated strategy has to clear. The underlying drifts upward over long periods. That is beta dressed as signal.
The noise floor gate removes strategies that cannot beat a coin flip on their own market. But it only addresses one form of inflation. The deeper problem is what happens when you test many configurations against the same market and the same time period.
The multiple-testing correction
Test one hundred configurations looking for profit factors above 1.5. Even with no edge, roughly five will clear that bar on a long enough history.
The data has features any pattern can overfit.
Scale to five million backtests and the apparent winners multiply accordingly.
You could publish a new backtest every week, every result looking like a real edge, and none would survive out of sample.
The correction is a permutation test. Instead of measuring a strategy against its own history, measure it against the distribution of results from thousands of random configurations on the same data, same market, same timeframe, same exit rules. A real edge scores in the top fraction of that distribution. When a coin flip scores just as high, the finding is not evidence of anything.
I call this drift-adjusted significance. Daily equity markets trend upward over long periods, so a naive test rewards any strategy that stayed long. The drift adjustment strips that free upward push out of the comparison: a long-biased strategy has to clear the market's own motion, not ride it. Skip that step and a lucky long looks like skill.
This is not a retail-trader problem. Every empirical science eventually runs into it. Medicine, psychology, genomics, and machine learning all had to invent ways to tell a real discovery from the residue of repeated searching. This is mine.
What survives
Over two thousand calibrated candidate edges cleared my baseline filter. A small set reached the permutation bar. The ones that did were not the ones with the best in-sample numbers. A strategy scoring implausibly high on historical data gets flagged, not promoted. Extreme results indicate overfitting. The candidates under forward observation are the ones that cleared the bar without topping the charts.
The surprise runs the other way too. Some of the best-looking backtests were the first to die, and a few unremarkable ones survived. Impressive in-sample numbers were a warning, not a promise.
The graveyard behind this newsletter includes strategies with good backtests that failed the permutation test. They looked real by standard metrics. Tested against the right null, they were exactly what the probability math predicted: luck with good timing.
If you have ever opened a folder of old strategies and wondered why none of them still work, you have already met this trap.
Most published strategies are not wrong so much as unknowable. You never saw how many failures came before the one they printed.
This gets worse, not better. A machine can propose ten thousand strategies before lunch. When ideas cost nothing, the only scarce thing left is the rejection.
The filter is not the backtest result. It is whether the result is distinguishable from luck at the scale I tested it. The more chances you give chance to fool you, the more evidence it owes you.
Most candidates never pay.
— Collin
