Introduction

A backtest evaluates one strategy; a search evaluates thousands and reports the best. These are not the same experiment, and the statistic that is honest for the first is a lie for the second. When you fit no parameters and test a single pre-registered rule, a positive out-of-sample Sharpe with p<0.05 is weak but real evidence. When you sweep a grid, rank by out-of-sample score, and report the winner, the very act of taking the maximum manufactures a positive Sharpe out of nothing: with enough trials, the luckiest of N zero-edge strategies is expected to sit several standard deviations above zero [3]. The winner's number is large because it was selected, and a test that does not know a selection happened will call the shadow of the search an edge.

The out-of-sample split does not save you. Practitioners reach for a held-out window as a talisman against overfitting, and against look-ahead leakage it works — data the search never fit cannot leak into it. But an OOS window used to choose the winner is clean of look-ahead and dirty with selection: you picked the survivor partly by how it did there. A second, truly untouched holdout is a stronger wall, yet a single search winner carried onto it can still clear it by luck — a coin that came up heads twice is not a biased coin. The only instrument that separates “won a lottery” from “has an edge” is one that prices in how many tickets were bought [5, 2, 6].

This paper is a controlled calibration of that instrument against a case where the correct answer is known in advance. We generate one price series that is a driftless geometric random walk: by construction it contains no trend, no predictability, no edge of any kind. We then run a realistic dual-timeframe moving-average-cross search over it, select a winner by out-of-sample Sharpe, and carry that winner onto an untouched holdout — reproducing, at controlled scale, the exact shape of a tempting result (a large positive OOS return and a small positive holdout return, the calibrated analogue of an ETHUSDT +16.35\%~/~+2.62\% single-symbol study). Because the ground truth is “no edge,” every apparatus we apply has a known target: it must reject. We report what four readings say — the naive per-strategy test, the DSR against the effective trial count, the DSR against the raw trial count, and an empirical best-of-N pure-noise null — and show that the naive test is fooled while the three multiplicity-aware readings all reject, in agreement, from independent directions.

Scope and honesty constraints.

Two constraints bound the claims. First, the DGP has no planted edge (meta.price_drift~=0); this is a demonstration that a plausible OOS+holdout winner from a realistic search is statistically indistinguishable from noise, not a market prediction or a tradable strategy. Second, the study is a single seed and a single DGP: it calibrates the multiple-testing machinery on a known negative, and the specific numbers (+49.00\%, \mathrm{DSR}=0.3538, and so on) are properties of that one seeded run, not universal constants. All statistics are reimplemented from scratch on NumPy [4] against the primary sources — no black-box library — with methods in scripts/search.py and scripts/stats.py and the harness in scripts/run_all.py.

Setup: a no-edge DGP, a realistic grid, and an honest split

The data-generating process.

The single symbol's close is a geometric random walk of n=3000 bars, \log p_t = \log p_0 + \sum_{i\le t}\varepsilon_i with \varepsilon_i \sim \mathcal{N}(0,\sigma^2) i.i.d., per-bar volatility \sigma=0.012, and zero drift. Zero drift is the entire point: there is no exploitable structure, so any strategy's apparent OOS profit is pure selection. The path is drawn once from numpy.random.default_rng(16).

The search grid.

We sweep a dual-timeframe MA-cross family: a fast/slow crossover on a trading timeframe, gated by a slower “trend” MA (long only when price is above trend). The grid is \text{fast}\in\{2,3,5,8,13,21\}, \text{slow}\in\{34,55,89,144\}, \text{trend}\in\{100,150,200,250\}, over two MA families (simple and Hull), enumerated under \text{fast}<\text{slow}<\text{trend}, giving N=180 valid configurations. Execution is honest and causal: the signal is decided on the close of bar i and the position held over bar i{+}1's return, with a one-bar lag and no look-ahead.

The split.

Of the 3000 bars, a 250-bar warm-up is reserved for MA initialization; the 2750 usable bars are split into in-sample (IS, 50\%, 1375 bars), out-of-sample (OOS, 30\%, 825 bars), and an untouched holdout (the remainder, 550 bars). The winner is the configuration with the best OOS Sharpe; it is carried once onto the holdout, which no selection ever touched. Table 1 collects the configuration.

The winner and its tempting numbers

Selecting by best OOS Sharpe crowns a simple-MA configuration (fast =8, slow =55, trend =100), trial index 49 of 180. Its numbers are exactly the kind that get capital allocated (Table 2). On the out-of-sample window — data used only to score, never to fit — it returns +49.00\%, an OOS Sharpe of 0.0593 per bar (annualized 0.941). Its in-sample Sharpe is actually slightly negative (-0.0250, IS return -22.93\%), which only sharpens the illusion: the winner did not look special in-sample and “came alive” out-of-sample, the textbook signature practitioners read as robustness. Then the second wall: carried onto the untouched holdout, it holds a positive +2.66\%. A clean walk-forward, a positive holdout, an honest one-bar execution model, and no look-ahead leak — stop here and you ship it.

The return distribution offers no warning either: the OOS stream has mild negative skew (-0.077) and heavy tails (non-excess kurtosis 5.45), unremarkable for a bar-level long/flat series. Nothing in the winner's own statistics reveals that it is the maximum of 180 draws from a process with no edge. That information lives not in the winner but in the search.

Pricing the multiplicity

We now price the search four ways: the naive per-strategy test that ignores it, the DSR against the effective and raw trial counts, and a direct best-of-N noise null.

The naive test (fooled).

The single-hypothesis, one-sided t-test on the winner's Sharpe treats it as if it were the only strategy ever evaluated: \begin{equation} \label{eq:naive} t = \mathrm{SR}\,\sqrt{n_{\mathrm{oos}}}, \qquad p = 1-\Phi(t), \end{equation} with \Phi the standard-Normal CDF. Here t = 0.0593\sqrt{825} = 1.703 and p = 0.0443 < 0.05: significant. This is precisely the test that lies on the survivor of a search — it has no term for N.

The noise ceiling (False Strategy Theorem).

With N trials whose per-bar Sharpes have cross-trial mean \mu_{\mathrm{SR}} and variance v_{\mathrm{SR}}, the expected maximum under zero edge is, to the extreme-value approximation of Bailey and López de Prado [3], \begin{equation} \label{eq:fst} \mathrm{SR}_0 = \mu_{\mathrm{SR}} + \sqrt{v_{\mathrm{SR}}}\, \Big[(1-\gamma)\,\Phi^{-1}\!\big(1-\tfrac1N\big) + \gamma\,\Phi^{-1}\!\big(1-\tfrac1{N e}\big)\Big], \end{equation} with \gamma the Euler–Mascheroni constant. The search's Sharpes have \mu_{\mathrm{SR}}=0.02251 and standard deviation 0.01827, so \mathrm{SR}_0 rises with N: this is the bar the winner must clear, and it climbs the harder you look.

The Deflated Sharpe Ratio.

The DSR is the Probabilistic Sharpe Ratio [1] evaluated at \mathrm{SR}^\star=\mathrm{SR}_0 instead of at zero, correcting for sample length and the return shape (g_3,g_4) (skew and non-excess kurtosis): \begin{equation} \label{eq:dsr} \mathrm{DSR} = \Phi\!\left(\frac{(\mathrm{SR}-\mathrm{SR}_0)\sqrt{n_{\mathrm{oos}}-1}} {\sqrt{1 - g_3\,\mathrm{SR} + \tfrac{g_4-1}{4}\,\mathrm{SR}^2}}\right), \end{equation} and a discovery requires \mathrm{DSR} \ge 1-\alpha = 0.95. For reference, the undeflated PSR against zero is 0.9549 — itself just clearing the bar, which is why the naive test fires. Deflation moves the benchmark up to \mathrm{SR}_0 and the probability collapses.

Effective vs. raw trials.

A dense MA grid is highly correlated: neighboring configurations are near-duplicate strategies, so the raw count N=180 over-counts the independent looks. We estimate an effective count from the average pairwise correlation \bar\rho of the per-trial OOS return streams, N_{\mathrm{eff}} = N/[1+(N-1)\bar\rho] clipped to [1,N], which yields N_{\mathrm{eff}}=1.37. We report the DSR against both counts. Deflating against N_{\mathrm{eff}} is the charitable reading — the smallest defensible trial count, the one least likely to reject a genuine edge — and even it fails: \mathrm{DSR}_{\mathrm{eff}}=0.8415 < 0.95. Against the raw N=180 the noise ceiling is higher still and the DSR falls to 0.3538. Both are below the gate, so deflated_says_edge is false either way (Table 3).

The best-of-N pure-noise null.

Finally we check the theorem empirically, making no distributional assumption about the winner. We simulate 5000 independent searches, each drawing N strategies of 825 zero-edge Normal returns and keeping the best per-bar Sharpe, and ask where the real winner falls in that best-of-N distribution. Against the raw count N=180 the null's central 95\% band is [0.0715,\,0.1261] with mean best-of-N Sharpe 0.0944; the winner's 0.0593 sits at quantile 0.0004 — below essentially every pure-noise search's best draw, and comfortably inside (in fact beneath the center of) the band. The winner is not merely consistent with best-of-noise; it is a slightly unlucky best-of-noise. winner_inside_null is true.

Results

Table 4 states the whole verdict. The naive test says edge; the deflated / multiplicity-aware verdict says no edge; they disagree, and that disagreement is the finding. All three multiplicity-aware readings agree with each other and with the known ground truth. The winner sits inside the pure-noise best-of-N null. The verdict is no robust edge.

The three readings converge from independent directions. The effective-N DSR (0.8415) is a parametric statement about the winner alone, deflated against the most favorable trial count we can defend — and it still falls short of the gate. The raw-N DSR (0.3538) prices the full look count and drops further, showing the verdict is not an artifact of how we count trials: it holds across the entire [\,N_{\mathrm{eff}}, N\,] = [1.37, 180] range. The best-of-N null is non-parametric and assumption-free: the winner's OOS Sharpe is smaller than the best draw of essentially every one of 5000 pure-noise searches (quantile 0.0004, mean best-of-N Sharpe 0.0944 vs. the winner's 0.0593). Parametric and non-parametric, effective and raw, all four numbers point the same way, and that way is the way the DGP was built.

Discussion

Why the holdout also passed by luck.

The tempting reading of the +2.66\% holdout is that it is independent corroboration — a second wall the strategy cleared, so surely something is there. It is not. The holdout is a single new window evaluated once for the one configuration the search already crowned. A driftless random walk over 550 bars has a return distribution centered near zero but with real dispersion; a long/flat strategy on it posts a positive total return with probability not far from one-half. Getting a positive holdout after a positive OOS is a coin coming up heads twice, not evidence of bias. The selection pressure that inflated the OOS number does not touch the holdout, but nothing deflates it toward zero either: it is one noisy draw that happened to land positive and small, exactly as likely under no-edge as under edge. A holdout can falsify (a strongly negative holdout would have killed the winner outright), but a mildly positive holdout confirms nothing. Corroboration would require the edge to appear on many independent windows or instruments; here we have one window and it is uninformative.

A search is a machine for finding the luckiest of N.

The winner's in-sample Sharpe is negative and its OOS Sharpe is positive not because it “learned” anything but because, among 180 draws from a zero-mean process, some draw is largest on any given window and the search returns it. The False Strategy Theorem quantifies exactly how large that lucky draw is expected to be, and the best-of-N null realizes it directly: the winner is not even an above-average best-of-noise, it is slightly below the pure-noise mean. The entire apparent edge is the shadow of taking a maximum over N.

The right benchmark is the noise ceiling, never zero.

The naive test fails for one reason: it compares the winner to zero. Every statistic that compares it to \mathrm{SR}_0 instead — the expected best of N zero-edge trials — rejects. This is the single transferable lesson: after any search, the question is never “is the winner's Sharpe positive?” (of course it is; you took the maximum) but “is it beyond what the luckiest of N coin-flippers would post?” Only the second question has a defensible answer, and here that answer is no.

Publish the negative.

Because the DGP has no edge, the correct result is the negative one, and reporting it is the apparatus working as designed. The value of a rigorous “no” is the capital never deployed against noise — the drawdown never taken. A worse-instrumented pipeline would have shipped the +49.00\%: a clean walk-forward, a positive holdout, no leak, and a plausible story, all of it wrong.

Limitations

• No planted edge, by design. The DGP is a driftless geometric random walk with no edge, chosen so the ground truth is known. The paper demonstrates that a plausible OOS+holdout winner is indistinguishable from noise; it makes no claim that any real market is efficient, that MA crosses cannot work, or that this or any strategy should (or should not) be traded.

• Single seed and single DGP. All numbers (+49.00\%, \mathrm{DSR}=0.3538, quantile 0.0004, and the rest) are properties of one seeded path (default_rng(16)) and one grid. A different seed yields a different winner and different magnitudes; what is invariant is the structure — a search over no-edge data produces a selection-inflated winner that the deflated statistics reject. The specific figures are a calibration point, not universal constants.

• An effective-N estimate, not a law. The effective trial count uses the average-correlation (mean variance-reduction) estimator, a reviewer-legible approximation, not the last word on independent looks. We report it as the charitable bound and also give the raw-N DSR; the verdict holds across the whole [N_{\mathrm{eff}}, N] range, so it does not hinge on the estimator.

• Modeled statistics. The normal CDF/inverse and all of PSR, the False Strategy Theorem, the DSR, and the null are reimplemented from scratch in NumPy (erf/erfinv rational approximations for \Phi,\Phi^{-1}). These are faithful to the primary sources to the stated tolerances but are not a peer-reviewed library; the tests in tests/ pin their identities.

Conclusion

A parameter search is a machine for finding the luckiest of N strategies, and its winner's number is large because it was selected. We demonstrated this on a driftless random walk with no edge: a 180-config dual-timeframe MA-cross search crowned a winner with a tempting +49.00\% out-of-sample return (Sharpe 0.0593, annualized 0.941) and a still-positive +2.66\% holdout, which the naive per-strategy test duly flagged significant (t=1.703, p=0.0443). Pricing the search reversed the verdict from every multiplicity-aware direction at once: the Deflated Sharpe Ratio failed the 0.95 gate against both the charitable effective trial count (\mathrm{DSR}=0.8415, N_{\mathrm{eff}}=1.37) and the raw count (\mathrm{DSR}=0.3538, N=180), and the winner landed at the raw-N best-of-noise null quantile 0.0004, below the pure-noise mean (0.0944) and squarely inside the band. Naive and deflated disagree, and the disagreement is the entire lesson: a clean out-of-sample number clears look-ahead, but only deflation clears selection, and the holdout that also “passed” passed by the same luck. Because the DGP has no edge, the correct verdict is the negative one — no robust edge — and producing it is the apparatus working exactly as designed. After a search, benchmark the winner against the noise ceiling of the luckiest of N, never against zero; if it does not clear that bar, the honest result is no.

Reproducibility.

All numbers derive from one seeded run. scripts/run_all.py regenerates results/results.json from seed 16 (Python 3.12.3, NumPy 2.5.1) deterministically; the search, MA family, and IS/OOS/holdout split are in scripts/search.py, and the from-scratch statistics (Sharpe, PSR, expected-max Sharpe, DSR, effective-N, best-of-N null) in scripts/stats.py. The price series is a driftless geometric random walk from numpy.random.default_rng(16) (n=3000 bars, per-bar volatility 0.012, zero drift); the best-of-N null uses default_rng(999) over 5000 simulations. tests/ contains deterministic invariant tests for the statistics, the search, and the naive-vs-deflated disagreement.

References