The Problem With Most Backtests

A backtest that trains a model on 2008–2022 data and then reports performance over 2008–2022 is not a backtest. It is a lookup table. The model memorized the answers and recited them back.

This practice — in-sample contamination — is the single most common fraud in quantitative finance marketing. Every system that reports “30% annual returns backtested over 15 years” without specifying walk-forward methodology is, at best, measuring the model’s ability to memorize history. At worst, it is deliberate deception.

The distinction matters because in-sample performance tells you nothing about future performance. A model that perfectly fits past data has learned the noise, not the signal. It will fail on new data.

How OVRWCH Handles This

OVRWCH uses strict walk-forward validation. The protocol:

  1. Train the model on all data available up to month T-1
  2. Generate predictions for month T only — data the model has never seen
  3. Record the out-of-sample result
  4. Advance to month T+1, retrain on all data through month T
  5. Repeat for every month in the evaluation period

The model retrains monthly. It never scores data it was trained on. The reported performance numbers come exclusively from predictions on unseen data.

The Honest Number

OVRWCH’s initial walk-forward result was a CAGR significantly below what an in-sample backtest of the same system would report. The gap between the two numbers is the size of the overfitting problem.

We publish the walk-forward number. Not the in-sample number. The honest number is lower, but it is real. An investor evaluating OVRWCH can trust that the performance they see is what the system would have actually generated in live trading, after costs.

A tuned version of the model with adjusted risk parameters is currently being evaluated. Results will be published when the walk-forward analysis is complete.

Why This Matters

When evaluating any systematic strategy — ours or anyone else’s — ask one question: “Is this performance number from data the model has seen, or data it hasn’t?”

If the answer is unclear, the number is worthless.

OVRWCH publishes walk-forward out-of-sample results exclusively. Specific model parameters and training procedures are proprietary.