Skip to content

0006. Walk-forward evaluation, leakage guards and the leaderboard

  • Status: Accepted
  • Date: 2026-10-05
  • Phase: T3

Context

AGENTS.md requires every evaluation to report the random-walk baseline and a Diebold-Mariano test, to publish honestly when baselines win, and to treat leakage tests as mandatory. T2 already had an honest replay for calibration (collect_scores). T3 needs the same honesty for end-to-end evaluation of models that have to be fitted (the regime router and the calibrator), at the scale of a leaderboard.

Decision

  1. One record type. Walk-forward output is a ScoreSet, the same record calibration uses (log-return quantiles, realised outcomes, optional sample paths, outcome publication times). Building it is shared code (scoreset_from_forecasts), so calibration and evaluation cannot disagree about what a forecast and an outcome are.
  2. Factories, not fitted objects. A walk-forward run takes a factory that builds a forecaster from training history and the time that history was known. Plain forecasters use a constant factory; the ensemble and calibrated forecasters refit router and calibrator inside the factory. The engine, not the model, decides what history it may see.
  3. Guards at three points. Every predict is wrapped (history published by the origin, as_of equals the origin); every training history is checked to be published by its fit time; and the training history must end at least embargo bars before the first origin it serves. The convention is available_at <= origin: a bar known exactly at the origin is usable, which a strict < would wrongly discard.
  4. A poisoning canary. Tests replace all data after an origin with garbage and require earlier forecasts to be bit-identical, for the baselines, the ensemble and both calibrated variants, plus a test that the canary fails on a cheating forecaster.
  5. Overlap handled, then avoided. The Diebold-Mariano test uses rectangular truncation at ceil(h / stride) - 1 lags (with a Bartlett fallback if the estimate goes negative) and the HLN correction; a test shows ignoring overlap inflates the rejection rate from 5% to over 15%. The benchmark additionally rejects overlapping origins, so the pooled test is valid.
  6. A verdict that is hard to win. "Beats the random walk" needs both one-sided tests (squared error and CRPS) below 0.05; "worse" needs both above 0.95. Otherwise "indistinguishable".
  7. CRPS honesty. Exact fair CRPS from sample paths where they exist; for quantile-only models a trapezoid integral of pinball loss, flagged, and never mixed with the exact value in one comparison.
  8. Harness outside the package. The engine, metrics, guards and cost model ship in tycheon.backtest. The runner, dataset manifest, model configs and leaderboard renderer live in benchmarks/ (not installed): they are an experiment harness, not library API.
  9. Results are versioned data; the page is generated. Each run writes benchmarks/results/<name>/<version>.json with provenance (version, commit, config and manifest hashes, environment). The leaderboard is rendered from those files; no number is typed by hand, and skipped or failed models are listed with their reasons.
  10. Data. Only synthetic series are generated by default. Real data is the user's own licensed files, never redistributed; results from it carry a not-redistributable warning, and a static real universe carries a survivorship warning.

Consequences

  • Fitting-heavy models cost n_origins replays per refit per series; the small config is sized for CPU, the full one for a GPU box.
  • The guard sees the history passed to predict. A forecaster that reads data from elsewhere bypasses it; the canary is the defence, and it is only as good as the models it is run on.
  • Pooled DM assumes independent series. Real, correlated universes need a cross-sectional correction that is not implemented in v0.1.

Alternatives considered

Expanding-window cross-validation without an embargo (leaks through overlapping labels); blocked or combinatorial purged CV (valuable for model selection, but the leaderboard needs a single causal timeline a reader can audit); reporting only MAE (hides distributions and is trivially gamed by a constant forecast).