Leaderboard¶
For research and risk analytics. Not investment advice.
Every model is scored by the same leakage-guarded walk-forward engine at the same origins and read against the random walk, with a Diebold-Mariano test. Results are published whichever way they fall; see the methodology for the leakage controls, the baselines and the limits of what this shows.
This page is generated by python -m benchmarks.render from the versioned JSON in benchmarks/results/. No number on it is typed by hand.
Benchmark: simulated-csv¶
user-csv-daily on simulated fat-tailed GARCH series (CPU, minutes).
| Tycheon version | 0.4.0 |
| Run | 2026-10-13T17:06:57+00:00, commit 4fdaa97192 (uncommitted changes) |
| Dataset | user-csv-daily (real, user-licensed), 4 series |
| Config / manifest hash | 4d960a64ec2d / 4cfa619fbe91 |
| Environment | Python 3.12.15, cpu, Windows AMD64, torch 2.14.1+cpu |
| Runtime | 275.5 s |
Warning
survivorship bias: the universe of 4 symbols is static (chosen with hindsight), so delisted and failed names are absent and every model's results are flattered. Use a point-in-time universe before drawing conclusions.
Warning
this dataset is not redistributable: publish these results only if your data licence allows it
Horizon 5 bars¶
48 forecast origins per series, pooled across 4 series, non-overlapping. MASE is the model's MAE over the random walk's MAE on the same origins (below 1 beats it). CRPS skill is 1 minus the ratio to the random walk's CRPS. DM p is the one-sided Diebold-Mariano p-value that the model has lower loss than the random walk (HLN-corrected). * CRPS approximated from the quantile grid (the model has no sample paths).
| Model | vs random walk | MASE | RMSE (%) | CRPS (%) | CRPS skill | Direction | IC | DM p, sq. error | DM p, CRPS |
|---|---|---|---|---|---|---|---|---|---|
| random-walk (baseline) | baseline | 1.019 | 2.333 | 1.233 | - | 47.9% | -0.039 | - | - |
| drift | = RW | 1.074 | 2.379 | 1.275 | -3.4% | 45.3% | 0.023 | 0.871 | 0.990 |
| seasonal-naive | = RW | 1.033 | 2.350 | 1.251 | -1.4% | 45.8% | -0.084 | 0.714 | 0.919 |
| arima | = RW | 1.063 | 2.380 | 1.280 | -3.8% | 42.2% | -0.067 | 0.859 | 0.991 |
| garch | = RW | 1.033 | 2.349 | 1.211 | 1.8% | 47.4% | -0.086 | 0.795 | 0.048 |
| tycheon-ensemble | = RW | 1.026 | 2.340 | 1.217 | 1.3% | 50.0% | -0.015 | 0.630 | 0.080 |
| tycheon-calibrated | worse than RW | 1.110 | 2.495 | 1.310 | -6.2% | 47.4% | -0.102 | 0.999 | 0.997 |
Where the baselines win
- Of 2 non-baseline models, 0 beat the random walk (both Diebold-Mariano tests, one-sided p < 0.05), 1 were statistically indistinguishable from it and 1 were worse.
- Best MASE: random-walk, a baseline.
- Best RMSE: random-walk, a baseline.
- Best CRPS: garch, a baseline.
Interval coverage (achieved share of outcomes inside the central interval; closer to nominal is better, narrower is better at equal coverage)
| Model | 50% interval | 80% interval | 90% interval | 90% width (log %) |
|---|---|---|---|---|
| random-walk | 56.8% | 82.8% | 91.1% | 7.60 |
| drift | 54.2% | 83.3% | 90.6% | 7.72 |
| seasonal-naive | 58.3% | 86.5% | 90.6% | 8.04 |
| arima | 56.8% | 84.4% | 92.7% | 8.09 |
| garch | 54.2% | 80.7% | 90.6% | 7.26 |
| tycheon-ensemble | 52.6% | 81.2% | 90.1% | 7.39 |
| tycheon-calibrated | 48.4% | 77.1% | 88.0% | 7.73 |
Per series (MASE and verdict)
| Model | SIM-A | SIM-B | SIM-MR | SIM-C |
|---|---|---|---|---|
| drift | 1.171 (= RW) | 1.028 (= RW) | 1.090 (= RW) | 1.036 (= RW) |
| seasonal-naive | 1.073 (= RW) | 1.022 (= RW) | 1.026 (= RW) | 1.017 (= RW) |
| arima | 1.078 (= RW) | 1.038 (= RW) | 1.037 (= RW) | 1.084 (= RW) |
| garch | 1.033 (= RW) | 1.019 (= RW) | 1.034 (= RW) | 1.044 (= RW) |
| tycheon-ensemble | 1.057 (= RW) | 0.995 (= RW) | 1.072 (= RW) | 1.010 (= RW) |
| tycheon-calibrated | 1.147 (= RW) | 1.129 (= RW) | 1.050 (= RW) | 1.094 (worse than RW) |
Cost-aware diagnostic (naive one-horizon sign strategy with spread and slippage; a research diagnostic, not a recommendation)
| Model | gross return / period | net of costs | turnover / period | hit rate |
|---|---|---|---|---|
| random-walk | -0.183% | -0.202% | 0.90 | 47.9% |
| drift | -0.106% | -0.114% | 0.42 | 45.3% |
| seasonal-naive | -0.013% | -0.035% | 1.10 | 45.8% |
| arima | -0.145% | -0.159% | 0.66 | 42.2% |
| garch | -0.069% | -0.085% | 0.76 | 47.4% |
| tycheon-ensemble | 0.025% | 0.012% | 0.61 | 50.0% |
| tycheon-calibrated | -0.068% | -0.076% | 0.41 | 47.4% |
Reproduce: python -m benchmarks.run --config benchmarks/configs/simulated-csv.yaml --execute. The raw document is benchmarks/results/simulated-csv/0.4.0.json.