Every model ships with a reproducible backtest. Regenerate all of them with:
uv run dugout-models backtest all| Model | Report | Protocol | Headline result |
|---|---|---|---|
| Match predictor | predictor.md · validation | Walk-forward, refit every ~4 gameweeks; design chosen on 2019-20..2023-24, tested once on 2024-25 and 2025-26 | Beats Elo on RPS, log loss and Brier in both test seasons; bookmaker closing odds remain better |
| Player quality (VAEP, xT) | player_ratings.md | VAEP probability models on 380 held-out games (StatsBomb PL 2015/16 + Wyscout PL 2017/18) | AUC 0.75 (scores), 0.78 (concedes); both beat base-rate Brier |
| Roles & similarity | roles.md | Own-previous-season retrieval; bootstrap role stability | ~3x random retrieval; stability ARI ~0.7 |
| Manager fingerprint & style similarity | fingerprint.md | Style similarity vs minutes share for players who changed club | Descriptive only: style similarity does not predict minutes (ρ ≈ 0.07) |
| Tactical fit: playing-time outlook | opportunity.md | Summer transfers, train 2016-22, test 2023-25 | R² 0.26 vs 0.03 baseline; Spearman 0.55; top vs bottom quartile 47% vs 7% of minutes |
| FootballBERT embeddings | embeddings.md | Leakage-guarded fit test on 4,966 transfers | Not adopted: apparent signal was leakage |
| Valuation | valuation.md | Train < July 2024, test after | Beats "fee = market value" on MAE and median error |
| Transfer realism | realism.md | Train summer windows 2015-23, test 2024-25 | Mobility (sold for a fee) AUC 0.76, Brier 0.067 (0.750 / 0.070 before the club-on-the-day fix, 2026-10-17); destination AUC 0.73 (the earlier 0.75 counted squads by today's club: hindsight) |
| Injury risk | injury.md | Train 2022-24, test 2024-25 | AUC 0.64; top-decile lift 2.3x; shown as risk tiers (calibration limited by season-to-season reporting shifts) |
| Player potential | potential.md | Time split: train snapshots to 2021 (2 y) / 2019 (4 y), test 2023-24 / 2021-22 | Beats age-curve and persistence baselines at both horizons; median error 42% (2 y), 54% (4 y); wide bands shown |
| Fact-Check NLI | factcheck_nli.md | 48 hand-labelled football premise/claim pairs | DeBERTa-v3 base and large: no false verifications; base 3x faster (used) |
| Agents (end to end) | templates · LLM | 56 hand-labelled questions | See reports: 0 untraceable numbers is the hard target |
| Lineup optimiser | lineup.md | Trust weight chosen on 2023-24, test 2024-25 | Predicted XI 78% overlap vs 77% heuristic |
| Rumour track record | rumours.md | Guardian Rumour Mill 2004-25 (34k player-club links) vs what happened; test 2025-26; feature choice on 2023-24 and 2024-25 | Within 6 months: Brier 0.135 vs 0.151 (base rate since 2020-21), log loss 0.432 vs 0.479, AUC 0.72, roughly calibrated. Outlets' shrunk records barely differ and do not help (negative result); repeat reports, clubs named and the window do. Source labels (record vs similar reports, 80% interval): 3 reliable, 1 speculative, the rest about average; too few later reports to confirm them |
| Lineup shape (formation) | lineup_shape.md | Formation from the manager's last 6 XIs, scored on 2025-27 | Back line right 75% vs 61% for the previous rule (= base rate) |
| Squad needs: played positions | needs.md | No ground truth: top-two needs per PL club with listed vs played positions; needs as of 1 June 2026 checked against summer 2026 signings (Wikipedia lists) | Top two change at 11 of 20 clubs today. Per need, a signing came in that group 72% (played) vs 75% (listed), random 74%: no measurable gain (negative result); shipped for describing squads where players start |
Negative and weak results are published on purpose: the site shows each model's evidence next to its output.