A Reproducible Backtest Checklist Bettors Actually Follow

A win rate nobody can check is a story, not a track record. If a model claims it hits 56% on NFL spreads, the right response is "show me exactly which games, which odds, and let me run it again and get the same number." Most touts cannot produce any of that. This is the checklist we follow when we build a model that bets a game for you in Studio and write up how it actually did, and it is what you should demand from anyone selling picks. It is written for a bettor who has a system — or follows one — and wants to know whether the edge is real or just a hot streak.

The 12-item checklist

Print this. Tape it next to your monitor. A backtest is not done until every line is true.

1. Lock the games and odds. Save the exact dataset of games, scores, and prices you tested on. Anyone re-running gets the same input — or sees instantly that it changed.
2. Same settings every time. Run it with identical settings so the win rate does not drift between runs. If two runs disagree, you are looking at luck, not a result.
3. Freeze your dice roll. Models that train have a bit of built-in randomness. Pin it down once and write it in the log so the same model comes out every time.
4. Pick test seasons by date. Decide which seasons are training and which are the test up front, by calendar — e.g. "learn on 2015 through August 2023, test on the 2024 season." Never let the model train on games it is being graded on.
5. No peeking at the future. Every input for a game must come from before that game's kickoff, and you should have a check that fails the moment it does not.
6. Bet real prices. Settle every bet at the actual line you would have gotten (closing or opening — just be consistent), after the juice. -110 wins +0.909 units, a loss costs 1 unit.
7. Charge yourself for the line you miss. You rarely get the exact number on the screen. Dock a small tax (about a penny on moneylines, a tenth of a point on sides) so the backtest matches reality, not the best-case line.
8. Test across at least 3 seasons. Use three non-overlapping test windows. A single "2024 season" test is one season of luck dressed up as a system.
9. Check it knows what it knows. Group your picks by how confident the model was and see if the confidence held up. If the games it called 60% only won 51%, it is overconfident — and betting bigger on "locks" will wreck you.
10. Report the full scorecard. Win rate, ROI, closing line value, and the honesty scores (Brier, log loss) — not just whichever number looks best.
11. Save the whole model. The model and everything it needs to make a pick travel together. Half a model makes no picks.
12. One-button rebuild. A single command that pulls the data, trains, grades the bets, and spits out the numbers. If you cannot reproduce your own win rate with one click, neither can anyone else.

Why each item earns its slot

Locking the data catches silent rewrites

nflfastR refreshes weekly. ESPN's box-score feed has been known to quietly retro-correct snap counts after the fact. If the file has the same name but different numbers than last month, your "re-run" is testing a different set of games. Saving the exact dataset is the receipt that you tested on the same thing.

Split your test seasons by the calendar

If you just shuffle all the games together, the model ends up studying for the test — a Week 17 result helps it "predict" Week 5. Always split by date, and set aside a test season the model never touches while you are tweaking it. If you tuned your bet threshold on the same season you brag about, that season is not really a test.

Three seasons kill the "lucky year" problem

Scoring environments swing season to season — a model that crushes a high-scoring year can fold in a low-scoring one, and the two net out to nothing. But if you only tested on the friendly year you would call it a winner. Testing across three non-overlapping seasons is the minimum to expose this. Five is better.

Confidence honesty beats raw win rate

Two models can both hit 56% straight-up. One calls its winners 65% and its losers 45% — its confidence is honest, the edge is real. The other calls its winners 85% and its losers 15% — it is wildly overconfident, and betting bigger on its "locks" will blow up your bankroll. The honesty scores (Brier, log loss) catch the second one; raw win rate never will. That is why the honesty-score explainer and the confidence-check walkthrough are required reading before you size a real bet.

What a verifiable model package looks like

If you build or download a model, this is everything it should ship with so anyone can check the win rate. Copy it.

nfl-spread-v3/
  README.md           # data SHA, seed, split dates, headline metrics
  requirements.txt    # pinned
  uv.lock
  data/
    games.parquet     # frozen snapshot, SHA listed in README
    games.parquet.sha256
  src/
    features.py       # pure functions, no IO
    train.py
    backtest.py
    settle.py         # -110 payouts, slippage, walk-forward folds
  artifacts/
    weights.bin       # tf.js binary or .pt
    scaler.json       # mean/std per feature, in column order
    feature-order.json
    config.json       # seed, train/val/test dates, model hparams
  reports/
    metrics.json      # brier, logloss, hit-rate, roi, clv per fold
    calibration.png
    fold-pnl.png
  Makefile            # make backtest = single-command rebuild

That package is the receipt. If someone selling picks cannot show you the equivalent, you cannot check their edge — and an edge you cannot check is just a story.

The line you miss: small number, big effect

Docking yourself a tenth of a point on NFL sides looks tiny. It is not. Each half point near the spread is worth roughly 3% — so missing the number by a tenth on average costs about 0.6% ROI per bet. Over 500 plays a season that is a 3-percentage-point haircut, enough to turn a 4% winner into a 1% one. Charging yourself for the line you miss is not pessimism; it is the difference between paper and reality. Tracking closing line value is the real-world test of whether your assumption was honest.

Testing across seasons, in practice

Three test seasons for 2022, 2023, and 2024 — learn on everything before each season, grade on that season:

FOLDS = [
    { 'train_end': '2022-08-31', 'test_start': '2022-09-01', 'test_end': '2023-02-15' },
    { 'train_end': '2023-08-31', 'test_start': '2023-09-01', 'test_end': '2024-02-15' },
    { 'train_end': '2024-08-31', 'test_start': '2024-09-01', 'test_end': '2025-02-15' },
]
for fold in FOLDS:
    train = df[df.kickoff < fold['train_end']]
    test  = df[(df.kickoff >= fold['test_start']) & (df.kickoff <= fold['test_end'])]
    model = fit(train, seed=SEED)
    metrics_for_fold = evaluate(model, test, prices='closing', slippage_pts=0.1)

Average the seasons for your headline number, and show each season too — if the model is +5% one year and -1% the next, that "2% average" is misleading and you owe people the swing.

Common ways a "winning" backtest is fake

1. "It worked when I ran it." Usually the settings or the dataset quietly changed between runs. Lock both and it holds still.
2. Bet size tuned on the test season. If you hunted for the "best bet threshold" using your test season's ROI, that ROI is contaminated. Tune on a separate season, lock it, then grade on the test season you never touched.
3. Juice quietly removed. Some tools default to no-juice payouts and flatter the model. Always show the actual per-bet payout you used.
4. Mismatched lines. An "opening line" from one site and a "closing line" from another can disagree by half a point or more. Pick one source, write it down.
5. Cherry-picked best run. If you tried 50 versions and only published the luckiest, that is cheating. Lock one version up front, or report the average and the spread across all of them.

What good looks like

An honest write-up is short and boring. Example:

nfl-spread-v3.2 — learned on 2015 through August 2023, tested on the 2024 season (held out the whole time), across three separate seasons. Hit rate 54.1%, ROI after the juice +2.6%, average closing line value +0.31 pts; honesty scores Brier 0.232 (beating the Vegas-implied 0.241) and log loss 0.661. One command reproduces every number in 4 minutes on a 2020 laptop.

That paragraph beats any "60% lock of the night" pitch — because every claim in it is checkable. The point of the checklist is to make publishing that paragraph easy, and ducking it embarrassing.

Run it on your model today

If you build a model in the browser, Studio already locks the settings and grades your bets the moment you click Backtest, so the win rate comes back the same every time. Use it, test across multiple seasons, and the next time someone challenges your record you have a one-line answer: run it again, watch the same number come back. The bettor desk then tracks your live closing line value against what the backtest promised — that is the loop where a story turns into a track record.

Model calibration from graded predictions

Calibration points render only when a verified source binds prediction probabilities to settled outcomes for the same observations.

Expected value from graded outcomes

Expected-value cells render only when a verified source binds observed win outcomes to the price paid for the same bets.