A win rate nobody can check is a story, not a track record. If a model claims it hits 56% on NFL spreads, the right response is "show me exactly which games, which odds, and let me run it again and get the same number." Most touts cannot produce any of that. This is the checklist we follow when we build a model that bets a game for you in Tinker and write up how it actually did, and it is what you should demand from anyone selling picks. It is written for a bettor who has a system — or follows one — and wants to know whether the edge is real or just a hot streak.
The 12-item checklist
Print this. Tape it next to your monitor. A backtest is not done until every line is true.
- Lock the games and odds. Save the exact dataset of games, scores, and prices you tested on. Anyone re-running gets the same input — or sees instantly that it changed.
- Same settings every time. Run it with identical settings so the win rate does not drift between runs. If two runs disagree, you are looking at luck, not a result.
- Freeze your dice roll. Models that train have a bit of built-in randomness. Pin it down once and write it in the log so the same model comes out every time.
- Pick test seasons by date. Decide which seasons are training and which are the test up front, by calendar — e.g. "learn on 2015 through August 2023, test on the 2024 season." Never let the model train on games it is being graded on.
- No peeking at the future. Every input for a game must come from before that game's kickoff, and you should have a check that fails the moment it does not.
- Bet real prices. Settle every bet at the actual line you would have gotten (closing or opening — just be consistent), after the juice. -110 wins +0.909 units, a loss costs 1 unit.
- Charge yourself for the line you miss. You rarely get the exact number on the screen. Dock a small tax (about a penny on moneylines, a tenth of a point on sides) so the backtest matches reality, not the best-case line.
- Test across at least 3 seasons. Use three non-overlapping test windows. A single "2024 season" test is one season of luck dressed up as a system.
- Check it knows what it knows. Group your picks by how confident the model was and see if the confidence held up. If the games it called 60% only won 51%, it is overconfident — and betting bigger on "locks" will wreck you.
- Report the full scorecard. Win rate, ROI, closing line value, and the honesty scores (Brier, log loss) — not just whichever number looks best.
- Save the whole model. The model and everything it needs to make a pick travel together. Half a model makes no picks.
- One-button rebuild. A single command that pulls the data, trains, grades the bets, and spits out the numbers. If you cannot reproduce your own win rate with one click, neither can anyone else.
Why each item earns its slot
Locking the data catches silent rewrites
nflfastR refreshes weekly. ESPN's box-score feed has been known to quietly retro-correct snap counts after the fact. If the file has the same name but different numbers than last month, your "re-run" is testing a different set of games. Saving the exact dataset is the receipt that you tested on the same thing.
Split your test seasons by the calendar
If you just shuffle all the games together, the model ends up studying for the test — a Week 17 result helps it "predict" Week 5. Always split by date, and set aside a test season the model never touches while you are tweaking it. If you tuned your bet threshold on the same season you brag about, that season is not really a test.
Three seasons kill the "lucky year" problem
Scoring environments swing season to season — a model that crushes a high-scoring year can fold in a low-scoring one, and the two net out to nothing. But if you only tested on the friendly year you would call it a winner. Testing across three non-overlapping seasons is the minimum to expose this. Five is better.
Confidence honesty beats raw win rate
Two models can both hit 56% straight-up. One calls its winners 65% and its losers 45% — its confidence is honest, the edge is real. The other calls its winners 85% and its losers 15% — it is wildly overconfident, and betting bigger on its "locks" will blow up your bankroll. The honesty scores (Brier, log loss) catch the second one; raw win rate never will. That is why the honesty-score explainer and the confidence-check walkthrough are required reading before you size a real bet.
What a verifiable model package looks like
If you build or download a model, this is everything it should ship with so anyone can check the win rate. Copy it.
nfl-spread-v3/
README.md # data SHA, seed, split dates, headline metrics
requirements.txt # pinned
uv.lock
data/
games.parquet # frozen snapshot, SHA listed in README
games.parquet.sha256
src/
features.py # pure functions, no IO
train.py
backtest.py
settle.py # -110 payouts, slippage, walk-forward folds
artifacts/
weights.bin # tf.js binary or .pt
scaler.json # mean/std per feature, in column order
feature-order.json
config.json # seed, train/val/test dates, model hparams
reports/
metrics.json # brier, logloss, hit-rate, roi, clv per fold
calibration.png
fold-pnl.png
Makefile # make backtest = single-command rebuild
That package is the receipt. If someone selling picks cannot show you the equivalent, you cannot check their edge — and an edge you cannot check is just a story.
The line you miss: small number, big effect
Docking yourself a tenth of a point on NFL sides looks tiny. It is not. Each half point near the spread is worth roughly 3% — so missing the number by a tenth on average costs about 0.6% ROI per bet. Over 500 plays a season that is a 3-percentage-point haircut, enough to turn a 4% winner into a 1% one. Charging yourself for the line you miss is not pessimism; it is the difference between paper and reality. Tracking closing line value is the real-world test of whether your assumption was honest.
Testing across seasons, in practice
Three test seasons for 2022, 2023, and 2024 — learn on everything before each season, grade on that season:
FOLDS = [
{ 'train_end': '2022-08-31', 'test_start': '2022-09-01', 'test_end': '2023-02-15' },
{ 'train_end': '2023-08-31', 'test_start': '2023-09-01', 'test_end': '2024-02-15' },
{ 'train_end': '2024-08-31', 'test_start': '2024-09-01', 'test_end': '2025-02-15' },
]
for fold in FOLDS:
train = df[df.kickoff < fold['train_end']]
test = df[(df.kickoff >= fold['test_start']) & (df.kickoff <= fold['test_end'])]
model = fit(train, seed=SEED)
metrics_for_fold = evaluate(model, test, prices='closing', slippage_pts=0.1)
Average the seasons for your headline number, and show each season too — if the model is +5% one year and -1% the next, that "2% average" is misleading and you owe people the swing.
Common ways a "winning" backtest is fake
- "It worked when I ran it." Usually the settings or the dataset quietly changed between runs. Lock both and it holds still.
- Bet size tuned on the test season. If you hunted for the "best bet threshold" using your test season's ROI, that ROI is contaminated. Tune on a separate season, lock it, then grade on the test season you never touched.
- Juice quietly removed. Some tools default to no-juice payouts and flatter the model. Always show the actual per-bet payout you used.
- Mismatched lines. An "opening line" from one site and a "closing line" from another can disagree by half a point or more. Pick one source, write it down.
- Cherry-picked best run. If you tried 50 versions and only published the luckiest, that is cheating. Lock one version up front, or report the average and the spread across all of them.
What good looks like
An honest write-up is short and boring. Example:
nfl-spread-v3.2 — learned on 2015 through August 2023, tested on the 2024 season (held out the whole time), across three separate seasons. Hit rate 54.1%, ROI after the juice +2.6%, average closing line value +0.31 pts; honesty scores Brier 0.232 (beating the Vegas-implied 0.241) and log loss 0.661. One command reproduces every number in 4 minutes on a 2020 laptop.
That paragraph beats any "60% lock of the night" pitch — because every claim in it is checkable. The point of the checklist is to make publishing that paragraph easy, and ducking it embarrassing.
Run it on your model today
If you build a model in the browser, Tinker already locks the settings and grades your bets the moment you click Backtest, so the win rate comes back the same every time. Use it, test across multiple seasons, and the next time someone challenges your record you have a one-line answer: run it again, watch the same number come back. The bettor desk then tracks your live closing line value against what the backtest promised — that is the loop where a story turns into a track record.
Named modeling examples
A model page is more useful when the feature examples are concrete. Josh Allen rushing attempts, Ja'Marr Chase target share, Nikola Jokic assist rate, Tarik Skubal strikeout projection, Igor Shesterkin starter confirmation, and Islam Makhachev control time are all different prediction problems. A single “player form” feature cannot explain them all, so the model needs sport-specific inputs and review notes.
- NFL: separate route participation, pressure rate, and red-zone role from box-score volume.
- NBA: separate usage, minute projection, pace, and back-to-back fatigue.
- MLB: separate starter skill, handedness, park, weather, and lineup confirmation.
- NHL and UFC: late confirmations and fight-week news can matter more than a season average.
Model inputs worth naming
Use names as evidence, not decoration. The useful SEO win is that Josh Allen, Ja'Marr Chase, Bijan Robinson and Puka Nacua and Rams, Chiefs, Bills, Eagles and Lions appear inside decisions, thresholds, and internal links instead of being dumped into a keyword list.
- NFL model: route participation for Ja'Marr Chase, rushing attempts for Josh Allen, pressure rate allowed by the Bengals, and red-zone carry share for Jonathan Taylor should be separate features.
- NBA model: usage, projected minutes, rest, and pace should move Nikola Jokic or Shai Gilgeous-Alexander props differently than a one-number power rating.
- MLB model: Tarik Skubal strikeout projection, Coors Field park factor, lineup confirmation, and bullpen rest need their own columns.
- Review loop: grade entry price, closing price, bet result, and model error separately so lucky results do not hide bad forecasts.
Build or audit the workflow in Tinker and review it with CLV.
Research note board
Use this model-audit board to keep features, validation, and bet sizing from collapsing into one confidence score.
| Model layer | What to inspect | Example input | Downgrade when |
|---|---|---|---|
| Feature | Whether the variable maps to the sport and market | Josh Allen role data or closing line value price movement | The feature is a proxy for something you can measure directly |
| Validation | Out-of-sample error, CLV, calibration, missing data | Rams market movement after injury news | Wins come without beating the close or improving calibration |
| Sizing | Bankroll, confidence interval, correlation, market limit | CLV exposure compared with related tickets | Multiple bets repeat the same thesis at full stake |
Bet responsibly — set limits, never chase losses.
Model calibration: predicted vs observed
Predicted win probability bucket vs the empirical win rate inside that bucket on the test set. Points on the y=x reference line are perfectly calibrated; points below mean the model is overconfident in that bucket.
EV per $100 across win rate × odds grid
Expected value of a $100 stake at each combination of true win rate and market odds. Anywhere the cell is positive you have a long-run profitable bet; the magnitude shows how aggressive Kelly will size it.
