Week 1 makes every offseason model feel smarter than it is. New rosters arrive, old assumptions get fresh uniforms, and the betting board finally offers somewhere to point the output. That is precisely when the model needs a backtest, not a pep talk.
A backtest is a pre-flight check. It asks whether the target, feature timing, market mapping, and grading behaved coherently on history where the answers are known. It cannot promise the new season will look like the old one. It can stop broken plumbing from reaching a live wager.
Name the live decision first
Write the market question in plain language. Are you predicting scoring margin for a spread, combined scoring for a total, or a player outcome for a prop? The historical test must grade that same target.
This sounds obvious until a convenient metric enters the room. A team-strength score is evaluated against wins, discussed as a spread model, then used to justify a player prop. The model did not become versatile. The question changed without permission.
If the live bet is a spread, preserve the historical spread decision. If it is a total, test the scoring target and market definition you intend to trade. A backtest should make target drift embarrassing.
Run history forward, never backward
At each historical decision point, the model may use only information that existed before that event. Later games, final-season summaries, revised participation, and closing prices cannot leak into earlier features.
Rolling features are common offenders. A season-to-date field must represent what “to date” meant then. A player-role feature cannot use future snaps to describe an earlier week. A market feature captured after your intended bet time is a comparator, not an input.
If the pipeline cannot reproduce point-in-time inputs, stop. “Close enough” is not a launch state.
Keep training earlier than evaluation
A model evaluated on the rows it learned from can look polished while merely remembering. Use a forward holdout, rolling retraining, or another time-aware design that keeps fitted history earlier than graded history.
The exact split depends on the data and model. The invariant does not: the evaluated outcomes were unavailable to the fitted model. Randomly scattering future games into training may satisfy a generic machine-learning template while violating the betting timeline.
Do not spend the holdout twice
A holdout stops being unseen when you inspect it, change the model in response, and inspect it again. Repeated tuning teaches the researcher what the model never directly saw.
Use development history for iteration. Preserve a later segment for a final check. When that final check informs another redesign, call it development data from then on and reserve new evidence. The label matters less than the refusal to pretend surprise survived repeated viewing.
Audit the market timestamp
Historical odds can leak too. If the intended wager happens before the close, the closing line cannot appear in the feature set. Keep it as a later benchmark for grading or CLV analysis.
Record which market snapshot is an input, which is a comparator, and when each exists relative to the wager. The market name alone is not enough. An opening number, an accepted price, and a closing reference answer different questions.
Test the system around the model
The estimator is only one part of the live decision. Reproduce the full path:
- Data availability: every required input exists before the historical wager time.
- Feature construction: rolling and lagged values stop at the right boundary.
- Model version: the tested code is the version intended for live use.
- Market mapping: the prediction points to the same market definition.
- Grading: pushes, voids, and missing markets remain explicit states.
- Output capture: the prediction and market snapshot can be audited later.
If training and live scoring use different preprocessing, you tested a laboratory artifact. If the grading script drops awkward rows, you tested a highlight reel.
Try to break the result before reading the edge
Look for duplicated events, impossible features, selective missingness, target leakage, and suspicious jumps. Remove the feature that looks too good. Shift the cutoff. Inspect the rows the pipeline excluded.
This order matters. Edge hunting turns every defect into a promising slice. Failure hunting asks whether the result deserves interpretation at all.
Use the existing workflow as the inspection surface
Studio's backtest panel keeps the target and historical replay inspectable together, with the live decision path visible beside the backtest.
The value is not one magic score. It is a reproducible run that names the target, data window, feature timing, market snapshot, grading rule, and model version -- the same discipline in the reproducible backtest checklist.
Week 1 is an operations test, not a verdict
The first live games can expose broken feeds, stale assumptions, and market mismatches. They cannot instantly settle the model’s long-run quality.
Record predictions before the games. Preserve the available market. Grade under the declared rule. Compare the live feature distributions with the backtest. If the model suddenly flags nearly every game or loses a key input, that is useful evidence even before outcomes accumulate.
Do not promote a model after an opening win or bury it after an opening loss. First ask whether the same process ran.
Know what stops the launch
- A required feature cannot be reproduced point in time.
- Evaluation overlaps training in a way that exposes outcomes.
- The live market differs from the tested market.
- Historical missingness removes a material class of events without explanation.
- The workflow depends on manual fixes that are absent from live scoring.
- The result collapses when one suspicious feature is removed.
Keeping the model in research is not a failed launch. It is the backtest earning its keep.
The Receipts Drawer
Before the season, record what the model predicts, what history it used, which inputs are available live, which market it maps to, what limitations remain, and what condition pauses deployment.
That note prevents goalposts from wandering after the first result. When the model changes, preserve the old predictions and create a new version. Never let a revised Week 1 model claim the original call.
Bottom line
Backtest before live because the season boundary is where stale assumptions meet new data. Match the target. Respect chronology. Hold out unseen history. Audit market timing. Rehearse the full pipeline. Try to break it.
Then use Studio to carry the same definitions into live observation. The goal is not to make a model look invincible. The goal is to prove it is answering the question you think you asked.
Model calibration from graded predictions
Calibration points render only when a verified source binds prediction probabilities to settled outcomes for the same observations.
Expected value from graded outcomes
Expected-value cells render only when a verified source binds observed win outcomes to the price paid for the same bets.
