A backtest is a suspect until chronology survives inspection. A smooth curve can still memorize noise or borrow facts from the future.
No source-linked run is attached, so no performance claim is published. The honest state is no reproducible, chronology-safe evaluation is attached.
Provenance tier: model-validation methodology; no graded result or model-performance statistic is published.
The model is on trial, not on stage
A backtest is a reconstruction of decisions that could have been made. That wording matters. The job is not to explain old results with every fact now available. The job is to rebuild the information set that existed before each decision, run a fixed rule against it, and preserve the rows that rule accepted or rejected.
That makes the first question brutally simple: could a bettor have known every input at the stated cutoff? If the answer is uncertain, the row is uncertain. If the answer is no, the row is contaminated. A polished model cannot rescue a contaminated row because the failure sits upstream of the fit.
The second question is whether the rule was fixed before the evaluation. A model chosen because it looked best on the final window did not pass a test. It won an audition in the room where the answers were posted.
Overfitting starts before the algorithm
People picture overfitting as an enormous model memorizing the training set. That can happen, but the quieter version is more common. The analyst tries a feature, dislikes the result, changes the cutoff, drops an awkward season, narrows the market, and keeps turning knobs until the story looks clean. Each choice may sound reasonable alone. Together they turn the development process into a search for flattering noise.
The danger grows whenever flexibility is hidden. A hand-built filter can overfit as easily as a neural network. A simple average can overfit when its window was selected after the result was visible. A supposedly obvious injury adjustment can overfit when only the successful examples are remembered. Complexity lives in the number of choices the analyst was free to make, not merely in the number of parameters printed on a model card.
Keep an experiment ledger. Record the hypothesis, eligible rows, feature definition, split, baseline, decision rule, and result for every serious attempt. Failed ideas stay in the book. Otherwise the final winner looks like the only idea ever considered, which is how a long search disguises itself as a clean discovery.
Leakage wears respectable clothes
Look-ahead bias rarely arrives labeled “future data.” It arrives as a season summary with no availability timestamp, a corrected roster file that overwrote the original snapshot, a closing quote joined to an earlier decision, or a rolling feature calculated before the table was sorted. The values look legitimate because they are legitimate facts. They are simply facts from the wrong time.
File creation time is not enough. A source can be published late, corrected later, or copied into a warehouse long after the event. Store the time the source made the fact available, the time the system received it, and the cutoff the model used. Those clocks answer different questions and should not be collapsed into one convenient timestamp.
Targets need the same discipline. A settled score belongs on the outcome side of the boundary. A final injury designation, closing market, or postgame correction may be useful for grading, but it cannot leak into a feature snapshot that claims to represent the pregame world.
The join is where many backtests break
A feature table can be individually clean and still leak when joined badly. A broad season key may attach a year-end value to every earlier event. A “latest row” join may select the newest correction rather than the newest row available before the cutoff. A duplicate event key may quietly multiply one result and give that game extra weight.
Make the join contract explicit: durable event identity, source version, availability time, strict cutoff comparison, duplicate policy, and missing-row policy. Failed joins should return a reason with a cure. “No eligible feature row before cutoff; repair the source lineage or exclude the event” is useful. Carrying the nearest row forward is not.
Write an availability contract for every feature
A feature definition is incomplete until it says when the value becomes usable. Team strength, travel, weather, injuries, rest, tracking data, and market context all update on different schedules. The model should not guess. Each feature needs a source, grain, transformation, release rule, and availability timestamp.
Rolling values must be built from events completed before the predicted event. Sort first, window second, shift before attachment, and test the boundary with a planted row that would leak if the shift disappeared. That regression matters more than a screenshot because it fails the moment the chronology rule is removed.
Corrections should be versioned. A later official correction can improve future training, but it should not rewrite the exact snapshot used by an earlier published decision. Keep both the original decision artifact and the corrected research artifact so the record stays auditable.
Split time in the direction deployment moves
Sports do not arrive as shuffled cards. Rosters change, rules change, market behavior changes, and data vendors change definitions. A random split can scatter those shifts across training and evaluation until both sides look artificially similar. Chronological splits expose the harder question: can the model move from an earlier world into a later one?
Use an earlier block to fit, a later development block to choose among declared alternatives, and a final block that remains sealed. The exact windows depend on the sport and target; the principle does not. Future events must never help shape a model that claims to predict the past.
When a season boundary, coaching change, or source revision creates a distribution break, label it. Do not delete the awkward period merely because the model struggles there. That struggle may be the most realistic preview of deployment.
Validation is a workshop; the final test is a vault
The development window is where ideas are allowed to fail. Compare feature sets, repair bugs, inspect calibration, and challenge assumptions there. The final evaluation is different. Once opened, it can answer the frozen question only. It cannot remain a test while also guiding another round of tuning.
If the final result disappoints, publish or retain it and start a new version. The next model needs a fresh future window. Quietly changing the threshold and rerunning the same events converts disappointment into hindsight and destroys the meaning of the original score.
The first-model workflow is useful only when the split and cutoff survive outside the interface. The model artifact, source manifest, and evaluation ledger should be reproducible without relying on a remembered sequence of clicks.
Multiple comparisons need an experiment ledger
Trying many ideas is not a sin. Pretending the winning idea was preordained is. Every feature family, filter, architecture, threshold, and subgroup viewed during development spends some of the evidence. The more choices explored, the less surprising the best-looking result becomes.
There are formal ways to account for repeated testing, but the practical defense starts with disclosure. Preserve the search path. Separate exploratory findings from confirmatory tests. Promote an interesting split into a new hypothesis, freeze it, and carry it forward rather than retroactively declaring victory on the archive that revealed it.
This is where sharp writing helps. “Interesting development split; not independently confirmed” tells the truth. “The model found an edge” skips the only question that matters: did the edge survive data it could not influence?
Use a baseline that can embarrass the model
A complex model should face a baseline that represents a credible alternative, not a straw target chosen for easy applause. Depending on the problem, that may be a market-derived probability, a plain historical estimate, or a smaller model using fewer inputs. The baseline must use the same eligible rows and the same evaluation window.
Lock the comparison before viewing the final result. Changing the baseline after the candidate loses is another form of tuning. When the data coverage differs, report the mismatch instead of comparing scores from different populations.
The glass-box and black-box guide makes the tradeoff visible: added complexity owes the reader an observable benefit on untouched data and a clear account of the failure modes it introduces.
Calibration and selection are separate jobs
A model can rank outcomes sensibly while attaching poor probabilities. It can also produce decent probabilities while a betting selector uses them badly. Keep those layers apart. Evaluate the forecast on the target it was trained to predict. Then evaluate the decision rule against the real market available at the cutoff.
Calibration needs counts as well as curves. Sparse probability bands should remain visibly sparse. Smoothing them into certainty is another kind of salesmanship. When coverage is thin, merge bands under a declared rule or report that the calibration view is unresolved.
For spread selections, the public record should be ATS wins and losses, hit percentage, evaluation window, sample size, and push handling. A financial summary requires the actual prices for every decision and belongs in a separate source-backed analysis. Missing prices do not become a standard quote by convention.
Closing information is a grader, not a time machine
The closing market can be a useful later comparison, but only when its role is declared. A model that makes an earlier decision may compare that decision with a later close as an execution diagnostic. It may not use the later close as an input and still claim the earlier timestamp.
Store the quote actually available at decision time and the later reference quote separately. Match market, side, line, book or consensus rule, and settlement terms. “The market moved our way” is not enough when the records came from different products or different timestamps.
The closing-line value guide covers that audit boundary. It does not turn a favorable move into proof that a model is calibrated, nor does an unfavorable move erase a well-specified forecast. Each metric answers its own question.
Make no-result states first-class
A credible pipeline can return no eligible rows, no pregame quote, unresolved identity, stale features, an invalid split, or an unreproduced artifact. Those are not exceptions to hide. They are results with cures.
Name the failure at the boundary. “Evaluation unavailable because the source snapshot lacks availability timestamps; rebuild the snapshot with source release times” is actionable. “Something went wrong” is not. A blank chart is worse because it invites the reader to assume there was nothing to worry about.
Do not backfill the surface with demo games or hand-entered prices. A model trained on invented rows may compile, but it no longer answers a real betting question.
The red-team pass before publication
- Can every feature be traced to a source version and availability time?
- Does every rolling value exclude the event it predicts?
- Did any final-window result influence features, thresholds, exclusions, or model choice?
- Do candidate and baseline use the same eligible rows?
- Are duplicate, missing, corrected, and unresolved rows visible?
- Can the run be reproduced from a clean checkout and named artifact?
- Does every public decision point to the model version and market quote used?
- Would removing the chronology guard make a planted regression fail?
Use the analyst workflow to inspect source timing, and use Studio only after the row contract is explicit. The interface should enforce the audit, not distract from it.
What earns the backtest a hearing
A backtest becomes evidence when its inputs were knowable, its design was frozen, its comparison was fair, its final window stayed untouched, and its artifact can be reproduced. Even then, it is evidence about a declared window and target, not a lifetime warranty.
What to watch: the first clean artifact whose feature manifest, availability cutoffs, split lineage, baseline, eligible-row ledger, and final evaluation agree. Until those pieces arrive together, the correct label is unverified backtest.
Model calibration from graded predictions
Calibration points render only when a verified source binds prediction probabilities to settled outcomes for the same observations.
Expected value from graded outcomes
Expected-value cells render only when a verified source binds observed win outcomes to the price paid for the same bets.
