A calibration plot is where a betting model has to explain what its confidence means. When the model labels a group of picks at sixty percent, that group should settle near sixty percent over a sufficiently large, comparable held-out sample. If it does not, the probabilities are miscalibrated even when the ranking looks smart.
The chart is simple: predicted probability on the horizontal axis, observed frequency on the vertical axis. The diagonal is the promise. Points below it show overconfidence for the plotted event; points above it show underconfidence. The distance is useful only when the bin contains enough independent observations to distinguish signal from noise.
Build the plot from predictions the model did not fit
Calibration must be evaluated out of sample. Training predictions are flattering because the model has already seen the labels. A calibration mapping fitted on those same rows learns the model's training errors, not its future probability behavior.
Use separate roles for data. One window fits the base model. Another fits any recalibration mapping. A final untouched window evaluates the full pipeline. In time-ordered sports data, those windows move forward rather than being randomly mixed across seasons.
Preserve the prediction timestamp and feature-availability boundary. A beautifully calibrated chart built from post-start information is still leakage. The plot cannot detect when the probability was made; the data contract has to do that.
Choose bins that reveal shape without inventing certainty
Equal-width bins divide the probability scale into fixed ranges. Equal-frequency bins place a similar number of observations in each group. The first makes positions easy to compare; the second avoids nearly empty tails when predictions cluster near the middle.
Neither choice is universally correct. Show the count and an uncertainty interval for every bin. If a point moves dramatically when the grouping changes, the finding is not stable enough to support a precise correction.
Do not interpret a jagged line as a complicated calibration defect until sampling noise has been ruled out. Sparse bins can produce theatrical shapes from ordinary variance. Combine bins, collect more live-pregame predictions, or state that the tail is inconclusive.
Read the pattern, not just the average gap
A curve consistently below the diagonal at high probabilities suggests overconfidence in that region. A curve consistently above it suggests the model is too timid. An S-shaped pattern suggests that the mapping from raw score to probability is distorted differently in the middle and tails.
Now split the plot by meaningful populations: market type, favorite status, season phase, book source, or model version. A model can look calibrated overall while two opposing errors cancel. Only use slices with enough observations to support the view, and declare the slice before reading the outcomes.
Keep the prediction distribution beside the plot. A model may look excellent in a region where it almost never predicts. The sample bar tells you whether the calibration claim describes the model's actual workload.
Platt scaling is the restrained correction
Platt scaling fits a sigmoid that maps a raw model score or probability to a recalibrated probability. Its limited shape is a feature when the main defect is smooth overconfidence or underconfidence. The mapping has fewer ways to chase noise.
Fit it on the calibration window only. Save the parameters with the model version and apply them unchanged to the final test. If the test does not improve in the intended way, reject the mapping rather than tuning it against the test.
Platt scaling cannot repair every shape. If the error bends differently across the range, the sigmoid may under-correct one region and over-correct another. That is a reason to inspect alternatives, not to add hidden adjustments by hand.
Isotonic regression buys flexibility with variance
Isotonic regression learns a monotone step function. It can follow calibration shapes a sigmoid cannot, while preserving the ordering of predictions. That flexibility also lets it memorize a small calibration set.
Use minimum group sizes, cross-validation inside the development window, and a separate final test. Inspect whether the learned steps are supported by repeated data or by a handful of outcomes. A perfect staircase on the fit window is not the goal.
Temperature scaling and other mappings belong to the same family of decisions: choose a constrained correction, fit it away from the final test, and retain it only when the untouched evidence supports the change.
Calibration does not create a betting edge
A calibrated model can be no better than the market. A model that repeats the base rate can be well calibrated and offer no separation. Calibration makes the probabilities interpretable; discrimination and price comparison determine whether the model has useful information.
Compare the recalibrated model with a devigged closing-market baseline on the exact same events. Use the Brier score or another proper scoring rule to evaluate the full probability output. Keep ATS results separate and report wins, losses, win rate, window, sample, grading rule, and provenance tier when making a spread-record claim.
Do not turn a calibration gap into a stake recommendation. Sizing requires an available price, a validated probability, uncertainty treatment, bankroll state, and correlation limits. This article stops at probability quality.
Watch for drift after publication
Calibration can change when the population changes. New rules, roster behavior, market sources, feature pipelines, and model updates can all move the relationship between forecast and outcome. A mapping fitted once is not permanent truth.
Lock model versions and plot live-pregame predictions in event order. The graded-at field is a batch stamp, not the time axis; use the prediction and event timestamps to preserve chronology. Compare recent behavior with the validated range without fitting every short wobble.
When drift appears, diagnose before recalibrating. Check data freshness, feature definitions, close source, and population mix. A new mapping should not hide a broken input pipeline.
A publication contract for the chart
A trustworthy calibration plot names the model version, event window, sample, market, prediction timestamp rule, binning method, count per bin, uncertainty method, and whether the view is raw or recalibrated. It also names the data used to fit the mapping and keeps the final test separate.
Studio is the modeling surface already associated with this article, and the browser backtest guide covers the replay contract. The chart should be treated as evidence generated by that contract, not as decorative proof that the model is sophisticated.
The watchpoint
The model earns stronger probability language when the reliability shape remains stable on untouched, live-pregame data and across defensible binning choices. Until then, say where the chart is uncertain. A probability model is credible when its confidence survives being counted, not when the line looks smooth.
Expected value from graded outcomes
Expected-value cells render only when a verified source binds observed win outcomes to the price paid for the same bets.
Model calibration from graded predictions
Calibration points render only when a verified source binds prediction probabilities to settled outcomes for the same observations.
