Calibration Plots: Why Your 60% Picks Should Hit 60%

A calibration plot is the single best diagnostic for telling a useful betting model from a confident-sounding fraud. It answers one question — when your model says 70%, does the event happen 70% of the time? — and the answer determines whether you should bet your model at full Kelly, half Kelly, or not at all. This post walks through what calibration plots show, how to read them, and how to fix the common ways models miscalibrate, with a working example you can run live on Tinker.

What a calibration plot actually plots

X-axis: predicted probability bin (e.g., 0–10%, 10–20%, …, 90–100%). Y-axis: observed frequency of the event in that bin. A perfectly calibrated model puts every point on the diagonal — when you say 70%, the event actually happens 70% of the time, so the point at x=0.7 sits at y=0.7.

Real models deviate. The deviation pattern tells you exactly what's wrong:

• Below the diagonal in the top half (60%+): overconfident on favorites. Your "80% lock" is actually a 70% pick. Most XGBoost models without explicit calibration fall into this pattern.
• Above the diagonal in the bottom half (under 40%): overconfident on dogs. Your "20% upset" is actually 30%. Common in neural nets without dropout regularization.
• S-curve crossing twice: too timid in the middle, too bold at the edges. Common in shallow models trained on small datasets.
• Inverted S: too bold in the middle, too timid at edges. Rare, usually a calibration-of-calibration artifact.
• Diagonal but jagged: well-calibrated on average, but bin-level noise. Larger sample needed before recalibrating.

Why this matters more than Brier alone

A model that's badly miscalibrated at 80%+ but well-calibrated elsewhere can have a competitive overall Brier — the well-calibrated bins compensate. But the bets you'd actually place at scale are the high-confidence ones (that's where edge times Kelly is biggest). If your high-confidence bin is the miscalibrated one, your bankroll experience will be worse than the Brier number suggests. Always look at both. The Brier explainer covers the score side in depth; this post covers the calibration side.

A worked example

Take a real-pattern NFL spread model trained on the last three seasons. After training, you score it on the held-out 2024 season and bin its predictions into 10 bins. Hypothetical but realistic output:

• 0–10% bin: 12 predictions, 1 covered (8.3%). Slightly under-confident. Small bin, ignore.
• 10–20% bin: 22 predictions, 4 covered (18.2%). Calibrated.
• 20–30% bin: 31 predictions, 9 covered (29.0%). Calibrated.
• 30–40% bin: 38 predictions, 16 covered (42.1%). Slightly overconfident.
• 40–50% bin: 41 predictions, 19 covered (46.3%). Calibrated.
• 50–60% bin: 39 predictions, 22 covered (56.4%). Calibrated.
• 60–70% bin: 34 predictions, 21 covered (61.8%). Slightly underconfident.
• 70–80% bin: 27 predictions, 18 covered (66.7%). Overconfident.
• 80–90% bin: 18 predictions, 13 covered (72.2%). Overconfident.
• 90–100% bin: 10 predictions, 7 covered (70.0%). Badly overconfident.

The top three bins are systematically below the diagonal. This model believes its big picks more than the data supports. A bettor staking at full Kelly using the raw model probabilities will overbet the favorite-side picks and lose more than the model predicts. The fix is recalibration.

Fixing miscalibration

Two standard techniques, both implemented as one-button options in the model publish flow on Workshop:

Platt scaling

Fit a sigmoid p_calibrated = 1 / (1 + exp(A × p_raw + B)) on a held-out calibration fold (separate from your training fold and your test fold). The two parameters A and B do all the work. Smooth, parametric, very robust on small samples.

Pros: low variance, can't overfit calibration. Cons: only fixes monotonic miscalibration. If your model is overconfident in the middle but well-calibrated at edges, Platt won't help.

Isotonic regression

Fit a non-parametric monotonic step function from raw to calibrated probabilities. Outputs a piecewise-constant mapping.

Pros: flexible, handles non-monotonic patterns within reason. Cons: needs more data (500+ predictions per calibration fold), can over-bin if you don't regularize.

When to use which

• Under 500 calibration samples: Platt.
• 500–2,000 samples, monotonic-looking miscalibration: Platt.
• 500–2,000 samples, S-curve miscalibration: isotonic with bin-count regularization.
• 2,000+ samples: isotonic, full bins.
• Time-series data with non-stationary calibration drift: refit Platt or isotonic monthly.

Don't calibrate on the training fold

The most common mistake: fitting the calibration on the same data the base model trained on. The base model has already memorized the labels, so calibration looks perfect — but it'll be broken on new data.

The right pattern is a three-fold split:

1. Fold A: train the base model.
2. Fold B: fit the calibration mapping (Platt or isotonic) using base-model predictions on this fold.
3. Fold C: evaluate the calibrated model's Brier and calibration on this fully unseen fold.

This is the same nested cross-validation pattern used in serious ML evaluation. Workshop's calibration UI does this automatically when you toggle "validated calibration" in the publish settings.

Reading calibration in the wild

Three patterns we see most often on bettor-built models:

Pattern 1: tree models overconfident on favorites

XGBoost trained on betting data with class imbalance (favorites win 55–60% of NFL spreads against the line in some training sets due to data quirks) tends to push high-probability predictions even higher. The fix is Platt scaling, which pulls the top bin back toward the diagonal. Expect Brier improvement of 0.003–0.008 after calibration.

Pattern 2: deep nets miscalibrated everywhere

Neural nets trained with cross-entropy loss are notoriously overconfident at both extremes, especially when trained for many epochs without dropout. The fix is either temperature scaling (a one-parameter sigmoid) during inference, or isotonic recalibration if you have enough data. Brier improvement: 0.005–0.015.

Pattern 3: ensembles that drift after stacking

When you stack two models with a meta-learner, the resulting probabilities are often slightly miscalibrated because the linear combination doesn't preserve calibration. Always recalibrate the stack output. The two-step recipe is in the stacking guide.

Calibration as a betting-time filter

Even after recalibration, model probabilities have inherent uncertainty in each bin (because each bin contains a finite number of historical samples). A practical heuristic for betting:

• Only bet picks where the recalibrated probability is at least 3% above the implied probability from the line (e.g., the model says 60% on a -110 line where break-even is 52.4%).
• Reduce Kelly fraction by 50% on picks in the highest-confidence bin (90%+) until you have 30+ bets in that bin to validate the calibration.
• Skip bets entirely in any bin where the calibration plot showed >10% deviation from the diagonal on the most recent eval.

This is risk management on top of the probabilistic model. Your model can be sharp and still recommend bets in a bin where the calibration is bad — the filter exists to catch that.

Calibration plots on Tinker

The calibration plot for every published model on Tinker updates as new outcomes land. The plot shows:

• Raw model calibration in light gray (pre-recalibration).
• Recalibrated model in solid color.
• Vegas closing-line calibration (which is, unsurprisingly, very close to the diagonal — Vegas is well-calibrated).
• Per-bin sample counts as bar heights below the plot.
• A 95% confidence band around the diagonal.

The right read is: are my model's points inside the confidence band? If yes, calibrated for this sample size. If no, recalibrate or wait for more data.

What miscalibration costs you in dollars

Concrete example: model says 75% on a +120 pick (break-even is 45.5%). True probability is 65% (model is 10 points overconfident in this bin). Full Kelly stake based on the model: 26.7% of bankroll. True EV at 65% true probability: 65% × 1.20 − 35% = 0.43 expected return per unit staked. Looks fine.

But Kelly based on the wrong probability over-sizes. If you repeat 100 such bets, your bankroll geometric growth rate is lower with the overconfident sizing than it would be with the correct probability. Over a season, this is the difference between +18% bankroll and +6% bankroll. Calibration is dollars, not academic hygiene.

Calibration drift over a season

A model that was well-calibrated in September can be miscalibrated by December. Causes:

• Sample shift: late-season games skew toward playoff-contending teams, different from the regular-season mix.
• Injury cascades: rotation changes invalidate per-team priors.
• Coaching adjustments: opponents stop playing certain tendencies, your model's features lose predictive power.
• Vegas getting sharper: closing lines tighten, your model's edge against the close compresses.

Best practice: re-fit calibration monthly during the season using only the last 60–80 games. The recent fit captures whatever drift is happening; the older data dilutes the signal.

Drift-detection alarm

Set a rolling Brier check: if the rolling 30-game Brier exceeds the long-run average by more than 0.010, the calibration likely drifted. Recompute the calibration plot, identify the affected bins, and refit. This is cheaper than a full model retrain and usually sufficient.

Calibration across sub-populations

A model can be well-calibrated overall and badly miscalibrated in slices that matter to your betting. Common slice splits:

• Home vs road: home-team probabilities often biased higher than they should be.
• Favorites vs dogs: dogs often over-credited because the public bets favorites.
• Primetime vs daytime: primetime games have different attention dynamics, different line movement.
• Division vs non-division: division games are typically more variance-driven; many models overrate the favorite.
• Early-season vs late-season: feature stability differs.

Plot calibration separately for each slice you have a non-trivial sample for. Flag any slice where the deviation from the diagonal exceeds 8% in two adjacent bins.

When to recalibrate per slice

If a slice has >200 predictions and shows consistent miscalibration, fit a slice-specific calibration mapping on top of the base recalibration. This is essentially a two-level calibration: global, then slice-adjustment. Workshop supports this through the "stratified calibration" toggle.

The cost of fitting too aggressively

Over-calibration is a real failure mode. If you fit isotonic regression on 100 predictions, you'll end up with a step function that perfectly matches the holdout — and is wildly wrong on new data. Symptoms:

• Calibration plot looks perfect on the calibration fold.
• Calibration plot looks worse than the raw model on the test fold.
• Brier on test improves marginally while ROI gets worse.

Fix: use Platt instead of isotonic for small calibration folds, regularize isotonic with a minimum-bin-size constraint (typically 15–25 samples per bin), or use cross-validated calibration (fit on K-1 folds, evaluate on the held-out fold, average across K).

Visualizing calibration: design tips

A well-designed calibration plot makes the diagnostic obvious. Recommendations we use across the platform:

• Diagonal reference line in a soft gray, full opacity 0.4.
• Model points plotted in the brand accent color, sized proportional to bin sample count.
• 95% confidence band as a translucent ribbon around the diagonal — easy to spot whether a point is "inside" or "outside" the noise band.
• Below the plot, a small bar chart showing per-bin sample counts. Critical for sanity-checking that "miscalibrated" bins aren't just empty.
• Hover tooltips showing exact predicted-bin range, count, observed frequency, and confidence interval.

The Tinker renderer uses this template for every model's calibration card. You can compare your model's plot side-by-side with the Vegas-close baseline plot on the same hover panel.

Calibration as a publishing gate

Before publishing a model to other users on Workshop, set a minimum calibration bar. Our default gate:

1. No bin's observed frequency may differ from the bin's predicted-probability midpoint by more than 10 percentage points, given at least 30 samples in the bin.
2. The Hosmer-Lemeshow goodness-of-fit p-value must be greater than 0.05 (not a brilliant test, but cheap and reasonable as a sanity check).
3. Brier score must beat the Vegas-close baseline by at least 0.005 on the test fold.

Models that fail any of the three get a warning badge on their card, advising users to recalibrate before betting. Sharing a miscalibrated model is worse than sharing no model.

Bottom line

Calibration plots tell you whether your probabilities mean what they say. Plot before you bet. Recalibrate with Platt for monotonic miscalibration and small samples; isotonic for bigger samples and weirder shapes. Always use a separate calibration fold. Pair calibration with the Brier score for a complete model-quality picture, then size with Kelly based on the calibrated probability, not the raw one. The live calibration plot on Tinker updates every week — bookmark it.

Bet responsibly — set limits, never chase losses.

Named modeling examples

A model page is more useful when the feature examples are concrete. Josh Allen rushing attempts, Ja'Marr Chase target share, Nikola Jokic assist rate, Tarik Skubal strikeout projection, Igor Shesterkin starter confirmation, and Islam Makhachev control time are all different prediction problems. A single “player form” feature cannot explain them all, so the model needs sport-specific inputs and review notes.

• NFL: separate route participation, pressure rate, and red-zone role from box-score volume.
• NBA: separate usage, minute projection, pace, and back-to-back fatigue.
• MLB: separate starter skill, handedness, park, weather, and lineup confirmation.
• NHL and UFC: late confirmations and fight-week news can matter more than a season average.

Research note board

Use this model-audit board to keep features, validation, and bet sizing from collapsing into one confidence score.

EV per $100 across win rate × odds grid

Expected value of a $100 stake at each combination of true win rate and market odds. Anywhere the cell is positive you have a long-run profitable bet; the magnitude shows how aggressive Kelly will size it.

Model calibration: predicted vs observed

Predicted win probability bucket vs the empirical win rate inside that bucket on the test set. Points on the y=x reference line are perfectly calibrated; points below mean the model is overconfident in that bucket.