The miss comes first: a smooth accuracy number can hide a model that does not mean what it says. A forecast probability is a promise about frequency. A calibration curve checks that promise by comparing predicted probability with observed outcomes.
The most useful sentence a model can say about itself is not “I was right.” It is “when I predicted this probability, the outcome happened at that frequency.” That sentence separates ranking skill from probability honesty. A model can rank games well and still be overconfident. It can also look cautious because it rarely leaves one narrow prediction band.
The reliability diagram in one sentence
A reliability diagram groups forecasts into probability bins. For each bin, the horizontal position represents the model’s average predicted probability and the vertical position represents the observed frequency of the target outcome. The diagonal is the calibration reference: prediction and observation agree there.
Points below the diagonal indicate overconfidence. The model predicted a higher probability than the outcome delivered. Points above the diagonal indicate underconfidence. The outcome occurred more often than the model’s probability suggested.
The chart does not say whether every individual pick was correct. It asks whether repeated probability statements deserve their labels. A single forecast cannot be calibrated by itself. Calibration is a property of a collection of comparable forecasts.
How to read predicted probability against observed frequency
Start with the bin, not the line connecting bins. Ask which forecasts entered the bin, how many observations it contains, and whether its average prediction matches its observed frequency. The connecting line is a visual aid. It can imply continuity that the data does not contain.
Then compare the point with the diagonal. The vertical gap is a calibration error for that bin. The direction identifies overconfidence or underconfidence. The magnitude matters only with the sample size and the uncertainty around the observed frequency.
Finally, inspect coverage. A diagram with points across many bins describes a model that expresses a range of probabilities. A diagram with one populated band describes a model that mostly repeats the same level of confidence. Both can be calibrated inside the observed band, but only one has demonstrated behavior across a broad probability range.
A worked illustrative calibration sentence
In this illustrative example, suppose one bin contains forecasts averaging 55% predicted probability and the target is observed 50% of the time; at -110, 52.4% is the betting break-even, although calibration itself compares probability with frequency rather than profit. In this illustrative example, “predicted 55%, observed 50%” is the central sentence because it states both the model’s claim and the empirical answer.
In this illustrative example, the point sits below the diagonal by 5 percentage points, so the model is overconfident in that bin. In this illustrative example, the result does not prove that every 55% forecast was poor; it says the collection behaved like 50% outcomes instead of the 55% frequency the model announced, while -110 still requires 52.4% to break even.
In this illustrative example, recalibration could map future raw 55% outputs closer to 50% until new evidence supports a stronger probability; at -110, the 52.4% break-even shows why that correction can turn an apparent betting edge into NO PLAY. In this illustrative example, that is useful information, not an embarrassment: the model has identified where its confidence outruns its evidence.
Overconfidence is a probability error, not a personality
Overconfidence has a precise meaning on the chart. Predicted probabilities exceed observed frequencies. It can arise from overfitting, weak regularization, stale training data, target leakage in development, distribution shift, or a decision layer that selects only the most extreme raw outputs.
The source matters because the repair differs. If the ranking remains useful but probabilities are too extreme, a calibration transform may help. If the features no longer describe the current environment, remapping the output can hide a deeper model failure. If selection changes the evaluated sample, the curve must be computed on the same population the user sees.
Underconfidence is the mirror image. Observed frequency exceeds predicted probability. It may look safer, but it still means the probability labels are wrong. A model that repeatedly understates strong forecasts can suppress valid decisions and distort bet sizing. Calibration is not a penalty for optimism. It is a demand that the labels match the frequencies.
Sample size belongs next to every bin
A calibration point without its sample size is incomplete. Observed frequency is an estimate. A small bin can swing after one outcome, especially near the edges where models make fewer forecasts. A large visual gap may therefore be weak evidence.
Bin counts also reveal selection. A model may generate many ordinary forecasts and only a few extreme forecasts. The extreme point attracts attention because it is far from the diagonal, but the count may show that the point represents a fragile tail rather than a stable pattern.
The right reading combines location, count, and uncertainty. Location says whether the bin is above or below the diagonal. Count says how much data supports the point. Uncertainty says how far the observed frequency could move under ordinary sampling variation. Removing any one of those turns a diagnostic into decoration.
Uncertainty can be shown with intervals, resampling, or a minimum-count warning. The display method matters less than the refusal to treat every point as equally stable. A model owner should be able to distinguish “the bin is off the diagonal” from “the available sample establishes a persistent calibration error.” The first is a chart observation. The second is an inference that needs support.
Why nine empty bins can be the main finding
In this illustrative ten-bin diagram, suppose nine bins are empty and every forecast lands in one narrow probability band. In this illustrative example, the chart is not saying the model is calibrated everywhere; it is saying calibration can only be evaluated where the model actually predicts.
A one-band model may be behaving cautiously. It may also be compressed by heavy regularization, weak features, a clipped output layer, or a post-processing rule. The empty bins are evidence about range, not accuracy. They show that the model has not demonstrated what a stronger or weaker probability means.
This matters for product language. A model that only predicts in one band should not be described as if it distinguishes broad confidence tiers. The interface should show the occupied range and the count. Users need to know whether the probability scale is expressive or mostly nominal.
Bin design can create or hide a pattern
Reliability diagrams require a binning rule. Equal-width bins divide the probability scale into fixed ranges. Equal-count bins place a similar number of forecasts in each group. The first preserves the meaning of the horizontal scale; the second gives each point more comparable support. Neither choice is neutral.
Wide bins can average away local overconfidence. Narrow bins can produce unstable points with thin samples. Boundaries can split nearly identical forecasts into adjacent groups and make the connecting line look jagged. A responsible chart states the binning rule and shows the actual mean prediction inside each bin.
The diagram should also resist decorative smoothing. A fitted curve may help summarize a large sample, but it can conceal empty regions and imply estimates where no forecasts exist. The raw bin points and counts remain the evidence. Smoothing is interpretation layered on top.
Use out-of-sample and time-ordered forecasts
A calibration curve is only credible when each probability was produced without access to its outcome. Training-set calibration can look excellent because the model has already adapted to those rows. Random validation can also leak future structure when the real deployment problem moves through time.
For a weekly betting model, the cleanest evaluation preserves chronology. The model version, features, and transformation used at decision time should be frozen with the forecast. Later outcomes grade that frozen probability. Retraining after the result must create a new version rather than rewriting the old prediction.
Selection belongs in the definition too. A curve built from every raw forecast answers a different question from a curve built only from published picks. If a confidence floor removes uncertain cases, the selected sample may calibrate differently from the full output. The chart label should name the population so the reader knows which promise is being tested.
Recalibration is not permission to ignore drift
Probability mapping can correct a stable tendency toward overconfidence or underconfidence. It cannot repair missing features, broken labels, changed market structure, or a model that no longer ranks cases. Recalibration should therefore follow diagnosis, not replace it.
Compare the reliability diagram across model versions and time windows. A consistent shape may support a mapping adjustment. A shape that changes abruptly after roster, data, or pipeline changes points to drift. In that case the probability error is a symptom.
The safest response is often narrower use. Restrict the model to the region where it has support, raise the confidence floor, or publish NO PLAY while the new regime accumulates grades. Honest calibration includes knowing where the sample cannot support a probability claim.
Any correction must be evaluated on forecasts produced after the correction was chosen. Fitting a mapping and grading it on the same observations only shows that the mapping can describe history. The useful question is whether the adjusted probabilities retain their meaning on later, untouched outcomes. That separation turns calibration from a cosmetic fit into an operating control.
Calibration and discrimination answer different questions
Discrimination asks whether higher-scored cases occur more often than lower-scored cases. Calibration asks whether the score values match actual frequencies. A model can discriminate well and calibrate poorly. It can order games correctly while assigning probabilities that are too extreme.
The reverse can also occur. A model can be calibrated after collapsing most forecasts into a narrow band, yet provide little separation between games. The probabilities may be honest but not useful for ranking. That is why calibration should sit beside, not replace, other diagnostics.
For betting, the distinction reaches the decision boundary. Ranking may identify the preferred side. Calibration estimates whether the stated edge is large enough to clear the price and the confidence floor. An uncalibrated probability can convert a lean into a false sense of margin.
What to inspect on the Insights calibration block
- Bin count. Confirm how many graded forecasts support each point.
- Predicted mean. Use the actual average forecast in the bin, not only the bin label.
- Observed frequency. Compare the empirical result with the predicted mean.
- Diagonal gap. Read direction before magnitude: below is overconfidence, above is underconfidence.
- Coverage. Empty bins show where the model made no forecasts.
- Population. Verify that the chart uses the same sport, market, model version, and graded decision set being discussed.
The live figures belong on Insights. That surface can show the current bins, counts, model version, and grading population. This article supplies the reading rule: prediction against observation, gap against uncertainty, and coverage against the claims made about the model.
Sources and limits
This article uses no live calibration rows, model outputs, graded picks, or performance figures. Every numerical probability and bin count is explicitly illustrative. For current reliability data, open the Insights calibration block rather than treating this mechanism explainer as a live report.
Bet responsibly — probabilities are estimates, not outcomes, and losses remain possible.
Model calibration: predicted vs observed
Predicted win probability bucket vs the empirical win rate inside that bucket on the test set. Points on the y=x reference line are perfectly calibrated; points below mean the model is overconfident in that bucket.
EV per $100 across win rate × odds grid
Expected value of a $100 stake at each combination of true win rate and market odds. Anywhere the cell is positive you have a long-run profitable bet; the magnitude shows how aggressive Kelly will size it.
