Brier Score for Bettors (Plain English + Live Plot)

Brier score is the single most useful number for evaluating a probabilistic sports model, and it's also the one most retail bettors don't compute. The formula is one line of math. The interpretation is intuitive. The pitfalls are real but small. This post is the explanation we wish we'd had when we started — what Brier is, what to compare it against, when it lies, and how to use it on Tinker to compare your model against the market.

The formula in one sentence

Brier score for binary outcomes is the mean squared difference between your predicted probabilities and the actual outcomes (1 for happened, 0 for didn't):

Brier = (1/N) × Σ (p_i − y_i)²

Where p_i is your model's predicted probability of the event (Chiefs win, over hits, fighter wins by KO) and y_i is 1 if it happened, 0 if it didn't. Lower is better. Best possible is 0 (perfect prediction every time). Worst possible (for a probability bounded in [0,1]) is 1.

A worked example with five bets

Suppose your model made these five NFL spread predictions and the games played out:

• Game 1: model says 0.65 Eagles cover, Eagles cover (y=1). Squared error: (0.65−1)² = 0.1225.
• Game 2: model says 0.55 Bills cover, Bills don't cover (y=0). Squared error: (0.55−0)² = 0.3025.
• Game 3: model says 0.72 Chiefs cover, Chiefs cover (y=1). Squared error: (0.72−1)² = 0.0784.
• Game 4: model says 0.48 Ravens cover, Ravens cover (y=1). Squared error: (0.48−1)² = 0.2704.
• Game 5: model says 0.30 Lions cover, Lions don't cover (y=0). Squared error: (0.30−0)² = 0.09.

Brier = (0.1225 + 0.3025 + 0.0784 + 0.2704 + 0.09) / 5 = 0.853 / 5 = 0.171.

That's a great Brier on five games, but five games is far too small a sample to mean anything. Real Brier evaluation needs hundreds of predictions to stabilize.

What to compare your Brier against

Raw Brier numbers don't mean much without baselines. The honest comparison set:

• Random guessing (always predict 0.5): Brier = 0.25 exactly. This is the floor a model has to beat to add value.
• Vegas closing line (devigged): on near-coin-flip markets like NFL spreads, the devigged close lands just under 0.25 — in the low 0.24s against actual cover outcomes. The open is a touch worse than the close; the close is the sharpened version.
• Naive home-team prior (always predict the home team covers): barely better than random, because the home-team cover rate hovers near 50%.
• Always betting the favorite at -110: also barely better than random. The public hammers the favorite and the favorite covers near 50% of the time; not useful.
• A good public NFL spread model: edges below the devigged Vegas close, but not by much — ATS is close to a coin flip, so the gap is small.
• A genuinely sharp custom model: sits a bit below the Vegas close on out-of-sample games. Small absolute gaps here are real edges.
• An overfit model that won't survive next season: posts a suspiciously low Brier when graded on the same games it learned from, then snaps back toward the 0.25 random floor once next season actually plays out.

The takeaway: any model that beats the devigged Vegas close on a full NFL season it never trained on is competitive with Vegas. Clearing the random 0.25 floor means you're adding some value; clearing the Vegas close means you're beating the market. When a model dips far below the close, double-check you didn't accidentally let it peek at the answers.

Sport-specific Brier baselines

The exact number you should expect depends on how close to a coin flip the market is. Two rules of thumb cover most cases:

• Near-coin-flip markets — NFL and NBA spreads, NFL totals, binary player o/u props — sit just under the 0.25 random floor, because cover/over rates hover near 50% by design. A sharp model beats the devigged close by a small absolute margin; large gaps are a red flag for leakage.
• Lopsided markets — UFC moneylines especially — score meaningfully lower than 0.25 even for the book, because favorites win well above 50% of the time, so confident-favorite predictions are usually right. Don't compare a UFC moneyline Brier to an NFL-spread Brier; compare each to its own random-and-close baseline.

In every sport, the comparison that matters is the same: your out-of-sample Brier versus the devigged closing line on those exact games, not against a universal target number.

Brier decomposition: where the score comes from

Brier score breaks into three components: reliability (is your confidence honest?), resolution (do you take real stands instead of hedging to 50/50?), and uncertainty (how much of the outcome is just luck nobody can predict).

Brier = Reliability − Resolution + Uncertainty

• Reliability: how well-calibrated are your probabilities? When you say 70%, does it happen 70% of the time? Lower is better.
• Resolution: how often does your model deviate from the base rate? Higher is better — a model that always says 50% has zero resolution.
• Uncertainty: how unpredictable is the outcome inherently? Fixed by the data, not your model.

This decomposition matters because a model can have low Brier through two very different paths: trustworthy and willing to take a stand (low reliability, high resolution = ideal), or always-50/50 (low reliability, low resolution = useless for betting). The does-my-confidence-hold-up guide covers the reliability side in depth.

Computing Brier on Tinker

On Tinker, you build a model that projects these games for you, and every model shows a live Brier score on games it didn't train on, updated as new outcomes land. The "Brier vs Vegas" chart shows your model's per-week Brier next to the closing line's Brier. A typical good model:

• Week-over-week Brier hovers between 0.220 and 0.245.
• Beats the Vegas close in roughly 9 of 17 NFL regular-season weeks.
• Season-long Brier ends 0.005 to 0.012 below the Vegas close.
• A model that wins on weekly Brier but loses on the season is almost always overfit to recent weeks.

What to look for in the time series

Real model patterns versus pathological ones:

• Healthy: noisy week-over-week, season-long trend at or below Vegas, no extreme weeks.
• Overfit recovery: starts hot at Brier 0.21 in week 1, drifts up to 0.245 by week 8. The model learned last year's noise and this season is calling its bluff.
• Overconfident: Brier looks reasonable but its big swings don't pay off — when it screams 80%+, the side cashes far less than 80% of the time. Dial the confidence back toward what actually happens before you bet it.
• Timid: Brier near 0.245 (just below random). The model predicts everything close to 50%. It has no edge; don't bet it.

Brier vs other metrics

Why Brier beats common alternatives for bettors:

• vs accuracy: accuracy ignores confidence. A 56% accurate model at 51% average confidence is far less bettable than a 53% accurate model at 60% average confidence — accuracy can't see that, Brier can.
• vs log loss: log loss is more punishing of overconfident wrong predictions. For betting, both are good. Brier is slightly easier to explain to non-technical bettors. Log loss is what you should optimize during gradient training; Brier is what you report at the end.
• vs ROI: ROI is the bottom line, but it's noisy. Two models with the same Brier can show wildly different ROI in a small sample. Brier grades every prediction, while ROI only reflects the subset you bet and how the odds paid — so Brier is a lower-variance estimator and converges on the truth meaningfully faster than ROI on the same set of games.

When Brier lies

1. Tiny sample sizes. Under 100 predictions, Brier is noisy. Don't compare two models with under 100 bets each.
2. Class imbalance. If your event happens 80% of the time (e.g., favored UFC fighters), the random baseline is way below 0.25. Always compute against the actual base rate, not the 0.25 default.
3. Survivorship bias in evaluation. If you drop games where the model predicts close to 50% from the evaluation, you've inflated your Brier artificially. Score every prediction.
4. Grading the model on games it already saw. Brier on the games a model trained on tells you nothing about how it'll do next Sunday. Always score it on games it never saw while learning, the way the real season unfolds — see the how-to-test-on-past-seasons checklist for the right pattern.
5. Multi-outcome events scored as binary. If you're predicting NFL totals as O/U binary, that's fine. If you're predicting UFC method of victory (KO/sub/decision), use multi-class Brier, not binary. The mean-squared-error generalizes naturally.

Slotting Brier into your workflow

Practical workflow for a serious bettor:

1. Build a model on Tinker and grade it only on games it didn't learn from.
2. Report Brier on those games. Compare to the Vegas closing line's Brier on the same games.
3. Check that your confidence holds up — when the model says 70%, does it cash near 70%? If not, dial it toward reality before betting.
4. Bet at Kelly fractions using those corrected probabilities.
5. Track how every bet did against the closing line via closing line value and re-score Brier monthly.
6. If month-over-month Brier creeps up above the Vegas close, the model is rotting — rebuild or replace.

Pair this with the does-my-confidence-hold-up guide for the visual companion to the Brier number.

Brier for multi-class problems

The binary Brier generalizes to multi-class predictions like UFC method of victory (KO win, sub win, decision, KO loss, sub loss, decision loss):

Brier_multi = (1/N) × Σ Σ (p_ij − y_ij)²

where the inner sum is over all K classes and y_ij is a one-hot indicator. Lower is better. Max value depends on K — for 6 classes the worst case is 2 × (1 − 1/K) = 1.67 if you predict completely wrong.

Practical baselines for multi-class:

• Uniform prior across K classes: Brier_multi = (K − 1) / K. For K=6 (UFC methods): 0.833.
• Empirical-prior baseline (predict class frequencies): generally 0.55–0.70 depending on imbalance.
• A useful UFC method model: 0.42–0.48.
• An excellent UFC method model: 0.36–0.40.

Use multi-class Brier wherever your prop has more than two non-trivial outcomes. NFL quarter-by-quarter scoring, NBA leading-scorer picks, MLB winning-margin buckets — all multi-class.

Reporting Brier honestly

When you share your Brier publicly (on the leaderboard, in release notes, in a tweet), always include:

• The number of predictions in the score (sample size).
• The time window covered.
• The Brier of the Vegas closing line on the same predictions, for comparison.
• Whether the model had ever seen those games before, or you graded it on the games it learned from.
• Whether you corrected the confidence levels before scoring, and how.

"My model scores 0.225 Brier" is not a number. "My model scores 0.225 Brier on 472 NFL spread picks across the 2024 season — games it never trained on — vs the Vegas close at 0.236, after correcting its confidence on a separate slice of games" is a number. The first is a tweet; the second is evidence.

Brier for in-play models

Live betting models update probabilities in real time. The Brier of a live model is harder to interpret because each "prediction" is a snapshot at a specific game-state, and the same game contributes multiple snapshots. Two clean ways to handle this:

1. Snapshot-Brier: compute Brier over all in-play snapshots equally. Useful but biases toward late-game snapshots where the prediction is closer to the eventual outcome.
2. Win-probability-trajectory Brier: compute Brier at fixed times (start of each quarter, end of each half) and average. Less biased, harder to compute.

For live-betting models, snapshot-Brier with a small bias correction is standard.

Long-running Brier monitoring

A model in production should have its Brier monitored continuously, with alerts when it drifts:

1. Compute a rolling 60-game Brier each week.
2. Compare to the rolling Brier of the Vegas close on the same games.
3. If your model's Brier exceeds Vegas's for 4 consecutive weeks, the model is rotting.
4. Investigate before rebuilding — sometimes the model has just gone wrong on one slice (say, home favorites), and fixing that one spot is cheaper than starting over.
5. Always rebuild on the most recent data after the diagnosis is complete.

The Tinker leaderboard for each model shows rolling Brier next to the Vegas baseline. Bookmark your model's page and check it weekly.

Brier vs ROI: when they disagree

Sometimes a model with worse Brier has better ROI in a small sample. Why? Brier weights every prediction equally; ROI only counts the bets you actually placed (the ones that cleared your edge threshold). A model with slightly higher Brier overall but very accurate calls on its biggest-edge plays will print money in actual betting even if Brier looks middling.

The right interpretation: trust Brier on samples >500 predictions, trust ROI on samples >500 bets. If they disagree on smaller samples, prefer Brier — it stabilizes faster. If they disagree on larger samples, dig into how the model did at each edge level to understand why.

Bottom line

Brier score is one mean-squared-error away from being the cleanest number you can compute on a betting model. Compare it to the Vegas closing line and to the coin-flip baseline of 0.25. Don't trust it without checking that the model's confidence actually holds up, don't compare it across fewer than 300 predictions, and don't chase a Brier below 0.220 if you can't reproduce it on games the model never saw. Run yours live on Tinker and let the number, not the gut, decide which model gets the action.

Bet responsibly — set limits, never chase losses.

Research note board

Use this model-audit board to keep features, validation, and bet sizing from collapsing into one confidence score.

Model calibration: predicted vs observed

Predicted win probability bucket vs the empirical win rate inside that bucket on the test set. Points on the y=x reference line are perfectly calibrated; points below mean the model is overconfident in that bucket.

EV per $100 across win rate × odds grid

Expected value of a $100 stake at each combination of true win rate and market odds. Anywhere the cell is positive you have a long-run profitable bet; the magnitude shows how aggressive Kelly will size it.