Two tracks: Live vs Backtest
Sports outcomes are public, so any backtest is gameable in principle — a determined submitter can hard-code "if game_id == X return correct answer." We address this by running two parallel tracks.
Live track (primary): a pipeline commits a feature_commit hash for each game before kickoff via POST /api/leaderboard/commit. At
settlement time the worker recomputes the feature vector from the pipeline's declared inputs
as of the commit timestamp and verifies the hash matches. Any mismatch is a strike and the
pipeline is removed from the live track for that window.
Backtest track (secondary): forced holdout (last 6 weeks NFL, 30 days NBA, season-final for MLB/NHL). Backtest ratings are capped at the 50th percentile of the live distribution until a pipeline accumulates ≥ 200 live events.
Bayesian Bradley-Terry pipeline ratings
Each pipeline has a posterior N(μ, σ²) on an Elo-scale rating. When two pipelines
have predictions over an overlapping event set, we observe their metric difference and update
both posteriors via a closed-form normal-normal conjugate update. The 95% credible interval (μ ± 1.96·σ) appears next to every rating.
A brand-new pipeline starts at the prior N(1500, 400²) — wide enough that the leaderboard
shows it as provisional until data narrows the band.
Hierarchical kit ratings
Each kit has its own posterior N(μ_b, σ_b²) derived from the pipelines that
include it. The math is partial-pooling Bayesian ridge regression: each pipeline's rating is a
noisy observation of α + Σ β_b + ε, and we fit the β's with a Gaussian prior N(0, 100²). A kit in many high-rated pipelines moves toward a high μ_b with tight σ_b; a kit in one pipeline stays near the prior with wide σ_b. We tag kits with σ_b > 50 as "provisional."
Enforced isotonic calibration
Log-loss and Brier conflate ranking quality with calibration quality. To separate them, we fit a leaderboard-owned isotonic regression on a held-out calibration slice (the 20% of events between train-end and live-start) and apply it to every submission's predictions before computing log-loss / Brier. ATS% and CLV are threshold-based and don't need recalibration.
Pareto frontier — best deals
For each (sport, market) we compute the Pareto frontier over (rating, −credits_per_eval). A pipeline is on the frontier if no other pipeline
strictly dominates it on both axes. UI surfaces frontier members with a "★ deal" badge. This
way a tiny logistic regression on a 5-credit budget can win "best deal" even when it doesn't
win raw rating.
Fairness rules
Every entry runs through a rule walker before display. Initial rules:
- R1 — No future data (feature-commit hash must verify)
- R2 — Holdout integrity for backtest entries
- R3 — Calibration enforced (isotonic recalibration applied)
- R4 — Compute reported (credits_spent > 0)
- R5 — Minimum sample size (n ≥ 30 for top tier)
- R6 — Trivial-output detection (variance ≥ 1e−6)
- R7 — Backtest cap at p50 until 200 live events
- R8 — No duplicate-submission farming
- R9 — Dataset version pinned
- R10 — Reproducibility seed
A self-paced audit loop walks these rules; if a current top-3 pipeline violates one we strike
+ document in tasks/leaderboard/fairness-rules.md.
Nightly recompute
At 04:00 UTC every day we rerun the top-50 pipelines on the latest dataset versions, recompute all posteriors from scratch, refresh the Pareto frontiers, and deploy. The community publish path also enqueues affected pipelines on every stat insert so community-submitted kits appear on the board within an hour of publish.
Full spec: docs/superpowers/specs/2026-05-13-leaderboard-design.md. Rationale: docs/adr/0013-leaderboard-fairness.md.