Calibration (Brier score + reliability report)¶

Audience: operators wanting to know whether the router's / judge's probability outputs are calibrated against real outcomes.

Overview¶

The calibration log records every probability output the router or judge produced alongside the eventual binary outcome. From that log Bernstein computes:

Brier score - mean squared error between predicted probability and the actual outcome; lower is better, range [0, 1].
Expected calibration error (ECE) - bucketed gap between predicted probability and observed frequency.
Reliability diagram data - per-bucket counts, predicted mean, observed rate.

Source: src/bernstein/eval/calibration.py, src/bernstein/cli/commands/eval_benchmark_cmd.py (eval calibration group).

Log path: .sdd/metrics/calibration.jsonl (configurable).

Logging¶

Decisions are logged by the routing layer through bernstein.eval.calibration.log_decision(). Each record carries:

Field	Use
`decision_kind`	e.g. `model_route`, `judge_pass`
`predicted`	probability in `[0, 1]`
`outcome`	observed binary outcome (`0` / `1`); `null` until known
`ts`	epoch seconds

The log is append-only JSONL. Malformed lines are skipped at read time with a debug log, never crashing the reader.

CLI¶

bernstein eval calibration report
  [--since DURATION]       e.g. "30m", "24h", "7d"
  [--kind DECISION_KIND]   e.g. model_route
  [--log-path PATH]        override .sdd/metrics/calibration.jsonl
  [--bins N]               reliability buckets; default 10
  [--output FILE]          write JSON to file (stdout if omitted)

The command emits a JSON report. Empty windows return {"decisions": 0, "brier": null, ...} instead of crashing.

Examples¶

Weekly Brier check from cron:

bernstein eval calibration report --since 7d \
  --output .sdd/audit/calibration-$(date +%Y%m%d).json

Drill into model routing only:

bernstein eval calibration report --since 24h --kind model_route

Adjust bucket resolution for a sharper reliability diagram:

bernstein eval calibration report --since 30d --bins 20

Reading the report¶

Brier	Interpretation
`< 0.05`	Calibrated; predicted probabilities track outcomes closely
`0.05 - 0.15`	Acceptable; tune retry / cascade thresholds at the margin
`> 0.15`	Miscalibrated; router scoring needs re-grounding against fresh traces

expected_calibration_error is bounded [0, 1]; same scale as Brier. The reliability diagram exposes whether the miscalibration is over-confident (predicted > observed) or under-confident (predicted < observed) per bucket.

Troubleshooting¶

decisions: 0 with a populated log. Either --since is too tight or --kind filtered everything out. Drop the filters first and re-narrow.

Brier looks fine but routing still feels off. Brier rewards sharpness and calibration; a router that always predicts 0.5 can post a misleadingly low score on a balanced dataset. Look at the reliability diagram buckets, not just the headline number.

Invalid --since value. Accepted suffixes are s, m, h, d. Anything else raises a ValueError before reading the log.