Skip to content

dew.eval.calibration

Calibration of a classifier’s forecasts over K options, and the scores of it: the math any model’s calibration reads.

A forecast is a row of logits, or of probabilities, over its real options. Each function reads held-out forecasts as arrays, so a model keeps its own policy of which forecasts it calibrates together (dew.decision.calibration’s buckets of a question type and its option count):

The two binnings are their references’: a histogram map’s bin is floor(top * bins), as a binning calibrator’s (Laya’s fit_binning_map), and a reliability bin is closed on the right, as the expected calibration error’s (Guo et al. 2017, Laya’s ece_score), so a top probability on an edge falls in the upper bin of one and the lower of the other.

NameSummary
abstention_thresholdThe smallest top probability at which the forecasts at or above it err at most target_error, a group of tied forecasts counted as a whole (tied_prefixes); with conservative, the error is estimated as (errors + 1) / (accepted + 1).
expected_calibration_errorThe gap between confidence and accuracy, per bin weighted by its forecasts (reliability_bins).
fit_temperatureThe temperature minimizing mean log loss of logits [N, K] against the gold distributions targets [N, K] over the real options [N, K], by SciPy’s bounded L-BFGS-B on its inverse, at least 1e-3, in which the loss is convex: its derivative is the mean over forecasts of E_p[z] - E_gold[z].
histogram_mapThe held-out accuracy of each of bins equal bins of the top probability on [0, 1], a forecast in the bin floor(top * bins) (the last holds 1); an empty bin keeps its midpoint.
histogram_mappedtop mapped through a histogram’s bins (histogram_map).
reliability_binsPer bin of bins equal bins on (0, 1], closed on the right, the forecast count, the summed top probability tops and the number correct, [3, bins]: additive, so passes merge by sum.
risk_coverage_areaThe area under the risk-coverage curve: the error rate among the forecasts at or above each confidence, weighted by how many forecasts that confidence adds, so a group of equal confidences counts as a whole in any order.
softmaxThe options’ probabilities of logits at temperature, over the last axis, in float64; an option of logit -inf takes no mass.
tied_prefixesEach distinct confidence, highest first, with how many forecasts are at or above it and how many of those are correct: a group of tied forecasts counts as a whole, in any order.

function source

def abstention_threshold(
tops: np.ndarray,
correct: np.ndarray,
*,
target_error: float,
conservative: bool = True,
) -> float

The smallest top probability at which the forecasts at or above it err at most target_error, a group of tied forecasts counted as a whole (tied_prefixes); with conservative, the error is estimated as (errors + 1) / (accepted + 1). 1.0 where no threshold meets the target.

function source

def expected_calibration_error(binned: np.ndarray) -> float

The gap between confidence and accuracy, per bin weighted by its forecasts (reliability_bins).

function source

def fit_temperature(logits: np.ndarray, targets: np.ndarray, real: np.ndarray) -> float

The temperature minimizing mean log loss of logits [N, K] against the gold distributions targets [N, K] over the real options [N, K], by SciPy’s bounded L-BFGS-B on its inverse, at least 1e-3, in which the loss is convex: its derivative is the mean over forecasts of E_p[z] - E_gold[z]. Each gold row is read as a distribution, to unit mass: a batch packs it in float32, a little short of unit mass, and the shortfall would scale with the logits’ common offset.

function source

def histogram_map(tops: np.ndarray, correct: np.ndarray, bins: int) -> tuple[float, ...]

The held-out accuracy of each of bins equal bins of the top probability on [0, 1], a forecast in the bin floor(top * bins) (the last holds 1); an empty bin keeps its midpoint. A top probability maps through it as histogram_mapped reads it.

function source

def histogram_mapped(values: tuple[float, ...], top: float) -> float

top mapped through a histogram’s bins (histogram_map).

function source

def reliability_bins(tops: np.ndarray, correct: np.ndarray, bins: int) -> np.ndarray

Per bin of bins equal bins on (0, 1], closed on the right, the forecast count, the summed top probability tops and the number correct, [3, bins]: additive, so passes merge by sum.

function source

def risk_coverage_area(tops: np.ndarray, correct: np.ndarray) -> float

The area under the risk-coverage curve: the error rate among the forecasts at or above each confidence, weighted by how many forecasts that confidence adds, so a group of equal confidences counts as a whole in any order. Lower is better; it rewards a confidence that ranks right answers above wrong ones.

function source

def softmax(logits: np.ndarray, temperature: float = 1.0) -> np.ndarray

The options’ probabilities of logits at temperature, over the last axis, in float64; an option of logit -inf takes no mass.

function source

def tied_prefixes(
confidence: np.ndarray,
correct: np.ndarray,
) -> tuple[np.ndarray, np.ndarray, np.ndarray]

Each distinct confidence, highest first, with how many forecasts are at or above it and how many of those are correct: a group of tied forecasts counts as a whole, in any order. Abstention’s thresholds and the risk-coverage area read their accepted forecasts off these.