Skip to content

dew.nn.categorical

The categorical distribution over a class axis: the one owner of its log-likelihood in Dew.

A model’s logits over a vocabulary, a classifier’s over its classes, a decision’s over a question’s options and a sampler’s over its next token are each a categorical distribution over the last axis. Every log-likelihood Dew takes of one composes on this module: the normalized log probabilities (log_probs), a label’s cross entropy (cross_entropy), a soft target’s (soft_cross_entropy), a divergence from a reference distribution (kl_divergence), the entropy (entropy), the proper scoring rules’ charges of a forecast against a target distribution beside the logarithmic one (brier, spherical, ranked_probability) and a target smoothed toward the even one (smoothed), the divergence among a group’s members over the classes they share (group_divergence), and, for a head that reads its vocabulary in blocks (dew.objectives.lm.chunked), the running partition it combines block by block (rescaled), a label’s cross entropy from it (target_loss) and the log partition of each segment of a ragged row (segment_log_partition), a support’s.

Each computes in at least float32 (dew.nn.precision.at_least_fp32): a bf16 log partition over a vocabulary carries bf16’s 8-bit mantissa into every score, error in the second decimal of a log probability near -10, while float64 logits stay float64. where masks the classes a row does not hold, as a decision’s padded option slots: they take no probability, and a class of no probability adds no term to a sum over the classes.

NameSummary
brierThe quadratic score as a charge, sum_k (p_k - t_k)^2 over the last axis, between 0 and 2.
cross_entropy-log p[labels] over the last axis, labels of logits.shape[:-1]: the log partition less the label’s logit, each taken from the row’s maximum (target_loss), which reads the label’s logit without building the normalized row.
entropy-sum_k p_k log p_k over the last axis, a class of no probability (a -inf logit, or one where excludes) adding no term, in value or gradient.
group_divergenceEach member’s share of its group’s generalized Jensen-Shannon divergence, and its share of the group, [..., M] arrays whose members sum to each group’s divergence and to one a group.
kl_divergenceKL(q || p) over the last axis, p the categorical of logits and q that of the reference logits (Optax’s kl_divergence of the two), the classes where excludes adding no term.
label_derivativeThe derivative of -log p[labels] in its row’s logits, from the row’s probability over the last axis: each other class’s probability, and at the label label_entry’s.
label_entryThe derivative of -log p in the label’s own logit, from the label’s share of its row and the others’ share summed: share - 1 while the label holds at most half the row, else minus the others’ sum (arXiv:2610.31642, eq.
label_logitEach row’s logit at its label over the last axis, 0 for a label outside the row’s columns, which picks none: a padding id past the vocabulary, or another tile’s or shard’s column.
label_termsEach row’s label cross entropy over the last axis and its log partition, of logits taken as they are (widened by the callers): the one rule of the family for both, whose derivatives are the label’s (label_derivative) and the row’s probabilities, the probabilities taken once from the row’s partition (partition).
log_probsThe normalized log probabilities over the last axis (jax.nn.log_softmax).
partitionEach row’s maximum over the last axis, the sum of its exponentials scaled to it, and those exponentials: the terms every whole-row label loss takes its log partition from (target_loss; cross_entropy, dew.objectives.lm.chunked’s whole head).
probabilitiesThe probabilities over the last axis (jax.nn.softmax).
ranked_probabilityThe ranked probability score of classes that are ordered levels: the squared gaps between the forecast’s and the target’s cumulative distributions over the K - 1 cuts between a row’s K levels, sum_k (P_k - T_k)^2 / (K - 1) over the last axis, the levels where holds leading the row.
rescaledA sum of exponentials moved from its own maximum to the larger one, as a blockwise partition combines its blocks: where the two agree the shift is exactly 0 and the sum unchanged, its derivative still the moved sum’s, so two blocks whose maxima tie each move by their own maximum’s tangent less the larger’s; two sums whose maximum is -inf add as they are rather than turn NaN.
segment_log_partitionThe log partition [count] of each segment of a ragged row of logits [C], segments naming each entry’s (count for an entry of none), and whether each segment holds an entry: its maximum, held off the gradient as jax.nn.logsumexp holds it, plus the log of its exponentials’ sum scaled to it, a segment of no entry’s log partition 0.
smoothedtargets over the last axis with share of their mass spread evenly over the classes where holds, (1 - share) t_k + share / K for a row of K classes (optax.smooth_labels), and nothing on the others; integer targets in the floating type a Python number promotes them to (float64 under x64).
soft_cross_entropy-sum_k t_k log p_k over the last axis against the distribution targets, a class of no target mass contributing nothing, whatever its log probability.
sphericalThe spherical score as a charge, `1 - (t .
target_lossA label’s cross entropy from its row’s maximum peak, the sum of exponentials scaled to it mass, and its label’s logit: log(mass) - (target - peak), as cross_entropy takes it, so neither term is rounded at the logits’ magnitude before the subtraction that cancels it does.
token_log_probslog_softmax(logits)[..., tokens], reduced in at least fp32 (cross_entropy’s negation).

function source

def brier(
logits: jax.Array,
targets: jax.Array,
where: jax.Array | None = None,
) -> jax.Array

The quadratic score as a charge, sum_k (p_k - t_k)^2 over the last axis, between 0 and 2.

function source

def cross_entropy(
logits: jax.Array,
labels: jax.Array,
where: jax.Array | None = None,
) -> jax.Array

-log p[labels] over the last axis, labels of logits.shape[:-1]: the log partition less the label’s logit, each taken from the row’s maximum (target_loss), which reads the label’s logit without building the normalized row. Optax’s softmax_cross_entropy_with_integer_labels subtracts the label’s logit from the whole log partition, maximum included, which rounds at the logits’ magnitude rather than the loss’s: mamba-tiny’s drawn tokens’ raw scores sat 1.47 times as far from float64 as log_softmax’s, and 1.00 times so taken here. Its derivative is Barron, Gnanapragasam and Tschannen’s (arXiv:2610.31642, eq. 7): each other class’s probability, and at the label minus their sum, so the row’s derivative sums to zero. The label’s p - 1 collapses to 0 once its margin over the other classes passes the dtype’s precision (eq. 4; 16.6 in float32), while the others’ probabilities stay until they underflow. A label outside the row takes the log partition, its derivative the row’s probabilities (label_terms).

function source

def entropy(logits: jax.Array, where: jax.Array | None = None) -> jax.Array

-sum_k p_k log p_k over the last axis, a class of no probability (a -inf logit, or one where excludes) adding no term, in value or gradient.

function source

def group_divergence(
logits: jax.Array,
groups: jax.Array,
classes: jax.Array,
where: jax.Array | None = None,
) -> tuple[jax.Array, jax.Array]

Each member’s share of its group’s generalized Jensen-Shannon divergence, and its share of the group, [..., M] arrays whose members sum to each group’s divergence and to one a group.

logits [..., M, K] are M members’ categoricals, each over its own K classes (where masks those it does not hold); groups [..., M] names each member’s group among the M (-1: none); and classes [..., M, K] each class’s shared class, in [0, K) (-1: none), as questions that ask one thing with their options re-ordered or reworded share their options. A member’s answer is its probabilities (probabilities) over its classes that map to a shared one, summed onto them; a group’s divergence is the mean of each member’s KL divergence from the members’ mean answer, zero exactly where they agree, at most the log of the member count, computed in at least float32 and finite at any finite logits. A shared class of no probability adds no term, in value or gradient; the gradient is autodiff’s, through the probabilities’ own derivative, which a confident member keeps. A member in no group, or with no class that maps, has share zero and takes no part.

function source

def kl_divergence(
logits: jax.Array,
reference: jax.Array,
where: jax.Array | None = None,
) -> jax.Array

KL(q || p) over the last axis, p the categorical of logits and q that of the reference logits (Optax’s kl_divergence of the two), the classes where excludes adding no term.

Its derivative in logits, p - q, is taken from the two rows’ difference in log space, as Barron, Gnanapragasam and Tschannen take it (arXiv:2610.31642, eq. 9), not by subtracting two softmaxes (eq. 8), which near each other keep only their own rounding: logits equal to the reference’s have derivative exactly zero, and near them it keeps the difference’s precision. A row where a class is -inf in one and not the other takes p - q, a row holding no class 0; the derivative in reference is autodiff’s.

function source

def label_derivative(probability: jax.Array, labels: jax.Array) -> jax.Array

The derivative of -log p[labels] in its row’s logits, from the row’s probability over the last axis: each other class’s probability, and at the label label_entry’s. A label outside the row one-hots to nothing, and its row’s derivative is the probabilities.

function source

def label_entry(share: jax.Array, others: jax.Array) -> jax.Array

The derivative of -log p in the label’s own logit, from the label’s share of its row and the others’ share summed: share - 1 while the label holds at most half the row, else minus the others’ sum (arXiv:2610.31642, eq. 7). Past a half share - 1 cancels, and rounds to 0 once the share rounds to 1, where the others’ sum keeps it and the row’s derivative sums to zero in any rounding; below a half 1 - share is at least a half and share - 1 rounds once, where the others’ sum carries each class’s share of the partition’s rounding, which left an unconfident label’s entry two to three times as far from float64. Every head’s label entry is this one rule (label_derivative, dew.objectives.lm.chunked).

function source

def label_logit(logits: jax.Array, labels: jax.Array) -> jax.Array

Each row’s logit at its label over the last axis, 0 for a label outside the row’s columns, which picks none: a padding id past the vocabulary, or another tile’s or shard’s column.

function source

def label_terms(logits: jax.Array, labels: jax.Array) -> tuple[jax.Array, jax.Array]

Each row’s label cross entropy over the last axis and its log partition, of logits taken as they are (widened by the callers): the one rule of the family for both, whose derivatives are the label’s (label_derivative) and the row’s probabilities, the probabilities taken once from the row’s partition (partition). cross_entropy and the whole-logits head (dew.objectives.lm.chunked.logits_cross_entropy) read it. A label outside the row (label_logit) takes the log partition, its derivative the probabilities.

function source

def log_probs(logits: jax.Array, where: jax.Array | None = None) -> jax.Array

The normalized log probabilities over the last axis (jax.nn.log_softmax).

function source

def partition(logits: jax.Array) -> tuple[jax.Array, jax.Array, jax.Array]

Each row’s maximum over the last axis, the sum of its exponentials scaled to it, and those exponentials: the terms every whole-row label loss takes its log partition from (target_loss; cross_entropy, dew.objectives.lm.chunked’s whole head).

The maximum is finite, as jax.nn.logsumexp takes one: an infinite maximum shifts by 0, so a row holding +inf scores +inf, and an all -inf row’s partition is -inf, never inf - inf.

function source

def probabilities(logits: jax.Array, where: jax.Array | None = None) -> jax.Array

The probabilities over the last axis (jax.nn.softmax). Their derivative is softmax’s, p (dz - sum p dz), but at the most probable class, which takes the other classes’ derivative summed, negated, so a row’s derivative sums to zero: softmax’s own subtracts two near-equal terms there once its probability nears one, keeping only their rounding (as a label’s p - 1 does, arXiv:2610.31642, eq. 4), while the others’ sum keeps their precision, in either mode.

function source

def ranked_probability(
logits: jax.Array,
targets: jax.Array,
where: jax.Array | None = None,
) -> jax.Array

The ranked probability score of classes that are ordered levels: the squared gaps between the forecast’s and the target’s cumulative distributions over the K - 1 cuts between a row’s K levels, sum_k (P_k - T_k)^2 / (K - 1) over the last axis, the levels where holds leading the row.

function source

def rescaled(maximum: jax.Array, total: jax.Array, larger: jax.Array) -> jax.Array

A sum of exponentials moved from its own maximum to the larger one, as a blockwise partition combines its blocks: where the two agree the shift is exactly 0 and the sum unchanged, its derivative still the moved sum’s, so two blocks whose maxima tie each move by their own maximum’s tangent less the larger’s; two sums whose maximum is -inf add as they are rather than turn NaN.

function source

def segment_log_partition(
logits: jax.Array,
segments: jax.Array,
count: int,
) -> tuple[jax.Array, jax.Array]

The log partition [count] of each segment of a ragged row of logits [C], segments naming each entry’s (count for an entry of none), and whether each segment holds an entry: its maximum, held off the gradient as jax.nn.logsumexp holds it, plus the log of its exponentials’ sum scaled to it, a segment of no entry’s log partition 0. An entry’s log probability within its segment is its logit less its segment’s log partition.

function source

def smoothed(
targets: jax.Array,
where: jax.Array | None = None,
share: float = 0.0,
) -> jax.Array

targets over the last axis with share of their mass spread evenly over the classes where holds, (1 - share) t_k + share / K for a row of K classes (optax.smooth_labels), and nothing on the others; integer targets in the floating type a Python number promotes them to (float64 under x64).

function source

def soft_cross_entropy(
logits: jax.Array,
targets: jax.Array,
where: jax.Array | None = None,
) -> jax.Array

-sum_k t_k log p_k over the last axis against the distribution targets, a class of no target mass contributing nothing, whatever its log probability. Its derivative in logits, p T - t for target mass T, is p (T - t) - t (1 - p), each class’s 1 - p by the label’s rule (label_entry: past a half, the other classes’ probability summed), and the class of the most target mass’s T - t the other classes’ targets summed: a one-hot target’s derivative is cross_entropy’s, bit for bit (arXiv:2610.31642, eq. 7), where p - t rounds to 0, and targets as confident as the logits keep its sum 0.

function source

def spherical(
logits: jax.Array,
targets: jax.Array,
where: jax.Array | None = None,
) -> jax.Array

The spherical score as a charge, 1 - (t . p) / |p| over the last axis, between 0 and 1.

function source

def target_loss(peak: jax.Array, mass: jax.Array, target: jax.Array) -> jax.Array

A label’s cross entropy from its row’s maximum peak, the sum of exponentials scaled to it mass, and its label’s logit: log(mass) - (target - peak), as cross_entropy takes it, so neither term is rounded at the logits’ magnitude before the subtraction that cancels it does.

function source

def token_log_probs(logits: jax.Array, tokens: jax.Array) -> jax.Array

log_softmax(logits)[..., tokens], reduced in at least fp32 (cross_entropy’s negation).

tokens is logits.shape[:-1] of ids and the result has that shape.