Skip to content

dew.objectives.supervised

Any Flax model, trained on a loss over its outputs and the batch.

Supervised(model, loss, metrics=(), inputs="x") applies the model to the batch field “x”, at the per-example shape the run’s data gives it, with no subclass of Objective; inputs=InputSpec(Field("x", (4,))) states the shape instead. loss(outputs, batch) reads the model’s output as it returns it, an array or any tree of them (an autoencoder’s reconstruction and latent, say), and returns one loss per example, any trailing axes averaged too; the objective is their mean over the batch’s rows (Objective.row_mean), so an evaluation batch’s repeated rows count for nothing. Each metric is called the same way and reported as its own mean, under its function’s name or its class’s.

A loss or a metric is a module-level function or a configured callable object (CrossEntropy(labels="label")): a run’s record names it by import path (ObjectiveConfig), and writing the record refuses a lambda.

NameSummary
Accuracy1 where the largest of the logits at output (selected) is the label in labels, else 0.
CriterionA loss or a metric: the model’s outputs and the batch, to one value per example.
CrossEntropyThe softmax cross entropy of the logits at output within the model’s outputs (selected) over their last axis, against the integer labels in labels, in float32 or wider as the logits are.
OutputsWhat a model returns: an array, or a tree of them.
SupervisedTrains model, a Flax module or a loaded source whose weights training starts from, on loss, reporting metrics beside it.
selectedThe array at the path output within the model’s outputs: an index into a tuple or a list, a key into a mapping, each in turn; the outputs themselves for an empty path.

dataclass source

class Accuracy(labels: str = 'label', output: tuple[int | str, ...] = ())

1 where the largest of the logits at output (selected) is the label in labels, else 0.

attribute source

A loss or a metric: the model’s outputs and the batch, to one value per example.

dataclass source

class CrossEntropy(labels: str = 'label', output: tuple[int | str, ...] = ())

The softmax cross entropy of the logits at output within the model’s outputs (selected) over their last axis, against the integer labels in labels, in float32 or wider as the logits are.

attribute source

What a model returns: an array, or a tree of them.

class source

class Supervised(
model: nn.Module | Source[nn.Module],
loss: Criterion,
metrics: Sequence[Criterion] = (),
*,
inputs: InputSpec | str,
mode: str | None = None,
work: Callable[[Batch], float] | None = None,
)

Trains model, a Flax module or a loaded source whose weights training starts from, on loss, reporting metrics beside it.

inputs is the batch field the model reads: its name, whose per-example shape a run reads off its first batch as the model reads it, after the loader’s device steps, before the model initializes (Objective.named_inputs), or an InputSpec whose sample states the shape too, which the trainer checks the first batch against. The model’s collections besides its parameters update as it trains, a BatchNorm’s running statistics among them. A model with no starting weights initializes on integer zeros of that shape, which a model that embeds ids reads as ids and any other promotes to its own dtype.

mode names the keyword the model’s call reads its mode from, train for a call __call__(x, train=False) whose BatchNorm and Dropout depend on it: the model is called with it True while it trains and False when it is initialized and when a validation pass scores it, which then writes nothing. None calls the model on the sample alone, in whatever mode its call defaults to.

work is the analytical FLOPs of training on a batch, its rows’ forward and backward, as the batch’s host rows hold them: the numerator of the run’s MFU (train/flops, train/mfu). None counts the model’s own (Objective.training_flops), which a module Dew holds no count for leaves unknown, never estimated from the compiled program.

def training_flops(model, batch: Batch) -> float

The batch’s training FLOPs as work declares them, or the model’s own count.

def fresh_variables(key: jax.Array, held: Variables | None) -> Variables
def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux]
def scored_units(
variables: Variables,
batch: Batch,
step: Step,
) -> tuple[Ratio, jax.Array]

loss’s statistics, from the one scoring loss makes, with each example’s share of their total: the sum of its losses, its row weighted as row_mean weights it.

function source

def selected(outputs: Outputs, output: tuple[int | str, ...]) -> jax.Array

The array at the path output within the model’s outputs: an index into a tuple or a list, a key into a mapping, each in turn; the outputs themselves for an empty path.