dew.objectives.supervised
Any Flax model, trained on a loss over its outputs and the batch.
Supervised(model, loss, metrics=(), inputs="x") applies the model to the
batch field “x”, at the per-example shape the run’s data gives it, with no
subclass of Objective; inputs=InputSpec(Field("x", (4,))) states the shape
instead. loss(outputs, batch) reads the model’s output as it
returns it, an array or any tree of them (an autoencoder’s reconstruction
and latent, say), and returns one loss per example, any trailing axes
averaged too; the objective is their mean over the batch’s rows
(Objective.row_mean), so an evaluation batch’s repeated rows count for
nothing. Each metric is called the same way and reported as its own mean,
under its function’s name or its class’s.
A loss or a metric is a module-level function or a configured callable
object (CrossEntropy(labels="label")): a run’s record names it by import
path (ObjectiveConfig), and writing the record refuses a lambda.
| Name | Summary |
|---|---|
Accuracy | 1 where the largest of the logits at output (selected) is the label in labels, else 0. |
Criterion | A loss or a metric: the model’s outputs and the batch, to one value per example. |
CrossEntropy | The softmax cross entropy of the logits at output within the model’s outputs (selected) over their last axis, against the integer labels in labels, in float32 or wider as the logits are. |
Outputs | What a model returns: an array, or a tree of them. |
Supervised | Trains model, a Flax module or a loaded source whose weights training starts from, on loss, reporting metrics beside it. |
selected | The array at the path output within the model’s outputs: an index into a tuple or a list, a key into a mapping, each in turn; the outputs themselves for an empty path. |
Accuracy
Section titled “Accuracy”class Accuracy(labels: str = 'label', output: tuple[int | str, ...] = ())1 where the largest of the logits at output (selected) is the label in labels, else 0.
Criterion
Section titled “Criterion”A loss or a metric: the model’s outputs and the batch, to one value per example.
CrossEntropy
Section titled “CrossEntropy”class CrossEntropy(labels: str = 'label', output: tuple[int | str, ...] = ())The softmax cross entropy of the logits at output within the model’s
outputs (selected) over their last axis, against the integer labels in
labels, in float32 or wider as the logits are.
Outputs
Section titled “Outputs”What a model returns: an array, or a tree of them.
Supervised
Section titled “Supervised”class Supervised( model: nn.Module | Source[nn.Module], loss: Criterion, metrics: Sequence[Criterion] = (), *, inputs: InputSpec | str, mode: str | None = None, work: Callable[[Batch], float] | None = None,)Trains model, a Flax module or a loaded source whose weights training
starts from, on loss, reporting metrics beside it.
inputs is the batch field the model reads: its name, whose per-example
shape a run reads off its first batch as the model reads it, after the
loader’s device steps, before the model initializes (Objective.named_inputs),
or an InputSpec whose sample states the shape too,
which the trainer checks the first batch against. The model’s
collections besides its parameters update as it trains, a BatchNorm’s
running statistics among them. A model with no
starting weights initializes on integer zeros of that shape, which a model
that embeds ids reads as ids and any other promotes to its own dtype.
mode names the keyword the model’s call reads its mode from, train
for a call __call__(x, train=False) whose BatchNorm and Dropout depend
on it: the model is called with it True while it trains and False when
it is initialized and when a validation pass scores it, which then
writes nothing. None calls the model on the sample alone, in whatever
mode its call defaults to.
work is the analytical FLOPs of training on a batch, its rows’ forward and backward, as the
batch’s host rows hold them: the numerator of the run’s MFU (train/flops, train/mfu). None
counts the model’s own (Objective.training_flops), which a module Dew holds no count for leaves
unknown, never estimated from the compiled program.
Supervised.training_flops
Section titled “Supervised.training_flops”def training_flops(model, batch: Batch) -> floatThe batch’s training FLOPs as work declares them, or the model’s own count.
Supervised.fresh_variables
Section titled “Supervised.fresh_variables”def fresh_variables(key: jax.Array, held: Variables | None) -> VariablesSupervised.loss
Section titled “Supervised.loss”def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux]Supervised.scored_units
Section titled “Supervised.scored_units”def scored_units( variables: Variables, batch: Batch, step: Step,) -> tuple[Ratio, jax.Array]loss’s statistics, from the one scoring loss makes, with each example’s share of their
total: the sum of its losses, its row weighted as row_mean weights it.
selected
Section titled “selected”def selected(outputs: Outputs, output: tuple[int | str, ...]) -> jax.ArrayThe array at the path output within the model’s outputs: an index
into a tuple or a list, a key into a mapping, each in turn; the outputs
themselves for an empty path.