dew.state
The train state and the accumulation record that jitted steps take and return.
| Name | Summary |
|---|---|
CHECKPOINTED | The train-state fields a checkpoint holds, by their names; a field its declaration marks checkpointed=False is rebuilt on resume. |
MODEL_STATE | The train-state fields that say which model a step holds, its counters, variables and EMA: what restore_best returns of the best step and a weights-only snapshot keeps. |
Accumulation | Holds the microbatches pooled so far, until the optimizer commits them. |
TrainState | Holds everything a run must checkpoint to resume where it stopped. |
CHECKPOINTED
Section titled “CHECKPOINTED”CHECKPOINTED = tuple(field.name for field in dataclasses.fields(TrainState) if field.metadata.get('checkpointed', True))The train-state fields a checkpoint holds, by their names; a field its
declaration marks checkpointed=False is rebuilt on resume.
MODEL_STATE
Section titled “MODEL_STATE”MODEL_STATE = ('step', 'microstep', 'updates', 'variables', 'ema')The train-state fields that say which model a step holds, its counters, variables and EMA:
what restore_best returns of the best step and a weights-only snapshot keeps.
Accumulation
Section titled “Accumulation”class Accumulation(tokens: jax.Array | None = None)Holds the microbatches pooled so far, until the optimizer commits them.
It keeps only sums, with no forward residuals and no Jacobians of the
statistics, so it can be saved in a checkpoint. gradient is the
window’s from its first microbatch (microstep % accumulation == 0),
which replaces what the last window left there. statistics and
effects hold their array leaves in canonical tree order, and the
objective’s traced result supplies their tree structure.
The replay buffers, batches and variables, have a leading dimension
of window slots, followed by the original batch or mutable-collection
dimensions. Only the collections that Aux.variables rewrites need a
snapshot of what each record read. tokens is the real tokens of the
window’s accepted microbatches, as two uint32 words, which its commit
counts toward a run’s tokens; None where the objective counts none.
Accumulation.replays
Section titled “Accumulation.replays”def replays(other: Accumulation | None) -> boolWhether this window’s replay slots hold batches of the shapes other’s do, so a step
compiled for other reads it: a batch ramp’s stage changes them, and a pooled mean,
which keeps no batch, replays none. A batch’s few fields are compared leaf by leaf, without
building a tree, as a ramped run’s runner does each step.
TrainState
Section titled “TrainState”class TrainState( variables: Variables = struct.field(metadata={'variables': True}), ema: Variables | None = struct.field(metadata={'variables': True}), accumulation: Accumulation | None = None, consumed: jax.Array | None = dataclasses.field(default=None, metadata={'optional': True}), deferred: jax.Array | None = dataclasses.field(default=None, metadata={'optional': True}), inner: Replicas | None = dataclasses.field(default=None, metadata={'optional': True}), compute: Variables | None = struct.field(default=None, metadata={'checkpointed': False, 'variables': True}),)Holds everything a run must checkpoint to resume where it stopped.
It keeps three separate counters. step counts attempted steps, and
together with the root key, which never changes, and a microbatch’s
place in its step, it determines each training draw. microstep counts
accepted microbatches, which a step’s batch is read in one or more of
(Trainer.compile), and the objective’s schedules read it per step; a
dropped window (dropped_window) takes back the ones it held.
updates counts committed updates, and the optimizer’s and the EMA’s
schedules read it.
The loss scaler (scale) and the partial accumulation window
(accumulation) are numerical state too, so they are saved in the same
checkpoint as the parameters.
consumed: jax.Array | None-
The source examples the run’s dispatched batches consumed, int32, in a run bounded in examples (
dew.batch.EXAMPLES_CONSUMED): each step adds what its microbatch’s real rows were the last rows of, whether or not the update it joins commits, so the run’s schedules and cadences read its progress exactly, the same on every process; None in a run counted otherwise, which a checkpoint that holds none restores (OPTIONAL). deferred: jax.Array | None-
The real tokens of the calls whose updates the optimizer holds back (
dew.held_updates.update_applied), two uint32 words (dew.training.model_flops._TokenCount), committed with the update that applies them; None for an optimizer that applies every update, which a checkpoint that holds none restores. inner: Replicas | None-
DiLoCo’s replicas’ inner state (
dew.training.diloco), where the variables, the EMA and the optimizer state above are the outer model’s and its outer optimizer’s; None in a run of one copy, which a checkpoint that holds none restores. compute: Variables | None-
The lower-precision copies of the parameters that the forward pass reads, such as bf16 copies of fp32 weights, written by the update (
dew.training.narrow).They are derived from
variables, so no checkpoint holds them and a restored state starts without them. averaged: Variables-
The objective’s EMA leaves merged into the live variables.
Raises
ValueErrorwhen the objective keeps no EMA. window: Accumulation | None-
The pooling window a step reads: the run’s own, or under DiLoCo its groups’, stacked (
Replicas.accumulation). pending_tokens: jax.Array | None-
The real tokens an update’s open window holds (
Accumulation.tokens, two words), which a run’s clocks count pending (TokenClock): its own window’s, or under DiLoCo the first group’s, each group counting the whole batch’s tokens as the run commits them; None for no window or no count.
TrainState.with_window
Section titled “TrainState.with_window”def with_window(window: Accumulation | None) -> TrainStateThis state with window as the window its step reads (window).