Skip to content

dew.state

The train state and the accumulation record that jitted steps take and return.

NameSummary
CHECKPOINTEDThe train-state fields a checkpoint holds, by their names; a field its declaration marks checkpointed=False is rebuilt on resume.
MODEL_STATEThe train-state fields that say which model a step holds, its counters, variables and EMA: what restore_best returns of the best step and a weights-only snapshot keeps.
AccumulationHolds the microbatches pooled so far, until the optimizer commits them.
TrainStateHolds everything a run must checkpoint to resume where it stopped.

attribute source

CHECKPOINTED = tuple(field.name for field in dataclasses.fields(TrainState) if field.metadata.get('checkpointed', True))

The train-state fields a checkpoint holds, by their names; a field its declaration marks checkpointed=False is rebuilt on resume.

attribute source

MODEL_STATE = ('step', 'microstep', 'updates', 'variables', 'ema')

The train-state fields that say which model a step holds, its counters, variables and EMA: what restore_best returns of the best step and a weights-only snapshot keeps.

dataclass source

class Accumulation(tokens: jax.Array | None = None)

Holds the microbatches pooled so far, until the optimizer commits them.

It keeps only sums, with no forward residuals and no Jacobians of the statistics, so it can be saved in a checkpoint. gradient is the window’s from its first microbatch (microstep % accumulation == 0), which replaces what the last window left there. statistics and effects hold their array leaves in canonical tree order, and the objective’s traced result supplies their tree structure.

The replay buffers, batches and variables, have a leading dimension of window slots, followed by the original batch or mutable-collection dimensions. Only the collections that Aux.variables rewrites need a snapshot of what each record read. tokens is the real tokens of the window’s accepted microbatches, as two uint32 words, which its commit counts toward a run’s tokens; None where the objective counts none.

def replays(other: Accumulation | None) -> bool

Whether this window’s replay slots hold batches of the shapes other’s do, so a step compiled for other reads it: a batch ramp’s stage changes them, and a pooled mean, which keeps no batch, replays none. A batch’s few fields are compared leaf by leaf, without building a tree, as a ramped run’s runner does each step.

dataclass source

class TrainState(
variables: Variables = struct.field(metadata={'variables': True}),
ema: Variables | None = struct.field(metadata={'variables': True}),
accumulation: Accumulation | None = None,
consumed: jax.Array | None = dataclasses.field(default=None, metadata={'optional': True}),
deferred: jax.Array | None = dataclasses.field(default=None, metadata={'optional': True}),
inner: Replicas | None = dataclasses.field(default=None, metadata={'optional': True}),
compute: Variables | None = struct.field(default=None, metadata={'checkpointed': False, 'variables': True}),
)

Holds everything a run must checkpoint to resume where it stopped.

It keeps three separate counters. step counts attempted steps, and together with the root key, which never changes, and a microbatch’s place in its step, it determines each training draw. microstep counts accepted microbatches, which a step’s batch is read in one or more of (Trainer.compile), and the objective’s schedules read it per step; a dropped window (dropped_window) takes back the ones it held. updates counts committed updates, and the optimizer’s and the EMA’s schedules read it.

The loss scaler (scale) and the partial accumulation window (accumulation) are numerical state too, so they are saved in the same checkpoint as the parameters.

consumed: jax.Array | None

The source examples the run’s dispatched batches consumed, int32, in a run bounded in examples (dew.batch.EXAMPLES_CONSUMED): each step adds what its microbatch’s real rows were the last rows of, whether or not the update it joins commits, so the run’s schedules and cadences read its progress exactly, the same on every process; None in a run counted otherwise, which a checkpoint that holds none restores (OPTIONAL).

deferred: jax.Array | None

The real tokens of the calls whose updates the optimizer holds back (dew.held_updates.update_applied), two uint32 words (dew.training.model_flops._TokenCount), committed with the update that applies them; None for an optimizer that applies every update, which a checkpoint that holds none restores.

inner: Replicas | None

DiLoCo’s replicas’ inner state (dew.training.diloco), where the variables, the EMA and the optimizer state above are the outer model’s and its outer optimizer’s; None in a run of one copy, which a checkpoint that holds none restores.

compute: Variables | None

The lower-precision copies of the parameters that the forward pass reads, such as bf16 copies of fp32 weights, written by the update (dew.training.narrow).

They are derived from variables, so no checkpoint holds them and a restored state starts without them.

averaged: Variables

The objective’s EMA leaves merged into the live variables.

Raises ValueError when the objective keeps no EMA.

window: Accumulation | None

The pooling window a step reads: the run’s own, or under DiLoCo its groups’, stacked (Replicas.accumulation).

pending_tokens: jax.Array | None

The real tokens an update’s open window holds (Accumulation.tokens, two words), which a run’s clocks count pending (TokenClock): its own window’s, or under DiLoCo the first group’s, each group counting the whole batch’s tokens as the run commits them; None for no window or no count.

def with_window(window: Accumulation | None) -> TrainState

This state with window as the window its step reads (window).