Skip to content

dew.training

The trainer and the pieces it is built from.

dew.training contains nothing specific to a modality. It imports nothing from dew.diffusion, dew.inputs or dew.sampling at runtime, and imports wandb only when a WandbTracker logs.

Each name is imported from its own module when it is first used (dew._exports.lookup), so import dew.training.budget reads that module, not the trainer.

NameSummary
AuxEverything a loss returns besides its statistics. Documented in dew.objectives.base.
BestKeeps the top best evaluations by a metric, a recorded metric name, or a custom score.
CheckpointsManages the checkpoints of one run in one directory. Documented in dew.
CooldownsWhere a run branches into cooldowns, and how each decays. Documented in dew.training.cooldown.
DiLoCoDiLoCo’s schedule and outer optimizer: the groups meet every every windows of updates, and the outer step is Nesterov SGD of the outer gradient at rate with momentum, 0.7 and 0.9 by default (the paper’s, MaxText’s diloco_outer_lr and diloco_outer_momentum).
EMASpecWhich leaves the EMA copy tracks, and how fast it follows them. Documented in dew.objectives.base.
EpisodesA split scored by flying a policy, not by reading records: score(variables, key) returns the split’s scores by name (dew.rl.evaluation.episodes), each declared with how it shows (scores, the split’s metrics, EpisodeScore).
EvalSuiteA validation split fit scores with its own metrics on its own cadence.
EvaluationThe result of one evaluation, with its previews kept on process 0.
InspectionWhat a run watches and how often (row 2317).
KeepKeeps the union of the latest steps, periodic steps, wall-time-spaced steps and a predicate’s steps. Documented in dew.
LayoutDecides how a train state is placed on a mesh. Documented in dew.distributed.placement.
LocalTrackerWrites each report to a tracking directory as it is logged.
MLflowTrackerAn MLflow run in experiment, opened on the first value logged into it.
MeshSpecDevice counts for the mesh’s sharding axes; the data axis takes the devices left over. Documented in dew.distributed.placement.
MetricReduces a validation pass to one scalar, on the host. Documented in dew.objectives.base.
ObjectiveDefines what is learned: the parameters, the loss and what evaluation produces. Documented in dew.objectives.base.
Observecall(state) at every every-th step of a fit and at its last, its result kept in values.
PassesProgressA custom training step (Trainer(step=)) that passes the run’s progress to a model that reads it (dew.nn.progress, Progress(state.updates)), as the objective’s own loss does: make, its factory, wrapped.
PauseEnds the run once it has trained fraction of its budget, its state and data position written in full, so a run resumed from that checkpoint trains on as one that never paused: the budget still sizes the learning-rate schedule. Documented in dew.training.budget.
PlateauStops training after evals eligible evaluations without an improvement larger than min_delta. Documented in dew.training.budget.
PreemptedRaised by Trainer.fit when it stops at a preemption notice. Documented in dew.training.runtime.
ProfileWindowA request for one profiler window per fit.
RolloutProduces a batch on the host, before the compiled step reads it.
StepWhat the trainer tells an objective about the current step. Documented in dew.objectives.base.
TensorBoardTrackerA TensorBoard event file in directory, opened on the first value logged into it.
Tracker
TrackersSends each report to several trackers.
TrainerRuns an Objective: gradients, sharding, EMA, checkpoints and logging.
WandbWhere a run reports to Weights & Biases.
WandbTrackerA Weights & Biases run, opened on the first value logged into it.
ema_updateUpdate the EMA leaves toward params, keeping each leaf’s storage dtype. Documented in dew.averaging.
inspectedThe scalars trainer’s inspection logs, of one sampled update over batch from the state the trainer places, its checkpoints’ latest where it has them, else its start: the view a live run logs, for a loaded run on one batch offline.
prepare_processSet the environment variables and XLA flags, raise the soft file descriptor limit, and join the JAX process pool. Documented in dew.training.runtime.
run_timestampReturn process 0’s wall clock as %Y-%m-%d_%H:%M:%S, on every process. Documented in dew.training.runtime.

dataclass source

class Best(
metric: str | Metric | TrainingScalar | Summary | types.MethodType | Callable[[Mapping], float],
top: int = 1,
mode: Literal['min', 'max'] | None = None,
threshold: float | None = None,
weights_only: bool = False,
split: str | None = None,
*,
_source: Metric | TrainingScalar | Summary | Callable[[Mapping], float] | None = None,
)

Keeps the top best evaluations by a metric, a recorded metric name, or a custom score.

A name refers to one of the metric objects passed to fit, or to a Summary as evaluation/summary/<name>. mode sets the direction, ‘min’ or ‘max’. When it is None, a custom score is minimized, and a named metric or a metric object takes its direction from its Shown declaration. An evaluation whose score is not better than threshold is not ranked.

def resolved(scored: Mapping[str, Sequence[Metric | Summary]]) -> Best

This selection bound to the split it ranks and the metric it names there: train for a training scalar, else the split of scored (each split’s metrics) it names, by split or as its recorded name’s prefix, or the only one, and the metric it names among that split’s own, so each suite’s instance of one metric is its own. Its direction, where it names none, is how that metric itself declares it is shown.

dataclass source

class DiLoCo(every: int, rate: float = 0.7, momentum: float = 0.9)

DiLoCo’s schedule and outer optimizer: the groups meet every every windows of updates, and the outer step is Nesterov SGD of the outer gradient at rate with momentum, 0.7 and 0.9 by default (the paper’s, MaxText’s diloco_outer_lr and diloco_outer_momentum). The groups are the mesh’s diloco axis (MeshSpec.diloco).

every: int

Windows of updates between the groups’ meetings (MaxText’s diloco_sync_period).

outer: optax.GradientTransformation

The outer optimizer: Nesterov SGD at rate with momentum.

def admits(host: Sequence[str], record: bool) -> None

Refuse what a DiLoCo run cannot express, before any state is made or supplied: a layout that keeps fields on the host (host, Layout.host), whose offload and host-master step place the outer model’s fields and run no group’s, and a record of each example’s draws (record), which no single tape holds of several groups’ own draws.

def started(state: TrainState, mesh: Mesh) -> TrainState

The state a DiLoCo run over mesh’s groups starts from: each group’s inner state a copy of state’s, the outer variables state’s with their parameters in float32, the outer optimizer’s state fresh.

def step(body: Callable, mesh: Mesh, split: int) -> Callable

body, a step over one group’s state and rows (Trainer._step_body), run by every group on its own share of the batch under jax.shard_map over the diloco axis alone, then the groups met where the step closes their everyth window (synced). Its loss, floating metrics and commit telemetry are the groups’ mean, the rest the first group’s (_agreed): its commits’ count and tokens, which each group counts of the whole batch.

def synced(state: TrainState) -> TrainState

state with its groups met: the outer gradient the outer parameters less the groups’ mean, stepped by the outer optimizer in float32, every group’s parameters the outer ones again; the other collections and the EMA the groups’ mean where they are floating, the first group’s otherwise. Each group keeps its inner optimizer’s state.

dataclass source

class Episodes(
score: Callable[[Variables, jax.Array], Mapping[str, float]],
scores: Mapping[str, Shown],
)

A split scored by flying a policy, not by reading records: score(variables, key) returns the split’s scores by name (dew.rl.evaluation.episodes), each declared with how it shows (scores, the split’s metrics, EpisodeScore). It goes where a reader goes (EvalSuite(Episodes(...))), and Evaluation.run scores it on the averaged weights where the run keeps them, on every process and agreed (a scorer that fails on one, or reports other scores than scores declares, fails the evaluation on all), process 0’s scores the pool’s, each of the metrics asked for (all of metrics unless a suite names fewer) named <split>/<name> as a data split’s; a warming pass flies to compile and keeps none. Its key is the evaluation’s event key (_event: the run’s key folded with ‘EVAL’ and the step) folded with ‘EPIS’. It reads no batches, so a run’s data metrics and loss are not its.

metrics: tuple[Metric, ...]

The split’s metrics: one EpisodeScore a declared score, made once, so a selection given one (Best(split.metrics[0])) reads the very metric the run declares.

dataclass source

class EvalSuite(
data: Split,
metrics: Sequence[Metric] = (),
every: int | str | Counted | None = None,
)

A validation split fit scores with its own metrics on its own cadence.

Empty metrics scores fit’s, and every None scores it on fit’s eval_every. Every suite is also scored at a checkpoint that ranks the run’s steps and at the end, so no ranking reads an earlier score. A split scored by its episodes holds an Episodes as its data.

dataclass source

class Evaluation(
step: int,
split: str,
scores: dict[str, float],
coordinated_batches: int,
records: int,
event_key: tuple[int, ...],
elapsed_seconds: float,
previews: tuple[Artifact, ...],
examples: int | None = None,
real_token_share: float | None = None,
)

The result of one evaluation, with its previews kept on process 0.

Every process holds the same scores, row counts and RNG identity (event_key). Each score’s name starts with the split, as in val/loss. coordinated_batches is the number of batches every process scored, and records the number of real rows in them, which covers every record of the split once. elapsed_seconds is process 0’s wall time, including closing the batch iterator. examples is the source examples the run had consumed when it evaluated, on the examples clock, its own coordinate there; None on another. The metric accumulators and the validation batches are not kept.

real_token_share: float | None

Of the tokens the split’s batches computed (each row’s Objective.row_tokens, padding and a drained rank’s repeats included), the share the loss scored (Objective.scored_tokens of its real rows); None where the objective counts no tokens or its rows declare none.

scalars: dict[str, float]

Return the metric values, plus the split’s counts if a batch was scored (evaluation/counts/<split>/records, each split’s its own at a step that scores several, apart from every summary’s, Summary.SPLIT, and its real_token_share where known), and on the examples clock the examples the run had consumed (EXAMPLES_SCALAR), a cooldown branch’s under its own name (COOLDOWN).

def branched() -> Evaluation

This evaluation as a cooldown branch’s: its split and every score named under COOLDOWN.

def run(
objective: Objective[Loss, Effects],
variables: Variables,
batches: Reader | Dataset | Episodes | None,
*,
key: int | jax.Array,
metrics: Sequence[Metric] = (),
step: int | jax.Array = 0,
averaged: Variables | None = None,
preview: bool = False,
mesh: Mesh | None = None,
split: str = 'val',
schedule_step: int | jax.Array | None = None,
loss: bool = False,
training: Mapping[str, jax.Array] | None = None,
warm: bool = False,
updates: int | jax.Array | None = None,
scored_to: str | os.PathLike[str] | None = None,
) -> Evaluation

Evaluate variables on the batches every process can read, without an optimizer or a tracker.

Every process calls this with the same objective, metrics and numerical settings. batches is called with this process’s share of the split (DataPartition.of(mesh)) and returns a fresh iterator, which this call opens and closes; you can pass Loader.val directly, or a chain (Dataset), which compiles here with the objective’s own evaluation stages and reads in its own batches (dew.data.compile.held). Every batch holds VALID_ROWS (all True when the reader marked none), and the objective’s loss and the metrics count only the real rows. A process whose share runs out before the others’ scores a copy of its last batch in which every row is a repeat, so every process scores the same number of batches and the pass reads every record of every share. With no metrics, no loss and no preview, no iterator is opened and the objective does no work, and a preview alone reads at most the first batch. Only process 0’s preview flag counts, and the preview is made once per call.

variables is the complete Flax variables tree. averaged, when given, is the complete set of averaged weights the objective sees as Step.ema, not an optimizer state; to evaluate those weights directly, pass state.averaged as variables. step keys the evaluation’s RNG and labels the result. schedule_step is the step the objective’s schedules read, and defaults to step. When training attempts were rejected, pass the number of accepted microbatches (TrainState.microstep, as fit does) so the schedule follows only the work training accepted. updates is the updates training committed (Step.updates), and defaults to the schedule’s step. warm scores at most WARM_BATCHES batches and finalizes no score, so the evaluation holds none: a pass over them runs every program a longer pass does, the loss’s reduction included, which a run compiles this way before its first step (Trainer.fit), and no score of a part of the split is checked or read.

scored_to, a directory, takes every batch’s scoring artifacts, process 0 writing each batch’s real rows (a decision run’s Decisions) as <scored_to>/<split>-<batch>.npz (dew.training.tracker.write_artifact), so a pass’s metrics pool exactly offline, across batches and splits; a warming pass writes none. Process 0’s scored_to is the pool’s: every process scores the batches it writes, whatever it was given.

The scores are broadcast to every process. Previews stay on process 0, and there is at most one objective preview, however long the validation split is. Reporting the result is up to the caller. In a process pool, the callers must agree on whether reporting failed before they enter their next collective, and a device failure in flight still requires ending the distributed runtime.

dataclass source

class Inspection(
paths: tuple[str, ...] = (),
targets: tuple[Target, ...] = ('params', 'grads', 'updates', 'deltas'),
every: int = 100,
modules: tuple[str, ...] = (),
cotangents: bool = False,
histogram: bool = False,
)

What a run watches and how often (row 2317).

paths are globs over a parameter’s path under params (dew.optim.parameter_name: layers_0/attn/q_proj/kernel, *norm*), as a ParamGroup’s patterns are; each leaf a glob selects is summarized on its own, for each target: its mean, standard deviation, root mean square, largest magnitude, non-finite share and zero fraction. Every sampled step also reports each target’s global L2 norm, over every parameter, and the update-to-weight ratio, ||update|| / ||params||. every is the windows of the run’s accumulation steps between samples (samples): the update that commits in each, every every-th update of a run no step of which is rejected; the run’s logs carry the latest sample, so a cadence of the logs’ or a multiple of it reports each sample once.

modules: tuple[str, ...]

Modules whose outputs a sampled step summarizes, by a glob of their class name (*Attention, MLP) or their path as the log names it (layers_3/*, a layer by its place in the decoder whether the stack scans or pipelines it): each call’s output, its first array where it returns several, over a language model’s real tokens (dew.nn.inspection.watched_positions), one summary a layer in a scanned stack and one a microbatch in a pipelined one (dew.nn.inspection.watching), logged as inspect/activations/<module path>/<statistic>, a pipelined layer by its place in the stack.

cotangents: bool

Whether a sampled step also summarizes the loss’s cotangent at each watched output, over the same positions: its mean, standard deviation, root mean square, non-finite and zero shares (and histogram), from sufficient statistics a probe’s gradient carries (dew.nn.inspection.tapped), so no tensor of the output’s size is kept, logged as inspect/cotangents/<module path>/<statistic>. The sampled microbatch’s own, its pipeline’s microbatches summed; no parameter or update changes with it.

histogram: bool

Whether each summary also holds the share of its magnitudes in each decade (dew.nn.inspection.HISTOGRAM_EDGES), logged one scalar a bin.

watched: Watched | None

The activations a sampled step watches, where modules names any.

def samples(step: int, accumulation: int) -> bool

Whether the step that ends at step is sampled: each step of every every-th window of accumulation steps, the loop’s count of the steps it attempts, so the update that commits in it is sampled at the window’s end or early, where a run on the examples clock closes its last window short. A rejected step (a non-finite loss under a dynamic loss scale) drops its window and shifts the commits after it, which the loop sees only on the devices: a sampled window that then holds no commit skips its sample, as often as rejections so fall.

class source

class LocalTracker(directory: str | Path, *, plots: bool = False)

Writes each report to a tracking directory as it is logged.

The reports are JSONL journals (scalars.jsonl and records.jsonl), preview files and, with plots, plots. Nonfinite metrics are written to the journal as the strings NaN, +Inf and -Inf. Plots are drawn only by plot(), or by close when plots is set, as dew.figures curves, and they read the journal, so the tracker keeps no history in memory.

def log(scalars: Mapping[str, float], step: int) -> None
def artifact(value: Reported, step: int) -> None
def scalars() -> list[tuple[int, dict[str, float]]]

The scalars the journal holds, each report’s step and values in the order they were logged, nonfinite values read back as floats. A tracker over a run’s directory reads the run’s reports this way, its final validation’s scores among them (Trainer.fit logs them under the run’s last step); a journal no report wrote yet holds none.

def timed() -> list[tuple[int, float, dict[str, float]]]

scalars, each report with the wall-clock second it was logged at (time.time()) between its step and its values, which a run’s reports against the time it took read as a curve.

def plot(
*metrics: str = (),
smooth: int = 1,
y_label: str | None = None,
y_range: tuple[float, float] | None = None,
) -> Figure

The journal’s metrics as one figure (dew.figures.curve), each line at the steps its metric was logged: by default the training loss beside every split’s validation loss (each <prefix>/loss the journal holds). smooth draws each line’s running mean over that many points over it, and y_label names the axis, by default the metrics’ common last part (loss). A nonfinite value is left out of its line and counted on the figure; the journal keeps it. Raises KeyError for a metric the journal never logged, naming those it holds.

def plot_files() -> list[Path]

Draw each metric in the journal to its own SVG file and return the paths. A file is named for its metric, each character outside [A-Za-z0-9_.-] written -, with a digest of the whole name, so two names never share a file. The journal is not changed.

def close() -> None

class source

class MLflowTracker(experiment: str, name: str | None = None, *, uri: str | None = None)

An MLflow run in experiment, opened on the first value logged into it.

Scalars are logged as the run’s metrics, and a preview is uploaded as the files LocalTracker writes for it. A record is uploaded as a JSON artifact and not as a parameter, because MLflow refuses a second value for a parameter and a run reports several records of the same type.

run: tuple[MlflowClient, str]

The MLflow client and the run id; the first access creates the run, and the experiment if it does not exist.

def log(scalars: Mapping[str, float], step: int) -> None
def artifact(value: Reported, step: int) -> None
def close() -> None

dataclass source

class Observe(
every: int,
call: Callable[[TrainState], Any],
values: list[tuple[int, Any]] = list(),
)

call(state) at every every-th step of a fit and at its last, its result kept in values.

The state is lent for the call alone: what call returns is copied to the host, so a returned tree of the state’s own arrays (lambda state: state.averaged) is a snapshot, not a view of buffers the next step reuses. A failure in call on any process fails the fit on all of them, at that step. values holds (step, value) in step order; a fit that resumes from a checkpoint drops what it holds past the restored step, which a failed run observed and the resumed one retrains, and a fit that finds the run already complete observes its final state once.

def opened(step: int, state: TrainState, *, complete: bool) -> None

A fit begins at step, restored or fresh: agree the cadence and forget later values. A fit with nothing to train, its run complete at step, observes the state there where any process holds no value for it, every process alike.

def at(step: int, state: TrainState) -> None

After step, where the cadence asks for it.

def last(step: int, state: TrainState) -> None

The run ended at step, done or stopped by a preemption: observe it unless this fit just did.

def take(step: int, state: TrainState) -> None

call(state) on every process, its result copied home and kept as step’s value. A fit’s steps only rise past what opened kept, so the value is the ledger’s new tail or replaces it.

dataclass source

class PassesProgress(
make: Callable[[Objective[L, E], optax.GradientTransformation], Callable[[TrainState, Batch], tuple[TrainState, jax.Array, Aux]]],
)

A custom training step (Trainer(step=)) that passes the run’s progress to a model that reads it (dew.nn.progress, Progress(state.updates)), as the objective’s own loss does: make, its factory, wrapped. A custom step not so declared refuses such a model, which it would otherwise train at the run’s end.

dataclass source

class ProfileWindow(
directory: str,
steps: int,
warmup: int = 2,
capture: telemetry_profile.CaptureOptions = telemetry_profile.CaptureOptions(),
)

A request for one profiler window per fit.

It traces steps steps into directory after warmup steps have run and the run’s first step has compiled, so the trace shows the loop and not the compile. A fit under JAX’s AutoPGLE, which profiles programs in sessions of its own, refuses a window (PGLE_CAPTURE).

The capture records what capture asks (dew.telemetry.profile.CaptureOptions): by default host events at level 2, no Python calls (the Python tracer slows Python-heavy host work several times over), and on GPU at most its activity_cap CUDA activity events.

dew.Profiler is the other way to capture a trace, as a context manager around any code; a fit refuses to schedule a window inside one. The loop reports the window it wrote as the ProfileTraced record of dew.telemetry.records.

class source

class Rollout(Protocol)

Produces a batch on the host, before the compiled step reads it.

Sampling has side effects and cannot be traced, so it runs outside jit. The trainer calls the rollout with the state, the prefetched batch and a key folded from the run key and the step, then reshards the result with shard_batch. The returned batch must hold arrays of fixed shapes, so the step still compiles once per run.

A rollout may have metrics, a mapping of names to floats about its latest call (rewards, lag, truncation); each logging interval sends them to the tracker and the display as rollout/<name>. It may declare how the display shows them in shown, as an objective does.

A rollout whose draws_further is true may read data batches past the step’s, as DAPO’s dynamic sampling does until its training batch is full: the trainer calls it with further, a read of the training stream’s next batch, None where it ran out (FitRun.further), every process reading as many, whose position the step’s checkpoint saves as any read’s.

A rollout whose calls change more than the batch they return (its metrics, a log, a journal, rollouts in flight) may have branch(name), a context manager a cooldown branch’s calls run inside, on the branch’s states, name the branch’s own (cooldown-<step>, the step it branches at), after which the rollout stands as the run left it (_branched). One without it is called alike there.

A rollout whose next batches depend on state it carries between calls (its environments mid episode, a collection it serves in minibatches, a replay) may have saved(), that state as a mapping of arrays (host arrays of 32 bits or fewer a leaf, which the trainer places), None before its first call; saved_shapes(), its leaves’ shapes and dtypes, readable before any call; restored(tree), which carries on from a saved tree; and branch(name), so a cooldown branch leaves that state as the run left it. Each checkpoint of the run holds the run’s rollout’s tree beside its state (dew.training._fit._rollout_saved), and a resume hands it back, placed as the run’s key is (_rollout_resumed), so a resumed run samples the batches the run that never stopped did. It is the pool’s: a run of several processes with a checkpointer is refused one as its fit starts (_rollout_admitted). The contract is the post-training spec’s (docs/design/post-training.md, §14).

class source

class TensorBoardTracker(directory: str | Path)

A TensorBoard event file in directory, opened on the first value logged into it.

Scalars are written as scalar summaries. A record is written as the JSON LocalTracker journals for it, in a text summary under reporting/<type>. Image and video grids are written as image summaries, with a video as an animated GIF; text samples as text; and representations as a histogram of their spread. Other previews raise TypeError.

writer: EventFileWriter

The event-file writer, which the first access opens in directory.

def log(scalars: Mapping[str, float], step: int) -> None
def artifact(value: Reported, step: int) -> None
def close() -> None

class source

class Tracker(Protocol)
def log(scalars: Mapping[str, float], step: int) -> None
def artifact(value: Reported, step: int) -> None
def close() -> None

class source

class Trackers(*trackers: Tracker = ())

Sends each report to several trackers.

Every tracker receives every report, even when an earlier tracker fails. Then the first failure is raised, with the later ones attached as notes.

def log(scalars: Mapping[str, float], step: int) -> None
def artifact(value: Reported, step: int) -> None
def close() -> None

class source

class Trainer(
objective: Objective[Loss, Effects],
optimizer: optax.GradientTransformation | OptimConfig,
*,
key: int | jax.Array,
mesh: MeshSpec | Mesh = _DEFAULT_MESH,
layout: Layout = _DEFAULT_LAYOUT,
microbatches: int | None = None,
accumulation: int = 1,
split: int | None = None,
dynamic_scale: bool | dynamic_scale_lib.DynamicScale = False,
checkpoints: Checkpoints | None = None,
tracker: Tracker | None = None,
step: Callable[[Objective[Loss, Effects], optax.GradientTransformation], StepFn] | None = None,
rollout: Rollout | None = None,
profile: ProfileWindow | None = None,
lower_for: str | None = None,
ahead: AheadOfTime | None = None,
tune_kernels: bool = False,
record: bool = False,
links: Mapping[str, Link] | None = None,
inspection: Inspection | None = None,
param_dtype: str | None = None,
diloco: DiLoCo | None = None,
)

Runs an Objective: gradients, sharding, EMA, checkpoints and logging.

key: jax.Array

The run’s PRNG key: the one it was given, or its seed’s, made when first read, which every path that builds state does, after readying the process (ready_process) and outside any trace: constructing a trainer opens no backend before its flags and cache are set.

device_mesh: Mesh

The device mesh (device_mesh), built on first use.

microbatches: int

The microbatches a pipelined step feeds through the stage axis (stage_microbatches).

def initial_state(
initializer: Initializer | None = None,
key: int | jax.Array | None = None,
) -> TrainState

Build the state a fresh run starts from, a DiLoCo run’s with its groups’ copies (DiLoCo.started), every layer whole, which a stage-local step stacks by stage (_staged_start). It is pure, so fit traces it once for its shapes and once, sharded, for its values.

Both inputs default to the run’s own, and place passes them explicitly so that what it compiles takes them as arguments. A held checkpoint then reaches the device as an argument instead of a constant embedded in the executable. Passing None means “use the configured input”, which is what a call without arguments does. This is the only implementation of the state, so a subclass overrides it here and every path sees the override.

def staged(state: TrainState) -> TrainState

state (initial_state’s, stored’s) held and placed as the step holds it, as fit starts.

def stored(state: TrainState) -> TrainState

state with a stage-local run’s layers back on layers_N, whole on every stage, every mirror alike: for a reader by layer name (an export), at the cost of every layer on every device.

def reading(
loader: Loader,
prefetch: int = PREFETCH_DEPTH,
*,
examples=None,
steps=None,
) -> None

Take loader’s device steps, prefetch and clock (clocked), as fit and compile-ahead do.

def shardings(state: TrainState) -> Placement[TrainState]

Return where each field of state is placed, on the axes that suit its kind.

Parameter gradients follow parameters, replay records follow batches, and the layout’s host-resident fields go in pinned host memory. Under a CPU-owned state, the frozen collection is the exception: it stays where the realization reads it (execution.resident) for the whole run.

def validate(
batch: Batch | None = None,
*,
steps: int | None = None,
capacity: int | None = None,
) -> Validation

What fit checks of the state before it places it (Layout.validate), read without placing anything, so a run is checked on a CPU host before a costly launch.

The state is a fresh run’s: its parameters, optimizer state and EMA, and, given one global batch as the run reads it, the accumulation its steps pool where accumulation is above 1, read in the first of the splits its plan may take whose devices hold it, the trainer’s own where it fixes one (splits). It leaves out the narrow copies a step may read (TrainState.compute), which the plan drops first where the step does not fit with them. An OptimConfig is built over steps, as fit(steps=...) builds it. capacity is the bytes each device has, for a mesh whose devices report none: a CPU mesh of --xla_force_host_platform_device_count devices standing in for GPUs.

Raises what fit raises before placing the state: LayoutRefused for a state over a device’s memory or a layout that replicates too much of the model, and ValueError for rules that name an axis the mesh lacks.

def place() -> tuple[TrainState, Placement[TrainState], bytes | None]

Put the state on the mesh, fresh or restored, once validate’s checks pass.

It returns the state with its shardings and the data position a resume continues from.

def compile(
state: TrainState,
batch: Batch,
trials: Trials | None = None,
*,
capacity: int | None = None,
) -> CompiledStep

Compile a transaction over the state and one already-produced global batch.

The step consumes its state: new = step(old, batch) donates the old buffers, keeping one copy of parameters and optimizer state. Keep no reference to the old state after stepping. Orbax copies a checkpoint to host before save returns, provided Checkpoints names neither prioritized keys nor a concurrent transfer limit. The loader’s batch is not donated.

The plan (dew.training.planning) takes the fewest microbatches at least recomputation, a transaction pooling split x accumulation. Batches split as the first plan or the pooling state’s window holds; later shapes climb the ladder alone. No admitted plan compiles or times optional alternatives, even with trials: speed selection is offline. Where JAX’s AutoPGLE is on (jax_enable_pgle), the plan admitted runs its first runs on a trial, trials’ or, where none are given, a copy of the state beside it, refused where the devices hold no room for it (Trials.profiled), and its program compiles again guided by their profile (Planned.guided), the executable it replaces released with its program (Planned.released). A device that states no memory, a compile-only GPU’s, holds capacity bytes, as a step compiled ahead states them.

def fit(
dataset: Dataset | Loader | Handed,
*,
steps: int | None = None,
epochs: int | None = None,
tokens: int | None = None,
examples: int | None = None,
log_every: int | str | Counted = 100,
eval_every: int | str | Counted | None = None,
checkpoint_every: int | str | Counted | datetime.timedelta | None = None,
checkpoint_final: bool = True,
metrics: Sequence[Metric] = (),
preview: bool = False,
scored_to: str | os.PathLike[str] | None = None,
best: str | Metric | TrainingScalar | types.MethodType | Best | Sequence[Best] | None = None,
stop: Plateau | Pause | None = None,
validation: HandedSplits | None = None,
summaries: Sequence[Summary] = (),
restore_best: bool = False,
state: TrainState | None = None,
observe: Observe | None = None,
device_prefetch: int = PREFETCH_DEPTH,
resume: Resume = 'auto',
cooldowns: Cooldowns | None = None,
) -> TrainState

Train until the step or real-token cap is reached, resuming the latest checkpoint as resume says; return the state as the run holds it, which its model reads off the mesh (off_stages).

resume="auto" continues the checkpointer’s latest checkpoint where it holds one and otherwise starts afresh; “require” refuses to start afresh, for a job that must continue, and “never” refuses a directory that already holds a run (Checkpoints.resumed). An explicit state is the run’s initial state, in place of initializing one and of restoring a checkpoint (a float64 truth’s widened copy of another run’s, say), not a continuation, which is a checkpoint’s alone: its input reader starts at the position dataset supplies.

Every log_every steps, the tracker receives the loss, the objective’s metrics and the throughput. tokens stops at the first microbatch that reaches this real-token budget, from the same counter as train/tokens, persisted in checkpoints. Padding never counts. With both caps the first reached ends the run; with steps=None, a decaying schedule needs its own step horizon. examples trains on the chain’s first source examples, each once with all its rows (Loader.example_steps).

Every eval_every steps, and at the end, the validation split is scored. The objective’s artifacts go to the tracker and to metrics, whose reductions are logged as val/<name>. validation is that split as a Dataset, or names several, each a Dataset, a reader scored as dataset.val would be or an EvalSuite with its own metrics and cadence, logged as <name>/<metric>. Each of summaries weighs those scores into one, evaluation/summary/<name>, formed at a step that scored every split it weighs, which best and Plateau rank by.

restore_best=True returns the model the run ranked best: the final state with the counters, variables and EMA of the best step (MODEL_STATE), read from its best checkpoint, a weights-only one included, or without Checkpoints from the copy it held (HeldBest). Its optimizer state and the rest stay the final step’s, so it is the best model, not a point to resume training from, which the run’s checkpoints are. cooldowns branch the run into cooldowns, each scored by every suite as cooldown/<split>/<metric> and discarded (dew.training.cooldown.Cooldowns).

Every checkpoint_every steps (or “5%”, FitPlan.cadences), the state and data position are written; checkpoint_final=True also writes the final state, independently of that cadence. A duration is checked at each log_every step, so they are written at the first log step after that much time, and every checkpoints.local_every steps to the local directory. A preemption notice (a scheduler’s SIGTERM; PreemptionNotice) stops the run at the next step every process agrees on. That step’s state and data position are written, the final validation is skipped, and fit raises Preempted, which ends the program with status 143 unless caught. Run it again and fit resumes from there.

Previews are generated only when preview=True and a tracker receives them; scalar reporting never triggers preview work. scored_to, a directory, takes every validation batch’s scoring artifacts at each evaluation, <scored_to>/step-<step>/<split>-<batch>.npz (Evaluation.run), a cooldown branch’s under cooldown/ there. observe takes values of the state at a cadence of its own while the run trains (Observe). device_prefetch training batches are placed on the devices ahead of the step, which the step’s memory plan holds room for. The process is readied first (dew.training.runtime.ready_process), as prepare_process readies it: a program that never called that gets Dew’s log handler, and one whose JAX opened on the CPU beside an NVIDIA GPU stops.

def scheduled(updates: int | None) -> None

Build the configured optimizer (an OptimConfig) for updates updates, as fit and validate do for their run’s length; its state holds the rate it logs. A trainer given an Optax transformation keeps it.

dataclass source

class Wandb(
project: str,
entity: str | None = None,
offline: bool = False,
name: str | None = None,
)

Where a run reports to Weights & Biases.

Setting it turns tracking on; the entity and the offline switch have no effect without a project. name is the run’s display name there, apart from the name its checkpoints are kept under; None shows it as the run is named.

class source

class WandbTracker(
project: str,
name: str | None = None,
*,
entity: str | None = None,
config: Mapping[str, object] | None = None,
offline: bool = False,
id: str | None = None,
)

A Weights & Biases run, opened on the first value logged into it.

def log(scalars: Mapping[str, float], step: int) -> None
def artifact(value: Reported, step: int) -> None
def close() -> None

function source

def inspected(
trainer: Trainer[LossT, EffectsT],
batch: Batch,
*,
steps: int | None = None,
) -> dict[str, float]

The scalars trainer’s inspection logs, of one sampled update over batch from the state the trainer places, its checkpoints’ latest where it has them, else its start: the view a live run logs, for a loaded run on one batch offline. A trainer that accumulates steps to the end of its window, from where its state stands in it (a fresh window or a restored part of one), batch each of its microbatches, and reports the step that commits. An OptimConfig is built over steps, the run’s length, as fit(steps=...) builds it. The stepped state is discarded.