dew
Dew: one objective, one trainer.
Each name exported here is imported from its own module when you first
access it, not at import dew. So import dew.training loads only the
training layer, with no modality, encoder or tracker backend. Importing
dew opens no JAX backend and loads no optional dependency; encoders,
decoders and datasets load what they need when they are built.
import dew does set XLA flags, before anything it imports loads jaxlib, so
they hold for every XLA runtime the process loads (dew.devices.flags_hold):
one everywhere and, when JAX’s CUDA plugin is installed, two more.
--xla_allow_excess_precision=false makes XLA round values declared in a
narrow dtype such as bf16 where the program rounds them. On CUDA,
--xla_gpu_enable_allocator_spatial_partitioning=false keeps a preallocated
GPU pool in one piece for a step’s temporaries, and
--xla_backend_extra_options=-nvptx-prec-divf32=2 divides fp32 to the nearest
float, as a CPU and the references Dew is held to do (dew.devices). If you
set any of them yourself in XLA_FLAGS, your value is kept. Where jax was
imported before Dew, XLA may have parsed its flags already and Dew’s are
unproven; where the JAX backend has already opened, they cannot take effect,
and Dew logs a warning.
import dew also makes the JAX user contexts the kernel autotuner and kernel
choices key traces on (dew.cache.KERNEL_TUNES, KERNEL_PINS,
KERNEL_CHOICES): JAX makes one only beside no other JAX call, so before any
thread starts.
import dew leaves the process’s loggers alone and asks nothing of its
hardware. Dew’s log handler, and the refusal of a machine with an NVIDIA GPU
whose JAX opened on the CPU, come with the process’s setup: prepare_process,
or the first Trainer.fit of a program that never called it
(dew.training.runtime.ready_process).
import dew marks absl’s flags parsed when nothing has parsed them, as
absl.app would: Grain’s worker processes read absl flags as they start, and
a program that never ran absl.app would stop them there. A program that
parses its flags itself, through absl.app or FLAGS(argv), still does.
| Name | Summary |
|---|---|
CFG | Interval-limited classifier-free guidance. Documented in dew.sampling.guidance. |
Aux | Everything a loss returns besides its statistics. Documented in dew.objectives.base. |
Best | Keeps the top best evaluations by a metric, a recorded metric name, or a custom score. Documented in dew.training. |
Checkpoints | Manages the checkpoints of one run in one directory. |
Condition | Names one conditioning input: its encoder, the batch field holding its tokens, and the raw datum for the unconditional branch. Documented in dew.inputs. |
Dataset | Records a run reads: a store (Files, Hub, TFDS, Memory, Torch), or a store with the steps the verbs added, in the order they are written. Documented in dew.data.pipeline. |
EMASpec | Which leaves the EMA copy tracks, and how fast it follows them. Documented in dew.objectives.base. |
EvalSuite | A validation split fit scores with its own metrics on its own cadence. Documented in dew.training. |
Evaluation | The result of one evaluation, with its previews kept on process 0. Documented in dew.training. |
Field | A batch field and its per-example shape, such as Field("image", (128, 128, 3)). Documented in dew.inputs. |
ImageGrid | Images in [-1, 1], [N, H, W, C], with the text each was conditioned on where there was any. Documented in dew.artifacts. |
InputSpec | Names the sample field and the conditions, keyed by the model keyword each is passed under. Documented in dew.inputs. |
Keep | Keeps the union of the latest steps, periodic steps, wall-time-spaced steps and a predicate’s steps. |
Layout | Decides how a train state is placed on a mesh. Documented in dew.distributed.placement. |
LocalTracker | Writes each report to a tracking directory as it is logged. Documented in dew.training. |
MLflowTracker | An MLflow run in experiment, opened on the first value logged into it. Documented in dew.training. |
Mean | Averages a metric over a validation pass, from per-example values or additive (total, count) pairs. Documented in dew.eval. |
MeshSpec | Device counts for the mesh’s sharding axes; the data axis takes the devices left over. Documented in dew.distributed.placement. |
Objective | Defines what is learned: the parameters, the loss and what evaluation produces. Documented in dew.objectives.base. |
Observe | call(state) at every every-th step of a fit and at its last, its result kept in values. Documented in dew.training. |
Pause | Ends the run once it has trained fraction of its budget, its state and data position written in full, so a run resumed from that checkpoint trains on as one that never paused: the budget still sizes the learning-rate schedule. Documented in dew.training.budget. |
Plateau | Stops training after evals eligible evaluations without an improvement larger than min_delta. Documented in dew.training.budget. |
Process | Pairs a noise schedule with what the model predicts on it and how the loss is weighted. Documented in dew.diffusion.process. |
ProfileWindow | A request for one profiler window per fit. Documented in dew.training. |
Profiler | Captures a JAX profile of this process, as a context manager or through start and stop. Documented in dew.telemetry.profile. |
Representations | Encoder outputs [N, D] and the labels of the records they came from, for a probe to score. Documented in dew.artifacts. |
Step | What the trainer tells an objective about the current step. Documented in dew.objectives.base. |
Supervised | Trains model, a Flax module or a loaded source whose weights training starts from, on loss, reporting metrics beside it. Documented in dew.objectives.supervised. |
TensorBoardTracker | A TensorBoard event file in directory, opened on the first value logged into it. Documented in dew.training. |
TextSamples | Generated token rows, with optional decoded preview text and prompt. Documented in dew.artifacts. |
TokenScores | Teacher-forced per-token losses [N, L] and the weight of each target. Documented in dew.artifacts. |
Tracker | Documented in dew.training. |
Trackers | Sends each report to several trackers. Documented in dew.training. |
TrainState | Holds everything a run must checkpoint to resume where it stopped. Documented in dew.state. |
Trainer | Runs an Objective: gradients, sharding, EMA, checkpoints and logging. Documented in dew.training. |
VideoGrid | Clips in [-1, 1], [N, T, H, W, C], with the text each was conditioned on where there was any. Documented in dew.artifacts. |
WandbTracker | A Weights & Biases run, opened on the first value logged into it. Documented in dew.training. |
pipeline | Load the inference task for source, with its weights placed once. Documented in dew.inference. |
sample | Run the reverse process from x_T with solver and return the final sample. Documented in dew.sampling. |
Checkpoints
Section titled “Checkpoints”class Checkpoints( directory: str, *, keep: int | Keep = Keep.latest, local_directory: str | None = None, local_every: int | str | None = None,)Manages the checkpoints of one run in one directory.
Constructing one opens nothing; the orbax managers are created on first
use. The directory keeps the latest keep steps, so a resume has
something recent, plus the step with the lowest loss metric a save
reported. A save without metrics can never become the best step.
local_directory names a path on every host’s own disk where the run
keeps one more checkpoint, the latest, which fit writes every
local_every steps, or at each share of the run it names, such as 5%,
as fit resolves its other cadences (dew.training.budget.every_step).
Each process writes the shards its devices hold under its own
subdirectory, so the same path works for a pod and for one host running
several processes. latest is the newest step every process can
read, local or persistent, and restore reads it from wherever it is. A
local checkpoint restores onto the placement it was written with, because
no process holds another process’s shards; the persistent checkpoint
restores onto any mesh.
local_path: str-
Return this process’s own local directory.
latest: int | None-
Return the newest committed step a resume can read, local or persistent.
Asking opens and writes nothing: until this process writes, Orbax’s step format lists the committed steps read-only, and once it has, its manager’s view, which holds this process’s own saves, does.
Checkpoints.would_keep
Section titled “Checkpoints.would_keep”def would_keep(ranking: Ranking | Sequence[Ranking]) -> boolReturn whether an evaluation would enter the best-K set of any of its trackers.
Checkpoints.committed
Section titled “Checkpoints.committed”def committed() -> list[int]The steps whose checkpoints have committed, oldest first, read from the store at each call
without opening it or writing to it, as latest reads it: what a process beside the one
that trains (an evaluation worker) reads as the run saves, a save in flight listed once it
commits. kept is this process’s own view, its retained steps and their metadata.
Checkpoints.kept
Section titled “Checkpoints.kept”def kept() -> list[Kept]Return the committed retained steps, oldest first; their immutable metadata is cached.
The steps are those this process’s store saw open or saved itself; a process that reads a
run another saves into lists its new steps with committed.
Checkpoints.python_run
Section titled “Checkpoints.python_run”def python_run(reasons: Sequence[str]) -> NoneWhy no record rebuilds the run these checkpoints keep, each part of it a run’s record cannot
state, which PYTHON_RUN carries where a run defined in Python first saves (Trainer.fit
names them from its own fields) and RunConfig.read shows.
Checkpoints.cooldown
Section titled “Checkpoints.cooldown”def cooldown(step: int) -> Mapping[str, object] | NoneThe cooldown branch the run began at step and has not finished, as it was resolved there
(dew.training.cooldown); None where none is (COOLDOWN_FILE).
Checkpoints.cooling
Section titled “Checkpoints.cooling”def cooling(pending: Mapping[str, object] | None) -> NoneKeep pending as the cooldown branch the run has begun, or with None that it has none
unfinished; process zero writes it, beside the steps (cooldown).
Checkpoints.artifact
Section titled “Checkpoints.artifact”def artifact(step: int | str | None = None) -> JSONReturn the selected step’s inference declaration.
It is None when the step’s objective declares no inference record.
Checkpoints.control
Section titled “Checkpoints.control”def control(step: int) -> dictCheckpoints.pinned
Section titled “Checkpoints.pinned”def pinned(step: int | str | None = None) -> intThe exact step step selects: itself, the best by a ranking, or, for
None or latest, the latest committed now, which a later save does not move.
Checkpoints.resolve
Section titled “Checkpoints.resolve”def resolve(step: int | str | None) -> int | NoneCheckpoints.resumed
Section titled “Checkpoints.resumed”def resumed(resume: Resume = 'auto') -> int | NoneThe step a run continues under resume, the latest, or None where it starts afresh:
“auto” continues one where the directory holds it; “require” refuses a directory with none,
so a job that must continue never trains from scratch; “never” refuses one that holds a run,
which a run started afresh would write beside.
Checkpoints.path
Section titled “Checkpoints.path”def path(step: int) -> strCheckpoints.source
Section titled “Checkpoints.source”def source(step: int) -> strReturn the directory restore reads step from.
That is this process’s local directory when the step is the local one every process holds, and the persistent directory otherwise; a local step this process lost is read from the first process’s copy that holds it.
Checkpoints.save
Section titled “Checkpoints.save”def save( step: int, state: TrainState | Mapping[str, object], saved: bytes | None = None, metrics: Mapping[str, float] | None = None, *, share: DataPartition | None = None, ranking: Ranking | Sequence[Ranking] | None = None, control: dict | None = None, weights_only: bool = False, primary: str | None = None, rung: JSON = None, artifact: JSON = None, boxed: Mapping[tuple[str, ...], LogicalAxes] | None = None, rollout: Mapping[str, object] | None = None,) -> NoneWrite state under step, asynchronously, a train state with its rollout’s tree of arrays where
it holds one (rollout), which rollout_state reads back.
state is a run’s TrainState, or a mapping of arrays in its place
for a state that is not a training run’s, such as a simulation’s,
which has no optimizer, average or loss scale. boxed is the axes the
model’s modules box on their own parameters (dew.nn.sharding.boxed_axes),
which the state holds unboxed; variables places with them. A mapping is
written as it is, with metrics, ranking and control as a train state’s are,
and restore reads it back through a mapping template; it takes no
data position, share, weights-only split, rung or artifact. It is
written whole at every step, without the store a train state’s
static collections go to (FROZEN_STORE), so large arrays no step
changes are better kept out of it.
Sharded arrays go straight to Orbax. Gathering them onto the host
first would serialise the whole state through one process and defeat
the purpose of an async checkpointer. A stream reports its position as
JSON bytes, which tensorstore has no dtype for, so the bytes are stored
as uint8 rows, one per process, next to the data share that process
read. A global position and a share’s offset are stored the same way
and told apart on restore. A position without its share is refused,
since no reader could be matched to it. A failed write surfaces from
wait, which deliberately does not catch it: a checkpoint that did
not land is lost data.
A state with arrays in pinned host memory is written before this returns, because the write reads those arrays’ own buffers and the next step donates them. Giving Orbax a copy instead would make the next step wait only for the copy, but it would hold a second copy of that state in pinned host memory until the write lands: 12 bytes a parameter for fp32 Adam moments and EMA, 84 GB at 7B parameters, on hosts that keep the state there because device memory is short.
Checkpoints.profile_steps
Section titled “Checkpoints.profile_steps”def profile_steps() -> list[int]Return the complete checkpoints that hold post-hoc EMA snapshots, oldest first.
Checkpoints.profile_metadata
Section titled “Checkpoints.profile_metadata”def profile_metadata(step: int) -> tuple[int, tuple[float, ...]]Return a snapshot’s recorded updates and relative standard deviations, without its averages.
Checkpoints.restore_profiles
Section titled “Checkpoints.restore_profiles”def restore_profiles(step: int) -> tuple[Variables, ...]Read only a snapshot’s averages, as host arrays, from its retained checkpoint.
Checkpoints.posthoc_ema
Section titled “Checkpoints.posthoc_ema”def posthoc_ema(std: float, step: int | None = None) -> VariablesReturn the post-hoc EMA with relative standard deviation std at checkpoint step.
step defaults to the latest snapshot. The result is host arrays in
the params’ structure and dtypes. It sums every snapshot up to step,
from every tracked profile, with the weights coefficients solves for,
reading one snapshot at a time and accumulating in fp32 or wider. Use
the result where the run’s params go:
merge(variables, {"params": checkpoints.posthoc_ema(...)}).
Checkpoints.save_local
Section titled “Checkpoints.save_local”def save_local( step: int, state: TrainState, saved: bytes | None, *, share: DataPartition | None = None, control: dict | None = None, rung: JSON = None, artifact: JSON = None, boxed: Mapping[tuple[str, ...], LogicalAxes] | None = None, rollout: Mapping[str, object] | None = None,) -> NoneWrite state under step to this process’s local directory, asynchronously, with its
rollout’s tree where it holds one (rollout, replicated: every process writes it whole).
The new step replaces the previous local step. As with save, a state
with arrays in pinned host memory is written before this returns. The
placement is saved with the state, so a resume onto a different
placement raises before it reads shards from directories that do not
hold them.
Checkpoints.rung
Section titled “Checkpoints.rung”def rung(step: int) -> JSONReturn the rung of fit’s ladder that the state at step trained on.
The value is what save recorded, read from the directory restore
reads the step from. It is None for a state saved outside fit,
which trained on no ladder.
Checkpoints.boxed
Section titled “Checkpoints.boxed”def boxed(step: int | str | None = None) -> dict[tuple[str, ...], LogicalAxes]The axes the model at step boxed on its own parameters, as save
recorded them; a checkpoint saved without them has none.
Checkpoints.variables
Section titled “Checkpoints.variables”def variables( *, step: int | str | None = None, ema: bool | None = None, mesh=None, layout=None, param_dtype: jax.typing.DTypeLike | None = None, parameter_roots: tuple[tuple[str, ...], ...] = PARAMETER_ROOTS, unread: tuple[tuple[str, ...], ...] = (),) -> VariablesRead the selected step’s live or averaged variables onto the requested layout.
Parameter storage conversion applies only to the owner’s parameter
roots; other collections retain their recorded dtypes and placement.
The subtrees at unread’s paths are neither placed nor read
(dew._records.UNREAD).
Checkpoints.materialized
Section titled “Checkpoints.materialized”def materialized( template: Mapping[str, Variables], placement: Mapping[str, Variables], *, step: int, param_dtype: jax.typing.DTypeLike | None = None, parameter_roots: tuple[tuple[str, ...], ...] = PARAMETER_ROOTS,) -> VariablesThe variables at step whose leaves template names (stored’s, under "variables", and
under "ema" the averaged ones, merged over them), restored onto placement’s shardings, no
other leaf read; the floating leaves under parameter_roots read as param_dtype where one is
given, the rest in their stored dtypes. Every load of a run’s variables restores through it: a
whole run’s onto its layout (variables) and a bank’s rows onto theirs
(dew.inference.banks.CheckpointBanks).
Checkpoints.stored
Section titled “Checkpoints.stored”def stored(step: int | str | None = None) -> VariablesReturn what the checkpoint at step holds, without reading its values.
step defaults to the latest. Each state field comes back as a
shape/dtype tree, and an unset field as None.
Checkpoints.accumulation_template
Section titled “Checkpoints.accumulation_template”def accumulation_template(step: int)Return the persisted pending-array shapes, a DiLoCo run’s groups’ stacked, without reading them.
Checkpoints.position
Section titled “Checkpoints.position”def position(step: int, share: DataPartition) -> bytes | NoneThe data position the reader of share resumes from at step (read_position), read
alone, before any state is restored: a run opens its data there and reads its first batch
while its state is still unplaced (Trainer.fit). None where the step saved no position.
Checkpoints.rollout_state
Section titled “Checkpoints.rollout_state”def rollout_state(step: int, template: Mapping[str, object]) -> dict | NoneThe rollout’s tree the step at step holds (save’s rollout), read alone and placed as
template’s leaves (jax.ShapeDtypeStructs) are; None for a step saved without one.
Checkpoints.restore
Section titled “Checkpoints.restore”def restore( template=None, step: int | str | None = None, *, share: DataPartition | None = None, served: Sequence[ProjectionGroup] = (),)Restore the state at step and the data position of share.
template is a pytree of jax.ShapeDtypeStruct naming the state
leaves to restore. When a leaf has a sharding, the array is placed
there, so a checkpoint written on one mesh restores onto whatever mesh
this run uses. None restores every leaf as a host array. A template
leaf that the checkpoint lacks raises an error naming it, unless the
template holds it as a concrete array, which is then kept as it is.
A step that is the local one every process holds is read from the
local directory, onto the placement it was written with; any other
step is read from the persistent directory. The data position is
returned as the bytes the reader of share resumes from (see
read_position). Without a share, as for a caller that reads weights
and no data, it is None.
served names the projection groups a step may hold packed where the
template holds their members (Objective.served_groups): each such
leaf, in any of the state’s trees, is read packed and split back into
its members (dew.checkpoints.served.unpacked). Carried to 1.0, when it is deleted (ledger
1552).
Checkpoints.close
Section titled “Checkpoints.close”def close() -> NoneWait for pending writes, then close the Orbax managers and the threads and metadata stores
they hold, each even where another, or its own wait, failed: Orbax reports a failed
asynchronous write once, from the wait, and its close still releases the manager. The first
failure is raised with the rest noted on it. A closed Checkpoints reopens its managers on
its next use.
Checkpoints.wait
Section titled “Checkpoints.wait”def wait() -> NoneBlock until pending async writes have landed on disk.
Saving is async so it stays off the training loop’s critical path; anything that reads a checkpoint back has to call this first.
class Keep( latest: int = 2, every: int | None = None, interval: datetime.timedelta | str | None = None, where: Callable[[Kept], bool] | None = None,)Keeps the union of the latest steps, periodic steps, wall-time-spaced steps and a predicate’s steps.
interval keeps checkpoints at least this much wall time apart; it does
not keep every old checkpoint. where receives a retained-step record
with its metrics.