Core API
This page describes the main interfaces and the contracts between them, grouped by task. The site generates the signature and field blocks below from source. Every public module also has a page generated from its docstrings, listed at the end of this page and in the sidebar. The Quickstart uses the core interfaces in one script.
Objective
Section titled “Objective”Import Objective, Aux, Step and Ratio from dew.objectives. Ratio.mean() reduces a ratio statistic, and objective.scalar_loss(variables, batch, step) evaluates and reduces a loss for direct differentiation. dew.Supervised trains any Flax model on a per-example loss(outputs, batch) without a subclass (Custom objectives).
def Objective.init(key: jax.Array, variables: Variables | None = None) -> Variables
def Objective.loss( variables: Variables, batch: Batch, step: Step,) -> Loss | tuple[Loss, Aux[Effects]]
def Objective.reduce_loss(stats: Loss) -> tuple[jax.Array, jax.Array]
def Objective.with_gradients(stats: Loss, gradients, params: Variables) -> Loss
def Objective.apply_effects(variables: Variables, effects: Effects) -> Variables
def Objective.evaluate(params: Variables, batch: Batch, step: Step) -> Artifacts | None
def Objective.preview( params: Variables, batch: Batch, step: Step, *, scored: Artifacts | None = None,) -> Artifacts | None
def Objective.optimizer( tx: optax.GradientTransformation,) -> optax.GradientTransformation
def Objective.averages(update: jax.Array) -> jax.Array
def Objective.scalar_loss( variables: Variables, batch: Batch, step: Step,) -> tuple[jax.Array, Aux[Effects]]
class Supervised( model: nn.Module | Source[nn.Module], loss: Criterion, metrics: Sequence[Criterion] = (), *, inputs: InputSpec | str, mode: str | None = None, work: Callable[[Batch], float] | None = None,)| Member | Contract |
|---|---|
init | Return a Flax variables mapping with a params collection. Pure; the trainer traces it once for shapes and once for values. variables is a held tree the caller supplies, which is how the trainer passes it as data; with None the objective uses its own (DiffusionObjective.held_variables, for example). An objective that holds nothing ignores it. |
loss | Return additive statistics and Aux. Use Ratio(total, mass) for a shared denominator; a scalar denotes a unit-mass term. |
reduce_loss | Return (value, has_data). Override for an objective-owned composite Flax PyTree. |
with_gradients | Return stats whose derivative in params is gradients, for a loss that states its own gradient rule. |
apply_effects | Return nonparameter replacements from additive accepted-window observations. Required when the objective emits effects. |
evaluate | Return an artifact, a tuple of artifacts, or None. The base method returns None. |
preview | Return display artifacts for a tracker, or None. The base method reuses scored, the first scoring artifacts. |
ema | Optional EMASpec; the base objective uses None. |
optimizer | Return the optimizer Trainer steps params with, from the one it was handed; the base method returns tx. Per-network optimizers use optax.multi_transform and optax.conditionally_mask. |
averages | Return whether the update after update committed ones moves the EMA; the base method averages every update. |
class Step( training: bool = struct.field(pytree_node=False, default=True), committed: jax.Array | None = None, example_key: jax.Array | None = None, example_index: jax.Array | None = None, example_count: int | None = struct.field(pytree_node=False, default=None), tape: Mapping[str, jax.Array] | None = None, recorder: Recorder | None = struct.field(pytree_node=False, default=None), record: bool = struct.field(pytree_node=False, default=False), witness: Witness | None = struct.field(pytree_node=False, default=None), qk_stats: bool = struct.field(pytree_node=False, default=False), equations: bool = struct.field(pytree_node=False, default=False), watched: Watched | None = struct.field(pytree_node=False, default=None),)
class Aux( variables: Variables | None = None, qk_stats: Variables | None = None, effects: Effects | None = None, inspected: Variables | None = None, record: Evaluated | Recorded | None = None,)Step.step counts accepted microbatches. Its key derives from consumed attempts, including rejected ones. ema holds selected averaged leaves overlaid onto the complete variables mapping, or None.
Aux holds training measurements, sequential mutable replacements, QK maxima and additive deferred effects. The trainer applies the effects once per supported optimizer commit. objective.scalar_loss(variables, batch, step) returns a scalar and the same Aux for direct JAX differentiation.
Categorical log-likelihoods
Section titled “Categorical log-likelihoods”dew.nn.categorical is the one owner of a categorical distribution’s log-likelihood over the last axis: a model’s logits over its vocabulary, a classifier’s over its classes, a decision’s over a question’s options.
def log_probs(logits: jax.Array, where: jax.Array | None = None) -> jax.Array
def probabilities(logits: jax.Array, where: jax.Array | None = None) -> jax.Array
def cross_entropy( logits: jax.Array, labels: jax.Array, where: jax.Array | None = None,) -> jax.Array
def soft_cross_entropy( logits: jax.Array, targets: jax.Array, where: jax.Array | None = None,) -> jax.Array
def kl_divergence( logits: jax.Array, reference: jax.Array, where: jax.Array | None = None,) -> jax.Array
def entropy(logits: jax.Array, where: jax.Array | None = None) -> jax.Array
def rescaled(maximum: jax.Array, total: jax.Array, larger: jax.Array) -> jax.Array
def target_loss(peak: jax.Array, mass: jax.Array, target: jax.Array) -> jax.Array
def segment_log_partition( logits: jax.Array, segments: jax.Array, count: int,) -> tuple[jax.Array, jax.Array]Each computes in at least float32 and keeps float64 in float64. where excludes the classes a row does not hold; they take no probability and add no term, in value or gradient. rescaled, target_loss and segment_log_partition are the pieces of a partition read in blocks or over ragged supports, as dew.objectives.lm.chunked reads its vocabulary.
Collections and EMA selection
Section titled “Collections and EMA selection”A variables tree is a nested mapping. Its outer keys name collections such as params and batch_stats; leaves are arrays such as a dense kernel or a running mean. The optimizer updates the params collection. A mutable Linen call returns replacement state collections, which the objective supplies through Aux.variables. See the BatchNorm example.
class EMASpec( decay: optax.Schedule, select: PathFilter = everything, progressed: bool = False,)EMASpec comes from dew.objectives.base. decay maps completed optimizer-update count to a scalar. select accepts a tuple of keys naming a leaf; under("params", "context_encoder") selects that subtree, and everything selects all leaves. EMA arithmetic uses at least fp32 and preserves explicit fp64, then rounds each result to the initialized EMA leaf dtype. Unit decay selects the frozen leaf exactly. Router bias updates also retain the initialized bias dtype; integer load comparisons avoid converting large counts to floats.
Input descriptions
Section titled “Input descriptions”Import Field, Condition, and InputSpec from dew.inputs or dew.
class Field(key: str, shape: tuple[int, ...], variable_axes: tuple[int, ...] = ())
class Condition( encoder: ConditionEncoder, field: str = 'text', unconditional: str | float | Mapping[str, object] = '',)
class InputSpec( sample: Field, conditions: Mapping[str, Condition] = dict(), mask: Field | None = None,)Field.shape is the shape of one sample, excluding batch. A Condition connects a condition encoder to its tokenized batch field. InputSpec.conditions maps the model keyword, such as textcontext, to a Condition. Each condition must read a distinct batch field. mask is an optional Field for a binary image mask, used for explicit masked-image latent conditioning. InputSpec.tokenize(captions) returns tokenized fields for the conditions whose encoder reads captions (reads_captions); an empty condition mapping returns no fields. An encoder of another modality, such as HFAudio, reads the field its dataset writes. Encoders define tokenization, parameters, and encoded output types.
Mesh and layout
Section titled “Mesh and layout”Import these from dew.training:
class MeshSpec( fsdp: int = 1, expert: int = 1, tensor: int = 1, sequence: int = 1, stage: int = 1, replicas: int = 1, diloco: int = 1, explicit: tuple[str, ...] = (),)
class Layout( rules: LogicalAxisRules | Mapping[str, MeshAxes] = DEFAULT_RULES, min_shard: int = 2 ** 16, tolerance: float = 0.02, host: tuple[str, ...] = (), host_parameters: tuple[str, ...] = (), frozen_whole: bool = False, stage_local: bool = True, data_axes: tuple[str, ...] = (),)
def MeshSpec.build(devices: Sequence | None = None) -> MeshMeshSpec.build uses the supplied devices, or JAX’s visible devices. The specified factors must divide the device count, and data parallelism takes the remaining factor. Trainer(microbatches=...) sets a pipelined step’s microbatches, a positive multiple of a stage axis above 1; a trainer also takes a native jax.sharding.Mesh in place of a MeshSpec (distributed). With replicas above 1, build makes a hybrid mesh whose data axis spans that many groups of granules (TPU slices, GPU hosts or NVLink domains, or processes whose devices all share one slice), and every other axis stays inside a group; see training on several nodes. A sequence axis above 1 splits every attention call’s positions, and each call chooses between the all-to-all and the gather exchange from its shape.
Layout has these fields:
rulesis an ordered sequence of(logical axis, mesh axes)pairs, or a mapping, which becomes those pairs in its order. Either replaces the default table,dew.mesh.DEFAULT_RULES, whole. When dimensions compete for one mesh axis, rule order decides which one gets it, and a dimension the axis size does not divide cannot use that axis. A rule may name any axis of the mesh, a native mesh’s own names included. A parameter splits over any axis butdataandstage, whose placementshardingsrefuses; what the rules put onfsdpalso splits oversequence(distributed). A rule naming an axis the mesh lacks is refused, unless it is one of the default table’s own rules, whose axes a mesh without them treats as size 1. To change one name’s placement and keep the rest, drop the defaults’ rules for that name and put yours ahead of the others, as the distributed page shows. The defaults name some logical axes more than once, as fallbacks, and a mapping holds one rule per name, so a mapping built from them drops those fallbacks.data_axesnames the axes a batch’s rows split over, ahead of the rules’ ownactivation_batchentries; the run’s data shares, its batch placement and its row counts read them. Empty leaves the rules as they are.min_shardis the size, in elements and not bytes, below which a parameter stays replicated, because sharding it would cost more in collectives than it saves in memory.toleranceis the fraction of shardable parameter elements a layout may leave replicated beforecheckrefuses it.hostnames train-state fields out ofvariables,opt_stateandema; any other name is refused. A namedopt_stateoremastays in pinned host memory between steps, and the step fetches it to the device. Namingvariablesinstead puts the wholeTrainStateon the CPU, including optimizer state, EMA and accumulation. The optimizer transaction then runs on a CPU companion of the mesh, and on every process the runtime CPU device count must match the accelerator count before JAX initializes.host_parametersholds glob patterns over logical parameter paths (params/layers_*) that an inference placement keeps in pinned host memory. Only theoffloadedplacement reads them, andcheckrefuses a layout that names them for any other placement.
shardings(mesh, tree) returns a matching tree of placements, and check(params, shardings, mesh) checks for excessive replication. validate(mesh, state, capacity=None) reads a state as it will be placed, its leaves arrays or jax.ShapeDtypeStructs that carry their shardings, and returns a Validation: each leaf’s spec, shard shape and memory kind, the bytes each device holds between steps by device id, the axes a batch’s rows split over, the collectives a step runs over each mesh axis by the role the placement gives it, and the logical axes no rule names. Rules naming an axis the mesh lacks are refused there as in shardings, and it raises LayoutRefused when a device would hold more than its capacity, the device’s allocator bytes_limit unless capacity states it. See distributed training.
Trainer
Section titled “Trainer”Import Trainer from dew or dew.training.
class Trainer( objective: Objective[Loss, Effects], optimizer: optax.GradientTransformation | OptimConfig, *, key: int | jax.Array, mesh: MeshSpec | Mesh = _DEFAULT_MESH, layout: Layout = _DEFAULT_LAYOUT, microbatches: int | None = None, accumulation: int = 1, split: int | None = None, dynamic_scale: bool | dynamic_scale_lib.DynamicScale = False, checkpoints: Checkpoints | None = None, tracker: Tracker | None = None, step: Callable[[Objective[Loss, Effects], optax.GradientTransformation], StepFn] | None = None, rollout: Rollout | None = None, profile: ProfileWindow | None = None, lower_for: str | None = None, ahead: AheadOfTime | None = None, tune_kernels: bool = False, record: bool = False, links: Mapping[str, Link] | None = None, inspection: Inspection | None = None, param_dtype: str | None = None, diloco: DiLoCo | None = None,)objective is an initialized objective object, and optimizer is an Optax gradient transformation or a dew.optim.OptimConfig. fit builds a config over the optimizer updates its steps make, steps // accumulation, so the config’s schedule spans that run; OptimConfig.weight_decay says which parameters its decay spares. The required JAX key seeds initialization and the run. mesh, a MeshSpec or a native jax.sharding.Mesh, and layout describe placement; microbatches is how many a pipelined step feeds through the mesh’s stage axis, one a stage when unset. The optional objects turn on checkpoints, tracking, host-side rollouts and profiling.
accumulation counts accepted microbatches per effective window. Shared means use a weighted gradient accumulator with at least fp32 precision, and keep float64 when it is enabled in JAX. A TPU has no float64 (XLA rewrites it into pairs of float32, which are not IEEE doubles), so a trainer whose parameters are stored in float64 on a TPU mesh is refused when it places the state. Each finalized gradient enters Optax in its parameter’s dtype, and partially accumulated gradients keep the wider working dtype.
Composite statistics keep independent normalizers, and the inputs for scalar-VJP replay. Parameters and the EMA stay fixed within the window, and sequential mutable replacements keep the snapshots they originally read. dynamic_scale=True persists the scale and its history, and rejects nonfinite working or optimizer-input gradients. A rejection drops its whole accumulation window, as torch’s AMP skips the whole update: the window’s accepted microbatches go with it, the next microbatch begins a window, and the mutable collections they wrote keep what they wrote. A batch ramp’s new stage drops a window a rejection left part filled, so every update pools one stage’s batches.
validate(batch=None, steps=None, capacity=None) runs the checks fit runs before it places the state, on the state a fresh run would place, without placing it. Given one global batch, the state includes the gradient accumulator accumulation pools. An OptimConfig is built over steps, as fit builds it. capacity states each device’s bytes for a mesh whose devices report none, such as CPU devices standing in for GPUs.
A custom step(objective, optimizer) factory is responsible for the accepted and update clocks, the scaler, the EMA and mutable writes. Its body returns (state, loss, aux), and the common compiled wrapper advances the attempt counter state.step. A host rollout produces the actual training arrays once per consumed attempt, before differentiation or replay.
def Trainer.fit( dataset: Dataset | Loader | Handed, *, steps: int | None = None, epochs: int | None = None, tokens: int | None = None, examples: int | None = None, log_every: int | str | Counted = 100, eval_every: int | str | Counted | None = None, checkpoint_every: int | str | Counted | datetime.timedelta | None = None, checkpoint_final: bool = True, metrics: Sequence[Metric] = (), preview: bool = False, scored_to: str | os.PathLike[str] | None = None, best: str | Metric | TrainingScalar | types.MethodType | Best | Sequence[Best] | None = None, stop: Plateau | Pause | None = None, validation: HandedSplits | None = None, summaries: Sequence[Summary] = (), restore_best: bool = False, state: TrainState | None = None, observe: Observe | None = None, device_prefetch: int = PREFETCH_DEPTH, resume: Resume = 'auto', cooldowns: Cooldowns | None = None,) -> TrainStatedatasetis aDataset.stepsis the final target, including a restored step count.log_everyis the interval between training log lines.eval_every=Nonedisables evaluation. With an interval, evaluation also runs at the end.checkpoint_everycontrols periodic saves when a checkpointer exists. The normal completion path can still save a final checkpoint when a checkpointer is present.- Each of the three, an
EvalSuite’s owneveryandCheckpoints(local_every=)may be a share of the run such as"5%", resolved once against its planned length: the whole updates nearest that share of its steps, epochs or examples, at least one, or for a token budget alone each time the committed tokens pass a multiple of that share. A schedule’s warmup (Cosine,PowerandLinear’swarmup_steps,OneCycle’swarmup_fraction) takes a share too, of the run’s updates. The run’s start records what each resolved to, and the warmups of the schedules its optimizer runs (FitStarted.cadences). metricsreduce evaluation artifacts and require evaluation to be enabled.cooldowns=Cooldowns(at=("20%", "40%"), length="20%")branches the run into cooldowns at those points (dew.training.cooldown): each branch trains on from the run’s state and data at its rate times a decay to zero, is scored by every validation suite ascooldown/<split>/<metric>at the step it branched from, and is discarded; the run trains on as one with none.--trainer.cooldowns.atsets it on a run’s command line.preview=Trueexplicitly requests generated/display artifacts during evaluation; adding a scalar tracker alone does not request them.
Checkpointable data must supply the consumed iterator position. A failed scaled transaction advances attempted work and scaler history while preserving the earlier accepted prefix. Completely inactive windows close accepted slots without an optimizer call, weight decay, EMA, or deferred effects. Auxiliary-only windows can still be active.
fit is responsible for closing the iterators the dataset returns. It closes training read-ahead before the final evaluation, and closes validation passes on success or failure. A restored fit that is already at its target does no data, evaluation, compilation or save work. On exit it stops its trace and waits for pending checkpoint writes, but it does not close the checkpointer or the tracker, which it only borrowed. Cleanup failures are attached to the primary error.
Initialize, restore, and compile
Section titled “Initialize, restore, and compile”initial_state() constructs an unplaced initial TrainState. place() returns (state, shardings, position), restoring from the configured checkpointer when available. Eager and placed initialization can differ in low floating-point bits across backends; compare the actual path used by your run.
An OptimConfig is built by fit over the run’s optimizer updates. If you need initial_state, place or compile before fit, pass OptimConfig(...).build(steps) to Trainer instead, where steps is the run’s length in optimizer updates.
OptimConfig.optimizer names one of Dew’s optimizers (adam, adamw, lamb, muon, muonclip, laprop) or any Optax optimizer factory by its import path, OptimConfig(optimizer="optax:sgd", optimizer_opts={"momentum": 0.9}), built as factory(learning_rate, **optimizer_opts) with the config’s b1, b2 and weight_decay where set, so a recorded run (--optim.optimizer optax:sgd) trains on the transformation an experiment ran. laprop is DreamerV3’s optimizer: the config’s b1 and b2 are its momentum’s and mean square’s decays, optimizer_opts carry its eps and its adaptive gradient clip agc (0 clips nothing), and it takes no weight decay.
def Trainer.initial_state( initializer: Initializer | None = None, key: int | jax.Array | None = None,) -> TrainState
def Trainer.place() -> tuple[TrainState, Placement[TrainState], bytes | None]
def Trainer.compile( state: TrainState, batch: Batch, trials: Trials | None = None, *, capacity: int | None = None,) -> CompiledStep
class CompiledStep(Protocol)compile returns a callable whose type is CompiledStep. The scaler is part of TrainState. loss_finite and accepted are separate, because a finite scalar loss can come with a nonfinite gradient that is rejected. The final value carries transaction commit telemetry, or None for a custom step. The callable consumes the state it is given (donate_argnums=0), and the returned state takes over its buffers, so write new = compiled(old, batch) and keep no reference to the old state. The batch is not donated, because the loader still owns it.
TrainState
Section titled “TrainState”Import TrainState from dew.state.
class TrainState( variables: Variables = struct.field(metadata={'variables': True}), ema: Variables | None = struct.field(metadata={'variables': True}), accumulation: Accumulation | None = None, consumed: jax.Array | None = dataclasses.field(default=None, metadata={'optional': True}), deferred: jax.Array | None = dataclasses.field(default=None, metadata={'optional': True}), inner: Replicas | None = dataclasses.field(default=None, metadata={'optional': True}), compute: Variables | None = struct.field(default=None, metadata={'checkpointed': False, 'variables': True}),)step counts completed attempted batches, including rejected work. microstep counts accepted microbatches, including finite zero-support slots. updates counts committed supported optimizer updates.
variables holds the complete Flax variables, including the inner params collection. opt_state is the underlying optimizer state, advanced only on commits. ema holds selected moving-average variables or None, and key is the immutable root training key.
scale is the dynamic loss scaler with finite-history state, or None. window_size is the persisted accumulation length, checked on restore. accumulation holds the actual partial gradient/statistic/effect/replay state, or None.
state.averaged overlays EMA leaves onto state.variables and raises when no EMA is configured. For fixed window length K, the accepted fill is microstep % K. A partial window is checkpointed without flushing.
Dataset, Loader and Loading
Section titled “Dataset, Loader and Loading”Import sources, Dataset, the step modules, presets and Loading from dew.data, and compile from dew.data.compile. Loader and DataPartition are in dew.data.dataset.
class Dataset
def lm( dataset: Dataset, tokenizer: str | SavedTokenizer | None, *, seq_len: int, batch: int | None, field: str = 'text', eos: bool | None = None, pack: Literal['first-fit', 'best-fit', 'concat'] | None = PACK_POLICY, bins: int = PACK_BINS, stride: int | None = None, cache: str | Cached | None = None, records: int | None = None, host: tuple[HostStep, ...] = (),) -> Pipeline
def images( dataset: Dataset, size: int, *, batch: int | None, augment: bool | image_steps.DeviceAugment | RandomStep | tuple[RandomStep, ...] = False, crop_scale: tuple[float, float] = (1.0, 1.0), field: str = 'image', staging_size: int | None = None, draft: bool = False, rejects: bool = True, caption: image_steps.Caption | None = None, cache: str | Cached | None = None, records: int | None = None, host: tuple[HostStep, ...] = (),) -> Pipeline
def compile( train: Pipeline, *, validation: Pipeline | None = None, host: tuple[HostStep, ...] = (), objective: ObjectiveData | None = None, read: Read | None = None, provision: bool = False, loading: Loading | None = None, seed: int = 0, rejections: Literal['drop', 'mask'] = 'drop', resume: Literal['exact', 'non-exact'] = 'exact', examples: int | None = None, widths: Mapping[str, object] | None = None, measured: _Measured | None = None, epochs: int | None = None,) -> Loader
class Loader( train: Reader, val: Reader | None, records: int | None, batch: int, ramp: Ramp | None = None, held_out: int = 0, counts: Mapping[str, CorpusCounts] = dict(), validation_counts: Mapping[str, CorpusCounts] = dict(), plan: Plan | None = None, statistics: Callable[[], Mapping[str, float]] = _data_statistics, batch_tokens: int | None = None, device: tuple[DeviceStep, ...] = (), widths: Mapping[str, object] = dict(), tail: Tail = 'dropped', order_records: int | None = None, examples: int | None = None, pass_examples: int | None = None, epochs: int | None = None, bound_ends: bool = True, token_ends: tuple[int | None, ...] | None = None,)
class DataPartition( index: int = 0, count: int = 1, readers: int = 1, reader: int = 0, row_multiple: int = 1, window: int = 1,)
class Loading( workers: int | Literal['auto'] = 0, threads: int = 16, read_buffer: int = 16, worker_buffer: int = WORKER_BUFFER, ram_budget_mb: int | None = None, max_workers: int | None = None, cache: bool = True,)A source is already a Dataset, and every verb returns a new one. A preset of dew.data.presets (images, lm, sft, prompts, preferences, canvas or clips) is a function from a dataset and its settings to a dataset, applied with pipe. It ends in batch, so its result goes to Trainer.fit, which compiles it into a Loader. Validation is a second dataset, fit(..., validation=), usually a part of holdout through the same preset; nothing is held out unless the chain says so.
batch(tokens=N) selects token-budget batching: rows of the fixed length a record step declares (TokenGeometry), as many as N tokens hold. The compiled Loader exposes batch_tokens. Row and token budgets are mutually exclusive. A run configuration takes the micro-batch from its chain and update_tokens from TrainerConfig, and records the derived update/micro accumulation.
A Dataset is immutable. Its methods add ordered stages: map, random_map, map_with_index, filter, filter_map, random_filter_map, flat_map, shuffle, repeat, slice, mix, interleave, windows, pack, batch, workers, prefetch, take and cache; a map after batch is a host step on the training batch. batch takes the global row batch. Host batch steps run after workers in the training process. compile(..., host=(step,)) applies common steps to both branches.
Sources implement Source: Memory holds records or columns, Files opens a format over URLs, Hub opens prepared Arrow or physical Hub Parquet shards, TFDS opens prepared ArrayRecords, and from_torch adapts map-style PyTorch samples with explicit tuple field names, as from_grain, from_hf and from_iter adapt Grain datasets, Hugging Face datasets and resumable iterators. A DataLoader keeps its own sampler and collator and is refused.
read="memory" materializes deterministic transforms and packing. read="indexed" requires a producer index or provisioning. read="streamed" reads physical shards. The mode applies to every leaf, including validation; with no read, the sources choose it. Trainer.fit provisions what the read needs, logging each artifact it prepares; under compile’s default, provision=False, a missing artifact is a named error.
Objective.data() returns ObjectiveData, holding train and evaluation record steps and host batch steps for each branch. ObjectiveData.pack optionally packs those records into fixed windows, and ObjectiveData.packed runs record steps on each window before batching. It does not change Objective.inputs. Metadata-dependent steps use Configured; the compiler binds their declared facts from the source.
holdout(records=N) reserves the source’s head, holdout(f) splits immutable record identities, and holdout(split=name) opens a named split. A second dataset of your own is also valid validation. A preset’s records= caps accepted records before the share cut.
train(partition) opens a new training iterator. val(partition) opens one finite pass, or is None without validation. Its final partial batch is filled with repeats marked by valid_rows=False. DataPartition.of(mesh) selects the process’s share; DataPartition() reads all rows.
records is the known accepted training-record count or None. batch is global. steps_per_epoch and epoch_steps require an authoritative finite count. The compiled plan records source snapshots, transforms, ordering, derived order and packing facts, cache generations and topology constraints.
A global count-preserving or memory-materialized plan resumes across process counts dividing its batch. A streamed, rejecting, expanding or online-packed plan saves exact Grain iterator state per share and refuses topology changes. Volatile fetches cannot claim exact resume before their bytes are persisted.
workers(n), threads(n), prefetch(n) and loading(Loading(...)) change concurrency without changing record order or draws. A chain that names none reads in the training process (workers=0); Loading(workers="auto") sizes the workers and read buffer from the CPUs the process may run on and the memory it may take. A multiprocessing program needs a main guard.
Each factory returns an exclusively owned iterator. Final close() must not race iteration or checkpoint operations. request_stop(), where supplied, is a nonblocking signal safe alongside both. The compiler’s host stages forward ownership and position.
DevicePrefetchIterator(iterator, mesh, depth=2, source_state=None) in dew.distributed.batches takes ownership after successful construction. It starts reading and restoring on the first next(); closing an unused iterator only finalizes it. Use it in a context manager or call close(timeout=5.0).
Depth is positive. The queue contains at most that many device batches plus one in flight. Saved source state names the last delivered batch, not speculative read-ahead. EOF or failure follows already queued batches; early close discards unread work and reports finalization errors.
The prefetch worker owns iteration, restore and finalization. A timeout leaves cancellation requested and permits retrying close; it does not kill a running source. Grain’s read executor does not promise to stop a blocked upstream read, so that remains outside Dew’s shutdown guarantee.
export writes committed Parquet or ArrayRecord shards and indexes. publish fills a deterministic cache prefix and certifies one generation. A filling run never adopts a newly published generation implicitly; a later run names Generation explicitly. See training data for layouts, sources and publication.
Checkpoints
Section titled “Checkpoints”Import Checkpoints from dew.
class Checkpoints( directory: str, *, keep: int | Keep = Keep.latest, local_directory: str | None = None, local_every: int | str | None = None,)
def Checkpoints.save( step: int, state: TrainState | Mapping[str, object], saved: bytes | None = None, metrics: Mapping[str, float] | None = None, *, share: DataPartition | None = None, ranking: Ranking | Sequence[Ranking] | None = None, control: dict | None = None, weights_only: bool = False, primary: str | None = None, rung: JSON = None, artifact: JSON = None, boxed: Mapping[tuple[str, ...], LogicalAxes] | None = None, rollout: Mapping[str, object] | None = None,) -> None
def Checkpoints.wait() -> None
def Checkpoints.restore( template=None, step: int | str | None = None, *, share: DataPartition | None = None, served: Sequence[ProjectionGroup] = (),)
def Checkpoints.path(step: int) -> strCheckpoints configures persistent storage and optional local emergency checkpoints. Local directory and cadence must be specified together. The object opens storage on use.
save schedules a persistent save. saved is the iterator position as bytes, or None, and share is the DataPartition that the stream read; a position requires a share. wait() waits for pending writes, and latest reports the newest eligible checkpoint. restore returns the restored state data and the iterator position that the reader of share resumes from, or None for the position when no share is given. A template controls structure and placement. path(step) gives the persistent checkpoint location.
This object does not write run.json; run configuration saving is separate. Use the complete resume example before adapting these lower-level calls.
def averaged( runs: Sequence[str | os.PathLike[str]], *, step: int | str | None = None, ema: bool | None = None, mesh=None, layout=None, trust: Sequence[str] = (),) -> Averaged
def averaged_pinned( runs: Sequence[SavedRun], *, ema: bool | None = None, mesh=None, layout=None,) -> Averaged
class Averaged( variables: Variables, model: ModelConfig, runs: tuple[tuple[str, int, bool], ...],)dew.checkpoints.averaged(runs, step=None, ema=None, mesh=None, layout=None) averages runs trained apart from one model, of any objective, in fp32, each read onto mesh as layout places it; a pool’s processes each take the mean of the shards they hold. A LoRA run’s factors are merged into its kernels first. It returns an Averaged: the mean variables, the model record they apply to (its adapter merged away), and each run’s directory, step and whether its EMA was read. Runs of another model are refused. averaged_pinned(runs, ema=None, mesh=None, layout=None) averages runs pinned already (SavedRun.pinned), as Decide.averaged does with the runs whose records it compared.
Language modeling and generation
Section titled “Language modeling and generation”Import LMObjective from dew.objectives.lm.
class LMObjective( model, seq_len: int, *, ema_decay: float | None = None, pad_id: int | None = None, head_chunks: int = 4, head_tile: tuple[int, int] | Literal['whole', 'tiled'] | None = None, samples: Samples | None = None, variables: Variables | None | Omitted = OMITTED, balance: ExpertBalance | None = None, loss_role: Role | None = None, prefix: int | Literal['prompt'] | None = None, mtp_weight: float | None = None, z_loss: float = 0.0, indexer: IndexerTraining | None = None, token_accuracy: bool = True, processor: Processor | None | Omitted = OMITTED, adaptation: ChunkAdaptation | None = None,)
class IndexerTraining(phase: Literal['warmup', 'sparse'], weight: float = 1.0)model may be a decoder bundle that Pretrained.load returned. The objective then reads the bundle’s model, initial variables and processor, and pipeline decodes with that processor; passing variables= or processor= as well overrides that part. An adapted bundle (bundle.adapt(LoRA(...), key=)) trains only the adapter’s factors, and its save and export merge them into the kernels. The model must implement Linen hidden_states(tokens, train=..., coordinates=...), its tokens’ layout one dew.batch.Coordinates (their documents, validity and positions), returning (B, S, D), and output_table(), returning the OutputTable (the stored vocabulary matrix and its orientation, bias, softcap and precision) that its logits contract (dew.nn.protocols). A decision head that reads the vocabulary’s rows (the JointSchemaHead’s lexical prior) needs vocabulary_rows() (VocabularyRows): the head’s [vocab, features] rows, OutputTable.rows where one matrix is the head, and past a nonlinear prediction head or an adapter on the head the rows its states meet. Prediction-depth training needs mtp_hidden_states and compatible prediction-depth configuration. Mutable router, QK and indexer collections are required when their options are enabled.
variables is the tree training starts from, either whole or split by dew.batch.freeze or an adapter into the part that trains and the part that stays frozen. loss_role requires aligned text_roles. pad_id masks matching targets. head_chunks controls vocabulary tiling, and head_tile the head’s backward tile ('whole', 'tiled' or a tile shape; None picks one for the objective); samples configures text previews. ema_decay defaults to None, which trains without an averaged copy; a decay such as 0.999 keeps one that evaluation and previews read, and 1.0 retains a frozen one. Routing balance, auxiliary loss, prediction-depth weight, and QK statistics require matching model computation. These interfaces make LMObjective specific to compatible decoders. z_loss adds PaLM’s auxiliary term, the coefficient times the squared log partition of every counted prediction; zero adds nothing. router_z_loss is the routers’ own z-loss (ST-MoE), and zero adds nothing. token_accuracy=False drops the token_accuracy metric and the pass over every logit it costs.
indexer trains DeepSeek-V3.2’s lightning indexer on a model whose mla mixer names index_n_heads and index_head_dim. IndexerTraining("warmup") needs a mixer without index_topk. The model then runs dense attention, init puts only the indexer in params and the rest of the tree under frozen (a variables tree may omit the indexer, as a dense checkpoint does), and the loss is the KL of the indexer’s softmax from the attention distribution, reported as indexer_kl. IndexerTraining("sparse") needs a mixer with index_topk. The whole tree trains: the cross entropy trains the main weights, and the KL over the selected keys trains the indexer, whose inputs are detached. A warm-up checkpoint’s split tree is accepted as variables. weight scales the KL term.
Import ExpertBalance from dew.objectives.routing.
class ExpertBalance( balance_rate: float | None = None, aux_loss_alpha: float | None = None, seq_aux: bool = True, router_z_loss: float = 0.0,)ExpertBalance holds how a run trains its routers. LMObjective(balance=...), MaskedDiffusionObjective(balance=...), GRPOObjective(balance=...), DPOObjective(balance=...) and PPOObjective(balance=...) compose it: balance_rate moves each balancing bias against its load at every committed update (it needs a mixture with bias=True), aux_loss_alpha and seq_aux add DeepSeek V2’s balance loss, and router_z_loss adds ST-MoE’s z-loss. Each term is optional.
Import generate, Sampling and Generation from dew.sampling:
def generate( model: nn.Module, params: Variables, inputs: ModelInputs | ArrayLike | Sequence[Sequence[int]], max_new_tokens: int, *, key: int | jax.Array, sampling: Sampling = _DEFAULT_SAMPLING, n: int = 1, logits: Transforms | None = None, stopping: Criteria | None = None, strategy: Strategy | None = None, trips: int | None = None, prefix: int | Literal['prompt'] | None = None, progress: Progress = END,) -> Generation
class Sampling( temperature: float = 1.0, top_k: int | None = None, eos_token_ids: int | tuple[int, ...] | None = None, pad_token_id: int | None = None, top_p: float = 1.0, min_p: float = 0.0, repetition_penalty: float = 1.0, presence_penalty: float = 0.0, frequency_penalty: float = 0.0, no_repeat_ngram_size: int = 0, min_new_tokens: int = 0, typical_p: float = 1.0, stop: tuple[str, ...] = (),)params is the complete variables tree. inputs is a ModelInputs from dew.nn.inputs, or an integer (B, P) array normalized to all-valid text. ModelInputs.token_fields["attention_mask"] identifies real token slots; there is no separate generation length argument. Every row needs a real token. Only real tokens count against model.max_seq_len. Conditioning arrays are batch-aligned and used during prefill; decode keeps the model’s cached logical positions. key is an integer seed or a JAX key; key=n is jax.random.key(n).
The compiled decoder uses one padded input shape, with a cache cursor per row and a fixed trip count. Finished rows keep their cached state. On a mesh, rows are split over the batch axes and the result keeps that sharding. Each process passes in its own rows, with the same row count and padded width on every process, reads them back with Generation.host(), and passes the same n. Keys fold in the global row index, so a pool draws what one process draws for the same rows. Invalid input on one rank raises on all ranks before device execution.
n is the number of continuations drawn per prompt and must be a positive integer. A prompt’s continuations share its prefill and then run one after another on the device, with the prompts batched as before for each continuation. So decode time grows with n, but each continuation reuses the cache working memory of the one before, and only the output storage grows with n.
A prompt’s key is its global row key. Continuation zero draws with that key, so n=1 and continuation zero of a larger request are the same draw. Continuation j folds j into the key, so raising n leaves the continuations already drawn unchanged. On a mesh, row padding pads the prompts before their continuations exist, so a prompt’s n rows stay together on the process that asked for them and host() drops only the padded prompts’ rows.
Sampling.eos_token_ids accepts an integer or a tuple of IDs, and any of them terminates a row; the value is normalized to an immutable tuple. Sampling holds the common generation controls as one value, and each default changes nothing. The controls are:
- the repetition penalty (Transformers’
RepetitionPenalty), over the prompt and the generated tokens; - vLLM’s presence and frequency penalties, over the generated tokens;
no_repeat_ngram_size;min_new_tokens, which holds back EOS and so needs aneos_id;- temperature, top-k, nucleus top-p, relative min-p and typical-p filtering.
transforms() compiles them in Transformers’ _get_logits_processor order, which dew.sampling.text.ordered_transforms defines, with vLLM’s penalties placed beside the repetition penalty where vLLM applies them. At least one token survives the filters. Zero temperature selects the argmax and runs no filter.
stop holds strings that end a row. They are compiled against a tokenizer’s vocabulary, so generate refuses them and a TextGeneration compiles them through its processor. A Sampling value is a convenience over the components below, and it compiles to those transforms plus the EOS criterion. When a task runs the policy, an eos_id or pad_id left None takes the task’s own value. generate on its own stops on no EOS unless one is named, and pads with 0.
Generation.tokens includes the original prompt and has shape (B * n, P + max_new_tokens), where B is the number of placed prompt rows. The rows hold prompt zero’s n continuations first, then the other prompts’ in request order. The other fields are aligned with the rows of tokens:
lengths, the count of response tokens including EOS;terminated, true when EOS ended the row and false when the token budget did (slots after termination holdSampling.pad_token_id);behavior_log_probsandraw_log_probs, of shape(B * n, max_new_tokens), the likelihoods of each action under the filtered distribution that drew it and under the unmodified policy.
rows counts this process’s real prompts times n. host() returns the record over host arrays of those rows, and text decodes them through the task’s processor, one string per row.
LMObjective.per_token_log_probs(params, tokens, left_padding=...) scores the raw policy. It left-aligns real tokens for the forward and restores the original next-token alignment. Unscored padding slots are zero. RolloutScheduler.sampled records the raw sampling-time values as old_log_probs and preserves actual draws as behavior_log_probs, in the packed layout GRPOObjective.packed_log_probs rescores. Reward text excludes EOS and padding. When a batch carries no old_log_probs, GRPO uses the behavior probabilities as the old policy and its behavior corrections follow verl’s bypass mode. The next-token objective refuses models declaring causal=False.
Decoding components
Section titled “Decoding components”Decoding has three extension points: logits transforms, stopping criteria and strategies. Import the protocols from dew.sampling and the built-ins from dew.sampling.decoding.
def LogitsTransform.__call__(state: StepState, logits: jax.Array, /) -> jax.Array
def Stopping.__call__(state: StepState, tokens: jax.Array, /) -> jax.Array
def Strategy.__call__( state: DecoderState, start: StepState, ops: DecodeOps, transform: Callable[[StepState, jax.Array], jax.Array], stopping: Callable[[StepState, jax.Array], Finish], budget: int, n: int, /,) -> Draws
class StepState( prompt_width: int = struct.field(pytree_node=False, default=0), budget: jax.Array | None = None,)StepState is everything a transform or a criterion receives. tokens is the fixed-capacity buffer of the prompt followed by the slots for generated tokens, [rows, prompt_width + max_new_tokens]. valid marks the slots that hold a real token, so a row reads its own history whatever padding its prompt batch needed. step counts the tokens a row has committed, active marks the rows still generating, and keys holds one PRNG key per row. state.history() returns each row’s real tokens left aligned with their count, prompt_history() returns the prompt region, and total() returns the real token count. A transform never sees model parameters or cache internals.
logits is the whole transform chain, in the order it runs. Left as None, it is the chain sampling compiles to. An explicit sequence replaces that chain entirely, and () runs no transform, so a caller who needs an order Sampling does not produce can write the order they want. A call’s logits replaces the task’s bound chain. An explicit sampling= on a call also clears a bound chain, because that chain was built for the policy the call just replaced.
stopping adds to the policy instead of replacing it. An explicit sequence runs beside the policy’s EOS criterion, so naming a criterion cannot drop EOS termination. Criteria combine with OR and run after every committed token. The token that fired a criterion is emitted with its likelihoods, and later slots hold pad_id with zero likelihood.
Built-in transforms are pytrees, so a configuration that holds arrays is passed as data and does not become part of a compilation cache key. A plain function works too, and jax.tree_util.Partial(fn, array) passes array configuration to one. Everything runs inside the compiled loop, with no host callback. Across a pool, the resolved components are compared by their structure and by the contents of their configuration arrays, so two ranks that ban different tokens are refused instead of each running its own policy.
import jaximport jax.numpy as jnpfrom dew.nn.backbones.causal_transformer import CausalTransformerfrom dew.sampling import Beam, Sampling, Speculative, decoding, generate
# An untrained decoder with one prediction depth, which Speculative drafts with.model = CausalTransformer(vocab_size=64, emb_features=32, num_layers=1, num_heads=2, mlp_features=64, max_seq_len=64, num_nextn_predict_layers=1)variables = model.init(jax.random.key(0), jnp.zeros((1, 8), jnp.int32))prompts = [[5, 6, 7], [8, 9, 10]]
def favor_short(state, logits): """Raise the end token's score once a row has drawn eight tokens.""" return logits.at[:, 2].add(jnp.where(state.step >= 8, 3.0, 0.0))
# The common controls are one value, compiled in Transformers' order.policy = Sampling(temperature=0.8, top_p=0.9, repetition_penalty=1.1, frequency_penalty=0.4, no_repeat_ngram_size=3, eos_token_ids=2, pad_token_id=0)drawn = generate(model, variables, prompts, 32, key=0, sampling=policy, stopping=(decoding.MaxNewTokens(24),))
# An explicit chain is complete: a transform of your own joins the policy's.nudged = generate(model, variables, prompts, 32, key=0, sampling=policy, logits=(favor_short, *policy.transforms()))
# The same request as a deterministic search over four beams, returning two.searched = generate(model, variables, prompts, 32, key=0, sampling=Sampling(eos_token_ids=2, pad_token_id=0), logits=(decoding.NoRepeatNGram(3),), strategy=Beam(width=4, length_penalty=1.0), n=2)
# Drafted by the model's own prediction depths and verified by the model:# tokens are distributed exactly as ordinary sampling under this call's# policy; accepted drafts save target forwards, which untrained depths rarely give.drafted = generate(model, variables, prompts, 32, key=0, sampling=Sampling(temperature=0.8, top_p=0.9, eos_token_ids=2), strategy=Speculative(block=4))print(drawn.tokens.shape, nudged.tokens.shape, searched.tokens.shape, drafted.tokens.shape)The transforms port transformers/generation/logits_process.py from Transformers 5.16.1, with each row reading its own unpadded history instead of the batch’s padded width.
| Transform | Reference | Notes |
|---|---|---|
Temperature(value) | TemperatureLogitsWarper | |
TopK(k) | TopKLogitsWarper | |
TopP(p) | TopPLogitsWarper | keeps the best token |
MinP(p) | MinPLogitsWarper | |
Typical(mass) | TypicalLogitsWarper | |
EpsilonCutoff(epsilon) | EpsilonLogitsWarper | |
EtaCutoff(epsilon) | EtaLogitsWarper | |
TopH(fraction=1.0) | TopHLogitsWarper | over the reference’s fixed head of 100 |
Greedy() | greedy search | zero on the argmax, -inf elsewhere; what temperature=0 compiles to |
Renormalize() | LogitNormalization | shifts every score by one constant, so no later filter and neither likelihood can see it |
RemoveInvalidValues() | InfNanRemoveLogitsProcessor | the only transform that repairs a broken distribution |
RepetitionPenalty(penalty) | RepetitionPenaltyLogitsProcessor | over the row’s valid prompt and drawn tokens |
PromptRepetitionPenalty(penalty) | EncoderRepetitionPenaltyLogitsProcessor | the prompt is the encoder input; the reference inverts the argument |
FrequencyPenalty(penalty) | vLLM model_executor/layers/utils.py | subtracts the penalty times each generated token’s count |
PresencePenalty(penalty) | vLLM model_executor/layers/utils.py | subtracts the penalty from every generated token |
NoRepeatNGram(size) | NoRepeatNGramLogitsProcessor | |
PromptNoRepeatNGram(size) | EncoderNoRepeatNGramLogitsProcessor | n-grams of the prompt |
sequence_bias(entries) | SequenceBiasLogitsProcessor | (token ids, bias) pairs compiled into one table |
bad_words(ids, eos_token_ids=None) | NoBadWordsLogitsProcessor | a -inf table; single-token EOS sequences are dropped |
SuppressTokens(tokens) | SuppressTokensLogitsProcessor | |
BeginSuppressTokens(tokens, offset=0) | SuppressTokensAtBeginLogitsProcessor | offset is the generated index to suppress at: 0 for the first drawn token, 1 when a forced BOS takes that slot |
ForcedBOS(token) | ForcedBOSTokenLogitsProcessor | |
ForcedEOS(eos, max_length=None) | ForcedEOSTokenLogitsProcessor | max_length counts prompt and generated tokens; None is the request’s own end, so a per-call budget moves it |
MinLength(length, eos) | MinLengthLogitsProcessor | suppresses EOS below a total length |
MinNewTokens(count, eos) | MinNewTokensLengthLogitsProcessor | suppresses EOS below a generated count |
ExponentialDecayLengthPenalty(start, factor, eos) | ExponentialDecayLengthPenalty | start counts generated tokens |
| Criterion | Reference | Notes |
|---|---|---|
EndOfSequence(eos) | EosTokenCriteria | what Sampling.eos_token_ids compiles to |
MaxNewTokens(count) | a budget below max_new_tokens | |
MaxLength(length) | MaxLengthCriteria | prompt and generated tokens together |
stop_strings(tokenizer, strings, vocab_size=None) | StopStringCriteria |
MaxNewTokens and MaxLength cut a row short rather than end it: a row they finish is not terminated, as one that runs out of its budget is not, so Generation.terminated marks the rows that ended on an EOS, a stop string or a criterion of your own. A criterion of your own that cuts rows short at a length says so by subclassing decoding.Truncates.
stop_strings reads the tokenizer once, on the host. It compiles where each token’s text can fall inside each stop string, and how many of the string’s trailing units the token’s start can cover. The criterion then runs entirely on device and never decodes. A string counts only when it touches the token just drawn. So a string produced earlier does not stop the row later, while a string spelled across several tokens or overhanging either end does stop it.
A byte-level or byte-fallback vocabulary is read through its byte spelling and matched over UTF-8 bytes, so a stop string still ends the row when two tokens split one of its code points. Every other vocabulary is read through convert_tokens_to_string behind an ordinary prefix, because a decoder adds or removes a leading space depending on what came before. tokenizer is a Transformers tokenizer or a task Processor holding one, and vocab_size sizes the table for a model whose head is wider than the vocabulary.
An active row can be drawn from only when every score is finite or -inf and at least one is finite. A NaN or a +inf beside a finite score makes the draw arbitrary, and a row with nothing finite has no distribution at all, so in both cases generate raises instead of returning an index. A zero-temperature policy does not collapse such a row onto an argmax; it passes the row through unchanged. The model’s own distribution must be defined too, because an undefined one cannot be reported truthfully. So generation raises there as well, rather than return a NaN likelihood, whatever a later RemoveInvalidValues() does to the scores. Add that transform to repair scores that the chain itself made invalid.
Strategies
Section titled “Strategies”class Sample(grammar: Grammar | None = None)
class Beam( width: int = struct.field(pytree_node=False, default=1), length_penalty: float = struct.field(pytree_node=False, default=1.0), early_stopping: bool | str = struct.field(pytree_node=False, default=False), stop_ids: int = struct.field(pytree_node=False, default=1),)
class Speculative( block: int = struct.field(pytree_node=False, default=4), confidence: float = struct.field(pytree_node=False, default=0.0),)Sample draws every row independently, and a request without a strategy runs it. A grammar from dew.sampling.guided (regex or json_schema) keeps every draw inside the grammar.
Beam is deterministic beam search, with the bookkeeping of _beam_search in Transformers 5.16.1. Each step keeps the best (1 + stop_ids) * width continuations, so that width live beams always remain. A stopping criterion moves a beam into the completed set, with its score divided by its generated length raised to length_penalty, and early_stopping takes the reference’s False, True and "never". The prompt is prefilled once and copied into width cache rows, which each step reparents, so a branched beam decodes exactly like a separately selected prefix. n is how many completed hypotheses to return, and n > width is an error.
A selected path comes from a search, so its behaviour log probability is zero, while its raw log probabilities are still the model’s own. Sampling with beams is refused, because the marginal probability of a selected beam is not the per-step candidate probability, so there is no correct behaviour likelihood to record.
Speculative drafts with the model’s own prediction depths and verifies with the model, following algorithm 1 of arXiv 2211.17192 as _speculative_sampling applies it. The first candidate of a block is an ordinary target draw, so it is always accepted, and each depth proposes the next candidate from the previous hidden state and the candidate’s embedding. A proposal is accepted with probability min(1, p(x) / q(x)), where p is the target’s post-transform distribution and q the draft’s actual one. The first rejection draws from the normalized positive part of p - q, and a block with nothing rejected draws a bonus token from p. So the emitted tokens have exactly the distribution Sample gives them, though not draw for draw at one seed.
Every emitted action records the target’s post-transform log probability as its behaviour value and the model’s own as its raw value. The draft’s q, the acceptance probability and the residual are never recorded. confidence stops the draft after the first candidate the draft is less sure of, as ConfidenceCriteria does. A candidate the draft never offered was not rejected, so the block then ends on an ordinary target draw. A model without prediction depths is refused, and there is no silent fallback.
The target cache is saved before a block and the accepted prefix is replayed into it, because a recurrent mixer keeps a running summary that no cursor can rewind. The prediction cache is rebuilt by the same rule. Depth d’s entry for token t reads depth d - 1’s hidden state at t - 1 and t’s own prepared embedding, at t’s own coordinate, and the target is depth zero’s predecessor. That is the input mtp_hidden_states trains the depths on, the one MTPCandidateGenerator corrects with, and the one Qwen3_5MultiTokenPredictor.forward takes. Each depth keeps its last state across blocks, so no entry is lost at a block boundary. The prompt seeds every depth from the embeddings the prefill already prepared, media replacements included, without running an encoder again.
A depth becomes usable after enough real tokens have preceded it. Rotary offsets and repeated image coordinates do not change that count. A newly available predecessor is retained even if its next depth cannot yet write a cache entry.
A model with a block drafter (DSpark, as DeepSeek-V4.1 and DeepSeek-V4-Flash-0731 ship it) drafts a block’s later candidates in one pass instead of chaining depths. The drafter reads the context its target layers record. V4.1’s averages each target layer’s streams at its attention input, and V4-Flash-0731’s at the layer’s output. The prefill seeds it with the prompt’s context, and each replayed block appends the context of its kept positions. As the pass reaches each position, the strategy draws that candidate from the drafter’s logits for the position, Markov bias included, so block - 1 may not exceed the drafter’s own block size. A standalone drafter checkpoint, such as RadixArk/Kimi-K3-DSpark, which reads another model’s hidden states and drafts through that model’s embedding and head, is refused by design (DrafterRefused).
A continuing block emits at least two tokens when the remaining budget allows it. An EOS or the budget can truncate the block earlier. Thus ceil(budget / 2) iterations bound the loop. Each active block uses two target forwards. Once every row has finished, later iterations run no model call. The predicate reduces the whole batch, so every rank skips the same blocks.
Existing JAX decoding
Section titled “Existing JAX decoding”T5X decoding provides JAX sampling and beam search with model callbacks and explicit cache state. Its upstream tests cover uneven prefixes, cached starts, callbacks and probability scores. The comparison below uses that pinned revision.
| Capability | T5X contract | Fit for Dew |
|---|---|---|
| Model callback | DecodingState and tokens_to_logits separate the loop from model execution. | Adapt this separation. Strategy receives model operations; parameters stay outside the row carry. |
| Likelihoods | temperature_sample returns one cumulative score per continuation. rescale_log_probs=False still scores logits after the logit callback. | Reference only. Rollouts need both original-model and actual-policy log probabilities for every emitted token. |
| Multiple continuations | Expansion and final sorting duplicate the cache and order each prompt’s samples by score. | Reference only. Dew preserves continuation identities and their row-owned keys. Sample shares prefill and reuses its working cache across continuations. |
| Cache reparenting | cache_map and cache_gather_beams use named exclusions and an axis offset for scanned caches. | Adapt the operation, not the leaf rules. Dew’s public cache has row axis zero, including GDN recurrence, MLA buffers and multimodal coordinates. |
| Beam ranking | brevity_penalty uses ((5 + length) / 6) ** alpha. | Reference only for source parity. Transformers uses generated length raised to length_penalty; the rankings can differ. |
| Ordered transforms | The logit callback precedes T5X’s temperature and top-k/top-p processing. | A complete callback chain can adapt this interface by disabling that processing. It does not supply Dew’s two likelihood tracks or speculative verification. |
Dew uses T5X as a reference and a test oracle, and adapts some of its state and cache operations. T5X is not a runtime dependency. Nothing in its JAX loops rules out ragged MoE, but Dew still has to handle row placement, inactive-row masks and collective agreement around model calls, and the T5X return contract does not cover them.
Source generation controls
Section titled “Source generation controls”A loaded source’s generation_config.json is data. Dew assigns every control Transformers 5.16.1 writes there to one consumer: the native policy, a transform, a criterion, a strategy or the task. Otherwise the control is provenance only, or PretrainedDecoder.text_generation() refuses it and says why. The common controls become the task’s Sampling value, task.sampling, so you change one with dataclasses.replace(task.sampling, ...).
A source that also sets a control Sampling does not hold gets task.logits, the complete chain. The same compiler (ordered_transforms) builds that chain in _get_logits_processor’s order, with the policy’s transforms in it. A source that runs beam search always gets a chain, which ends after the processors because the search picks its own continuations. An unset control, or one at the value where generate() adds no processor, criterion or search mode, has no effect. As upstream, beam-only and sampling-only controls are checked only when beam search or sampling is active.
Each source control has one rule for its consumer, neutral value, mode and refusal. Source value precedence remains generation_config.json, wrapper config, then text config. The source resolver selects the actual policy once. Sampling supplies convenience defaults at the request boundary; only resolved transforms, criteria, strategy and padding reach the compiled decoder. An explicit chain therefore has no unused sampling settings in its compilation or process-agreement identity.
| Control | Native mapping | Refused because |
|---|---|---|
do_sample, temperature, top_k, top_p, min_p, typical_p, repetition_penalty, no_repeat_ngram_size, min_new_tokens, stop_strings, eos_token_id, pad_token_id | Sampling (stop_strings as stop, compiled through the task’s processor) | |
max_length, max_new_tokens | the task’s token budget | |
num_return_sequences | the task’s n, independent of any sampling= override | |
bos_token_id, decoder_start_token_id | inapplicable to supplied-input causal decoding; the tokenizer prepares special tokens | |
max_cache_len | capacity assertion only; it does not resize the cache or limit the request | a length above the model’s max_seq_len |
encoder_repetition_penalty | PromptRepetitionPenalty | |
encoder_no_repeat_ngram_size | PromptNoRepeatNGram | |
sequence_bias | sequence_bias | |
bad_words_ids | bad_words | |
min_length | MinLength | |
forced_bos_token_id | ForcedBOS | |
forced_eos_token_id | ForcedEOS at the request’s own end, so a per-call budget moves it | |
suppress_tokens, begin_suppress_tokens | SuppressTokens, BeginSuppressTokens | |
exponential_decay_length_penalty | ExponentialDecayLengthPenalty | |
remove_invalid_values | RemoveInvalidValues | |
renormalize_logits | Renormalize | |
epsilon_cutoff, eta_cutoff, top_h | EpsilonCutoff, EtaCutoff, TopH | |
use_cache | native decoding always runs through its own cache | use_cache=False |
cache_implementation | the fixed-capacity static cache | any other implementation |
cache_config | quantized and offloaded caches are not implemented | |
prefill_chunk_size | the native prefill evaluates a prompt in one call | |
continuous_batching_config | continuous batching is not implemented in the task’s generate path; dew.inference.Server batches continuously as a separate object | |
compile_config, disable_compile | the native decoder owns its compilation and always runs compiled | |
low_memory | sequential beam evaluation is not implemented | |
output_attentions, output_hidden_states, output_scores, output_logits | generation returns tokens, lengths, termination and both likelihood arrays, and none of these | |
return_dict_in_generate | inapplicable to the native result type; generation always returns a record | |
transformers_version, _from_model_config, _commit_hash, tokenizer_name | provenance | |
num_beams, early_stopping, length_penalty | Beam(width, length_penalty, early_stopping), with stop_ids from the EOS ids | |
do_sample with num_beams | a selected beam’s marginal probability is not the per-step candidate probability, so no honest behaviour likelihood exists | |
num_beams with use_mtp | a request cannot run two strategies | |
num_return_sequences above num_beams | a search returns at most its width | |
num_beam_groups, diversity_penalty | diverse group beam search is not implemented | |
constraints, force_words_ids | constrained beam search is not implemented | |
max_time | a host clock cannot stop a coordinated device loop | |
token_healing | retokenizing the prompt is prompt construction, not decoding | |
guidance_scale | classifier-free guidance evaluates the model a second time per step | |
penalty_alpha | contrastive search is a decoding strategy that is not implemented | |
dola_layers | DoLa is a decoding strategy that is not implemented | |
watermarking_config | no watermarking transform is implemented | |
use_mtp, speculation_type, num_assistant_tokens | Speculative(block=num_assistant_tokens + 1), the drafted count plus the block’s own target draw | a checkpoint without prediction-depth weights, or a proposer other than the model’s own depths |
assistant_ensemble_weight below one | ensemble verification below one accepts a biased distribution | |
assistant_confidence_threshold | Speculative(confidence=...), which stops the draft without changing the block size | |
num_assistant_tokens_schedule | "constant" is the fixed block a device loop runs | any adaptive schedule |
assistant_early_exit | early-exit proposal is not implemented | |
assistant_lookbehind, target_lookbehind | translating between two tokenizers’ token spaces is not implemented | |
prompt_lookup_num_tokens, max_matching_ngram_size | prompt lookup proposal is not implemented | |
is_assistant | a source loads as a target model, not as another model’s assistant | |
| any other name | the native decoder does not know the control, so it refuses instead of ignoring it |
Inference tasks
Section titled “Inference tasks”Import pipeline, TextGeneration, BlockGeneration, MaskedGeneration, TextToImage, Images, DenoisingInputs and RunProcessor from dew.inference. dew.pipeline is the same function. Inference describes placement and the three workflows.
def pipeline( source: str, *, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: DTypeLike | None = None, param_dtype: DTypeLike | Literal['auto'] | None = None, ema: bool | None = None, step: int | str | None = None, revision: str | None = None, trust: Sequence[str] = (), options: SourceOptions | None = None, precision: Numerics | str | None = None,) -> TextToImage | TextGeneration | BlockGeneration | MaskedGeneration | SavedTask
def Objective.pipeline( state: TrainState, *, ema: bool | None = None, processor: Processor | None | Omitted = OMITTED,) -> Task | SavedTask
def Objective.build_task( variables: Variables, *, processor: Processor | None | Omitted = OMITTED,) -> Task | SavedTaskObjective.pipeline builds the objective’s task over state.averaged or state.variables. An omitted processor keeps the objective’s own; None clears it. build_task returns TextGeneration for LM, GRPO and PPO’s actor, MaskedGeneration for MDLM, BlockGeneration for block SFT and TextToImage for diffusion, which takes no processor. An objective whose saved_task builds from the settings its run records (from_settings, as Dew’s token tasks do) needs no build_task of its own: the default builds that task from its task_settings() over its model, as a saved run of it loads.
class TextGeneration( model: nn.Module, variables: Variables, processor: Processor | None = None, sampling: Sampling = Sampling(), max_new_tokens: int | None = None, max_length: int | None = None, n: int = 1, logits: tuple[LogitsTransform, ...] | None = None, stopping: tuple[Stopping, ...] = (), strategy: Strategy | None = None, prefix: int | Literal['prompt'] | None = None, progress: Progress = END, *, precision: Numerics | None = None,)
def TextGeneration.__call__( request: Request, max_new_tokens: int | None = None, *, key: int | jax.Array, n: int | None = None, sampling: Sampling | None = None, images: Media | None = None, logits: Transforms | None = None, stopping: Criteria | None = None, strategy: Strategy | None = None,) -> Generation
def TextGeneration.bind(variables: Variables, progress: Progress | None = None) -> Self
def TextGeneration.decode(generation: Generation) -> tuple[str, ...]
def TextGeneration.quantized( spec: Quantization, example: Rows = ((0,),),) -> TextGeneration
def SavedRunTask.from_run( directory: str, *, ema: bool | None = None, step: int | str | None = None, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: DTypeLike | None = None, param_dtype: DTypeLike | None = None, trust: Sequence[str] = (),) -> Self
def TextGeneration.from_saved( run: SavedRun, *, ema: bool | None = None, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: DTypeLike | None = None, param_dtype: DTypeLike | None = None,) -> TextGenerationclass BlockGeneration( model: nn.Module, variables: Variables, process: BlockProcess, processor: Processor | None = None, eos_token_ids: tuple[int, ...] = (), pad_token_id: int = 0, max_new_tokens: int | None = None, max_length: int | None = None, n: int = 1, *, precision: Numerics | None = None,)
def BlockGeneration.__call__( request: Request, max_new_tokens: int | None = None, *, key: int | jax.Array, n: int | None = None, process: BlockProcess | None = None, images: Media | None = None,) -> CanvasGeneration
def SavedRunTask.from_run( directory: str, *, ema: bool | None = None, step: int | str | None = None, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: DTypeLike | None = None, param_dtype: DTypeLike | None = None, trust: Sequence[str] = (),) -> Self
def BlockGeneration.from_saved( run: SavedRun, *, ema: bool | None = None, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: DTypeLike | None = None, param_dtype: DTypeLike | None = None,) -> BlockGeneration
def SavedRunTask.from_run( directory: str, *, ema: bool | None = None, step: int | str | None = None, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: DTypeLike | None = None, param_dtype: DTypeLike | None = None, trust: Sequence[str] = (),) -> Self
def MaskedGeneration.from_saved( run: SavedRun, *, ema: bool | None = None, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: DTypeLike | None = None, param_dtype: DTypeLike | None = None,) -> MaskedGenerationdef Pretrained.load( name_or_dir: str | Path, *, dtype: DTypeLike = jnp.bfloat16, param_dtype: DTypeLike | Literal['auto'] = jnp.float32, attention_impl: AttentionImpl = 'auto', max_seq_len: int | None = None, revision: str | None = None, subfolder: str | None = None, gguf_file: str | None = None, single_file: str | None = None, dduf_file: str | None = None, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, fallback: str | None = None, weights: bool = True, on_host: bool = False, modeling_overrides: Mapping[str, object] | None = None, trust_remote_code: bool | None = None, text_only: bool = False, prediction_depths: int | None = None, dex: DifferentialExtension | None = None, precision: Numerics | str | None = None, scan_layers: bool = False, step: int | str | None = None, ema: bool | None = None,) -> Self
def PretrainedDecoder.text_generation( *, sampling: Sampling | None = None,) -> TextGeneration
def PretrainedMaskedDecoder.text_generation() -> MaskedGeneration
def PretrainedBlockDecoder.block_generation() -> BlockGeneration
def PretrainedPipeline.text_to_image() -> TextToImage
def Pretrained.adapt(lora: LoRA, *, key: int | jax.Array) -> Self
def Pretrained.save( directory: str | Path, *, variables: Mapping[str, object] | None = None, max_shard_size: int | str = MAX_SHARD_SIZE,) -> None
def Pretrained.push_to_hub( repo_id: str, *, variables: Mapping[str, object] | None = None, private: bool = False, commit_message: str = 'Upload dew export', max_shard_size: int | str = MAX_SHARD_SIZE,) -> NonePretrained.load returns the kind it is called on, or the kind the source is. adapt returns the same kind, with the bound adapter as adapter.
class LoRA( rank: int, modules: tuple[str, ...], alpha: float | None = None, rslora: bool = False, dropout: float = 0.0, source: str | None = None, weight_name: str | None = None,)
def LoRA.apply( model: nn.Module, variables: Variables, *, key: int | jax.Array, layouts: Mapping[str, WeightLayout] | None = None, fp8_base: InputQuantization | None = None, base: AdapterBase | None = None,) -> Adapter
def LoRA.load( model: nn.Module, variables: Variables, path: str | FilePath, *, layouts: Mapping[str, WeightLayout] | None = None, base: AdapterBase | None = None,) -> Adapter
def Adapter.from_run( directory: str | FilePath, *, step: int | str | None = None, ema: bool | None = None, trust: Sequence[str] = (),) -> Adapter
def Adapter.merge(variables: Variables) -> Variables
def Adapter.save(variables: Variables, path: str | FilePath) -> Noneclass TextToImage( model: nn.Module, process: Process, inputs: InputSpec, variables: Variables, autoencoder: AutoEncoder | None = None, steps: int = 50, guidance: Guidance | None = None, solver: Solver[object] = DDIM(), grid: Callable[[int], tuple[Process, jax.Array]] | None = None, final_denoise: bool = True, finish: Callable[[Variables, jax.Array], jax.Array] | None = None, *, precision: Numerics | None = None,)
def TextToImage.from_objective( objective: DiffusionObjective, variables: Variables,) -> TextToImage
def SavedRunTask.from_run( directory: str, *, ema: bool | None = None, step: int | str | None = None, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: DTypeLike | None = None, param_dtype: DTypeLike | None = None, trust: Sequence[str] = (),) -> Self
def TextToImage.from_saved( run: SavedRun, *, ema: bool | None = None, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: DTypeLike | None = None, param_dtype: DTypeLike | None = None, variables: Variables | None = None,) -> TextToImage
def TokenScorer.policy( params: Variables, sampling: Sampling = _DEFAULT_SAMPLING, progress: Progress = END,) -> TextGeneration
def TextToImage.bind(variables: Variables) -> TextToImage
def TextToImage.quantized(spec: Quantization) -> TextToImage
def TextToImage.prepare( prompts: str | Sequence[str | Mapping[str, object]] | None = None, *, conditions: Mapping[str, Conditioning] | None = None, key: int | jax.Array, steps: int | None = None, unconditional: str | Sequence[str | Mapping[str, object]] | Mapping[str, Conditioning] | None = None, image: ArrayLike | None = None, image_latents: ArrayLike | None = None, mask: ArrayLike | None = None, noise: ArrayLike | None = None, initial: ArrayLike | None = None, times: ArrayLike | Sequence[float] | None = None, encode_key: int | jax.Array | None = None,) -> DenoisingInputs
def TextToImage.__call__( prompts: str | Sequence[str | Mapping[str, object]] | DenoisingInputs, *, steps: int | None = None, guidance: Guidance | float | None | Omitted = OMITTED, solver: Solver | None = None, key: int | jax.Array, decode: bool = True,) -> Images
class RunProcessor(tokenizer: Tokenizer)
def Server.from_task( task: TextGeneration, *, slots: int, capacity: int, admission: int | None = None, kv_cache: KVCache | None = None, chunk: int | None = None, prefix_cache: bool = False, prefix_states: int | None = None, prefix_state_every: int | None = None, decode_steps: int = 1,) -> ServerRunProcessor adapts a run’s ByteTokenizer or HFTokenizer as a task processor.
A task captures the variables mapping at construction and on bind. Replacing the caller’s mapping does not change the existing task. Array buffers remain shared; do not mutate, donate or delete them while a task uses them. Text requests need a processor. Numeric token rows remain integers: mixed text and token rows, floats, booleans and strings are refused; a resident jax.Array or ModelInputs reaches the model without a host copy.
max_new_tokens takes precedence over a source default. If only max_length is declared, the budget is that total minus the padded prompt width. Otherwise an LM run records its max_new_tokens and sampling value; an objective uses its Samples. With no limit the call must provide one. A call’s n takes precedence over the task’s bound count the same way, including n=1 over a source that asks for more; omitting it uses the bound count. Equal shapes and controls reuse the compiled executable across calls and bind.
LMObjective.policy(params) uses those parameters directly. DPO, GRPO and PPO pipelines publish the trained policy and leave out the frozen reference their loss uses; PPO also removes the critic. For other generative objectives, ema=True requires the moving-average state and raises if it is absent; use ema=False for live weights. objective.pipeline, dew.pipeline, Pretrained.from_run and every task’s from_run and from_saved default to ema=None, which uses the EMA copy merged over the live parameters when the state or run has one, and the live parameters otherwise. ema=True refuses a run without an EMA copy, and ema=False takes the live parameters.
TextToImage.from_run reads run.json and the latest checkpoint under one directory, pinning it once (SavedRun.pinned), and from_saved builds the task from a run pinned already. dew.pipeline is the one loader of a published run on the Hub: it pulls the run directory at the revision asked for, pins it, and picks the task class from the objective name in run.json. The three text tasks are constructed the same way for the model kinds they generate for. In pipeline, dtype selects the computation dtype and param_dtype the parameter storage. None keeps a run’s stored dtypes and uses FP32 masters for a source, and "auto" keeps the stored dtypes of either.
A source-default text task keeps the temperature, top-k, top-p, min-p, EOS and padding settings as its Sampling value. It holds the source’s complete chain as logits, its criteria as stopping and the strategy its config names, and takes its return count from num_return_sequences.
An explicit sampling= replaces the policy and the chain, and the controls behind them are then neither built nor checked, so a distribution control you just replaced cannot block the call. The criteria, the strategy and the return count still come from the source, and every control the task keeps is checked as usual. An unknown control name is always refused, because no consumer is defined for it. A control the native decoder does not implement raises when the default task is created, naming the control and the reason; the table above lists every one. Loading weights for training or export does not select a decoding policy.
BlockGeneration uses BlockProcess.generate. Its CanvasGeneration has lengths, termination and decoder-step counts but no autoregressive likelihoods, plus the same rows, host(), text and continuation rows. Its continuations refine the shared encoded prompt independently, each over the original prompt rows, so each row sees the same batch-wide canvas draw that a single continuation sees.
TextToImage keeps the objective’s or source’s steps, guidance and solver defaults. prepare encodes the prompts and draws their noise once, placed for the task’s mesh. Images.images is [rows, H, W, C] in [-1, 1], and host() reads a process’s rows back. For a source whose solver pairs its own sigma and model-time tables, grid(steps) returns the process and the explicit time grid that a trajectory of that length steps through. The noise prior follows that process, so prepare takes the same steps. final_denoise=False ends a trajectory at the last grid point without the final clean prediction. sample(denoise, x_T, steps=None, *, solver, guidance=None, key, times=None, final_denoise=True) in dew.sampling takes the same two controls; pass exactly one of steps and times, and an explicit grid decides the trajectory’s length.
For a source that ships a checker or an output transform, finish(params, images) runs it on the decoded images under the same placement. Each call encodes the task’s own unconditional prompt with the encoders it holds, as the training step encodes its blank (DiffusionObjective.blank_conditions). Rebinding keeps every task’s compilation identity.
Server.from_task(task, slots=, capacity=) serves a TextGeneration task with continuous batching over one resident KV cache. It holds slots rows of capacity cache slots each, admits queued requests into free rows, and runs one compiled step over every row per iteration. Every request runs the task’s bound policy and draws with the key it was submitted with, so a served request draws the same tokens as the same request run alone, except where the model’s products round at boundaries that two programs’ roundings cross, as a quantized decoder’s moving weights do (docs/concepts/inference.md). submit returns a ticket that resolves to the request’s Generation, step runs one iteration, and run steps until the queue and the rows are empty. Calling the server with a batch of prompts submits them and runs them.
External engine clients
Section titled “External engine clients”Dew does not run an HTTP server. Install [inference-clients] and inject the official client configured for your local or deployed engine. The adapter does not own the SDK client’s lifetime.
import ollamaimport openaifrom dew.inference import OllamaCompletion, OpenAICompletion
with ollama.Client(host="http://127.0.0.1:11434") as client: task = OllamaCompletion("my-exported-model", client) answer = task("Explain this result.", 64, key=7, options={"temperature": 0.8, "top_p": 0.9, "num_gpu": 0}) print(answer.texts)
with openai.OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="local") as client: task = OpenAICompletion("my-exported-model", client) answer = task("Explain this result.", 64, top_p=0.9, extra_body={"top_k": 40, "min_p": 0.05})The convenience call returns Completion(texts, finish_reasons, token_counts, usage, responses). Missing per-choice token counts and finish reasons remain None. Aggregate OpenAI usage stays separate from the per-choice fields. responses retains the SDK models, including the logprobs, token data and extensions the backend reported.
An explicit sampling=Sampling(...) sets the native policy controls that the selected backend supports. Ollama receives neutral repetition, presence and frequency penalties and explicit top-k, top-p and min-p values, so its hidden repeat_penalty=1.1 default does not change the request. Conflicting explicit options are refused.
OpenAICompletion(..., provider="vllm") or provider="sglang" turns on the translation of Sampling into the parameters both engines accept, including repetition_penalty=1.0 and optional EOS token IDs. Without a provider, the client refuses top_k, min_p and eos_id rather than drop them. Without a Sampling value, the provider’s defaults or the caller’s SDK options apply. A backend’s tokenizer can split the same prompt differently from the exported one, so prompt token counts agree more often than prompt IDs do.
stream returns native SDK response chunks. chat(messages, max_new_tokens, stream=..., **parameters) keeps SDK tools, tool-result messages, structured-output controls and media fields. For asynchronous execution, pass an AsyncClient or AsyncOpenAI and use acall, astream or achat.
Ollama requests go through the SDK’s public generate and chat methods, which handle request conversion, HTTP behavior, errors and line-stream framing. The adapter rejects negative token counts and request fields the installed SDK does not accept, and the SDK’s own parsing rejects unparsable values. OpenAI completions use the SDK’s public with_raw_response hook, so the choice and usage fields are checked on the wire before parsing. OpenAI request parameters go to its completion and chat resources, and vLLM-only parameters go explicitly in extra_body.
Provider extensions cannot override the task’s model, prompt, token budget, requested choice count or an explicit Sampling policy. The SDK writes extra_body over the named parameters, so a policy field in it must equal the policy, or the request is refused before any network call.
Pretrained.from_run(run_dir, *, ema=None, step=None).save(destination) writes a saved run in that same layout. from_run rebuilds the run’s model from its record as the right kind of bundle (PretrainedDecoder for a decoder), and save passes it to its family’s writer, which refuses by name a model with no published layout. dew export <run> <dest> does the same from the command line. bundle.push_to_hub(repo_id, private=, commit_message=) publishes what save writes. To publish in the form dew.pipeline pulls back, upload the run directory itself with huggingface_hub.HfApi().upload_folder.
A decoder trained through the LM recipe, exported with PretrainedDecoder.from_model(...).save and converted by ollama create answers a greedy request with Dew’s own greedy continuation, token for token, over the live daemon. A loaded source’s save and a from_model bundle’s leave the same files, so either export converts.
Diffusion and JEPA objectives
Section titled “Diffusion and JEPA objectives”class DiffusionObjective( model: nn.Module | PipelineSource, process: Process | Preset | None = None, inputs: InputSpec | None = None, *, autoencoder: AutoEncoder | None | Omitted = OMITTED, unconditional_prob: float = 0.12, ema_decay: float | optax.Schedule | None = 0.999, solver: Solver = _DEFAULT_SOLVER, guidance: Guidance | None = _DEFAULT_GUIDANCE, steps: int | None = None, variables: Variables | None | Omitted = OMITTED, uncertainty: int | None = None, alignment: Alignment | None = None, end_to_end: EndToEnd | None = None, grid: SourceGrid | None | Omitted = OMITTED,)
class JepaObjective( encoder: nn.Module, predictor: nn.Module, mask: MultiBlockMask, sample: Field, momentum: tuple[float, float] = (0.996, 1.0), momentum_steps: int | None = 100000, label_key: str = LABEL_KEY, encoder_variables: Variables | None = None, predictor_variables: Variables | None = None,)Import DiffusionObjective from dew.objectives.diffusion. model may be a pipeline bundle, as in DiffusionObjective(pipe). The objective then reads the bundle’s denoiser, process, conditions, autoencoder, initial variables and sampling policy, and any keyword you pass overrides that part. JepaObjective’s encoder_variables and predictor_variables are the trees each module starts from, which can be a pretrained or adapted encoder’s. process takes a preset such as Flow() or EDM(regime="pixel"), or a custom Process. A preset builds once and objective.process holds the resulting Gaussian process; masked-token presets are refused. FlowGRPOObjective accepts the same values. TextToImage.from_objective keeps the objective’s built process, and a manually built image task takes a Process. Its model accepts noisy arrays shaped (B, *latent_shape), model noise levels shaped (B,), and conditioning keywords from InputSpec. It returns a prediction with the sample’s channel/spatial geometry. The Process determines the training target and prediction conversion. The objective passes train=True and a dropout RNG during training. An autoencoder changes sample geometry and must expose compatible encode/decode operations. ema_decay=None keeps no averaged copy, so previews and evaluation read the live variables. steps, solver, and guidance configure preview sampling; they do not set the number of optimization steps.
Import JepaObjective from dew.objectives.jepa. The encoder receives normalized images/video and optional token indices plus train and RNG settings. It returns token features with the feature dimension last. The predictor consumes context features and context/target position indices and returns target features of the encoder width. Mask grid, patch geometry, and predictor dimensions must agree. momentum specifies the EMA schedule endpoints over momentum_steps optimizer updates; label_key identifies labels for representation evaluation. See the JEPA example.
Pretrained latent diffusion
Section titled “Pretrained latent diffusion”dew.interop.Pretrained.load also reads a diffusion checkpoint directory with model_index.json, component configurations, safetensors and tokenizer files. Seven denoisers load:
UNet2DCondition, reading one or two CLIP towers through cross attention;SD3Transformer, reading both CLIP towers jointly beside a T5 tower;FluxTransformer, reading a T5 sequence with a pooled CLIP vector;- Qwen-Image 2.1’s
QwenImageTransformer, reading its Qwen3-VL encoder’s prompt states; - FLUX.2’s
Flux2Transformer, reading stacked hidden states of a Mistral-3 ([dev]) or Qwen3 ([klein]) encoder; - Z-Image’s
ZImageTransformer, reading its Qwen3 encoder’s second-to-last layer; - Wan 2.1’s
WanTransformer, reading its UMT5 encoder’s states over a video latent.
The last six run on flow-matching schedules. The denoiser component in the directory selects the family, and the class it declares selects the model. The returned Pretrained holds the native model, its autoencoder behind the existing autoencoder interface (an AutoencoderKL, or Qwen-Image 2.1’s one-frame QwenImageVAE), native text conditioning, a Process, the native solver policy and the published pipeline’s own call policy. Model and scheduler implementations from other libraries run only in the reference tools.
source.text_to_image() builds the native image task. source.save(directory, variables=updated) writes the updated component weights back to their published layouts, retaining tokenizer files, image geometry and any safety-head parameters. Flax-declared components retain their source class and receive Flax msgpack files alongside the safetensors used by Dew; export does not relabel them as PyTorch models.
The source UNet configuration selects the normalization groups and epsilon, and the declared implementation selects the GEGLU semantics: exact GELU for PyTorch sources and approximate GELU for Flax sources. Spatial upsampling targets the actual shape of the next skip connection, including odd intermediate dimensions.
A published MM-DiT reads its own modulation order, joint attention, optional query-key normalization and second self-attention, and its stored sin/cos position buffer. That buffer goes into a buffers collection, so an optimizer and an EMA see only params and export writes the stored array back unchanged. The buffer is cropped centred on the latent’s patch grid.
Flux’s transformer has a different stack. Its double-stream blocks keep the image and text residuals apart except inside attention. Then single-stream blocks run over the concatenation, with attention and feed-forward sharing one output projection, and queries and keys rotated by a three-axis interleaved-real rotary table over the text and image IDs. The pipeline’s 2x2 latent packing is inside the model, so a caller works in latents. A guidance-embedded checkpoint reads the distilled guidance value as a model input and does not run two guided branches.
DiffusionConditioner builds the text conditioning for every family, in one of these compositions:
- one CLIP tower’s last hidden states;
- two towers’ penultimate states, concatenated, with the second tower’s projected pooled vector and the size and crop time IDs;
- those states padded out to the T5 width, with the T5 states after them along the sequence and both projections as the pooled vector;
- the T5 states alone, with one tower’s unprojected pooled vector, which is Flux’s composition.
Each text slot is encoded by the tower that its source pipeline uses for it. An SD3 source that declares no third encoder gets the zero segment its own pipeline writes, sized to the CLIP tokenizer’s window and not to the sequence length a call asks for. Flux has no such path, so a Flux composition without its T5 tower is refused.
Qwen-Image 2.1 has its own conditioner, QwenImageConditioner. It runs the Qwen3-VL encoder’s language model over the pipeline’s text-to-image chat template, reads the last layer’s output before the final norm, and drops the system turn. Rows are padded on the right to a token budget (tokens, 512 by default), and a prompt past the budget is refused. DenoisingCondition.mask marks the real tokens, so the transformer excludes padded keys. The encoder’s vision tower is not run for text-to-image, but its tensors are kept as stored so that an export writes the whole encoder back.
The transformer lays out its sequence image first, so each of its two attention calls ends a row’s keys at a length (key_value_seq_lengths on scaled_dot_product_attention). cuDNN applies that length as its padding mask and skips the padded keys. The transformer keeps the source’s block-causal attention, in which the text attends causally and the image attends to all the text and to itself, and it modulates the text from time zero. Each row starts its image’s rotary frame position after its own text. The source pipeline starts it after the longest prompt in the call, so a padded row here matches the source run on that prompt alone.
The 2.1 VAE is an image-only Wan-style autoencoder with four channels (RGBA) and 64 latent channels, normalized per channel with the config’s latents_mean and latents_std. Image editing is not supported, because it needs the Qwen3-VL vision tower with its deepstack mergers, which Dew does not implement.
FLUX.2’s conditioner is HiddenStatesConditioner. Its pipelines format a prompt with the encoder’s chat template ([dev]: a system turn and a user turn; [klein]: a user turn with thinking off) and pad every row on the right to 512 tokens. For each token they stack the encoder’s hidden states after layers 10, 20 and 30 ([dev]) or 9, 18 and 27 ([klein]). The transformer reads every row whole, so the padding positions hold whatever states the encoder gives them under the padding mask. The condition’s mask marks the real tokens, but FLUX.2’s transformer does not read it. [dev] embeds its guidance scale, 4.0 by default, and [klein] guides two branches at 4.0 against the empty prompt unless its index marks it step-distilled.
FLUX.2’s VAE (AutoencoderKLFlux2) folds each 2x2 block of its 32-channel latent into 128 channels and normalizes each channel by its batch norm’s running statistics, so the transformer’s latent is the autoencoder’s own, 16 times smaller on a side. The pipelines pass the scheduler linspace(1, 1/N, N) and their own shift, compute_empirical_mu of the token count and the step count, which the grid reproduces (empirical_mu). A tiny [klein] pipeline built from the published configs runs four steps to within 1e-5 of the source’s call, and a tiny Mistral-3 encoder under Mistral Small 3.1’s chat template gives [dev]‘s prompt states to 1e-5 (tests/test_flux2_source.py). [dev]‘s own files are gated, so its published tokenizer is not among the fixtures.
Z-Image (S3-DiT) uses the same conditioner, with the chat template’s thinking turned on. It reads the output of its Qwen3 encoder’s second-to-last layer, and only the real tokens, which the condition’s mask marks. Its transformer cuts the Flux VAE’s latent into 2x2 patches, pads the image and the prompt each to a multiple of 32 tokens with learned pad tokens, and refines each separately. It then runs one stream over both, image first, with rotary positions that follow each row’s padded prompt length.
The source is called with the time 1 - sigma and returns the negated flow, while ZImageTransformer takes Dew’s model time and returns the flow. Its pipeline guides as pos + 5.0 (pos - neg) against the empty prompt, which is Dew’s CFG(6.0). Tiny transformers match the source to 1e-5 of the largest output and 1e-4 in every gradient, and a tiny pipeline built from the published configs runs four steps to within 1e-5 of the source’s call (tests/test_z_image_source.py).
Wan 2.1 (text to video) conditions through WanConditioner. Its pipeline cleans each prompt with ftfy (the wan extra, uv pip install 'dewml[wan]'), unescapes HTML entities twice and collapses whitespace. It then pads the prompt to 512 UMT5 tokens, runs the encoder under the padding mask and zeroes every state past the prompt’s own tokens; the transformer reads all 512. UMT5 is the T5 tower with a relative position bias table in every layer (T5EncoderTransformer(per_layer_bias=True), which a umt5 config sets).
The Wan VAE encodes and decodes in its source’s chunks: the first frame alone, then four frames to one latent frame at a time, with each causal convolution’s last frames carried from one chunk to the next (chunked_moments, chunked_decode). So a clip’s length does not set the memory the VAE holds. A 480x832 decode peaks at 4.6 GiB on an RTX 4080 for 33, 49 or 81 frames, where the one-pass decode needs 10 GiB at 33 frames and 14 GiB of temporaries at 49.
WanTransformer cuts the Wan VAE’s latent into 1x2x2 patches. It self-attends with a three-axis rotary table over (frame, row, column), with queries and keys normalized across all heads, and cross-attends to the text. The sample is a clip, Field("video", (frames, height, width, 3)), of 81 frames at 480x832 unless the index or load_diffusion_source(size=(frames, height, width)) says otherwise. The frame count is 1 + 4k, which the VAE decodes whole. text_to_image() returns [N, frames, height, width, 3], and the pipeline guides at 5.0 against the empty prompt, for 50 steps over its UniPC grid with flow shift 3.0. Tiny transformers match the source’s float32 flow and gradients within twice its own RMS distance from float64, and a tiny pipeline run ends 2.8e-6 from the source’s latent (tests/test_wan_source.py).
SourceTask holds the published pipeline’s own call policy: the steps and guidance its __call__ defaults to, and the grid it prepares, with the sigma origin and latent geometry that pipeline uses. That way text_to_image() runs what the source runs. A directory that declares no pipeline takes its family’s reference pipeline. A declared pipeline Dew does not implement, or one that drives a different denoiser than the directory holds, is refused, so it never runs under another pipeline’s defaults.
SourceSchedule.from_config reconstructs eighteen published scheduler classes: DDIM, PNDM, DDPM, LMS, Euler, Euler ancestral, Heun, KDPM2, KDPM2 ancestral, DPM-Solver multistep, singlestep and SDE, DEIS, UniPC, EDM DPM-Solver, LCM, TCD and flow-matching Euler. The returned process, solver and grid use the controls and defaults of the declared class, and the solver is resolved once when the file is read.
Supported policies include source timestep spacing and offsets; Karras, exponential and beta sigma grids; finite lambda clipping; zero-terminal-SNR rescaling; terminal sigma selection; solver orders and correctors; DDPM fixed posterior variances; and clipping or dynamic thresholding in the source conversion order. DDIM and PNDM keep fixed training strides. LCM and TCD select their grids from original_inference_steps. Two-evaluation solvers keep the source’s stage sigma and model time.
EDM uses its own sigma range, data scale, rho, log-sigma model time and signed output preconditioning, and its training process draws EDM sigmas instead of using a VP beta table. A flow-matching file holds its static or resolution-dependent shift, its terminal stretch and both sigma origins its pipelines use. A resolution-dependent shift reads the calling pipeline’s latent token count through the task’s grid, which text_to_image() sets to the checkpoint’s own geometry.
Unimplemented active controls fail explicitly: Lu-lambda and flow-sigma grids, UniPC external predictors, nonlinear sigma interpolation, continuous model times, learned variances, DDPM log-space wide variance, and PNDM velocity-domain history. TCD clipping/thresholding and epsilon-prediction UniPC thresholding are also refused because the published update does not apply them. Sigma-indexed source schedulers whose initial model time repeats are refused when preparing their grid: the source starts at the second match and cannot complete its published evaluation list. Repeated noninitial stage and corrector times remain supported.
The tiny oracles in tools/diffusers_source_reference.py run actual Diffusers scheduler objects. Ordinary stochastic trajectories share explicit Gaussian draws. DPM-Solver SDE runs the actual torchsde tree, and the native solver is compared on its recorded increments; separate tests check the native Brownian bridge law.
All native source trajectories and VJPs run in float32. Trajectories use a 1e-4 scaled-error bound. Every source VJP is checked with tests/reference_error.py: its RMS distance from the exact source evaluation must be at most twice the float32 source’s. The exact evaluation builds every scheduler table and does the step arithmetic in float64, from the same initial float32 latent, cotangent and recorded random draws. Its gradient is kept in float64 and not rounded back to float32.
from dew.interop import PretrainedPipeline
source = PretrainedPipeline.load("./image-checkpoint", dtype=jnp.float32)images = source.text_to_image()(["a flower"], steps=20, key=0).host().imagesobjective = DiffusionObjective(source)Training batches hold uint8 NHWC images and source.inputs.tokenize(captions). A nine-channel inpainting source also specifies inputs.mask, binary NHWC masks with one channel, where white marks the region to repaint. Caption dropout keeps the mask and the masked-image latents. The mask only conditions the network, so it does not guarantee that the decoded unmasked pixels equal the original image.
Pretrained.load(..., mesh=, layout=) and load_diffusion_source(..., mesh=, layout=) stream a pipeline’s weights the way they stream a decoder’s. Each leaf is read from the mapped checkpoint one device shard at a time and placed before the next, so the host never holds the translated pipeline. Convolution kernels and a UNet’s per-head attention kernels are small, so they are read whole. The placed tree is identical to the host load’s, bit for bit.
load_diffusion_source(..., text=False) reads and downloads every component except the text encoder. Load the family’s conditioner separately to encode prompts, for example WanConditioner.from_pretrained(checkpoint, mesh=MeshSpec()). TextToImage.prepare(conditions=..., unconditional=...) accepts these encodings as {"conditioning": condition}, with one row per sample, in place of prompts. Each row gets the same noise as a prompted call, so both calls produce the same images. Without encoded unconditional, this call applies no guidance. A pipeline loaded with text=False refuses prompts, DiffusionObjective(pipe) and save, since they require the text encoder.
Prepared DenoisingInputs can supply encoded native conditions and initial latents. Explicit grids pass to sample(times=..., final_denoise=False) when the last latent is the result; without an explicit grid, the existing steps and final clean-prediction convention remain unchanged. Published tabulated PRK grids own their integer half-interval rule; ordinary native grids retain exact half-intervals.
Configuration and registries
Section titled “Configuration and registries”Code builds every model, preset, solver, dataset and metric from its class. A record names a class by its import path, {"class": "dew.nn.backbones.dit:SimpleDiT", "fields": {...}}, and a function as {"function": "optax:adamw"}; RunConfig.from_dict and ModelConfig.build import what a record names. Where a person writes a record or a flag, dew.registry maps short aliases (simple_dit) to those paths.
A run names its model as ModelConfig(name, fields) and its objective as ObjectiveConfig(name, fields): the class’s import path or alias and the fields or constructor arguments the run states. Each also records every other field or argument the class defaults, as defaults, so a default that changes in a later release leaves dew train <run.json> and a resume building what the run trained; a record without one takes the class’s default at the time. An argument the objective resolves from what it is built around (Objective.resolved), such as a diffusion objective’s sampling guidance, which a loaded pipeline supplies, is recorded as the run resolved it. ObjectiveConfig.build(**derived) constructs the objective with the arguments its caller builds (the model, what the data decides), and a class that is not a dataclass is read through its __init__ signature. dew.config.sweep.override(config, {"model.num_layers": 12}) and assigned(config, ["trainer.steps=2000"]) set fields by dotted path; dew train and sweeps go through them.
RunConfig.save writes the run configuration. It is separate from the state checkpoint. Recipes describes the configuration entry points and their side effects.
attention_impl is a model field that names the attention kernel. Its names, what each runs, and the order in which 'auto' takes them on each trace are declared once, by AttentionImpl in dew/nn/attention.py; each kernel’s conditions are its requirements in dew/nn/attention_kernels.py, which the attention selector reads.
'auto' and 'xla' both take 'reference' for float64 inputs, because JAX’s XLA attention narrows their softmax to float32, and for bf16 on a GPU older than sm80, which rejects the bf16 dot algorithm it names.
Splash’s mask is a block-sparse descriptor built along with the executable, so a causal or sliding-window sequence skips the blocks its mask empties and does not pay for the whole rectangle. The head width is unconstrained, because the kernel pads it, but the query and key lengths have to be multiples of 128 for its mask blocks to tile them.
An additive bias, a mask that is a value of the trace (a KV-cache decode mask, or the striped mask sequence parallelism builds), a mask past the published cell budget, or a length that is not a multiple of 128 sends an explicit 'tpu' call to the older Pallas flash kernel and keeps 'auto' on 'xla'. Splash applies a logit softcap (Gemma 2) and attention sinks (GPT-OSS) itself, and a packed batch reaches it as segment IDs instead of a mask. The flash kernel has neither a softcap nor sinks, so an explicit 'tpu' call that splash cannot describe refuses them. On a GPU, 'auto' runs both on 'xla', and 'cudnn' refuses them by name.
Below 512 keys 'auto' stays on 'xla', where XLA’s attention measured faster on a v6e (SPLASH_MIN_LENGTH in dew/nn/attention_kernels.py). The parameter tree does not change with the implementation, so checkpoints are interchangeable across hardware.
The README model list names which model configurations run the whole workflow; each task guide covers its own data and objective.
All modules
Section titled “All modules”Each module below has a page generated from its docstrings. A name a module re-exports links to the page of the module that defines it.
Top level
Section titled “Top level”| Module | Summary |
|---|---|
dew | Dew: one objective, one trainer. |
dew.batch | The batch contract every layer shares, the one module below them all that names it: a batch (Batch) and a variables tree (Variables), the columns a batch’s rows carry beside their fields (VALID_ROWS, EXAMPLE_ROWS) and the collection frozen weights stay in (FROZEN), how a batch is filled out with repeats of its own rows (padded_rows), how a batch’s token fields say which slots are real and where each token sits (Coordinates, real_slots), the token ids a field holds (token_ids) and which targets a language-model loss scores (scored_targets). |
dew.mesh | The mesh facts every layer reads, below the models and the data alike: the names of a mesh’s axes (MESH_AXES), the ones a batch’s rows split over (BATCH_AXES), the default table of rules that places each logical axis on them (DEFAULT_RULES, or a layout’s in flax’s context, axis_rules), the spec those rules give an array (logical_spec), which axes a batch’s rows split over on a mesh (row_axes, batch_axes), and the refusal of a layout that does not run (LayoutRefused). |
Training
Section titled “Training”| Module | Summary |
|---|---|
dew.training | The trainer and the pieces it is built from. |
dew.training.budget | How long a run trains and what ends it early: its one budget, in steps, epochs, examples or tokens (run_length, run_steps), the stop policies Trainer.fit takes beside it (Plateau, Pause), and its cadences resolved against it (every_step). |
dew.training.cooldown | Cooldown branches of a run (Hägele et al. |
dew.state | The train state and the accumulation record that jitted steps take and return. |
dew.optim | The optimizer a recipe builds from an OptimConfig. |
dew.laprop | DreamerV3’s optimizer (laprop): LaProp’s direction (scale_by_laprop) of the update clipped adaptively (clip_by_agc), at DreamerV3’s learning rate. |
dew.newton_schulz | Muon’s Newton-Schulz orthogonalization (dew.optim’s Muon and M3 iterate through it): optax’s own iteration (orthogonalized_leaf) over each update’s matrices (as_matrices), a head’s alone where a run names its width (headed), the matrices of a shape stacked and spread whole across the mesh’s devices (orthogonalized), in the run’s numerics or a stated dot algorithm (ns_operands), at Jordan’s coefficients or Polar Express’s (polar_express). |
dew.gauss_newton | Gauss-Newton’s update, damped as Levenberg-Marquardt’s, as an optimizer: an update’s step solves the system its objective states for the update’s residuals r and their Jacobian J in the trained parameters, |
dew.training.quantization | The training step that quantizes the modules an objective trains (dew.quantization). |
dew.training.runtime | Process setup that every recipe runs before it builds anything. |
dew.distributed.placement | Where a run’s arrays live: the device mesh (MeshSpec), the parameter and state layout over it (Layout, from the logical axes every module declares, dew.nn.sharding, and the rule table, dew.mesh), and what a placed tree holds (placed_bytes, placed_shapes). |
dew.distributed.batches | A run’s batches on the devices: the fields a batch carries to the host apart (split_fields, with_host), the shape a batch’s compiled step is keyed by (Signature), its shardings and its placement on a mesh (batch_shardings, shard_batch, first_reader_batch), its host fields gathered across processes by the rows’ origins (row_origins, shared_host_fields), and the prefetch that keeps the next batches in flight within a memory budget (DevicePrefetchIterator). |
dew.averaging | Weight averages: the EMA’s update, and post-hoc EMA’s averages of any length, reconstructed after training. |
Objectives
Section titled “Objectives”| Module | Summary |
|---|---|
dew.objectives | |
dew.objectives.base | What the trainer is optimizing. |
dew.objectives.lm | Language-model objectives, each imported from its module when first used, so importing one of the package’s modules, or a name from one, reaches only what that module imports. |
dew.objectives.diffusion | Diffusion objectives and their distillations, each imported from its module when first used, so importing one of the package’s modules, or a name from one, reaches only what that module imports. |
dew.objectives.routing | How a run trains its routers: the one policy every objective composes. |
dew.objectives.diffusion.adversarial | Adversarial diffusion distillation: LADD, with ADD’s distillation term. |
dew.objectives.diffusion.alignment | Representation alignment: REPA and iREPA. |
dew.objectives.diffusion.consistency | Continuous-time consistency distillation (sCM) combined with distribution matching (DMD2), which together make rCM. |
dew.objectives.diffusion.end_to_end | REPA-E: the autoencoder trained together with the diffusion model through REPA. |
dew.objectives.diffusion.few_step | Few-step generators trained from scratch: MeanFlow and shortcut models. |
dew.objectives.diffusion.guidance_distillation | Guidance distillation: a student that reads the guidance scale as an input. |
dew.objectives.jepa | JEPA: its objective, masks and probes, and the encoder and predictor it trains, each imported from its module when first used, so importing one of the package’s modules, or a name from one, reaches only what that module imports. |
dew.objectives.supervised | Any Flax model, trained on a loss over its outputs and the batch. |
dew.objectives.residuals | Objectives whose loss is a mean square of residuals, stepped by the gradient or by Gauss-Newton. |
dew.objectives.perturbation | Training without backpropagation: node perturbation, as Dust trains a transformer. |
dew.objectives.rl | Online RL objectives: rollouts, preference losses and group updates. |
dew.objectives.rl.flow | Flow-GRPO over recorded rectified-flow transitions. |
dew.objectives.rl.nft | DiffusionNFT: online diffusion reinforcement on the forward process (arXiv 2509.16117). |
dew.objectives.rl.control | PPO for continuous control in a dew.rl.env.Env: an actor-critic policy, its rollout and its loss. |
dew.objectives.rl.dreamer | DreamerV3 (Hafner et al., arXiv 2301.04104 v2 and Nature 2025) for vector observations, as its code computes it (danijar/dreamerv3 at e01491fad6434b2245a3b8ca201dd7faedcc458c, cited by file:line). |
dew.objectives.rl.tssm | TransDreamer’s transformer state-space model (Chen et al., arXiv 2202.09481): a Dreamer world model’s dynamics as a causal transformer over the latents and actions before each step. |
dew.objectives.world_model | Dreamer 4’s world model at a small scale (arXiv 2509.24527, docs/research/dreamer4.md): a causal tokenizer of video frames, and an interactive dynamics transformer over its latents trained by shortcut forcing (Eqs. |
dew.objectives.rl.scheduler | Collect rollouts and admit complete groups into the trainer’s batches, synchronously or ahead of it. |
dew.rl.sandbox | Run untrusted programs and tool sessions in bounded Linux subprocesses. |
dew.interop.gateway | The seam between an agent harness and the policy: rllm-model-gateway, recording every model call. |
dew.interop.harness | Agent harnesses as a session source: one harness command per sample, its model calls recorded. |
dew.interop.harbor | Harbor trials as a harness: each sample one harbor trials start, its model calls recorded. |
dew.interop.openenv | OpenEnv environments as episode environments: one OpenEnv session per episode, Dew drawing each turn. |
| Module | Summary |
|---|---|
dew.data | Datasets for training and evaluation: a store, and the steps that read it. |
dew.data.presets | Dataset functions that prepare one kind of record for a model, from the public steps, ending in batch. |
dew.data.pipeline | Immutable record pipelines, lowered to Grain by :func:compile. |
dew.data.holdout | Held-out records: a dataset split into the part a run trains on and the parts it scores. |
dew.data.compile | Compile the same pipeline to an elastic order or an exact per-share reader. |
dew.data.dataset | The compiled Loader, shared reader partitions and checkpoint contracts. |
dew.data.plan | Execution identities and the order a saved record count names. |
dew.data.mixing | Accepted-row accounting and mixtures measured in valid target tokens. |
dew.data.cache | Write-through artifacts with per-input completions and conditional publication. |
dew.data.storage | URL storage, named credentials, bounded transport caching and conditional puts. |
dew.data.sinks | Immutable exports to filesystem URLs: ArrayRecord shards Dew writes, and Parquet or any other format a native writer writes from one Arrow stream (Writer), all committed by one publisher. |
dew.data.sources | Storage capabilities and the source boundary every chain reads through. |
dew.data.sources.base | What a pipeline reads from: a store, its identity, and how it can be read. |
dew.data.sources.files | Indexed files and checkpointable physical-shard readers over filesystem URLs. |
dew.data.sources.memory | Sources whose records are already resident in the caller’s memory. |
dew.data.sources.hub | Prepared HF Arrow tables and physical Parquet shards from HfFileSystem. |
dew.data.sources.tfds | Prepared TFDS data sources, with explicit preparation and remote mirrors. |
dew.data.sources.pytorch | Map-style torch datasets, imported on use and read through the shared pipeline. |
dew.data.sources.hf | A Hugging Face datasets dataset read as a Dew store. |
dew.data.sources.grain | A Grain dataset read as a Dew store. |
dew.data.sources.iterator | Any resumable iterator read as a Dew store. |
dew.data.sources.codec | The ArrayRecord byte-map layout and the artifact record codec. |
dew.data.sources.urls | URL catalogs and volatile fetches, executed by the ordinary Grain worker path. |
dew.data.sources.av_utils | Frames and sound of a media file over a time window, which video.PickClip draws once for both. |
dew.data.steps.generic | Field selection, formatting and record filtering, steps any chain takes. |
dew.data.steps.text | Over-length policies, fixed-length checks and target counts of token records. |
dew.data.steps.windows | Overlapping causal windows, with the boundary target retained. |
dew.data.steps.chat | Block-diffusion response rows of rendered conversations, and prompt record encodings. |
dew.data.chat | Supervised fine-tuning data: conversations with a role on every token. |
dew.data.images | Image record steps: decoding, filters, resizing, crops, captions and augmentation, on the host or on the accelerator. |
dew.data.tokens | Document chunk tables, packed windows, first-fit plans and a row’s shares of its room. |
dew.data.text | Tokenizers for language-model data, utf-8 bytes or any HF tokenizer, and the text record steps: tokenizing, chunking, dropping, normalizing and filtering text, and curating it through DataTrove’s filters (Language, Quality, Classifier, Decontaminated) and its MinHash deduplication (NearDuplicates, which Dataset.snapshot applies; the curation extra). |
dew.data.video | Video records: a directory tree of clips, a clip window drawn once, and its frames. |
dew.data.audio | Sound records: a media file’s audio over the window video.PickClip drew. |
dew.data.preferences | Preference pairs for DPO: chosen and rejected rows with their masks. |
dew.data.prompts | Prompts for online RL: fixed-width rows with their reward context. |
dew.data.processors | Turns raw audio into the arrays a batch carries. |
dew.data.image_codec | Bounded RGB image decoding, shared by every source and image step. |
Models
Section titled “Models”| Module | Summary |
|---|---|
dew.registry | Short names for the classes a run is made of, and the records that rebuild them. |
dew.quantization | Quantization of a model’s matmuls through Qwix, for training, serving and loading. |
dew.nn.backbones | Model backbones. |
dew.nn.backbones.causal_transformer | The autoregressive transformer decoder that every language model in Dew trains. |
dew.nn.backbones.decoder_block | The decoder block that CausalTransformer stacks, and the parts it is built from. |
dew.nn.backbones.layer_plan | The layer plan of a CausalTransformer: each layer’s kind, its spec, and which layers scan together. |
dew.nn.backbones.layer_names | The module names of a decoder stack’s layers: a single layer’s layers_N and a scanned run’s layers_3_7, which every reader of a stack’s variables (a LoRA, a checkpoint’s reader, a host’s banks) keys by, apart from the stack’s plan (layer_plan), whose modules they need not import; and a stage-local store’s runs cut back onto those layers (stored_layers), as such a reader reads them. |
dew.nn.kv_cache | How a decode cache stores its keys and values: dense or paged, full precision or quantized. |
dew.nn.mixers.parallel | The parallel mixer kind: several token mixers reading one normed input side by side, their outputs summed into the layer’s one residual add, as a parallel hybrid’s layer runs them (Falcon-H1’s Mamba-2 beside its attention, FalconH1DecoderLayer.forward in modeling_falcon_h1.py). |
dew.nn.backbones.edm2 | EDM2’s magnitude-preserving U-Net. |
dew.nn.backbones.joint | The blocks shared by the published MM-DiT family: SD3, Flux, FLUX.2, Z-Image and Qwen-Image. |
dew.nn.backbones.flux | Flux’s transformer, as Diffusers 0.34.0’s FluxTransformer2DModel runs it. |
dew.nn.backbones.flux2 | FLUX.2’s transformer, as Diffusers 0.40.0’s Flux2Transformer2DModel runs it. |
dew.nn.backbones.qwen_image | Qwen-Image 2.1’s transformer, as Diffusers’ QwenImage21Transformer2DModel runs it at commit 6256aa76. |
dew.nn.backbones.sd3 | Stable Diffusion 3’s own MM-DiT, as the published transformer computes it. |
dew.nn.backbones.wan | Wan 2.1’s video transformer, as Diffusers 0.34.0’s WanTransformer3DModel runs it for text-to-video. |
dew.nn.backbones.z_image | Z-Image’s single-stream transformer (S3-DiT), as Diffusers 0.40.0’s ZImageTransformer2DModel runs it for text-to-image. |
dew.nn.mp | EDM2’s magnitude-preserving layers. |
dew.nn.fourier | Fourier feature pyramids: learnable periodic feature grids at coarse-to-fine resolutions, read at any coordinate by Fourier interpolation. |
dew.nn.diffusion_gemma | DiffusionGemma’s shared encoder and decoder, and its self-conditioning MLP. |
dew.nn.gemma3n | Gemma 3n’s residual stream: AltUp, the LAuReL block and activation sparsity. |
dew.nn.multimodal | Image and audio conditioning around the shared causal decoder. |
dew.nn.cross_attention | Gated cross-attention from a conditioning sequence into a decoder’s layers. |
dew.nn.dex | Differential extension (DEX, arXiv 2505.16333): a learned differential on chosen heads’ outputs. |
dew.nn.progress | The training step’s position, which a module whose computation changes over a run reads: an annealed coefficient, a schedule of its own. |
dew.nn.memory | In-context memory: a layer whose inner network’s parameters update over the sequence (ledger 1645). |
dew.nn.ngpt | The normalized transformer’s (nGPT, arXiv 2410.01131) arithmetic: hidden states on the unit sphere, each sublayer’s output joining the residual by a learned step along the sphere. |
dew.nn.hyper_connections | Manifold-constrained hyper-connections: the residual as hc_mult streams. |
dew.nn.engram | Engram: n-gram hash lookups written into the residual streams. |
dew.nn.autoencoders | Autoencoders a latent diffusion model reads and writes its samples through: the interface (AutoEncoder) and the KL and Stable Diffusion VAEs. |
dew.nn.autoencoders.pretrained | A published autoencoder, built as its own config names it. |
dew.nn.vision | Vision towers, projectors and reference checkpoint maps. |
dew.nn.categorical | The categorical distribution over a class axis: the one owner of its log-likelihood in Dew. |
dew.nn.inputs | Numeric model inputs shared by preprocessing, training and generation. |
dew.nn.mixers.attention | Grouped-query causal attention as a mixer kind. |
dew.nn.mlp | The feed-forward every family builds: GatedMLP, two maps with an activation between or three with a gate, which owns its maps’ names, placement, remat names, activation rounding and work count. |
dew.nn.protocols | What a model can be asked for, as structural protocols. |
dew.nn.kernels | Pallas kernels for the operations XLA does not fuse. |
dew.algebra | Dew’s algebraic rewrite of its own programs (docs/design/algebra.md): a jaxpr held as an e-graph of terms (terms), rules applied in named sets (rules), and each value’s cheapest form under a device’s prices (cost) evaluated in place of the program’s (program.rewritten). |
dew.nn.sharding | Where a parameter is split across devices, declared where the module creates it. |
dew.nn.wrapping | Models around a backbone: a module around one causal decoder, answering the decoder’s training, cache and serving hooks (dew.nn.protocols) as the decoder answers them, its decoder’s parameters under the field that holds it (a multimodal model’s language model, DiffusionGemma’s text decoder); and a backbone with a head over its final states, two modules over one variables tree (HeadedModel). |
dew.nn.scatter | Indexing along a sequence: a row’s entries at its own positions (gather_positions), and the index a scatter with mode='drop' gives the entries it must not write (DROPPED). |
dew.nn.precision | The precision a bf16 model’s fp32 boundaries multiply at. |
dew.nn.derivatives | Declared derivatives: a function whose derivative is not its value’s own. |
dew.nn.nested | A nested program: a function an outer program compiles as one sub-program of its own, traced once for its arguments’ types, so a function called many times within one trace (a contraction under Dew’s wide arithmetic, row 3142) is traced once. |
dew.lora | Low-rank adapters (LoRA) for Dew models. |
Diffusion and sampling
Section titled “Diffusion and sampling”| Module | Summary |
|---|---|
dew.diffusion | Forward processes, noise schedules and the parameterizations over them. |
dew.diffusion.process | The diffusion process a model is trained and sampled with, and the denoiser that runs a model on it. |
dew.diffusion.schedules | The noise schedules, one module per family. |
dew.diffusion.schedules.source | Published diffusion scheduler files, resolved once and sampled on their native grids. |
dew.diffusion.schedules.source_grids | The schedules that sampling on a published diffusion checkpoint’s grid uses. |
dew.diffusion.presets | Named diffusion conventions, stored as dataclasses in a run’s run.json. |
dew.diffusion.discrete | Masked (absorbing-state) discrete diffusion, with the interface of the Gaussian one. |
dew.sampling | The reverse process for diffusion, and decoding for language models. |
dew.sampling.solvers | Diffusion solvers, each taking one reverse step from t to t_next with the model’s prediction at t. |
dew.sampling.flow | Stochastic rectified-flow transitions and their Gaussian likelihoods. |
dew.sampling.guidance | Guidance as a wrapper around a denoiser. |
dew.sampling.decoding | Logit transforms and stopping criteria for the decode loop. |
Inference and interop
Section titled “Inference and interop”| Module | Summary |
|---|---|
dew.inference | Inference tasks: native generation with a model and its weights, and clients for external engines, each imported only when you use its API. |
dew.inference.banks | Build and place parameter banks for generation, one bank at a time. |
dew.inference.tasks | Tasks that hold a model, its weights and its host processing for repeated generation. |
dew.interop | Checkpoint readers and translators, each imported only when you use its API. |
dew.interop.diffusion_gemma | Assemble a DiffusionGemma for the shared pretrained-model loader. |
Decision models
Section titled “Decision models”| Module | Summary |
|---|---|
dew.decision | Decision models: one forward pass answers typed questions about a state. |
dew.decision.config | The decision run, as one typed record: a decision model fine-tuned on a table of labelled text, as laya-train --data tickets.csv does. |
dew.decision.data | Labelled requests: what a decision model trains and calibrates on. |
dew.decision.objective | DecisionObjective: training a decision model on labelled questions. |
dew.decision.rows | The decision objective’s training rows: examples fitted to the layout’s rows and grouped examples’ units (Fit), encoded with their augmentations (Encoding), packed into windows (PackedRows), and the fields a row carries (TRAINING_FIELDS); DecisionObjective.data assembles them. |
dew.decision.layout | How a state and its questions become the token rows a decision model reads. |
dew.decision.task | Decide: a decision model answering requests, in Dew’s types and in Jev’s wire form. |
Conditions and evaluation
Section titled “Conditions and evaluation”| Module | Summary |
|---|---|
dew.inputs | What a generative objective is fed: the sample field and its conditions (dew.inputs.spec), the condition encoders (dew.inputs.encoders, their contract dew.inputs.condition) and a diffusion model’s conditioner (dew.inputs.diffusion), each imported from its module when first used, so importing one of them, or a name from one, reaches only what that module imports. |
dew.inputs.condition | What every condition encoder is (ConditionEncoder) and the work it does (ConditionWork), apart from the encoders themselves (dew.inputs.encoders), so a spec of a model’s inputs, a protocol or a model reads the contract without the towers. |
dew.inputs.encoders | Encoders that turn raw conditioning data into the value a model keyword takes. |
dew.eval | Metrics: the image metrics FID, CLIPScore, PSNR, SSIM, LPIPS and CLIPDistance, and the calibration of a classifier’s forecasts. |
dew.eval.calibration | Calibration of a classifier’s forecasts over K options, and the scores of it: the math any model’s calibration reads. |
dew.eval.harness | Run a saved run as an lm-evaluation-harness model. |
Configuration
Section titled “Configuration”| Module | Summary |
|---|---|
dew.config | Typed configuration for a training run. |
dew.config.lm | The language-model run, as one typed record. |
dew.config.jepa | The JEPA run, as one typed record: I-JEPA over images, V-JEPA over video. |
dew.config.diffusion | The diffusion run as one typed record, and the one function that builds it. |
dew.config.world_model | Dreamer 4’s world model as a run over video (docs/design/post-training.md §14.4): its tokenizer, then its dynamics over the frozen tokenizer’s latents. |
dew.objectives.rl.config | The environment run as one typed record: the environment an agent acts in, the agent’s networks and objective, the rollout that collects its batches there, and a teacher that labels or demonstrates its steps (docs/design/post-training.md §14). |
dew.config.sweep | Hyperparameter search that trains each trial through the ordinary training path. |
Utilities
Section titled “Utilities”| Module | Summary |
|---|---|
dew.rl | Reinforcement learning on plain arrays and pure functions: its math, its environments, their physics and renderer, a replay and simulators. |
dew.rl.env | Environments as two pure functions over a pytree, and the one loop over them. |
dew.rl.simulator | A simulator outside JAX as an environment: the boundary any GPU simulator plugs into. |
dew.rl.quadrotor | A racing quadrotor’s physics in JAX, one drone at a time (vmap for a batch), every parameter drawn per episode for transfer from simulation. |
dew.rl.flight | First-person drone gate racing in JAX: one drone’s race as a dew.rl.env.Env (FlightEnv), on device and jittable whole, flown by dew.rl.quadrotor’s physics; and a batch of them seen through a renderer’s camera as a dew.rl.simulator.Simulator (GateRace). |
dew.rl.hyperlux | HyperLux’s batch ray tracer (rl-tools’ hyperlux 0.1.10, the hyperlux extra, on an NVIDIA GPU with OptiX) drawing each drone’s front camera of the gate race (dew.rl.flight.Renderer), its scene built from the GLB assets tools/fpv_assets.py writes. |
dew.rl.replay | A replay of environment steps held on the device, written and sampled by jittable functions. |
dew.rl.playground | MuJoCo Playground’s MJX environments as dew.rl.env.Envs. |
dew.rl.distributions | DreamerV3’s output transforms and distributions on plain arrays (arXiv 2301.04104 and its code, danijar/dreamerv3 at e01491fad6434b2245a3b8ca201dd7faedcc458c, embodied/jax/nets.py and outs.py). |
dew.rl.aggregation | A student’s own steps labelled by a privileged teacher and aggregated into what the student learns from: DAgger (Ross, Gordon and Bagnell, arXiv 1011.0686), for any teacher policy and any student policy over one environment. |
dew.rl.skydreamer | SkyDreamer’s race (Verraest et al., arXiv 2510.14783) flown from the camera’s RGB image by collective thrust and body rates: its task, reward, termination and starts, with Dew’s quadrotor and renderer. |
dew.rl.evaluation | A policy scored by the episodes it flies rather than by records it reads: episodes(policy, env, steps), the scorer dew.training.evaluation.Episodes runs on a run’s validation cadence, beside its data splits. |
dew.artifacts | Typed values that an objective’s evaluation produces. |
dew.distributed | What the processes of a pool do together on the host: agree a phase or a value before going on, so a failure on one rank reaches every rank rather than leaving the rest inside a collective (agreed, agree_process_phase, agreed_same), and bring a global tree to every rank or the first (collective_host), one rank’s value to all (broadcast_from_process_zero) or every rank’s to each (from_every_process). |
dew.cache | Where Dew keeps what it caches on disk, and JAX’s persistent compilation cache. |
dew.telemetry.profile | Process-local JAX profile captures, exported as XProf’s own reports. |
dew.telemetry.benchmark | Wall-clock timing of calls: samples, which every benchmark tool in Dew measures independent calls with, carried, which times calls that carry a state from one to the next, benchmark, which lays out many calls’ timings as a table, and compared, which reads two calls against each other by their paired ratio; and the process’s resident memory as the kernel reports it (resident_memory), which the memory benchmarks read. |
dew.figures | SVG figures for notebooks and docs, in colors that follow the page’s light or dark theme. |