Key concepts
A Dew training run is built from four objects, each responsible for one part of the work:
| Object | Responsible for | Written by you when |
|---|---|---|
Model, a flax.linen.Module | The forward computation and the shapes of its variables | You need a new architecture. The built-in ones are classes in dew.nn.backbones. |
Objective | Initializing the variables, the loss, evaluation outputs, and which weights keep a moving average | You need a loss Dew does not have. |
Dataset | The records, from a source through your steps and a preset, in batches | Always: a source and a preset, plus any steps of your own. |
Trainer | The device mesh, the compiled step, the optimizer update, the moving average, checkpoints and logging | Never; it is configured, not subclassed. |
Trainer.fit returns a TrainState, the numerical state of the run: the variables, the optimizer state, the root random key, the step counters and the moving-average copy. Continuing a run also needs the data position, which checkpoints store beside the state, and the configuration that built the objective and trainer (a recipe writes it to run.json).
Example
Section titled “Example”This trains a small decoder on one repeated sentence and generates from it. It downloads nothing and runs on a CPU.
import jaximport numpy as np
from dew import Trainerfrom dew.optim import OptimConfigfrom dew.data import ByteTokenizer, Memoryfrom dew.nn.backbones import CausalTransformerfrom dew.objectives.lm import LMObjectivefrom dew.sampling import Sampling, generate
tokenizer = ByteTokenizer()text = tokenizer.encode("dew trains jax models. " * 3)rows = np.tile(np.asarray(text[:65], np.int32), (8, 1))data = Memory({"text": rows}).repeat().batch(8)
model = CausalTransformer( vocab_size=tokenizer.vocab_size, emb_features=64, num_layers=2, num_heads=4, mlp_features=256, max_seq_len=128)objective = LMObjective(model, seq_len=64)trainer = Trainer(objective, OptimConfig(learning_rate=3e-3), key=jax.random.key(0))state = trainer.fit(data, steps=100, log_every=25)
prompt = [tokenizer.encode("dew")]out = generate(model, state.variables, prompt, max_new_tokens=40, key=jax.random.key(1), sampling=Sampling(temperature=0))print(tokenizer.decode(out.tokens[0]))Output on two cores of a shared workstation CPU:
Training CausalTransformer from step 0 to 100: 147,840 parameters, on 1 × cpu, batch 8, float32step 25/100 loss 0.03061 ce 0.03061 perplexity 1.031 token_accuracy 100.0% step_time_ms 23.68 samples_per_sec 337.9 accepted 100.0% 0:00:01 leftstep 50/100 loss 0.01018 ce 0.01018 perplexity 1.010 token_accuracy 100.0% step_time_ms 24.52 samples_per_sec 326.2 accepted 100.0% 0:00:01 leftstep 75/100 loss 0.006710 ce 0.006710 perplexity 1.007 token_accuracy 100.0% step_time_ms 24.68 samples_per_sec 324.1 accepted 100.0% 0:00:01 leftstep 100/100 loss 0.005113 ce 0.005113 perplexity 1.005 token_accuracy 100.0% step_time_ms 22.84 samples_per_sec 350.2 accepted 100.0%Trained 100 steps in 0:00:04: first step after 2.06 s, then 42.6 step/s53.0% of the wall time in steps, final loss 0.005113dew trains jax models. dew trains jax modelThis is the output with stdout piped to a file; on a terminal, fit draws the same numbers as one live panel with a progress bar and a sparkline per metric. Each step line shows the loss and the objective’s metrics of that step (for LMObjective: cross entropy, perplexity and token accuracy), and the step time and throughput over the interval since the previous line. Each row holds 65 byte ids. LMObjective(seq_len=64) feeds the first 64 to the model and scores its predictions of the 64 that follow, shifted by one position. generate returns the prompt followed by the new tokens; temperature=0 picks the most likely token at every step.
A model is a flax.linen.Module with init and apply. Its variables are a nested dictionary of arrays, and it has no knowledge of training. Build one from its class, from dew.nn.backbones import CausalTransformer. A saved run records the class by its import path, and a configuration file may name it by a short alias (causal_transformer) instead.
Your own model needs nothing to be recorded: its import path names it, and its constructor fields are its record. Put it in a module, here mymodels.py:
import flax.linen as nn
class ResidualMLP(nn.Module): """A denoiser `DiffusionObjective` can train: it maps a noisy sample, the noise level's embedding and optional text context to its prediction."""
features: int = 64
@nn.compact def __call__(self, x, temb, textcontext=None, train=False): return x + nn.Dense(x.shape[-1])(nn.gelu(nn.Dense(self.features)(x)))Every checkpoint a run of this model writes records "architecture": "mymodels:ResidualMLP" and "config": {"features": 64}, and TextToImage.from_run and dew.pipeline rebuild the model from that record in any process that can import mymodels. A decoder’s run loads the same way through TextGeneration.from_run and Pretrained.from_run. A composite model records each part inside its own record, and an adapted model records its base model with the adapter’s rank, alpha and modules.
A package outside Dew works the same way, with nothing to register: a field typed with a class takes the record {"class": "mypackage.layers:Gate", "fields": {...}} of any class that derives from it or function declared to return it, and a function as {"function": "mypackage.layers:gate"}. Where a person writes a record, the class may be one of Dew’s short aliases instead ("simple_dit", "mla"), which dew.registry lists by kind.
A package can bring a whole kind of run the same way. Its own RunConfig subclass builds its objective and data in prepare, and run.json names that class, so dew train runs/x/run.json --trust mypackage trains it again in any process. Its objective may score into a dataclass of its own (dew.artifacts.Artifact), which its metric reads by declaring reads = MyArtifact; the trackers show previews of Dew’s artifact types, so a package shows its own as one of them, a spike raster as an ImageGrid. Its modules may name logical axes of their own (nn.with_logical_partitioning(init, ("inputs", "neurons"))), which the run places by adding rules to its layout, Layout(rules=(*DEFAULT_RULES, ("neurons", "fsdp"))). A record arriving from elsewhere imports a package’s modules only once the reader trusts the package (trust=("mypackage",), --trust mypackage), as trust_remote_code does in transformers.
A model defined where no import path reaches it, in __main__ or inside a function, trains, checkpoints and resumes the same way. The first checkpoint warns, and loading the run says, that its model has no import path.
A model written in Flax NNX trains through Flax’s own bridge, flax.nnx.bridge.ToLinen, which turns an NNX module into a Linen one with its parameters, its other state (a BatchNorm’s batch_stats) and its random streams (the dropout stream an objective passes to apply). The bridge runs the module’s __call__, which is all DiffusionObjective calls on a denoiser: bridge.ToLinen(MyDenoiser, args=(channels,)) is the model. An objective that reads another method, such as hidden_states and logits for LMObjective, needs a Linen class that names it, a few lines of your own:
from flax.nnx import bridge
class MyTokens(bridge.ToLinen): causal: bool = True
def hidden_states(self, tokens, *, train=False, **fields): return self(tokens, nnx_method="hidden_states", train=train, **fields)
def logits(self, tokens, *, train=False, **fields): return self(tokens, nnx_method="logits", train=train, **fields)
model = MyTokens(MyNNXLanguageModel, args=(vocab_size,))Such a model checkpoints and resumes as any unregistered model does. LoRA cannot adapt it: the adapter intercepts flax.linen.Dense calls, and an NNX Linear behind the bridge makes none, so LoRA.apply refuses an NNX model rather than train factors the forward never reads. Adapt it in NNX before the bridge instead, with flax.nnx.LoRALinear in place of nnx.Linear or flax.nnx.LoRA beside it.
Dew’s modules name the logical axes of their parameters, such as embed, heads and mlp. The trainer maps those names onto the device mesh, so the model code does not change when the mesh does. Distributed training describes the mapping.
Objective
Section titled “Objective”An objective implements two methods:
init(key, variables=None)returns the model’s variables.loss(variables, batch, step)returns the loss as aRatio(total, mass)and anAuxwith metrics to log.
It can also implement evaluate for validation and preview for samples, and it names the weights that keep an exponential moving average (EMA) in its ema attribute. Its shown attribute maps metric names to Shown values that tell the training display how to show them: Shown(better="higher", percent=True), for an accuracy, colours a rise as progress and prints the value as a percentage.
Dew ships objectives for autoregressive language modeling (LMObjective), image and video diffusion (DiffusionObjective), masked and block diffusion over tokens (MaskedDiffusionObjective, BlockDiffusionObjective), JEPA (JepaObjective), preference and reinforcement learning (DPOObjective, GRPOObjective, PPOObjective, FlowGRPOObjective) and distillation (DistillationObjective). A new kind of model is a new module; a new kind of training is a new objective. Custom objectives writes one.
Dataset
Section titled “Dataset”A Dataset is one chain: a source (Memory, Files, Hub or TFDS), the steps that read its records, and a preset or batch(n) that ends it. Trainer.fit compiles it into a Loader of two functions: train(partition) opens training batches and val(partition) opens one finite validation pass. DataPartition names the process’s share of the global batch. The example’s chain is a Memory source over column arrays, repeated and batched, and checkpoints save its consumed position.
The presets in dew.data.presets prepare one kind of record for a model and end in batch: images arrive as uint8 arrays in [0, 255] and token windows as int32 IDs under "text". A custom order writes the chain’s verbs itself. Training data covers read modes, storage, mixtures and exact resume.
Trainer
Section titled “Trainer”Trainer(objective, optimizer, key=...) takes an OptimConfig or an Optax optimizer, and a JAX random key or integer seed. fit(dataset, steps=...) builds a configured optimizer over the run’s length, places the variables on the devices, compiles one training step and runs it until the step counter reaches steps. Between steps it logs every log_every steps, evaluates every eval_every steps, and writes a checkpoint every checkpoint_every steps when the trainer was given checkpoints=Checkpoints(directory).
The same call runs on one device or many. Trainer(..., mesh=MeshSpec(fsdp=4)) shards the parameters and optimizer state over four devices; the objective, the model and the data do not change.
The trainer logs the loss, the objective’s metrics, an injected learning rate and train/grad_norm: the mean global norm of the gradient each committed update hands the optimizer, before any clipping. A compiled step returns it as Committed.norm over Committed.count.
Training state
Section titled “Training state”TrainState keeps three counters. step counts attempts, and together with the root key it determines the next random draw. microstep counts accepted microbatches. updates counts optimizer updates, which differs from microstep when gradients are accumulated. state.variables holds the live variables, and state.averaged holds the same tree with the EMA weights in place.
After training, objective.pipeline(state) wraps the weights in an inference task, such as text generation or text-to-image, and Pretrained.save writes a checkpoint in its source’s own format. Checkpoints lists which artifact continues what.