Skip to content

Language models

LMObjective trains a decoder on next-token prediction. It reads rows of token IDs, feeds all but the last token to the model and scores its predictions against the following tokens with cross entropy. The decoder is usually CausalTransformer from dew.nn.backbones, and dew.sampling.generate draws text from the trained weights. This page also covers loading published checkpoints with Pretrained.load, and the two non-autoregressive language objectives, MaskedDiffusionObjective and BlockDiffusionObjective.

The example writes a small corpus of token records, trains a two-layer decoder on it and generates a continuation. It uses ByteTokenizer, which maps each UTF-8 byte to one of 256 IDs, so it downloads nothing.

import numpy as np
from dew.data import ByteTokenizer
from dew.data.sinks import export
tokenizer = ByteTokenizer()
stories = [
"Once upon a time, there was a little girl named Lily. She had a red ball.",
"One day, Lily saw a big dog in the park. The dog wanted to play.",
"Lily threw the ball and the dog ran after it. They played all day.",
"At night, Lily went home. She was happy and tired.",
]
ids = np.array(tokenizer.encode(" ".join(stories * 200)), dtype=np.int32)
held_out = len(ids) // 20
metadata = {"token_store": True, "tokenizer": "byte", "vocab_size": tokenizer.vocab_size,
"dtype": "int32", "eos_id": None}
export([{"text": ids[held_out:]}], "data/stories-byte/train", format="array_record",
metadata=metadata, length_field="text")
export([{"text": ids[:held_out]}], "data/stories-byte/val", format="array_record",
metadata=metadata, length_field="text")
print(len(ids) - held_out, "training tokens,", held_out, "validation tokens")
48830 training tokens, 2569 validation tokens

export writes each split as committed ArrayRecord shards with a manifest recording the tokenizer, vocabulary size and dtype. Each holds the token IDs as one stream, and the validation split is the head of it.

import jax
from dew import Trainer
from dew.optim import OptimConfig
from dew.data import presets
from dew.data.sources import TokenFiles
from dew.nn.backbones import CausalTransformer
from dew.objectives.lm import LMObjective, Perplexity
from dew.sampling import Sampling, generate
train, val = TokenFiles("data/stories-byte/train"), TokenFiles("data/stories-byte/val")
data = train.shuffle(seed=0).repeat().pipe(presets.token_windows, seq_len=64, batch=16)
held = val.pipe(presets.token_windows, seq_len=64, batch=16, records=64)
model = CausalTransformer(vocab_size=tokenizer.vocab_size,
emb_features=64, num_layers=2, num_heads=2, mlp_features=256,
max_seq_len=128)
objective = LMObjective(model, seq_len=64)
lm_state = Trainer(objective, OptimConfig(learning_rate=3e-3), key=jax.random.key(0)).fit(
data, validation=held, steps=100, log_every=50, eval_every=100, metrics=(Perplexity(),))
continuation = generate(model, lm_state.variables, [tokenizer.encode("One day, Lily")],
max_new_tokens=40, key=jax.random.key(1),
sampling=Sampling(temperature=0.0))
print(tokenizer.decode(continuation.tokens[0]))
Training CausalTransformer from step 0 to 100: 147,904 parameters, on 1 × cpu, batch 16, float32
step 50/100 loss 0.2531 ce 0.2531 perplexity 1.288 token_accuracy 95.6% step_time_ms 32.71 samples_per_sec 489.1 accepted 100.0% 0:00:01 left
step 100/100 loss 0.09353 ce 0.09353 perplexity 1.098 token_accuracy 97.2% step_time_ms 27.76 samples_per_sec 576.4 accepted 100.0%
eval val at step 100: perplexity 1.087 loss 0.08359 (40 records in 0.91 s)
Trained 100 steps in 0:00:10: first step after 6.46 s, then 35.6 step/s
27.2% of the wall time in steps, final loss 0.09353
One day, Lily saw a big dog in the park. The dog want

TokenFiles reads each store, and the token_windows preset cuts its stream into windows by index: seq_len=64 yields rows of 65 IDs under the key text. LMObjective(model, seq_len=64) feeds the first 64 to the model and scores the predictions against the next 64. Token IDs must be integers inside the model’s vocabulary. temperature=0.0 picks the most likely token at every step, and the returned row holds the prompt followed by the continuation.

The training loss is near zero because the same four sentences repeat, and validation on the held-out start of the stream scores just as low. Validation and the generation above both use the trained lm_state.variables, since the objective keeps no moving average unless ema_decay is set.

The same code trains a decoder on TinyStories, a corpus of short stories in simple English, with the GPT-2 tokenizer. Download the 22 MB validation file of TinyStories V2 and export two token stores; the script reserves the first 1% for validation:

Terminal window
hf download roneneldan/TinyStories TinyStoriesV2-GPT4-valid.txt \
--repo-type dataset --local-dir data
python - <<'PY'
from pathlib import Path
import numpy as np
from dew.data import HFTokenizer
from dew.data.sinks import export
tokenizer = HFTokenizer("gpt2")
ids = np.asarray(tokenizer.encode(Path("data/TinyStoriesV2-GPT4-valid.txt").read_text()), np.int32)
held = int(len(ids) * 0.01)
metadata = {"token_store": True, "tokenizer": "gpt2", "vocab_size": tokenizer.vocab_size,
"dtype": "int32", "eos_id": None}
export([{"text": ids[held:]}], "data/tinystories/train", format="array_record",
metadata=metadata, length_field="text")
export([{"text": ids[:held]}], "data/tinystories/val", format="array_record",
metadata=metadata, length_field="text")
PY

The measured run changes these settings from the example:

SettingValue
TokenizerHFTokenizer("gpt2")
DataTokenFiles("data/tinystories/train") and TokenFiles("data/tinystories/val"), each through presets.token_windows with seq_len=256, batch=32
Modelemb_features=256, num_layers=4, num_heads=4, max_seq_len=256, dtype=jnp.bfloat16
OptimizerOptimConfig(learning_rate=1e-3)
ObjectiveLMObjective(model, seq_len=256, ema_decay=0.999)
Runsteps=2000, log_every=500, eval_every=1000
Prompt"Once upon a time", 40 new tokens, temperature=0.0

On one Colab L4 GPU the run takes about three minutes. The training loss falls from 2.75 at step 500 to 1.95 at step 2,000, and validation perplexity reaches 8.7. Validation reads the EMA weights, and at step 1,000 their perplexity is 29.3. One run’s greedy continuation reads:

Once upon a time, there was a little girl named Lily. She had a big, red ball. Lily loved to play with her ball. One day, she saw a big box. She wanted to open it.

GPU reductions are not bitwise repeatable by default, so a second run can continue differently after the first sentence. Post-training continues a decoder with SFT, DPO and GRPO, and Inference covers generation tasks.

Use one tokenizer everywhere: data preparation, model construction, decoding and checkpoint export. ByteTokenizer has a vocabulary of 256 and uses ID 255 as its EOS. HFTokenizer(name) wraps a Hugging Face tokenizer with that model’s vocabulary and chat template, and downloads its files on first use.

ByteTokenizer and HFTokenizer have encode and decode. A source loaded with Pretrained.load has the checkpoint’s own processor instead, which has no encode; you call it on text. bundle.processor(["The capital of France is", "Hello"]) returns ModelInputs, whose tokens are the ID rows padded the way the tokenizer pads and whose kwargs() hold the attention mask and positions. bundle.processor.decode(rows) turns IDs back into strings, and bundle.processor.chat(messages) runs the checkpoint’s chat template when it has one.

RunProcessor(tokenizer) in dew.inference wraps an encode/decode tokenizer in the same kind of callable, which is what a task takes as processor=. A text-to-image run’s captions are tokenized by its condition encoder, for example CLIPText.tokenize in dew.inputs.

export writes token records as committed ArrayRecord shards with a producer manifest and length index. Use the same tokenizer for data, model construction, decoding and checkpoint export. dew data export writes one from the command line, from a store’s URL (a text file’s lines here), a chain’s .json record or a Python file’s data chain; --tokenizer and --eos tokenize the records as text.export_tokens does.

Terminal window
dew data export data/corpus.txt data/corpus-byte --tokenizer byte --eos --read memory

That command publishes the complete input. Prepare held-out records separately and read them as a second Files source. To preserve a token-axis head split, encode the stream once and export two slices:

from pathlib import Path
import numpy as np
from dew.data import ByteTokenizer
from dew.data.sinks import export
tokenizer = ByteTokenizer()
ids = np.asarray(tokenizer.encode("The cat sat on the warm mat.\n" * 256), np.int32)
held = int(len(ids) * 0.1)
metadata = {"token_store": True, "tokenizer": "byte", "vocab_size": 256,
"dtype": "int32", "eos_id": None}
export([{"text": ids[held:]}], "data/corpus-byte/train", format="array_record",
metadata=metadata, length_field="text")
export([{"text": ids[:held]}], "data/corpus-byte/val", format="array_record",
metadata=metadata, length_field="text")

TokenFiles("data/corpus-byte/train") and TokenFiles("data/corpus-byte/val") read those stores, and presets.token_windows with seq_len=64, batch=8 cuts each into windows. Indexed mode reads producer lengths without scanning token payloads.

  • token_windows, and presets.lm with pack=None, cut fixed windows of seq_len + 1 IDs, {"text": int32 [B, seq_len + 1]}. Each starts seq_len IDs after the last.
  • presets.lm with pack="first-fit", its default, keeps documents in padded windows and adds aligned segment IDs and positions; pack="concat" concatenates and splits aligned fields. Over a store’s documents it takes tokenizer=None.

LMObjective(model, seq_len, **options) reads batch["text"] of shape [B, seq_len + 1]. model may be a loaded decoder bundle in place of the model (see Pretrained checkpoints). Every auxiliary term is off until its argument is set.

ArgumentDefaultMeaning
ema_decayNoneDecay of an EMA copy, which evaluation and previews then read. None keeps no copy.
pad_idNoneToken ID whose targets carry no loss.
head_chunks4Vocabulary slices the tiled head scores in. 1 is the full pass.
head_tileNoneBackward tile of the head, or 'whole' / 'tiled'. None keeps the whole logits where they fit, and the fit ladder tiles them where they do not. The masked- and block-diffusion objectives take head_chunks and head_tile too: every token objective runs its head by this one policy (dew.objectives.lm.head).
samplesNoneSamples configuration for generated previews at evaluation.
variablesNoneThe tree to start from instead of a fresh init, whole or split by freeze or an adapter. A split tree stays split, so its params train and its frozen part does not change.
loss_roleNoneCount only targets whose text_roles entry equals this Role (SFT).
balanceNoneRouter training for mixture layers, an ExpertBalance (dew.objectives.routing): balance_rate, the aux-loss-free routing-bias update rate; aux_loss_alpha and seq_aux, DeepSeek V2’s expert balance loss and its per-sequence form, which balances each packed document alone (Megatron-LM and MaxText balance a packed row as one sequence); and router_z_loss, ST-MoE’s router z-loss coefficient.
mtp_weightNoneWeight of DeepSeek V3’s multi-token prediction loss.
z_loss0.0PaLM’s squared log-partition auxiliary.
token_accuracyTrueReport argmax accuracy.
prefixNoneA prefix run: each document’s first tokens (an int) or its prompt ("prompt") form one block the model reads whole, and no target inside it is scored (Prefix blocks).
adaptationNoneChunkAdaptation(window, rate, select, damped, carry, layers): step the model’s continuum levels, or its own feed-forward blocks of layers, in context by this loss (test-time training over the task loss, below). None reads the levels as trained.

The step reports ce, perplexity and token_accuracy. LMObjective refuses a model built with causal=False.

A decoder built with continuum=ContinuumLevel(hidden) (dew.nn.continuum) reads a small residual level, x + up(act(down(x))), on its final states before its head. up starts at zero, so the level starts as the identity and the decoder’s logits are its own. LMObjective(model, seq_len, adaptation=ChunkAdaptation(window=16, rate=0.01)) steps that level in context by the loss it scores, Nested Learning’s continuum memory (Behrouz et al., arXiv 2512.24695, section 7.1, equation 71): every window tokens of a document the level steps by rate times the summed gradient of those tokens’ weighted next-token losses, and the next window reads the stepped level. Each document, a packed row’s included, starts from the level the run trains, and training learns that start through every step. That is the default, carry="document", Titans’ and TTT’s reset. ChunkAdaptation(carry="run") keeps the level instead, as Nested Learning’s continuum memory does. Each row’s documents step one level in turn, so a packed row’s second document reads the level its first left. Each committed update carries the mean, over its rows, of how far their windows moved the level, the last window’s step applied. The run holds what it carried in the carried collection, starts it at zero, saves it with its state and resumes it bit for bit. The next update’s documents start from the trained level plus what is carried. Evaluation and validation score the same way. select names the level’s parts that step (down, up); the others hold their trained values.

A step whose rate times the loss’s curvature passes 2 diverges, as gradient descent does: over weights drawn at 0.5 N(0, 1), a rate of a half grows a row’s rounding 10^4-fold in three windows. ChunkAdaptation(damped=True) steps the selected parts’ kernels (down, up, or both when select is empty) at a rate d with d <= rate / (1 + rate * L(d)) instead, where L(d) bounds the window’s loss curvature in those kernels along a step of rate d, so d * L(d) stays below 1. The step settles d toward the largest such rate, the bound’s fixed point, rising to it from below. L combines two kinds of bound. The model’s bounds on a slot loss’s curvature and slope in the states its head reads (head_bounds) cover an affine head, a logit softcap, and a head transform, whose norm is bounded at the states the step can reach from the window’s (a norm states its bounds through dew.nn.protocols.BoundedNorm). The activation’s bounds on |act'| and |act''| are combined with the window’s weighted sums of |x|^2, |f|^2, |f| |x| and |x|, where f = act(down(x)), each read as its map multiplies it (a checkpoint’s quantizer of the map’s input applied), and with the kernels’ norms along the step. Under Qwix’s forward quantization each operand’s norm is the most its rounding can grow it to (twice itself where Qwix takes a small float16 group’s scale as 1), and a step follows Qwix’s straight-through gradient: the bound holds for the surrogate’s smooth part, and a step that crosses a rounding cell adds the quantization’s own jump. The rate is a constant of the step, so the outer gradient flows through the step’s gradient alone. An adapter’s factors hold. A damped step never overshoots, whatever the rate. Under an affine head with up alone stepping, the loss is convex, so every damped step is also non-expansive: two nearby levels move no further apart. A softcap, a head transform, or a step of down makes the loss non-convex. A step below 1 / L there can expand two levels’ distance by at most 1 + rate * L (under 2) a window, not the unbounded growth of an undamped step. Some cases have no bound, so they are refused, with the reason in the message: a step of down under ReLU, whose act'' is a point mass; a head map or an up that a method interceptor computes other than as a linear map (a checkpoint’s quantizer of a head map’s input, a frozen FP8 product, an adapter’s dropout), which makes the loss piecewise constant in what it reads; a head transform whose norm states no bounds; a Qwix weight calibration other than absmax or absmax,<clip>, whose clip no bound reads; and a Qwix rule that quantizes the backward (bwd_qtype), whose rounded gradient can reverse a small one, so no rate makes the step a descent. A quantizer of down’s input rounds the states, which no step moves, so every part still steps there.

A chain of levels, continuum=(ContinuumLevel(hidden=64), ContinuumLevel(hidden=256)), is the section’s multi-frequency memory (equation 70): each level reads the one before it, and ChunkAdaptation(window=(16, 64), rate=(1e-2, 2e-3)) steps each at its own window and rate, one for each level in the chain’s order (a single window or rate applies to every level). The windows, ordered, each divide the next, so a slower level steps on a faster one’s boundary. A token reads every level at the version its own window of that level opened with, and each window’s gradient in its level is its tokens’ own, read through the other levels as they read them. A damped chain bounds each level’s curvature through the levels after it, their Jacobians and second derivatives in the states they read, at the states and versions each token read, so every level’s step stays below 1 / L.

ChunkAdaptation(layers=(26, 27)) steps a decoder’s own feed-forward blocks instead, section 7.3’s ad-hoc level stacking: a pretrained model’s blocks are the levels, its trained weights their starts, and the model’s tree is unchanged. A block mid-stack reads every earlier token through attention, so the row runs chunk by chunk, a chunk the least window, through the decoder’s decode cache: each chunk reads the earlier chunks from the cache as they were, at the blocks’ versions they read, and each window’s gradient in its block sums its chunks’ loss gradients there, the cache holding the earlier chunks fixed. Continued pre-training through the steps trains the blocks’ starts, the outer loss’s meta-gradient. The decode cache keeps no document ids, so stacked blocks read one document a row; they collect no routing or attention statistics, and a damped step is refused there, as attention’s curvature in a mid-stack block’s output has no useful bound.

dew.eval.continual measures continual learning: continual_run(tasks, fit, accuracy) trains a sequence of tasks and scores every task after each, its AccuracyMatrix giving the final average accuracy, each task’s forgetting (its best accuracy less its last) and the backward transfer, and label_scores scores a language model’s choice among labels by the log-likelihood its objective gives each, in context or not.

No generation steps the levels yet, so an adapted objective takes no samples, refuses a rollout’s policy and records no inference task; its runs save and resume their trained state. A continuum level has no transformers counterpart, so a decoder with one does not export.

The vocabulary loss runs in float32. The head’s matrix product uses the compute dtype, as it does in torch autocast and MaxText. Under bf16 compute, the hidden states and the head multiply as bf16 and sum in float32, in the backward pass too, which is 1.5x faster on an L4 and 1.6x faster on an RTX 4080 than with fp32 operands. At the default matmul_precision (unset or "default"), the logits are then rounded to bf16 and their gradient is rounded to bf16 once, as torch autocast’s and MaxText’s bf16 logits are. The softmax and the loss read them in float32. An fp32 model multiplies in fp32 (Performance measurements).

That rounding costs nothing measurable in training. The comparison ran 2000 steps of wikitext-103 on an RTX 4080 against float32 logits and a float32 gradient at the same seed. Validation loss moved by at most 4.5e-4 for a 3-layer decoder trained from scratch and 1.6e-3 for a Qwen3-0.6B fine-tune, while two seeds of either rounding are 1.3e-2 and 2.9e-3 apart on average. The rounding makes the 3-layer decoder’s step 13% faster and Qwen3-0.6B’s 3% faster at 1 x 1024 tokens.

The rounding does show in layout parity. With the logits’ gradient rounded once, a bf16 run on 4 RTX 3090s read 1.75 times its layout-parity bound at a dense model’s final norm and 5758 times it at an MoE’s expert gate_proj, against 0.47 and 0.41 with the gradient kept in float32. So a run that compares layouts in bf16, or resumes onto a different mesh and expects the same numbers, should set matmul_precision="highest", which keeps the head in float32. tools/layout_parity.py sets it for bf16 decoders. An LM run recorded before this change resumes with the new rounding, which is within the rerun spread above but not bitwise the same as before.

token_accuracy=False (--objective.no-token-accuracy on the recipe) skips the argmax over every logit, which costs 0.77 ms of the head’s 8.0 ms on a TPU v6e. Chunking the head with head_chunks can lower peak memory but adds work, and the backend compiler may rewrite it; the saving depends on vocabulary size, sequence length, batch size and the compiled executable.

With bf16 compute, Dew keeps the residual stream and every norm and sublayer output in bf16, each rounded where transformers rounds a bf16 tensor, as MaxText and Megatron (with fp32_residual_connection=False) do. torch.autocast keeps the residual stream and the norms in float32 and rounds at each matmul’s input. To compare at Dew’s rounding points, the torch run uses autocast with the embedding output and every RMSNorm output cast to bf16 and the rotary table left in float32 (tools/reference_runs/torch_lm.py --precision autocast-bf16-residual). At those rounding points the first step’s loss depends on the attention kernel as much as on the framework. The table shows the first batch of a Qwen3-0.6B fine-tune (4 x 1024 tokens, one A100, tools/reference_runs/step0_attention.py):

RunAttention kernelFirst-step loss minus float32Final hidden states, relative distance from torch’s FlashAttention 2 run
torchFlashAttention 2-1.60e-40
torchcuDNN+8.17e-41.42e-2
torchmath-1.40e-41.42e-2
DewcuDNN+1.09e-31.38e-2
DewXLA+1.10e-31.40e-2

Plain torch.autocast with FlashAttention 2 is at +4.9e-4. Over 256 steps of the same fine-tune, every 32-step window of Dew’s loss and gradient norm stays within twice the distance from float32 of the torch run at Dew’s rounding points, or within twice that run’s run-to-run spread where the spread is larger.

LMObjective keeps no moving average unless ema_decay is set. Without one, state.ema is None, state.averaged raises ValueError, and previews and evaluation read the live variables. With ema_decay=0.999, state.averaged holds the EMA copy merged into the live variables, evaluation and previews read it, and the console labels the split val (ema). The copy costs one more set of trained parameters. A checkpoint written with one setting does not restore into the other.

To measure validation perplexity, give the dataset a validation split and set eval_every, as in the example (Evaluation and tracking). With a tracker and fit(preview=True), Samples adds one generated preview per evaluation event, separate from the teacher-forced scoring of the batch.

With gradient accumulation, the cross-entropy and multi-token-prediction (MTP) losses are weighted by the mass of supported main targets, including the target-role masks at each MTP depth. Sequence-level router losses normalize by rows. Global router losses first add up the selected-slot counts and the score sums, then take their product. The routing bias stays fixed during an accumulation window and is updated from the window’s total counts. A window with zero cross-entropy can still update when a router auxiliary loss is active. Accumulated and combined batches match only when their random draws are the same and mutable state follows its declared rules.

prefix trains a prefix run. Each document’s first prefix tokens, an int, or its tokens before the first assistant token, "prompt", form one block. A layer that reads blocks whole (AttentionMixer(bidirectional_blocks=True) on one of the model’s layer kinds) lets each token of the block see the block’s later tokens too, and every other token stays causal. The loss scores no target inside its source’s block, since the source would read that target off; each other target counts as usual.

from dew.nn.mixers import AttentionMixer
# The example's decoder with its one layer kind reading blocks whole.
prefixed = CausalTransformer(vocab_size=tokenizer.vocab_size, emb_features=64, num_layers=2, num_heads=2,
mlp_features=256, max_seq_len=128, layer_types=("full_attention",) * 2,
kinds={"full_attention": {"mixer": AttentionMixer(bidirectional_blocks=True)}})
prefix_objective = LMObjective(prefixed, seq_len=64, prefix=16) # or prefix="prompt" on chat rows

On the recipe, --objective.prefix 64 or --objective.prefix prompt. The model’s trunk must read blocks in at least one layer kind; a prediction depth’s kind does not count. The run refuses a prefix longer than seq_len, and mtp_weight, because a prediction depth’s layers read blocks beyond their own source and the exclusion does not cover them. A layer kind’s attention chunk (LayerKind.chunk) cuts a block wherever the block crosses one of its multiples, and a block’s places start wherever its tokens sit in their document (a window that continues a document, or a prompt after cached tokens), so a chunk of any length can cut one. The loss would drop targets that layer never saw, so a prefix, and any batch’s blocks, need a trunk layer that reads blocks with no chunk.

Serving reads the same blocks. A served prompt’s first prefix tokens form the block (the whole prompt under "prompt"), and its generated tokens stay causal, as in training. TextGeneration does this for a task built from a run’s settings, and a server does it per request. A server refuses a prompt shorter than prefix, since its trained block would cover tokens the prompt does not hold, and one that prefills in pieces (chunk) refuses a block longer than a piece, since a block read whole cannot be split across pieces. Shared prefix pages (prefix_cache) are shared only between prompts whose blocks hold the same tokens.

The data stage Blocks (dew.data.steps.blocks) makes blocks for any chain: Blocks(prefix=N), Blocks(prompt=True) on chat rows, or Blocks(given="field"), where the field gives each token a number (-1 for none) and each run of consecutive tokens with one number inside one document is a block. A batch carrying text_block_groups trains with those blocks whatever prefix says, and a model that reads no blocks refuses such a batch. A prefix run’s batch that carries none, as a precompiled loader or a raw stream gives it, takes the prefix’s blocks where the loss scores it and where the host counts its targets (prefixed_fields, the stage’s own rules), so a run trains on the blocks it serves whatever built its batches; a prompt prefix reads them off the rows’ text_roles, and refuses a batch without them.

A layer that reads blocks builds a dense mask and runs the XLA attention kernel. The in-block targets are not scored, so a prefix run counts fewer targets than the causal run over the same rows.

Pretrained.load(name_or_dir) reads a local Hugging Face directory or a Hub identifier into a Pretrained bundle that holds the native Flax model, its variables, the checkpoint’s processor or tokenizer, the source config and the generation defaults. The bundle’s class depends on the kind of source it read, and each class has the methods that work for that kind: PretrainedDecoder (text generation), PretrainedMaskedDecoder (LLaDA, Dream), PretrainedBlockDecoder (DiffusionGemma), PretrainedPipeline (latent diffusion, text_to_image) and PretrainedFallback (tier 3, transformers’ own forward; text generation where its attention is Dew’s). Every kind can take an adapter (adapt), and you can pass any of them in place of the model to the objective that trains it. Calling load on a specifi…

LMObjective(bundle, seq_len, **options) trains a decoder bundle from its loaded weights. With token files prepared using the checkpoint’s tokenizer (dew data export ... --tokenizer Qwen/Qwen3-0.6B), fine-tuning uses the same trainer as training from scratch:

from dew.data import presets
from dew.data.sources import TokenFiles
from dew.interop import PretrainedDecoder
data = TokenFiles("data/qwen3-tokens").shuffle(seed=0).repeat().pipe(presets.token_windows, seq_len=512, batch=4)
bundle = PretrainedDecoder.load("Qwen/Qwen3-0.6B", max_seq_len=512)
objective = LMObjective(bundle, seq_len=512)
state = Trainer(objective, OptimConfig(learning_rate=1e-5), key=jax.random.key(0)).fit(data, steps=100)

This downloads the Hub weights and needs memory for the model, gradients and optimizer. The bundle supplies the model, the initial variables and its processor, so objective.pipeline(state) takes text prompts. Passing variables= or processor= as well overrides that part of the bundle.

dew.lora.LoRA describes a low-rank adapter with PEFT’s own fields (rank, modules as PEFT’s target_modules, alpha, rslora, dropout). bundle.adapt(lora, key=) returns the same kind of bundle with the adapter attached: the adapted model, the variables with the factors under params and every base weight under frozen, and the bound adapter. Any objective over that bundle trains the factors and leaves every other weight as loaded. tuned.adapter.save writes PEFT’s adapter directory, and tuned.save writes the source’s layout with the factors merged into the kernels. Both take the trainer’s state.variables as they come back from fit:

from dew.lora import LoRA
tuned = bundle.adapt(LoRA(rank=8, modules=("q_proj", "v_proj")), key=0)
objective = LMObjective(tuned, seq_len=512)
state = Trainer(objective, OptimConfig(learning_rate=1e-4), key=0).fit(data, steps=100)
tuned.adapter.save(state.variables, "qwen3-adapter")
tuned.save("qwen3-merged", variables=state.variables)
print(objective.pipeline(state)("The capital of France is", 8, key=0).text[0])

adapt leaves bundle itself unchanged and returns a new value; adapting an already adapted bundle is refused. LoRA(...).apply(model, variables, key=) attaches the same adapter to any model and variables, including a model built from its class, and LoRA.load(model, variables, path, layouts=) reads an adapter that PEFT or Diffusers wrote. Both return the bound Adapter, whose model and variables you pass to an objective.

On the command line, a run takes the same spec as --lora.rank 8 --lora.modules q_proj v_proj. With --pretrained, the adapter goes on the loaded weights; without it, on a fresh draw of the model from the run’s key. Pretrained.from_run(run).adapter.save(...) writes the run’s factors under the source’s names from the run alone. To train part of a model without an adapter, split its starting variables with dew.batch.freeze(variables, filter). The leaves the filter keeps are trained and the rest stay frozen.

PretrainedDecoder.from_model wraps a trained CausalTransformer as a source bundle so you can export it, and the bundle’s save writes the Hugging Face layout. This exports the decoder from the example and loads it back:

import jax.numpy as jnp
import dew
from dew.interop import PretrainedDecoder
trained = PretrainedDecoder.from_model(model, lm_state.variables, tokenizer="byte",
generation_config={"do_sample": False, "max_new_tokens": 24})
trained.save("stories-decoder")
task = dew.pipeline("stories-decoder", dtype=jnp.float32)
print(task.sampling)
result = task([tokenizer.encode("At night")], key=0).host()
print(tokenizer.decode(result.tokens[0]), result.lengths)
Sampling(temperature=0.0, top_k=None, eos_token_ids=None, pad_token_id=0, top_p=1.0, min_p=0.0, repetition_penalty=1.0, presence_penalty=0.0, frequency_penalty=0.0, no_repeat_ngram_size=0, min_new_tokens=0, typical_p=1.0, stop=())
At night, Lily went home. She wa [24]

The directory holds config.json, generation_config.json and model.safetensors. generation_config.json sets the task’s defaults: do_sample: false becomes temperature=0.0, and max_new_tokens becomes the budget, so the call passes neither. The byte vocabulary has no Hugging Face tokenizer files, so the export records only tokenizer_name: "byte" and the loaded task takes token rows.

An export with an HFTokenizer writes the tokenizer’s own files (save_pretrained) and does not record its name, because for a local tokenizer the name would be a path on the exporting machine. So the directory loads on any other machine, and its task accepts strings and returns result.text. from_model refuses a model whose computation the exported config would not reproduce, and names the field.

A Hub name such as "Qwen/Qwen3-0.6B" loads the same way, and the published checkpoint needs enough host and device memory for its weights and cache. LMObjective.policy(params) returns the same kind of task for a training tree, and RolloutScheduler.sampled samples with it. To serve the weights with Ollama or vLLM, export them and point the runtime at the directory; the README shows the steps.

From the Hub, the loader fetches the configs and indexes first, then only the weight files the index’s weight_map names, or model.safetensors when there is no index. It does not download other copies of the weights in the repo, such as Mistral’s consolidated.safetensors, a pipeline’s fp16 variants and root single-file checkpoints. A tensor stored in two shards is refused. Pretrained.revision records the commit the Hub resolved, so you know exactly what you loaded even if the branch moves later. It is None for a local directory.

Pretrained.load’s param_dtype defaults to float32 master weights, and dew.pipeline’s to "auto". param_dtype="auto" keeps each tensor in the dtype the checkpoint stores it in, so a bf16 checkpoint’s fp32 state (Mamba’s A_log, a router’s correction bias) stays fp32; a quantized checkpoint’s packed weights decode to the dtype its config.json declares, or else the first floating tensor’s. Pretrained.load(..., mesh=MeshSpec(...), layout=Layout(...)), which dew.pipeline calls, places each weight on its devices directly from the memory-mapped checkpoint, one device shard at a time, so the host never holds the whole translated model (Distributed training has the measurements).

A config field Dew cannot express is refused by name. The exceptions are the fields in _INERT_FIELDS (dew/interop/hf_decoders.py), which transformers 5.16.1 neither declares nor reads, such as SmolLM2’s transformers.js_config, Qwen2.5’s use_mrope: false and the Mamba2 ports’ rms_norm. A family reads only the fields its transformers config class declares. A Llama config that names a sliding window is refused, because transformers’ Llama attends to every key; the window is read only for a model type that applies it, such as Mistral or Ministral. MLX quantization is refused by name.

The real prompt plus the continuation must fit in model.max_seq_len; input padding takes no room. Sampling holds temperature, top-k, top-p, min-p, EOS and the output padding ID. For a loaded checkpoint, the source’s generation_config.json fills it in, and text_generation(sampling=...) overrides it. Generation prepares the inputs on the host, then runs prefill and decode in one compiled call with one padded input shape and a cache cursor per row, so inputs with different valid lengths but the same padded shape reuse the executable.

The base checkpoints used below are not instruction-tuned chat models. With an instruction-tuned model, use the chat template its checkpoint documents. The Supported models page lists which decoder families load, train, generate and export.

  • GGUF. Pretrained.load(repo, gguf_file="Model-Q4_K_M.gguf") reads one llama.cpp file (uv pip install 'dewml[gguf]'). The file’s metadata becomes the config, and every tensor is dequantized to float32 on the host. The model then trains and generates like a safetensors load, and param_dtype sets how it is stored. When the repo has no tokenizer, the tokenizer comes from the file. The llama, qwen2 and qwen3 architectures load, and anything else is refused. Loading a repo that has only GGUF files without gguf_file is refused with a message that lists its files and names the argument that loads one.
  • PyTorch pickles. A repo with only pytorch_model.bin (or its index) also loads. If SFconvertbot has opened a safetensors pull request for that commit, Dew loads that refs/pr/N revision, logs which one, and records its commit in Pretrained.revision. Otherwise, with torch installed (uv pip install 'dewml[torch]'), Dew unpickles the files once with torch.load(weights_only=True), which runs no code from the file. It saves them as safetensors under ~/.cache/dew/converted/ ($XDG_CACHE_HOME/dew/converted/ when that is set), and later loads read that copy without importing torch. Without torch and without a pull request, the load is refused, and the refusal names the extra to install and the https://huggingface.co/spaces/safetensors/convert space. Local directories work the same way.
  • mamba_ssm checkpoints. state-spaces/mamba2-* and other checkpoints in mamba_ssm’s own format have a config.json with d_model, n_layer and ssm_cfg but no model_type. They load as the transformers Mamba2 port that transformers’ conversion script would produce, and save writes that port.

Loading works in three tiers.

  1. Native family. The model type is registered, and a test pins its numbers against transformers. It gets Dew’s attention kernels, sharding rules, cached generation and export.
  2. Verified mapping. Some unregistered model types compute a model Dew builds natively. Tier 2 first assembles the type’s own modules from known parts: each module’s code must be a known class’s code (Llama’s attention, Mixtral’s experts, DeepSeek V3’s latent attention), and each part reads its settings off the module itself. Granite (its multipliers), SmolLM3 (its layers without rotary positions), OLMo 2, OLMoE, StableLM, LFM2, LFM2-MoE, ERNIE 4.5, Hunyuan, HY-3 and GLM-4.7-Flash load this way. Hunyuan’s dynamic-alpha rotary is read as its static alpha base at every length, as vLLM serves it; past max_position_embeddings transformers recomputes plain dynamic-NTK frequencies from a batch’s longest position, which no serving stack reproduces. Otherwise each registered family whose mapping reads the config is tried, Llama’s first: CWM is the Llama block, Seed-OSS is Qwen2’s. Before downloading the weights, Pretrained.load builds transformers’ own class for that type from the same config at much smaller widths and, where transformers’ own code builds it, over the fewest first layers that hold every kind of layer its stated depth builds. It gives the instance random weights and runs it and each candidate in float32 beside the candidate’s own decoder in float64: a candidate matches where the source’s logits are no farther from that float64 truth than a measured factor of the candidate’s own distance to it, as two float32 roundings of one computation are, at any depth and width. An assembly from parts is also held to two training checks: with its dropout set to zero, the source’s training forward and gradients must equal its eval ones bit for bit (no hook or training-only branch that Dew’s decoder lacks), and each parameter’s gradient, pooled over its elements, must be no farther from the float64 truth than Dew’s own float32 gradient by more than eps^-1/2 (about 2896), the bound between two float32 roundings and a backward the forward does not show. The load goes ahead as a native Dew model through the first candidate that matches, and emits one VerifiedMappingWarning that names tier 2, the route (parts or the family), the transformers version and the measured ratio of the two distances. The model saves under the type’s own config and tensor names. Tensors the source’s own load leaves unread, such as GLM-4.7-Flash’s and HY-3’s multi-token prediction layer, are not read. The check needs torch (uv pip install 'dewml[torch]'); without torch the load is refused and the refusal names that extra. A type that no candidate computes goes to tier 3, with each candidate’s reason.
  3. Transformers’ own forward. An unregistered type that no registered family’s mapping computes loads through transformers’ PyTorch model, run by torchax, when it is a causal LM: before any weight downloads, the same small random instance must give each position logits that the later tokens leave unchanged, and its forward must use no operation whose output’s shape or value depends on its inputs (a nonzero, an item()), which a traced program cannot hold. The load emits one TransformersTierWarning that names tier 3 and why no mapping matched. A type whose small instance cannot be built or run (a context shorter than the probe’s tokens, BERT’s of 8 positions) is checked on its full configuration instead, with no weights: its forward is traced over abstract weights, each layer must attend through Dew’s attention with a causal mask, and a walk of the traced program must show that no output position reads a later token. No single trace proves every sequence length, so tier 3 then proves each length the model runs at before its first program there, and refuses a length where the program reads ahead. Pretrained.load(repo, fallback="torchax") takes this tier for any source, a registered one included. The tier builds transformers’ PyTorch model on the host and runs its forward through torchax. It compiles, differentiates and trains through Trainer and LMObjective. The variables keep the source’s torch names and are split into params, which the optimizer updates, and buffers, which it never touches. TorchLayout places them on a mesh by name. Where every decoder layer of the type asks transformers’ attention interface for its attention, with a mask Dew’s causal and windowed visibility reproduces (observed on the same small instance), the layers’ attention is Dew’s: attention_impl picks its kernel, and packed rows train each document within itself, rotated from its own first token. A type whose layers compute their own attention keeps transformers’, with no packing. Either way the tier has no cached generation (text_generation refuses) and reads float weights only. It pins the torch version (uv pip install 'dewml[torchax]'), and the load keeps the torch model in host memory too, so plan for at least twice the checkpoint size. The load warns and names the tier. Train it in float32 compute (dtype="float32", with weights stored at any param_dtype): in bfloat16 compute, JAX’s autodiff sums the gradient of a parameter broadcast over positions, a norm weight or a bias, in bfloat16 (jax-ml/jax#30310), and those gradients land about 2.5 times further from float64 than torch’s own bfloat16 training. Dew’s native layers sum them at float32.

A multimodal checkpoint loads as a MultimodalTransformer. Its variables hold the text decoder under language_model and the vision tower and projector under tower and projector; Gemma 3n and Gemma 4 also have an audio tower and its embedder, under audio_tower and audio_projector. The checkpoint’s processor resizes and normalizes the media, cuts them into patches and expands the placeholders, and Dew’s Processor runs it and lays out its outputs row by row.

To train or serve the language model alone, load it with text_only=True: Pretrained.load("Qwen/Qwen3.5-0.8B", text_only=True) builds Qwen3.5’s text decoder from the checkpoint’s model.language_model.* tensors and the tokenizer, and reads no vision tensor. It computes the multimodal model’s logits on text, and save writes Qwen3.5’s own text checkpoint (Qwen3_5ForCausalLM). A checkpoint whose config names a prediction depth it doesn’t hold, as transformers’ own re-saves of Qwen3.5 are, loads as its trunk. Its multi-token prediction depths, where it holds some, load too; prediction_depths=0 builds none and reads none of their tensors, and prediction_depths=n keeps the first n. A training run passes both as --source.text-only and --source.prediction-depths.

The next examples read the tiny checkpoints under tests/fixtures/hf in a Dew repository checkout. They run on CPU and show how the calls work; the models are too small to say anything about language quality. This one loads a tiny Gemma 3 and gives it images:

from pathlib import Path
import jax.numpy as jnp
import numpy as np
from dew.interop import PretrainedDecoder
fixtures = Path(dew.__file__).parents[2] / "tests" / "fixtures" / "hf"
source = fixtures / "gemma3-native-tiny"
bundle = PretrainedDecoder.load(source, dtype=jnp.float32, max_seq_len=64)
images = np.load(source / "raw_images.npy") # uint8 [3, 32, 32, 3]
inputs = bundle.processor(["token7 <start_of_image> token9",
"token5 <start_of_image> token8 <start_of_image> token6"],
images=[[images[0]], [images[1], images[2]]])
logits = bundle.model.apply(bundle.variables, inputs.tokens, **inputs.kwargs())
task = bundle.text_generation(sampling=Sampling(temperature=0))
for text in task(inputs, 3, key=1).text:
print(text)
token9token9token9
token6token6token6

ModelInputs has three parts:

PartContents
tokens[B, S] token IDs.
token_fieldsattention_mask, positions, image_indices and image_groups (the soft feature and the image behind each slot, -1 for text), audio_indices for audio slots, and for Qwen 3.5 the three-axis rotary_positions.
conditioningThe media, padded to the row with the most images or clips. pixel_values is [B, images, ...] in the processor’s own layout, with image_position_ids (Gemma 4) or image_grid_thw (Qwen 3.5) beside it. input_features and input_features_mask are [B, clips, frames, mel].

The two rows above have different numbers of images, so the processor left-pads the shorter one. inputs.token_fields["attention_mask"] is the only record of which slots are valid, and padded slots take no cache. A batch with nothing to pad has no attention_mask at all, and that absence is how the host says every slot is real. Without a mask, the model passes causality to the kernel as a flag and attention runs fused, which an all-true mask would prevent (Performance measurements). So code that handles both cases reads the field with token_fields.get("attention_mask").

Each process pads its own rows, so on a process pool the processes agree on whether the mask is present before anything is assembled. A generation request creates the field on every process if any process has it, and leaves it out if none does. A training batch placed by shard_batch has it on every process. For text-only prompts, leave out images. Gemma 3n and Gemma 4 take audio=[waveform, ...], one waveform per audio placeholder in reading order.

The processor checks token IDs, placeholder counts and media shapes on the host, and the compiled model does no checking. You pass the same ModelInputs to model.apply, the objective and generate. The model encodes the media at prefill, and the decode steps read the cache.

To keep training, build the bundle’s objective and feed the trainer {"text": inputs} batches:

import optax
from dew.data.dataset import Loader
from dew.training import Layout, MeshSpec
mm_objective = LMObjective(bundle, inputs.tokens.shape[1] - 1, pad_id=0)
rows = 2 * jax.device_count()
batch = inputs.take_rows(jax.numpy.arange(rows) % 2)
mm_data = Loader(train=lambda partition: iter([{"text": batch}]), val=None,
records=rows, batch=rows)
mm_trainer = Trainer(mm_objective, optax.sgd(1e-4), key=jax.random.key(3),
mesh=MeshSpec(), layout=Layout(min_shard=2**30))
mm_state = mm_trainer.fit(mm_data, steps=1, log_every=1)
bundle.save("gemma3-tiny-step1", variables=mm_state.variables)

bundle.save(directory, variables=...) writes the trained weights back under the source tensor names, together with the processor, so the directory loads again in both Dew and Transformers. The weights include Gemma 4’s frozen standardization and clipping buffers, which are kept in the constants collection and stay bit-for-bit unchanged through training. Past max_shard_size (default "5GB"), the weights are written as numbered shards with an index.

A dense decoder or a source-layout checkpoint builds each tensor on the host only when its shard is written. A quantized source, a from_model export of a Gemma 4 or GLM-5-next model, and a DiffusionGemma export build the whole export first. Saving over an earlier export replaces it in one step. The new shards get fresh names and the index is written last, so a save that stops partway leaves the previous export whole. Files the export did not write, such as model.fp16.safetensors, are left alone.

Each family’s processor emits what its reference implementation expects:

  • Gemma 3 gives one fixed-resolution image per placeholder block.
  • Llama 4 splits each image into local tiles and a global tile with separator tokens. It normalizes pixels in bfloat16, as the original implementation does, and the loader widens them to float32 exactly.
  • Gemma 4 emits padded patch streams with 2D patch positions. It expands video placeholders, which the decoder maps to the pad embedding.
  • Qwen 3.5 packs patches channel first, then time, with a grid per image. The loader derives the spatial rotary coordinates that the reference’s get_rope_index computes.
  • Gemma 3n uses the MobileNet-v5 encoder. It embeds its hard vision and audio vocabulary ranges through the multimodal embedders and keeps placeholder IDs for its per-layer inputs while masking the hard ranges, on both training and decode steps.
  • DeepSeek-V4.1 ships no processor. Its release resizes and pads each image in its own inference/image_processor.py, which Dew does not reproduce, so the caller builds the inputs it writes. pixel_values is [B, images, 3, H, W], normalized to [-1, 1], with H and W multiples of the patch size. Each image fills rows * (columns + 1) + 2 positions of image_token_id, where rows and columns count the aligner’s downsample_ratio squares over the patch grid. image_indices numbers those positions in reading order.

Audio clips have a mask that is True for valid frames. Gemma 4 inserts one placeholder per encoded frame. Gemma 3n inserts a fixed audio_soft_tokens_per_image per clip and fills the remaining slots with the embedder’s padding token.

The Gemma 3n and Gemma 4 audio encoders are dew.nn.audio.Gemma3nAudio and Gemma4Audio, with the tower aliases gemma3n_audio and gemma4_audio. audio_config reads the checkpoint’s audio_config record and rejects unknown computational fields. audio_weights converts the tower’s own tensors and keeps Gemma 4’s checkpointed clipping bounds in a frozen constants collection.

An encoder takes input_features shaped [B, T, F] and a boolean input_features_mask that is True for valid frames, and returns AudioEncoding(features, mask), with the mask subsampled to the encoder’s frame rate. A loaded source’s processor (Pretrained.load(...).processor) runs the checkpoint’s own Transformers feature extractor, which turns 16 kHz mono waveforms into those two arrays. Nothing resamples the audio. Gemma 3n projects audio through Gemma3nProjector.soft_embeddings, without the scaling used for vision. Gemma 4 reuses Gemma4Projector, with its input width taken from output_proj_dims.

Masked diffusion (MDLM, LLaDA, Dream) and DiffusionGemma’s block diffusion denoise a whole span at once rather than decode token by token; they have their own page, Diffusion language models.