dew.config.lm
The language-model run, as one typed record.
LMRunConfig is what an LM run parses from its command line and writes as
run.json next to the checkpoints. Every run records the model, the data,
the optimizer and the trainer. A decoder run also records the tokenizer its
ids came from and the preview policy it generates with.
TextGeneration.from_run and Pretrained.from_run read those two fields back
from that file, so a run that does not record them loads as weights with no
way to turn text into ids.
The three objectives a decoder can be trained under are one field, because
they share every other field. lm is the next-token loss, masked_diffusion
is MDLM over a bidirectional model, and block_diffusion is DiffusionGemma’s
own fine-tuning objective. run trains it on the token files dew tokenize
wrote, which decide the vocabulary.
| Name | Summary |
|---|---|
LMRunConfig | A decoder’s run (DecoderRunConfig) plus the settings specific to language models. |
LMRunConfig
Section titled “LMRunConfig”class LMRunConfig( model: ModelConfig = (lambda: ModelConfig('causal_transformer', {}))(), data: Annotated[Dataset | None, tyro.conf.Suppress] = None, validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress] = None, data_seed: int = 0, source: SourceOptions = SourceOptions(), optim: OptimConfig = (lambda: OptimConfig(learning_rate=0.0006, weight_decay=0.1, clip_grads=1.0))(), trainer: TrainerConfig = TrainerConfig(), objective: ObjectiveConfig = (lambda: ObjectiveConfig('lm'))(), lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None = None, val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None = None, val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])] = dict(), val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])] = (), loop_steps: int | None = None, loop_layers: tuple[int, int] | None = None, loop_backprop_steps: int | None = None, loop_remat: bool = False, tokenizer: str = 'byte', prompt: str = '', max_new_tokens: int = 128, sampling: Sampling = (lambda: Sampling(temperature=0.8, top_k=40))(), pretrained: str | None = None, rollout: Annotated[PromptRollout, tyro.conf.subcommand('rollout')] | None = None,)A decoder’s run (DecoderRunConfig) plus the settings specific to language models.
objective: ObjectiveConfig-
The objective and its arguments: lm, masked_diffusion (MDLM), block_diffusion (the official DiffusionGemma fine-tuning objective), or any objective by its import path that
LMObjective’s arguments build: the decoder,seq_len, its startingvariables,samplesandprocessor. A step whose optimizer clips by the attention logit maxima (muonclip) asks the objective for them (Step.qk_stats), whichever objective it is. tokenizer: str-
The tokenizer the ids were written with: ‘byte’, or an HF tokenizer name.
prompt: str-
The prompt that validation samples continue; an empty prompt continues a newline.
max_new_tokens: int-
The number of tokens generated per validation sample; 0 logs no text.
sampling: Sampling-
The preview policy, recorded with the run for inference.
pretrained: str | None-
The Hugging Face decoder to continue training from: a hub repo id,
repo@revision(a branch, tag or commit), or a local directory in that layout. A run records a hub repo asrepo@commit, with the commit it resolved to. The checkpoint sets the architecture, so of its fields —model.max-seq-len alone may be set. rollout: Annotated[PromptRollout, tyro.conf.subcommand('rollout')] | None-
The rollout a GRPO run trains on (
--rollout.reward), over its chain’s prompts (dew.data.presets.prompts): GRPO needs one, and another objective takes none. The model’s context is a prompt and its completion.
LMRunConfig.build_run
Section titled “LMRunConfig.build_run”def build_run() -> PreparedThe objective this run names over its token files.
The files decide the vocabulary; training and sampling decide the
context, since generation decodes into a cache sized once at build
time. A pretrained checkpoint decides the architecture instead, and
the run records it as the commit it was read at. A fresh model records
its vocabulary and context; a fine-tune records only its model choices,
since the source supplies its geometry on every load. The objective’s
inference record holds the built model for dew.pipeline. An adapter
binds to the model and the tree training starts from, the loaded weights
or a fresh draw from the run’s key, so the objective trains its factors
alone. Each validation pass scores val_metrics, perplexity by default.
LMRunConfig.default_metrics
Section titled “LMRunConfig.default_metrics”def default_metrics(*, validates: bool) -> tuple[MetricConfig, ...]Perplexity, where the run evaluates a validation split it holds.
LMRunConfig.tokens
Section titled “LMRunConfig.tokens”def tokens() -> TokensThe token windows this run trains on, as its chain states them (Dataset.facts): their
tokenizer, vocabulary and length. Raises ValueError for a chain that states none, which an LM
run cannot size its model for.
LMRunConfig.processor
Section titled “LMRunConfig.processor”def processor()The run’s tokenizer as its checkpoints record it for every loader.
LMRunConfig.samples
Section titled “LMRunConfig.samples”def samples() -> SamplesWhat the objective generates and decodes at every validation. The
policy is kept with no preview budget too: the run records it, and its
export and dew.pipeline decode with it.
LMRunConfig.pinned
Section titled “LMRunConfig.pinned”def pinned() -> LMRunConfigThis run with its Hub sources at their commits (RunConfig.pinned), pretrained among them
(SourceReference.pinned).
LMRunConfig.pretrained_source
Section titled “LMRunConfig.pretrained_source”def pretrained_source(written: str, vocab_size: int, context: int)The bundle pretrained names, at the commit the run pinned it to (pinned).
The checkpoint decides every architecture field, so the model flags may say only how far the KV cache reaches, the compute dtype and the attention kernel. The token files’ tokenizer has to be the checkpoint’s: continuing pretraining on ids from another vocabulary trains the embedding table against noise.
LMRunConfig.masked_objective
Section titled “LMRunConfig.masked_objective”def masked_objective(model, fields, variables, source=None)MDLM over a bidirectional model, its mask id from the model’s fields.
A pretrained diffusion checkpoint carries mask_token_id; a
from-scratch run names it beside —model.no-causal. A token window is
seq_len + 1 ids wide, and masked diffusion has no shift, so it
denoises the whole row. The validation text is the unmasked rows
decoded with the run’s tokenizer, or bare ids with no preview budget. A loaded source is
what the objective binds, so its end-of-sequence ids reach the task.
LMRunConfig.block_objective
Section titled “LMRunConfig.block_objective”def block_objective(model, variables, source=None)Each complete token-window row split into the clean prompt the run
names (--objective.prompt-length) and response canvases, over a loaded source
where there is one, whose end-of-sequence ids the task takes.