dew.nn.backbones.causal_transformer
The autoregressive transformer decoder that every language model in Dew trains.
By default the model has a token embedding, rotary positions, pre-norm blocks of grouped-query causal attention and a gated MLP, a final RMSNorm, and an fp32 head. Attention goes through the shared kernel path in dew.nn.attention, so a run picks reference/xla/cudnn/tpu the same way a diffusion run does, and decoding uses the same fixed-size KV cache helpers.
Parameter names follow the HF decoder layout: embed_tokens, layers_N.{input_layernorm, self_attn.{q,k,v,o}_proj, post_attention_layernorm, mlp.{gate,up,down}_proj}, norm, lm_head. Gemma’s two extra norms are the exception. HF calls them post_attention_layernorm and post_feedforward_layernorm even though they normalize sublayer outputs, so here they are attention_output_norm and mlp_output_norm, and the pre-norms keep their names. dew.interop.hf_decoders does that rename. A model family is supported only after its translator and a same-weight parity test against the reference have landed.
The block holds its token mixer in a slot. Any module called (x, call) -> x
with one dew.nn.inputs.MixerCall, as CausalSelfAttention is, reading its
tokens’ places and documents off the call metadata’s Coordinates and
declaring the masks it reads (reads), can be self_attn without changes to the
block, and that is where a linear-attention mixer goes.
| Name | Summary |
|---|---|
CausalTransformer | Decoder-only transformer over token ids: [B, S] int32 -> [B, S, vocab] fp32. |
CausalTransformer
Section titled “CausalTransformer”class CausalTransformer(LinearProducts)Decoder-only transformer over token ids: [B, S] int32 -> [B, S, vocab] fp32.
The defaults train a model from scratch: multi-head attention, swiglu,
tied embeddings, no softcap. Every setting that differs between open
decoders is a field here, so loading Qwen3 or Gemma3 is a mapping of config
fields and needs no subclass. The field comments name the family that sets
each one. A classic GPT block is norm_type='layer', norm_bias, an
ungated mlp, mlp_bias and learned positions; the learned positions
advance from each decode row’s count of its real tokens (DecodeCursor).
layer_types is the pattern, one kind per layer, and kinds says what
each kind does. kv_shared_layers lists the layers that share by index: a
trailing run of layers for Gemma 3n/4, or the layer after every indexer
layer for GLM’s IndexShare. A sharing layer reads what the last earlier
non-sharing layer of its kind stored: keys and values for attention, or the
indexer’s selection for MLA.
When attention dropout is active the model runs the reference kernel, and it refuses an explicitly chosen fused kernel that cannot drop probabilities.
Interleaved mRoPE (Qwen3.5’s mrope_section) is partial_rotary_factor for
text. With one position per token the three grids’ angles are equal, so
text-only input reduces exactly to the partial rope; image-grid positions
are not modelled. A mixer kind other than attention reads its own record,
but the GQA geometry fields are still validated, so a translation has to
fill them consistently.
MTP depth d pairs the previous depth’s state at position p with the embedding of the token at p + d, and scores the token that follows p + d (arXiv 2412.19437, section 2.2). Each depth is therefore one position shorter than the one before it.
With scan_layers, each run of like layers (layers whose resolved specs
give the same parameter shapes and computation) runs as one scanned body,
so compile time stops growing with depth. The variables tree stays the
unscanned one (StackView). init always draws a run of like layers under
the scan, once per run. When the mesh has a stage axis larger than one, the
stack runs as a pipeline.
value_head_dim: int | None-
The width of a value head; None is the query and key heads’ width. MiMo-V2-Flash attends 192-wide queries and keys over 128-wide values.
mlp_bias: bool-
Whether both feed-forward projections have a bias: none by default, as Llama’s published
mlp_bias: false, where Flax’s Dense adds one. Themlpactivations gelu, gelu_exact and relu give an ungated feed-forward. mlp_features: int | tuple[int, ...] | None-
The dense feed-forward width. None means four times
emb_features, a tuple gives one width per layer (Gemma 3n), and 0 means no feed-forward (Mamba-2). mlp_multipliers: tuple[float, float] | None-
The dense feed-forward’s gate and output multipliers (Falcon-H1’s
mlp_multipliers,GatedMLP.gate_multiplieranddown_multiplier); None for neither. The shared and routed experts take none. context_length: int | None-
The context a checkpoint states (max_position_embeddings), which an export writes back; None for
max_seq_len. The model computes nothing from it:max_seq_lenis its decode cache’s capacity, which a loader caps below a long context (decoder_parts.DEFAULT_MAX_SEQ_LEN). position_embedding: Literal['rotary', 'learned', 'alibi']-
How a layer whose kind names no mixer places its tokens (
layer_mixer): rotary attention, or unrotated attention after a learned table the embedding adds (‘learned’) or with ALiBi’s biases (‘alibi’). A mixer a model or a kind names places its own tokens; the learned table is added whatever the mixers. position_embedding_size: int | None-
The number of rows in the learned position table; None means
max_seq_len. It does not depend on the decode cache’s capacity. position_embedding_offset: int-
The number of reserved rows before learned position zero; OPT checkpoints have two.
rope_scaling: Scaling | None-
The model’s rope scaling: Llama 3.1’s ramp, LongRoPE’s factors or YaRN’s record (
rotary_freqs). narrow_rotary_tables: bool-
The rotary’s cos and sin round to the activations’ dtype before the fp32 rotation, as vLLM casts its cos/sin cache to the query’s dtype (
MixerContext.narrow_rotary_tables); False keeps them at the rotation’s precision. norm_eps: float-
Llama 2’s and 3’s and Mistral’s published
rms_norm_eps, where Flax’s RMSNorm takes 1e-6. norms: Mapping[str, NormSpec] | None-
The norms that differ from the model’s others, by module name (
decoder_block.NORM_SITES): K2 Horizon’s grouped statistics, an epsilon or a weight of a norm’s own. embedding_norm: bool-
Normalize token embeddings before the first block, as BLOOM and ModernBERT do.
embedding_norm_reads: Literal['inputs', 'lookup']-
What
embedding_normnormalizes: every input embedding, a caller’s own and a media encoder’s among them, as BLOOM’s and ModernBERT’s embedding norms read theirinputs_embeds(‘inputs’); or the table’s lookup alone, before media take their slots, as Muse Glimmer’s normed embedding module does (scaled_embeddings, ‘lookup’). first_attention_norm: bool-
Whether the first block norms its attention input. ModernBERT’s first block reads the embedding norm’s output as it is (modeling_modernbert.py, ModernBertEncoderLayer: attn_norm is Identity at layer 0).
embedding_features: int | None-
The token embedding’s width where it differs from the stream’s (OPT-350m’s word_embed_proj_dim), bias-free
project_intaking the lookup in andproject_outthe final states out to the head. parallel_residual: bool-
Whether attention and the feed-forward both read the same residual, as in GPT-NeoX.
shared_parallel_norm: bool-
Both parallel branches read one LayerNorm, as in Phi, Falcon-7B and GPT-J.
attention_bias: bool-
q/k/v biases, and o_proj’s unless
o_proj_biassays: none by default, as Llama’s publishedattention_bias: false, where Flax’s Dense adds one. value_scale: float | None-
The factor that multiplies the values as they are projected, before they are cached: MiMo-V2-Flash’s
attention_value_scale. key_multiplier: float | None-
The factor that multiplies the keys as they are projected, before they are rotated and cached: Falcon-H1’s
key_multiplier. embedding_multiplier: float-
The factor that multiplies the token embeddings before the first layer: muP’s m_emb in lm-engine, and GraniteMoeHybrid’s
embedding_multiplier. residual_multiplier: float-
The factor that multiplies every sublayer output before it is added to the residual stream: lm-engine’s m_residual, and GraniteMoeHybrid’s
residual_multiplier. residual_in_fp32: bool-
Whether the residual stream between layers sums in at least fp32 whatever the compute dtype, as mamba_ssm’s
residual_in_fp32keeps it (transformers’ Mamba, Mamba2 and FalconMamba blocks): each sublayer reads the stream in the compute dtype and its output adds to the stream in fp32 (stack). logits_scaling: float-
The divisor of the logits: lm-engine’s m_width (
lm_logits * (1 / m_width)), and GraniteMoeHybrid’slogits_scaling. The division is applied to the final states, in fp32, so every head that contracts them (output_table) gets the same logits that__call__returns. initializer_range: float | None-
The std of lm-engine’s initialisation; None keeps each module’s own initializer.
When it is set, the embedding table (and an untied head) is drawn from N(0, initializer_range^2), and every hidden matrix (attention and Mamba-2 projections, conv taps, router, experts, dense MLPs) from N(0, (initializer_range / sqrt(logits_scaling))^2). Biases start at zero and norms at one. With
logits_scalingset to m_width this is lm-engine’sinit_method="mup", and withlogits_scalingat 1 it is lm-engine’s"normal". depth_scaled_init: bool-
Whether, with
initializer_rangeset, the projections back into the residual stream (o_proj, out_proj, down_proj) divide their std by sqrt(2 * num_layers). This is lm-engine’suse_depth_scaled_init. ngpt: NGPT | None-
The normalized transformer (
dew.nn.ngpt.NGPT), NVIDIA’s Dory: every hidden state on the unit sphere, each block stepping its residual along it (BlockWiring.ngpt), the attention and feed-forward unit-norming their products and scaling them by learned gains, no norm with a weight anywhere, a unit norm between a loop’s passes, no norm before the head, and the logits times a learned per-token gain (logit_scale). None for none. head_transform: Literal['gelu', 'gelu_exact'] | None-
The activation of a BERT-style prediction head between the final norm and the vocabulary projection: a bias-free dense layer, this activation and a norm of the model’s kind (ModernBertPredictionHead, modeling_modernbert.py). None projects the final states directly. As in the feed-forward, ‘gelu’ is the tanh form and ‘gelu_exact’ the erf form.
hidden_statesstays the final norm’s output, the encoder’s. head_bias: bool-
A vocabulary bias added in fp32 after a tied or untied head’s product, as Phi’s, GPT-J’s and ModernBERT’s decoders add one (
output_table’sbias). embedding_zero_ids: tuple[int, ...]-
Placeholder ids that the embedding lookup reads as token zero. The labels keep the original ids.
padding_idx: int | None-
The id whose token embedding rows (
embed_tokens,embed_tokens_per_layer) the lookup reads but never trains, as a source’snn.Embedding(padding_idx=...)leaves them (Gemma 3n’s and Phi’s pad_token_id,TokenEmbedding.padding_idx); a tied head still trains the row. None trains every row. kv_cache: KVCache-
How attention layers store the decode cache: dense or paged, full or quantized (
dew.nn.kv_cache). The parameters do not depend on it. continuum: Levels | None-
A continuum level on the final states before the head, or a chain of them (
dew.nn.continuum): parameters an objective steps in context by its own loss (LMObjective(adaptation=...)), every head that reads the states,__call__’s and each prediction depth’s, reading the levels first. It starts as the identity. None gives the plain head. causal: bool-
Whether attention is causal. False gives full attention, the encoder that a masked diffusion language model denoises with. The parameter tree is the same either way.
per_layer_input_dim: int | None-
The width of Gemma 3n/4 per-layer inputs, or None for a model without them. Each layer reads its own slice of an extra table and adds it to its input through its own gate.
mtp_layer_type: str | None-
The layer kind the prediction depths build their mixer from. It need not occur in the trunk. None uses the full-attention kind when the pattern has one, and the first layer’s kind otherwise.
mtp_feedforward_type: str | None-
The layer kind whose MLP mixer (
MLPMixer) builds the prediction depths’ feed-forward, as Nemotron-H’s step pairs an attention block with a routed MLP block. None routes like the last layer, and is dense otherwise. mtp_hyper_connections: HyperConnections | None-
The residual streams of the prediction depths. None gives the depths plain residuals and normalized trunk inputs. Setting it opts the depth into streams, separately from the trunk’s
hyper_connections. attention_residuals: AttentionResiduals | None-
Kimi K3’s attention residuals (
dew.nn.attention_residuals): a softmax over finished blocks of layers that replaces the running residual. None disables them. engram: Engram | None-
DeepSeek-V4.1’s n-gram lookups or Qwen4-Exp’s per-layer embedding (
dew.nn.engram) before each named layer’s attention, over the compressed vocabularyconstantsholds (engram_hashes/token_map). dex: DifferentialExtension | None-
The chosen heads whose attention outputs take a learned differential (
dew.nn.dex), or None. cross_attention: CrossBlocks | None-
The layers that read a conditioning sequence through Flamingo’s gated cross-attention (
dew.nn.cross_attention), before their own sublayers; None for a model without them. A call passes the sequence as itsconditioning. dspark: DSpark | None-
DeepSeek’s block drafter (
dew.nn.dspark, V4.1 and V4-Flash-0731), or None for a model without one. Its stages are decoder blocks that attend through aDSparkAttentionbuilt from the V4 attention of itslayer_typekind. Its target layers sow the stream means that the drafter reads (draft_context). activation_sparsity_pattern: tuple[float, ...] | None-
Gemma 3n’s gaussian top-k activation sparsity, one fraction per layer.
mask_token_id: int | None-
The vocabulary id that a masked-diffusion objective corrupts tokens to. None means plain training.
shifted_logits: bool-
Whether masked diffusion reads position i’s prediction from position i-1’s logits, as a checkpoint continued from an autoregressive one predicts the token after each position (Dream-v0); False reads each position’s own (LLaDA). The forward ignores it: it says how the checkpoint’s logits line up, which
dew.diffusion.discrete.alignedapplies for the masked loss and the sampler alike. scan_layers: bool-
Runs of like layers under flax’s scan; a decode step over resident weights scans a period’s repeats as one body too (
scan_groups(periodic=True)), so its program grows with the period, not the depth. bank_layers: int | None-
The most layers one scanned run holds, which is also how many layers its parameter bank stacks. A longer run of like layers is split into consecutive runs of at most this many, each with its own bank under its own name. None keeps a whole run in one bank.
scan_layersandinitread it, and because init draws each run under one scan, it also changes what init draws for a given seed. The split bounds the memory needed to build a host-resident bank and read it back, so set it for a deep stack that is offloaded to the host. loop: Loop | None-
The layer stack run several times over its own output, its parameters shared (
Loop), as Ouro’s looped decoder runs it. None runs it once. remat: RematPolicy | None-
The rematerialization policy. Each block is recomputed in the backward pass, keeping its inputs, any K/V passed to later layers, and the residuals the policy names. A config gives a name from
REMAT_POLICIESor a record of residual names to save or offload. None recomputes nothing. Init and cached decode call the blocks directly, and the stored parameters have the same layout either way. layer_collections: tuple[str, ...]-
The collections a reader of layer states opens (
LayerStates). wiring: BlockWiring-
How every block wires its two sublayers (
BlockWiring), from the model’s own dials, each a field of its own (docs/design/api.md): the one value the stack’s blocks, the prediction depths, DSpark’s stages and the FLOP count build with. init_stds: tuple[float | None, float | None]-
The normal std of the hidden matrices and of the projections back into the residual stream. Both are None when
initializer_rangeis None, and the modules then use their own initializers. position_rows: int-
The rows of the learned position table (
position_embedding_size, elsemax_seq_len). position_capacity: int | None-
The places a row’s learned positions cover, the table’s rows past its reserved offset; None for rotary or ALiBi positions, which cover any (
PositionCapacity). mlp_widths: tuple[int, ...]-
Each layer’s dense feed-forward width.
hidden_features: int-
The model’s single feed-forward width. The routed experts fall back to it and the prediction depths use it. A model whose layers have different widths has none and raises
ValueError. hash_layers: set-
The sparse layers that route by the mixture’s token table.
sparse_layers: tuple[int, ...]-
The layers whose feed-forward routes to experts.
sharing_layers: tuple[int, ...]-
The sorted indices of the layers that read what another layer stored.
kv_sharing: dict-
A map from each sharing layer’s index to the provider layer it reads.
A sharing layer has no K/V of its own (and, for MLA, no indexer). It reads the last earlier non-sharing layer of its own type. The map is empty unless
kv_shared_layersis set, and a sharing layer with no earlier provider raisesValueError. in_context: bool-
Whether the model holds a continuum level (
SteppedInContext). in_context_levels: int-
The continuum levels the model holds, a chain’s count (
SteppedInContext). embedding_factor: float-
The factor
scaled_embeddingsscales an embedding by, unrounded, for a mixture of the table’s rows to take in its own arithmetic (dew.nn.diffusion_gemma.soft_embeddings). context_width: int-
The width of a context’s tokens (
CrossAttending): the spec’s, or the decoder’s own. context_dtype: jnp.dtype-
The dtype a context’s tokens enter in (
CrossAttending): the decoder’s compute dtype, at least float32, as its projections widen what they read. cache_rebuild_position: int | None-
Where LongRoPE’s factors switch from short to long, past which cached keys go stale: the original length of every trunk layer’s LongRoPE, its kind’s or the model’s (
kind_of), None where no layer takes LongRoPE. Kinds that switch at different lengths are refused, since a cache rebuilds at one. layer_mixer: MixerBase-
The mixer a layer whose kind names none builds: the model’s own, else attention rotated or not by its position embedding.
declared_mixers: tuple[MixerBase, ...]-
The mixer values it names: its own, then each layer kind’s.
CausalTransformer.kind_of
Section titled “CausalTransformer.kind_of”def kind_of(layer_type: str) -> ResolvedKindReturn the ResolvedKind of layer_type, with the model’s defaults where the kind is unset.
A kind’s rope scaling takes the model’s place.
CausalTransformer.mixer_context
Section titled “CausalTransformer.mixer_context”def mixer_context( kind: ResolvedKind, layer_type: str, kv_shared: bool, dex: LayerDex | None = None, dex_start: DexStart | None = None, routed: bool = False,) -> MixerContextReturn one layer’s mixer geometry as a MixerContext built from the resolved kind.
head_dim, rope_theta, window, the rope scaling, the partial
rotary, the sinks and the query-key norm take the layer kind’s resolved values
(kind_of), and the precision the model’s layers take (products). Every other field
is the model’s field of the same name. A kind builds its DecoderBlock factory
from this context and its own record, and setup chooses the mixer
there and nowhere else.
CausalTransformer.bank_sites
Section titled “CausalTransformer.bank_sites”def bank_sites(store: Variables) -> tuple[DecoderBank, ...]Where this decoder’s layer stack is stored in store, the variables its banks are drawn
from: scanned runs, or one layer per bank.
groups already records which of the two it is. A scanned stack
declares its runs and a plain loop declares one-layer runs, and
run_stack fetches a one-layer run the same way it fetches a longer
one. A run ends where the store’s rows differ (StackView.layer_structures:
layers a checkpoint quantizes differently, or adapts in part), so an
unbound decoder plans its runs from the store’s shapes as a decoder
bound to them reads them. scanned records whether those fetches are
sequenced, which a host layout requires.
CausalTransformer.stored_view
Section titled “CausalTransformer.stored_view”def stored_view(store: Variables) -> StackViewHow store holds this decoder’s layer stack (bank_sites), as a decoder bound to its shapes
plans its runs (SteppedFeedForward).
CausalTransformer.layer_kinds
Section titled “CausalTransformer.layer_kinds”def layer_kinds(types: Sequence[str]) -> dict[str, ResolvedKind]Resolve every kind the pattern names, plus the kinds of the prediction depths and the drafter.
Raises ValueError when the pattern does not have one kind per layer,
or when kinds describes a kind that no layer has.
CausalTransformer.model_flops
Section titled “CausalTransformer.model_flops”def model_flops(lengths) -> floatReal-row hidden-state work, summing the same block factories as setup (decoder_counts). A
context’s cross-attention work is cross_flops, which needs the context’s lengths.
CausalTransformer.parameter_flops
Section titled “CausalTransformer.parameter_flops”def parameter_flops(lengths) -> floatLearned matrices, separately from nonparameter attention/state work.
CausalTransformer.projections
Section titled “CausalTransformer.projections”def projections() -> dict[str, tuple[int, int]]The trunk’s own maps as {name: (inputs, outputs)}, the head
included: what setup builds and the counts read.
CausalTransformer.counted_blocks
Section titled “CausalTransformer.counted_blocks”def counted_blocks() -> list[DecoderBlock]Each layer’s block as setup builds it, unbound, which its counts read (decoder_counts).
CausalTransformer.prediction_blocks
Section titled “CausalTransformer.prediction_blocks”def prediction_blocks() -> list[MTPBlock]Each prediction depth’s block as setup builds it, unbound, which its counts read.
CausalTransformer.replicated_flops
Section titled “CausalTransformer.replicated_flops”def replicated_flops(stages: int, lengths) -> floatThe work of the layers a pipeline of stages runs on every stage rather than once
(stage_split): those before and after the run its stages repeat.
CausalTransformer.layer_passes
Section titled “CausalTransformer.layer_passes”def layer_passes(index: int) -> intHow many times a call runs layer index: a loop’s steps for a layer it repeats, else once.
CausalTransformer.logits_flops
Section titled “CausalTransformer.logits_flops”def logits_flops(lengths) -> floatThe vocabulary matrix and optional prediction transform, project_out’s included. A tied
head reads the embedding table, vocab_size rows of the embedding’s width.
CausalTransformer.prediction_flops
Section titled “CausalTransformer.prediction_flops”def prediction_flops(lengths, *, prediction_lengths=None) -> floatEach trained prediction depth shortens the real sequence by one token.
CausalTransformer.prediction_parameter_flops
Section titled “CausalTransformer.prediction_parameter_flops”def prediction_parameter_flops(lengths, *, prediction_lengths=None) -> floatThe same shortened depth forwards’ learned matrices, head included.
CausalTransformer.prediction_matrix_usage
Section titled “CausalTransformer.prediction_matrix_usage”def prediction_matrix_usage(path: tuple[str, ...], lengths, *, prediction_lengths=None)Each depth’s maps run once; the shared head runs on every shorter state.
CausalTransformer.matrix_usage
Section titled “CausalTransformer.matrix_usage”def matrix_usage(path: tuple[str, ...], lengths, *, logits: bool = False)Native matrix scopes declare head versus trunk work and repeated loop passes.
CausalTransformer.setup
Section titled “CausalTransformer.setup”def setup()Build the embeddings, the layers, the prediction depths mtp, the final
norm, and lm_head when the embeddings are not tied.
Each layer has one LayerSpec, is built by block, and is scanned in
groups.
CausalTransformer.mixer_of
Section titled “CausalTransformer.mixer_of”def mixer_of(index: int) -> nn.ModuleLayer index’s mixer, unbound, as its kind builds it over the layer’s context.
CausalTransformer.drawn_like
Section titled “CausalTransformer.drawn_like”def drawn_like(index: int) -> HashableWhat layer index draws as: its scan key and what its extension starts from (_dex_start), so
like layers draw under one scan. A method interceptor that gives a layer parameters of its own (an
adapter’s factors, dew.lora) keys it apart where they differ.
CausalTransformer.cross_flops
Section titled “CausalTransformer.cross_flops”def cross_flops(lengths, context_lengths) -> floatThe cross-attention blocks’ work, which model_flops leaves out (crossed_flops).
CausalTransformer.held_cross_flops
Section titled “CausalTransformer.held_cross_flops”def held_cross_flops(lengths, context_lengths) -> floatThe part of cross_flops its own matrices do, which train as its trunk does (crossed_flops).
CausalTransformer.states_and_logits
Section titled “CausalTransformer.states_and_logits”def states_and_logits(tokens, **kwargs={})Return the prediction-input states and the logits from one forward pass.
A speculative decoder verifies with both. The logits give the target distribution, and the states seed the next block’s prediction depths. The states are the normalized states, except for V4, where they are the uncollapsed residual streams.
CausalTransformer.states_and_logits_at
Section titled “CausalTransformer.states_and_logits_at”def states_and_logits_at(tokens, slots, **kwargs={})Return the prediction-input states and the logits at one slot per row, or at [rows, K] slots.
A prefill needs logits only at slots, the positions the first draw
reads. The head over every prompt position would be the largest array
the forward allocates, [rows, width, vocab], and a decoder keeps only
one row of it. Gathering the states before the head cuts the head’s
work to [rows, features].
CausalTransformer.head_bounds
Section titled “CausalTransformer.head_bounds”def head_bounds(states) -> HeadTermsWhat bounds a slot loss’s curvature and slope in the states head_logits reads (HeadTerms):
the bound rows (vocabulary_rows), softcap and transform, its norm read at the states
(BoundedNorm), each kernel as Qwix’s rounding can grow it (_head_growth).
CausalTransformer.in_context_level
Section titled “CausalTransformer.in_context_level”def in_context_level(variables)The continuum level’s parameters in variables, None without a level (SteppedInContext).
CausalTransformer.with_in_context_level
Section titled “CausalTransformer.with_in_context_level”def with_in_context_level(variables, level)variables holding level as the continuum level’s parameters (SteppedInContext).
CausalTransformer.continuum_states
Section titled “CausalTransformer.continuum_states”def continuum_states(hidden, index: int | None = None)Final states hidden past the continuum levels, the head’s input, or past level index alone
(chain_states), read at the levels’ own parameters (SteppedInContext).
CausalTransformer.level_terms
Section titled “CausalTransformer.level_terms”def level_terms(hidden, parts: tuple[str, ...], index: int = 0, passed: bool = False)What bounds a damped step of continuum level index’s parts at its states hidden
(Continuum.step_terms), its maps’ inputs as a checkpoint’s quantizers round them.
CausalTransformer.head_logits
Section titled “CausalTransformer.head_logits”def head_logits(x)The fp32 head over states past the continuum levels (continuum_states), or final states.
CausalTransformer.mtp_hidden_states
Section titled “CausalTransformer.mtp_hidden_states”def mtp_hidden_states( hidden, tokens, train: bool = False, input_embeddings=None, image_groups=None, block_groups=None, rotary_positions=None, coordinates: Coordinates | None = None, conditioning: Mapping[str, jax.Array] | None = None,)Return one final-normed state array per shifted prediction depth.
Each depth combines the preceding hidden state with the next token’s
embedding, including any media replacement, and uses the next token’s
positions. A pair counts only when both of its positions are valid and
belong to the same packed document, so padded positions in between
never become keys. A sequence no longer than the number of depths
raises ValueError. The depths keep no cross-attention: hidden, the
trunk’s states, already read the call’s conditioning, which they take
as the trunk does and leave unread.
CausalTransformer.mtp_logits
Section titled “CausalTransformer.mtp_logits”def mtp_logits( hidden, tokens, train: bool = False, input_embeddings=None, image_groups=None, block_groups=None, rotary_positions=None, coordinates: Coordinates | None = None, conditioning: Mapping[str, jax.Array] | None = None,)Return the shared language head’s logits for each prediction depth’s hidden states; a
call’s conditioning is the trunk’s (mtp_hidden_states).
CausalTransformer.mtp_step
Section titled “CausalTransformer.mtp_step”def mtp_step( hidden, tokens, *, depth: int = 0, input_embeddings=None, rotary_positions=None, decode: bool = False, prediction_phase: PredictionPhase = 'ordinary', coordinates: Coordinates | None = None,)Run one unshifted prediction step, optionally appending to the depth’s own KV cache.
Call init_mtp_cache before cached steps. hidden is the target
model’s preceding state, and tokens or input_embeddings give the
candidate next token, as in vLLM’s Qwen3_5MultiTokenPredictor. Returns
the step’s logits and hidden state; a chained draft’s next step reads
that state in place of the target’s. With
index_share_for_mtp_iteration, a cached “extend” step publishes its
index selections and a “draft” step reuses them, while “ordinary”
always recomputes them. Its tokens are laid out by their coordinates.
CausalTransformer.token_embeddings
Section titled “CausalTransformer.token_embeddings”def token_embeddings(tokens)Return the embeddings a prediction depth pairs with tokens.
mtp_hidden_states reads the unscaled table, so this method does that
lookup, with embedding_zero_ids read as token zero, and nothing else.
A drawn token is always text, so no media embedding replaces it.
CausalTransformer.stream_embeddings
Section titled “CausalTransformer.stream_embeddings”def stream_embeddings(tokens)token_embeddings at the stream’s width, through project_in past an embedding’s own: what a
prediction depth pairs with tokens, and a context’s tokens a cross-attention reads by the table.
CausalTransformer.scaled_embeddings
Section titled “CausalTransformer.scaled_embeddings”def scaled_embeddings(x)Return the token embeddings x scaled the way the first layer reads them.
x is normalized first where the norm reads the lookup alone
(embedding_norm_reads, Muse Glimmer), multiplied by
sqrt(emb_features) when embedding_scale is set (Gemma), then by
embedding_multiplier (lm-engine, GraniteMoeHybrid).
The decoder, the multimodal wrapper and the Qwen-Image conditioner all
scale here.
Gemma casts embed_scale to the embedding weight dtype. The lookup holds
that table in fp32, so the factor stays fp32 and only the product is
rounded (a bf16 factor would be 34.0 at hidden size 1152, not
33.941…). lm-engine multiplies in fp32, and scaled does the same.
CausalTransformer.draft
Section titled “CausalTransformer.draft”def draft( context, tokens, *, decode: bool = True, coordinates: Coordinates | None = None, choose=None,)Draft a DSpark block after each row’s last context position.
context [B, M, targets * D] comes from draft_context (the stream
means the target layers sow), its coordinates mark its real
positions (real_slots: its validity less a padding document, every
one where None; one document a row, as a draft continues one), and
tokens [B] are the
tokens drawn after it. Returns
(ids [B, block + 1], logits [B, block, vocab], confidence [B, block]).
Tokens are chosen by choose(index, logits), or greedily when choose
is None (dew.nn.dspark.draft). For a cached draft, call
init_draft_cache first. With tokens None the call only appends
context, and with context None it drafts after what the windows
already hold. A model without a drafter raises ValueError.
CausalTransformer.reach_drafter
Section titled “CausalTransformer.reach_drafter”def reach_drafter(tokens, dtype)Create the DSpark drafter’s parameters, which no forward pass reaches.
It drafts after tokens over a zero context as wide as the drafter’s
first stage projects.
CausalTransformer.draft_context
Section titled “CausalTransformer.draft_context”def draft_context(prediction_inputs: Mapping) -> jax.ArrayReturn DSpark’s context [B, S, targets * D] from prediction_inputs.
A forward pass sows prediction_inputs when the caller makes that
collection mutable. The context is each target layer’s stream mean,
concatenated in target_layers order.
CausalTransformer.init_draft_cache
Section titled “CausalTransformer.init_draft_cache”def init_draft_cache(batch_size: int)Allocate the drafter’s window caches, separately from the trunk’s cache.
CausalTransformer.init_mtp_cache
Section titled “CausalTransformer.init_mtp_cache”def init_mtp_cache(batch_size: int)Allocate the prediction depths’ caches, separately from the trunk’s cache.
CausalTransformer.hidden_states
Section titled “CausalTransformer.hidden_states”def hidden_states(tokens, **kwargs={})Return the final normalized states, without the vocabulary projection.
CausalTransformer.capture_layers
Section titled “CausalTransformer.capture_layers”def capture_layers(module: nn.Module, method: str) -> boolThe capture_intermediates filter that keeps every layer’s output (LayerStates).
It matches the __call__ of the stack’s layers_N modules, a scanned
run’s layers_3_7 and a pipeline stage’s layers_j included, and
StackView.unstack lays what those sow out per layer, as the plain loop
does; layer_state reads one layer back. Neither is a Flax method, so
the filter the capture calls on every module runs no module call.
CausalTransformer.layer_state
Section titled “CausalTransformer.layer_state”def layer_state(collections: Mapping[str, Mapping], index: int) -> jax.ArrayLayer index’s output out of what the forward kept (LayerStates): the last layer’s as
the stack returns it, and another’s as capture_layers kept it, the last call’s, a looped
layer’s last pass; a model that carries several residual copies (AltUp’s) returns them all
at every layer. index counts as a Python index does, -1 the last layer.
CausalTransformer.logits
Section titled “CausalTransformer.logits”def logits(tokens, *, train: bool = False, **fields={})Return __call__’s fp32 logits over hidden_states(tokens, **fields), which
write no cache unless fields ask to decode.
CausalTransformer.logits_from_hidden
Section titled “CausalTransformer.logits_from_hidden”def logits_from_hidden(hidden)Return the logits of final states hidden: the head __call__ and every MTP depth score with.
CausalTransformer.hidden_and_mtp_inputs
Section titled “CausalTransformer.hidden_and_mtp_inputs”def hidden_and_mtp_inputs( tokens, train: bool = False, decode: bool = False, input_embeddings=None, image_groups=None, block_groups=None, rotary_positions=None, attention_pairwise_mask=None, attention_key_positions=None, attention_key_runs=None, routed_experts=None, routed=None, media_mask=None, admitted=None, coordinates: Coordinates | None = None, conditioning: Mapping[str, jax.Array] | None = None, whole_blocks: bool = False, progress: Progress | None = None,)Return the final normalized states and the prediction depths’ input.
V4’s depth reads the raw residual streams from before the collapse head
and the final norm; other depths read the final normalized states.
The tokens’ layout is their coordinates, the one description of them
every layer reads; only __call__ takes flat layout fields and converts
them (token_coordinates, ledger 1122).
input_embeddings[B, S, D]replace the scaled token embeddings, for a caller that fuses in another encoder’s output. The token ids still feed the per-layer inputs and routing.image_groupsandblock_groups[B, S]number the tokens’ images and blocks, -1 outside any, which the attention layers that read them see whole (dew.nn.inputs.AttentionMetadata).attention_pairwise_maskis an explicit [B, queries, keys] visibility mask for ordinary attention, and the optionalattention_key_positions[B, keys] give key positions for local windows. Both describe the cached reads of this call only.attention_key_runs[B, queries, R, 2] replaces the mask with each query’s key intervals, whose union it reads, R of them, static, its key positions windowed within each run alone (the template’s spans,dew.nn.attention.scaled_dot_product_attention’skey_runs).routed_experts[B, S, layers, top_k](or one such a routed site, keyed by it,LayerInputs.experts, where a layer routes at several) replays a rollout engine’s routing (vLLM’srouted_experts, SGLang’smeta_info.routed_experts), androuted[B, S]marks the tokens it covers (dew.nn.protocols.Routes).media_mask[B, S] marks the positions a media encoder fills. Engram keeps them out of its n-grams, and a media-biased router selects for them.admittedmakes a cached call a serving step’s mixed call (dew.nn.inputs.Admitted).coordinatesare the tokens’ documents, validity and positions as the batch’s builder made them (dew.batch.coordinates, with the packer’smax_segments, andright_paddedwhere each row holds its real tokens first); None for one document a row, every token real.conditioningcarries the sequence thecross_attentionlayers read (dew.nn.inputs.CONTEXT) and its rows’ lengths (CONTEXT_LENGTHS). A cached call that carries it keeps its keys and values for the calls after it. A model without cross-attention takes none.whole_blocksstates that each ofblock_groups’ blocks is its document’s whole run of real tokens (AttentionMetadata.whole_blocks).progressis a training step’s (dew.nn.progress), which its DEX layers anneal over; None is the run’s end, outside a training step.
CausalTransformer.looped
Section titled “CausalTransformer.looped”def looped(x, **stack={})Run the layer stack over x once, or as loop says (looping.run_loop): its block’s later
passes under one scan, so a step’s program holds two passes whatever the loop’s count.
CausalTransformer.stack
Section titled “CausalTransformer.stack”def stack( x, *, train: bool, decode: bool, per_layer_input, attention_metadata=None, span: tuple[int, int] | None = None,)Run the layer stack over x as the plain loop, as scanned runs, or as a pipeline over the stages.
span runs layers [first, end) alone, as a Loop over a block runs its
prelude, its block and its coda; on one stage, with no banked store.
init draws like runs under the scan and then unstacks them, so the
tree is the plain loop’s; under a stage mesh or with decode, init
runs the plain loop. With scan_layers or a stage axis, the stack runs
under a StackView, which stacks the leaves while the loops run. The
loops carry one residual dtype, so the stream enters in the dtype it
settles in (residual_dtype). A store built bank by bank
(dew.inference.banks) already holds one array per run, and the view
leaves it alone. A pipeline runs the layers its stages repeat
(split_stages) and the layers before and after them on every stage,
and raises LayoutRefused for decode and for a banked store. With
residual_in_fp32 the stream enters widened, so every sublayer’s output
adds to it in fp32 whichever loop runs.
CausalTransformer.stage_view
Section titled “CausalTransformer.stage_view”def stage_view(stages: int) -> StackViewHow a pipeline over stages stacks the run of layers they repeat (split_stages): each
stage’s runs as its scan takes them, from the store’s rows (scan_keys).
CausalTransformer.stage_sites
Section titled “CausalTransformer.stage_sites”def stage_sites(store: Variables, stages: int) -> tuple[DecoderBank, ...]Where the run a pipeline over stages stacks is stored in store, the variables a
stage-local state is made from (stage_view): none for a looped stack, which no pipeline
runs.
CausalTransformer.banked_collections
Section titled “CausalTransformer.banked_collections”def banked_collections() -> tuple[str, ...]Return the collections whose layer subtrees the store holds as banks.
A store built per layer holds layers_0, and a store built bank by bank
holds each multi-layer run under the run’s name, such as layers_0_15.
A run of one layer looks the same either way. A collection that holds a
run’s bank beside its layers, with different leaves in each (a host
layout’s frozen leaves), is not banked, because its rows still stack
(StackView.stack). A bank and a row that hold the same leaf raise
ValueError, and so do some banks without the others when there are
no rows.
CausalTransformer.residual_dtype
Section titled “CausalTransformer.residual_dtype”def residual_dtype( x, *, train: bool, decode: bool, per_layer_input, attention_metadata=None,) -> jnp.dtypeReturn the dtype the residual stream settles in: x’s promoted with the first layer’s output.
A scan carries one dtype through, and each pipeline stage takes the dtype the stage before returns. Under a bf16 policy the dense feed-forward returns fp32, so the loops use fp32 from the start. The first layer runs abstractly in its own scope (no cache, nothing sown, a placeholder ‘params’ key, which flax draws any other stream from), and a banked store answers from its first bank’s first row without fetching it.
CausalTransformer.first_layer_shapes
Section titled “CausalTransformer.first_layer_shapes”def first_layer_shapes() -> dictReturn the stack’s first layer’s variables as shape/dtype structs.
These are layer 0’s variables, a bank’s first row with the layer axis
dropped, or layer 0’s variables completed by the bank’s
(banked_collections), or a stage-local store’s first stage’s first
row of the run that holds layer 0 (StackView.stage_form). Nothing is
read, fetched or sliced.
CausalTransformer.stage_split
Section titled “CausalTransformer.stage_split”def stage_split(stages: int) -> StageSplitHow the stack runs over stages pipeline stages (split_stages): the layers before and
after the run its stages repeat run on every stage.
CausalTransformer.per_layer_inputs
Section titled “CausalTransformer.per_layer_inputs”def per_layer_inputs(tokens, inputs_embeds)Return every layer’s input signal [B, S, L, P] (Gemma 3n/4 PLE).
The token-identity part is the packed table’s row for each token,
scaled like the main embedding. The context part is the input
embeddings projected down, scaled and normed. Each layer’s gate
multiplies in their sum divided by sqrt(2). A model without
per_layer_input_dim raises ValueError.
CausalTransformer.output_table
Section titled “CausalTransformer.output_table”def output_table() -> OutputTable | NoneReturn the head as the matrix logits_from_hidden contracts, read from the bound variables
(_head_table). None where no matrix alone is the head: past a head_transform or a continuum
level, or where lm_head’s or project_out’s parameters are not its kernel alone: dew.lora’s
factors beside it, which its interceptor adds, or no kernel, a frozen FP8 layer’s.
CausalTransformer.vocabulary_rows
Section titled “CausalTransformer.vocabulary_rows”def vocabulary_rows() -> jax.ArrayEach vocabulary token’s [features] row of the head (OutputTable.rows) where no exact
table is: past a head_transform the matrix’s, with an adapter on the head as it updates them
(applied_kernel), and past an embedding’s own width in the stream’s, through project_out.
CausalTransformer.init_cache
Section titled “CausalTransformer.init_cache”def init_cache(batch_size: int)Allocate a zeroed decode cache for batch_size sequences.
cache = model.apply(params, batch_size, method=CausalTransformer.init_cache, mutable=['cache'])[1]['cache']It runs the forward pass in decode mode on a single dummy token, and that token’s keys are never written: the first decode-mode call allocates the cache, and the calls after it write to it.
CausalTransformer.with_kv_cache
Section titled “CausalTransformer.with_kv_cache”def with_kv_cache(layout: KVCache) -> SelfThe same decoder over the cache layout layout (Layered).
CausalTransformer.with_cache_capacity
Section titled “CausalTransformer.with_cache_capacity”def with_cache_capacity(capacity: int) -> Selfmax_seq_len is the cache size its layers read (open_kv_cache);
a learned position table it sized keeps its rows.
CausalTransformer.with_passes
Section titled “CausalTransformer.with_passes”def with_passes( steps: int | None = None, layers: tuple[int, int] | None = None, *, backprop_steps: int | None = None, remat: bool | None = None,) -> Self | NoneThis model with its loop run steps times (None keeps its count), over the
block layers where given (None keeps its block). backprop_steps
sets how many final passes train through, and remat checkpoints
each pass; None keeps the loop’s own value for either.
A model with no loop gains one over layers, with no norm between its
passes, so one pass computes the model as it was; None when it has no
loop and the two do not both say what to add.
CausalTransformer.projected
Section titled “CausalTransformer.projected”def projected(params, *, updated: bool)ProjectsParameters: before a training step’s update, every AltUp layer’s prediction
coefficients clamped to altup.coef_clip, as the reference clamps its stored weight before
a training forward (clamped_predictions); the updated parameters as they are.
CausalTransformer.with_cross_attention
Section titled “CausalTransformer.with_cross_attention”def with_cross_attention(cross: CrossBlocks | None) -> SelfThis decoder with cross as its cross-attention layers (CrossAttending), None for none.
CausalTransformer.with_dex
Section titled “CausalTransformer.with_dex”def with_dex(extension: DifferentialExtension | None) -> SelfThis decoder with extension’s differential on its chosen heads (dew.nn.dex), None for none.
CausalTransformer.with_scanned_layers
Section titled “CausalTransformer.with_scanned_layers”def with_scanned_layers() -> SelfThis decoder running its like and periodic runs under the scan (scan_layers).
CausalTransformer.reads_progress
Section titled “CausalTransformer.reads_progress”def reads_progress() -> boolWhether its computation changes over a run (dew.nn.progress.ReadsProgress): its DEX layers’
lambda anneals over the run’s updates.
CausalTransformer.dex_exported
Section titled “CausalTransformer.dex_exported”def dex_exported(variables: Mapping[str, Mapping]) -> dict[str, Mapping]variables as an export writes them, each DEX layer’s differential folded into its output
projection at the operands its layers multiply in (dew.nn.dex.folded); ValueError names a
layer whose heads’ outputs pass through more before it, which does not fold.
CausalTransformer.dex_sites
Section titled “CausalTransformer.dex_sites”def dex_sites() -> tuple[tuple[str, ...], frozenset[int]]Its parameters at the root of params, and its softmax attention layers (Differential).
CausalTransformer.with_remat
Section titled “CausalTransformer.with_remat”def with_remat(policy: RematPolicy | None) -> SelfStart at policy, including policies outside DECODER_REMAT.
recompute_more stays at a policy off that ladder, so the trainer
cannot climb from it; None keeps every layer’s residuals.
CausalTransformer.recompute_record
Section titled “CausalTransformer.recompute_record”def recompute_record() -> JSONCausalTransformer.recompute_more
Section titled “CausalTransformer.recompute_more”def recompute_more() -> Self | NoneOne rung up DECODER_REMAT, past any rung that keeps nothing more of the names
these layers mark than the rung above it (MixerBase.marks), whose step would
compile the same; None at the top, off the ladder, or where the top keeps what
this rung keeps, so no rung above holds less.
CausalTransformer.recompute_offloaded
Section titled “CausalTransformer.recompute_offloaded”def recompute_offloaded() -> Self | NoneAt the ladder’s top (‘full’), the decoder that keeps what ‘minimal’ saves in pinned host memory
(‘minimal_offloaded’): as many device bytes as ‘full’ (tests/test_remat_rungs.py’s _saved), the
recompute of ‘minimal’ and its residuals over the host link (row 1932); else None.
CausalTransformer.recompute_half
Section titled “CausalTransformer.recompute_half”def recompute_half() -> Self | NoneThe decoder one rung up where that rung recomputes only the residuals it names
(RematPolicy.recompute, a delta rule’s memory), else None.
CausalTransformer.marked_residuals
Section titled “CausalTransformer.marked_residuals”def marked_residuals() -> frozenset[str]The RESIDUALS this decoder’s layers can mark: each layer’s mixer kind’s
(MixerBase.marks, every name where a kind does not say) and, where a layer, its
gated cross-attention, a prediction depth or the drafter has a feed-forward, its
(MLP_RESIDUALS, the dense MLP’s and the experts’ alike).
CausalTransformer.restore_recompute
Section titled “CausalTransformer.restore_recompute”def restore_recompute(record: JSON) -> SelfCausalTransformer.inference_projection_groups
Section titled “CausalTransformer.inference_projection_groups”def inference_projection_groups( variables: Mapping[str, Mapping],) -> tuple[ProjectionGroup, ...]Its layers’, depths’ and drafter’s groups, bound without tracing a forward, causal or not: a bidirectional model’s masked denoising reads its projections packed as a decoder’s steps do. The groups read which parameters the layers hold, never their values, so the layers are bound over the variables’ shapes: flax’s bound tree is cyclic, and bound over the weights themselves it would keep them alive after the caller drops them until a full collection.
CausalTransformer.block_chunks
Section titled “CausalTransformer.block_chunks”def block_chunks() -> tuple[int | None, ...]The attention chunk of each trunk layer that reads a block whole
(MixerBase.reads_blocks), None where a layer has none (BlockReading). Each layer’s
mixer is _layer_specs’ own, the kind’s or the model’s layer mixer, so a prediction depth’s or the
drafter’s kind, which the trunk never runs, reads none here.
CausalTransformer.mixed_admission_refusal
Section titled “CausalTransformer.mixed_admission_refusal”def mixed_admission_refusal() -> str | NoneA cache rebuilt per request, a reader beyond the token, a context
read beside the tokens, or a layer whose mixer, over the context it is
built with, would not run the mixed call as the separate calls would
(MixerBase.mixed_refusal).
CausalTransformer.generation_refusal
Section titled “CausalTransformer.generation_refusal”def generation_refusal() -> str | NoneWhy a generation loop cannot continue this decoder’s rows, or None
(dew.nn.protocols.Generating): its sparse layers’ mixture, or a mixer it
names (declared_mixers: a layer kind’s, a prediction depth’s or the
drafter’s), that refuses it (Mixture.generation_refusal,
MixerBase.generation_refusal).
CausalTransformer.split_prefill_refusal
Section titled “CausalTransformer.split_prefill_refusal”def split_prefill_refusal() -> str | NoneA cache rebuilt per request from its whole prompt, or a loop’s shared cache that a later
pass rewrites, whose earlier tokens hold the last pass’s keys: a prompt’s later piece would
read those where its whole prefill reads each pass’s own. One pass, or a block whose layers
keep no cache (MixerBase.keeps_cache), rewrites nothing.