dew.nn.mixers.attention
Grouped-query causal attention as a mixer kind.
The attention value builds this module: grouped-query projections, rotary
positions, q/k norms and a fixed-size KV cache.
| Name | Summary |
|---|---|
temperature_scale | Llama 4’s query multiplier at each absolute position (arXiv 2501.19399), log1p(floor((p + 1) / floor_scale)) * attn_scale + 1 in dtype, so the first floor_scale positions scale by 1. |
exclusive_self_attention | Remove each head’s component along its token’s own value vector. |
held_reads | The keys each row’s decode query reads from a paged cache: the row’s filled slots index [rows], or one where its query is padding (positions -1), as the dense path’s per-row key count reads. |
mixed_attention_refusal | Why attention over ctx cannot run a serving step’s mixed call (dew.nn.inputs.Admitted), or None: it reads a full-precision cache, dense or one page pool, whole and causal, and nothing a row carries besides. |
group_fields | The AttentionMetadata fields of groups a causal layer reads whole: image_groups where it reads images whole, block_groups where it reads blocks whole; a non-causal one reads none, since it sees its whole row already. |
CallVisibility | A call’s own visibility over its keys (call_visibility), which replaces a layer’s causality, documents, groups and validity: its mask [B, 1, queries, keys], its key runs [B, queries, R, 2] (the template’s spans), the window the runs are read under, and the places its keys are read at. |
call_visibility | A call’s own visibility, None where it states none: its pairwise mask (AttentionMetadata.pairwise_mask), a windowed layer keeping each query to the keys within its window of the query’s rotary position where the call gives the keys’ positions; else its key runs (AttentionMetadata.key_runs), a windowed layer windowing each run over the key positions the call gives, its queries read at them too where they are its own tokens, or a cached call’s at its rotary positions through a mask. |
dense_visibility | The [B, 1, queries, keys] mask of a call’s visibility as a kernel reads it: mask with its documents, causality, window, block and image groups, key validity valid and key runs folded in (visibility_mask, the window counting real tokens where validity is given, or the runs’ key places), joined with the layer’s chunks of chunk places, each query and key in the chunk of its counted place (places, key_places), never its cache slot. |
GATE_ACTIVATIONS | The output gate’s activations (MixerContext.output_gate_activation). |
ATTENTION_AXES | The logical axes of an attention layer’s projections, by name, as CausalSelfAttention declares them where it makes them; a packed projection (qkv_proj) declares none. |
CausalSelfAttention | Causal self-attention with grouped-query heads, rotary positions, qk RMSNorm and a fixed-size KV cache. |
AttentionMixer | Grouped-query attention with optional image- and block-bidirectional masking and M-RoPE. |
QSASelfAttention | CausalSelfAttention under QSA (qsa), which reads no pairwise mask or key runs: either may hide a key from one query that sees it from another, where QSA pools each key once (qsa_selection). |
QSAAttentionMixer | Qwen4-Exp’s qwen_sparse_attention layer: the attention reading the keys its QSA indexer picks (dew.nn.qsa), the indexer’s geometry by the release’s config names. |
temperature_scale
Section titled “temperature_scale”def temperature_scale( positions, floor_scale: float, attn_scale: float, *, dtype: DTypeLike,)Llama 4’s query multiplier at each absolute position (arXiv 2501.19399),
log1p(floor((p + 1) / floor_scale)) * attn_scale + 1 in dtype, so the
first floor_scale positions scale by 1.
exclusive_self_attention
Section titled “exclusive_self_attention”def exclusive_self_attention(attention: jax.Array, value: jax.Array) -> jax.ArrayRemove each head’s component along its token’s own value vector.
Exclusive self attention (arXiv 2603.09078) as lm-engine implements it
(SoftmaxAttention._compute_xsa_output, softmax_attention/module.py at
45b6b57b): y - (<y, v> / <v, v>) v per token and query head, in fp32
at least, with the key/value head repeated over its query group. attention is
[B, S, heads, D], value [B, S, kv_heads, D].
lm-engine divides by <v, v> with no guard, so a zero value vector
(padding under a values norm, or a zero-initialised projection) is NaN
there; here it leaves the output unchanged, which is the limit of the
projection onto a vanishing direction’s span being empty.
held_reads
Section titled “held_reads”def held_reads(index: jax.Array, positions: jax.Array) -> jax.ArrayThe keys each row’s decode query reads from a paged cache: the row’s
filled slots index [rows], or one where its query is padding
(positions -1), as the dense path’s per-row key count reads. A freed
row keeps its last request’s count until the next one is admitted, and
the paged kernel reads every key under it: on an A100 at 128 slots and
32 requests a second, with 40 to 50 rows decoding, that was 5.1 ms of
attention a step against cuDNN’s 2.0 over the dense cache.
mixed_attention_refusal
Section titled “mixed_attention_refusal”def mixed_attention_refusal( ctx: MixerContext, *, alibi: bool, bidirectional_images: bool, bidirectional_blocks: bool,) -> str | NoneWhy attention over ctx cannot run a serving step’s mixed call
(dew.nn.inputs.Admitted), or None: it reads a full-precision cache,
dense or one page pool, whole and causal, and nothing a row carries
besides.
group_fields
Section titled “group_fields”def group_fields(causal: bool, *, images: bool, blocks: bool) -> tuple[str, ...]The AttentionMetadata fields of groups a causal layer reads whole: image_groups where it reads
images whole, block_groups where it reads blocks whole; a non-causal one reads none, since it sees
its whole row already.
CallVisibility
Section titled “CallVisibility”class CallVisibility(NamedTuple)A call’s own visibility over its keys (call_visibility), which replaces a layer’s causality,
documents, groups and validity: its mask [B, 1, queries, keys], its key runs [B, queries, R, 2]
(the template’s spans), the window the runs are read under, and the places its keys are read at.
call_visibility
Section titled “call_visibility”def call_visibility( metadata: AttentionMetadata, rotary_positions, *, batch: int, queries: int, keys: int, key_places, sliding_window: int | None,) -> CallVisibility | NoneA call’s own visibility, None where it states none: its pairwise mask
(AttentionMetadata.pairwise_mask), a windowed layer keeping each query to the keys within its
window of the query’s rotary position where the call gives the keys’ positions; else its key runs
(AttentionMetadata.key_runs), a windowed layer windowing each run over the key positions the call
gives, its queries read at them too where they are its own tokens, or a cached call’s at its rotary
positions through a mask. key_places are where the keys are read otherwise.
dense_visibility
Section titled “dense_visibility”def dense_visibility( mask, documents, causal: bool, window: int | None, valid, runs, places, key_places, queries: int, keys: int, chunk: int | None, *, block_groups=None, image_groups=None,) -> jax.ArrayThe [B, 1, queries, keys] mask of a call’s visibility as a kernel reads it: mask with its
documents, causality, window, block and image groups, key validity valid and key runs folded in
(visibility_mask, the window counting real tokens where validity is given, or the runs’ key places),
joined with the layer’s chunks of chunk places, each query and key in the chunk of its counted place
(places, key_places), never its cache slot.
GATE_ACTIVATIONS
Section titled “GATE_ACTIVATIONS”GATE_ACTIVATIONSThe output gate’s activations (MixerContext.output_gate_activation).
ATTENTION_AXES
Section titled “ATTENTION_AXES”ATTENTION_AXES: dict[str, LogicalAxes] = {'q_proj': ('embed', 'heads'), 'k_proj': ('embed', 'kv'), 'v_proj': ('embed', 'kv'), 'gate_proj': ('embed', 'heads'), 'o_proj': ('attention', 'embed')}The logical axes of an attention layer’s projections, by name, as CausalSelfAttention declares them
where it makes them; a packed projection (qkv_proj) declares none.
CausalSelfAttention
Section titled “CausalSelfAttention”class CausalSelfAttention( reads: frozenset[str] = frozenset({'pairwise_mask', 'key_positions', 'key_runs', 'position_bias', 'admitted'}), qk_norm_weight: bool = True, temperature_tuning: tuple[float, float] | None = None, gate_projection: Literal['query', 'own', 'head'] = 'query', bidirectional_images: bool = False, bidirectional_blocks: bool = False, mrope_section: tuple[int, int, int] | None = None, rotary_axes: tuple[int, ...] | None = None, rotary_pairs: Literal['half', 'adjacent'] = 'half', rotary_per_axis: bool = False, qk_norm_after_rotary: bool = False, packed: bool = False, linear: Callable[..., Dense] = functools.partial(Dense, dot_general=float32_dot_general), nope: bool = False, alibi: bool = False, exclusive_self_attention: bool = False, qsa: QSA | None = None,)Causal self-attention with grouped-query heads, rotary positions, qk RMSNorm and a fixed-size KV cache.
decode=True runs against the cache: the first call writes the prompt and
each later call appends, so prefill and decode are one path. Keys are
rotated before they are cached, so rotary positions come from the cache
index. causal=False is full attention, which a masked diffusion model reads
its corrupted sequence with and an encoder reads its input with; with
bidirectional_window a window keeps the keys within window - 1 positions
on either side (dew.nn.visibility.window_sides). Decoding attends a bidirectional canvas
over the frozen prefix a causal prefill cached, and writes nothing back.
kv_shared marks a layer without K/V projections (Gemma 3n/4 cross-layer
sharing): it reads the keys, values and places its provider stashed in
kv_store, post rope and norm, and keeps no cache.
reads: frozenset[str]-
Every
MIXINGfield of its call (dew.nn.inputs.MixerCall): it masks by a pairwise mask or key runs, places keys, adds a position bias and runs a server’s mixed step. temperature_tuning: tuple[float, float] | None-
A (floor_scale, attn_scale) that scales each query by its position (
temperature_scale), rotated or not: Llama 4’s unrotated layers’. gate_projection: Literal['query', 'own', 'head']-
Where the output gate comes from: the doubled query projection (Qwen3.5), a
gate_projof its own over the layer’s input (Muse Glimmer, modeling_muse_glimmer.py:305, 351), or such agate_projwith one gate a head, which scales the head’s every value (LFM3, modeling_lfm3.py:229, 268-269). Either way the branch multiplies by its sigmoid. bidirectional_images: bool-
Whether a causal layer lets each token of an image (
AttentionMetadata.image_groups) see the image’s later tokens too, within its window (Gemma 3’s, and Gemma 4’s sliding layers’). bidirectional_blocks: bool-
Whether a causal layer lets each token of a block (
AttentionMetadata.block_groups) see the whole block, whatever its window (transformers’block_sequence_ids): a decision prompt’s blocks, read both ways on the full layers that train so, a unified model’s media. A layer reading image groups as well sees the union. rotary_axes: tuple[int, ...] | None-
The channel widths of a rotary over several axes (
dew.nn.rope.axis_tables), a vision tower’s patch grid, whose idsattention_metadata.rotary_positionscarries[B, S, len(rotary_axes)]; None rotates by the positions. rotary_pairs: Literal['half', 'adjacent']-
The channels each angle turns (
dew.nn.rope.rotate): the HF decoders’ halves, or the adjacent pairs of Llama 4’s vision tower. rotary_per_axis: bool-
Whether each of the
rotary_axesturns its own channels in halves, Gemma 4’s vision rope (apply_multidimensional_rope), where Qwen 3.5’s turns the axes’ angles together over the whole head. qk_norm_after_rotary: bool-
Whether each head’s queries and keys are normed after their rotation, not before it: Hunyuan’s (modeling_hunyuan_v1_moe.py, HunYuanMoEV1Attention), a cached key normed as it is written. The query’s logit scale then multiplies after its norm, where a rotation folds it in.
packed: bool-
Whether the queries, keys and values project through one kernel,
qkv_proj, at init as from a checkpoint that stores them so (the Qwen 3.5 and DeepSeek-V4.1 vision towers). linear: Callable[..., Dense]-
The projections’ module: Gemma 4’s vision tower clips their inputs and outputs (
dew.nn.vision.gemma4.Gemma4ClippableLinear). nope: bool-
No positional encoding: q and k enter the kernel unrotated, and the logits keep their scale (lm-engine’s
position_embedding_type="nope"). alibi: bool-
Linear key-position bias added to the logits, after any rotation of the queries and keys; BLOOM’s are unrotated (
nope). exclusive_self_attention: bool-
XSA (arXiv 2603.09078): each head’s output loses its component along the token’s own value vector before the output projection.
qsa: QSA | None-
Qwen4-Exp’s QSA (
dew.nn.qsa): the keys each query reads are those itsindexerpicks among the ones it sees; None reads them all. ctx: MixerContext-
The layer’s geometry and attention policies, as the backbone resolves them for its kind (
MixerContext). reads_train: bool-
Whether its call takes
train(ReadsTrain): it drops attention probabilities while training. value_dim: int-
The width of one value head.
reads_context: bool-
Whether it projects keys and values of its own to read a context with (
context_keys): a KV-sharing layer does not.
CausalSelfAttention.setup
Section titled “CausalSelfAttention.setup”def setup()CausalSelfAttention.model_flops
Section titled “CausalSelfAttention.model_flops”def model_flops(lengths) -> floatQuery/output maps, unshared K/V maps and DEX’s maps of its chosen heads, plus QK and AV.
CausalSelfAttention.parameter_flops
Section titled “CausalSelfAttention.parameter_flops”def parameter_flops(lengths) -> floatLearned QKV/output/gate maps, excluding data-dependent attention products.
CausalSelfAttention.head_outputs
Section titled “CausalSelfAttention.head_outputs”def head_outputs() -> tuple[int, int]Its heads’ count and each one’s output width, which DEX extends (dew.nn.dex.HeadOutputs).
CausalSelfAttention.dex_fold_refusal
Section titled “CausalSelfAttention.dex_fold_refusal”def dex_fold_refusal() -> str | NoneWhat its heads’ outputs pass through before o_proj, past DEX’s differential, which keeps
the differential out of the projection (dew.nn.dex.folded); None where nothing does.
CausalSelfAttention.projections
Section titled “CausalSelfAttention.projections”def projections() -> dict[str, tuple[int, int]]The attention’s maps as {name: (inputs, outputs)}, keys and
values absent where the layer shares them or the keys are the values, and values where routed
experts give them (_value_expert_flops):
what setup builds, the count reads and the packed group
(_group) concatenates.
CausalSelfAttention.projection_groups
Section titled “CausalSelfAttention.projection_groups”def projection_groups() -> tuple[ProjectionGroup, ...]Its group (_group) where its variables hold it (ProjectionSites),
which setup reads packed in the members’ place.
CausalSelfAttention.context_heads
Section titled “CausalSelfAttention.context_heads”def context_heads() -> tuple[int, int, int]Its key heads’ count and the widths of a key and a value head, as context_keys gives them.
CausalSelfAttention.context_query
Section titled “CausalSelfAttention.context_query”def context_query(x)Its query heads over x [B, S, E] as it reads another sequence (attend_context):
projected and normed as its own call does them, scaled to the logit scale its checkpoint asks
for, and placed nowhere: a context has no positions to rotate by, nor a temperature a place
tunes. With the output gate, None where it gates none.
CausalSelfAttention.context_keys
Section titled “CausalSelfAttention.context_keys”def context_keys(context)Its key and value heads over a context [B, C, E], projected and normed as its own keys
and values (_projected_kv), unrotated. A KV-sharing layer projects none of its own.
CausalSelfAttention.attend_context
Section titled “CausalSelfAttention.attend_context”def attend_context( query, gate, key, value, *, lengths, queries=None, segments=None, max_segments=None, progress=None,)The layer’s own attention of query heads (context_query) over a context’s key and
value heads (context_keys), each row’s first lengths keys real (none for a row without
context, whose read its block leaves out) and, where given, each
query reading its own document’s keys alone (queries [B, S], segments [B, C]): its
unit sphere under nGPT, its sinks, logit softcap and kernel, unmasked by causality, then its
output gate and projection. A context holds no token’s own value, so exclusive self
attention excludes none.
CausalSelfAttention.context_flops
Section titled “CausalSelfAttention.context_flops”def context_flops(lengths, context_lengths) -> floatIts read of a context’s work over rows of lengths tokens reading context_lengths
(attend_context): its projections’ (context_projection_flops) and the attention between
them.
CausalSelfAttention.context_projection_flops
Section titled “CausalSelfAttention.context_projection_flops”def context_projection_flops(lengths, context_lengths) -> floatThe part of its read of a context done by its own matrices: the query, gate and output
projections over the rows of lengths tokens, the key and value projections over the context’s
context_lengths.
AttentionMixer
Section titled “AttentionMixer”class AttentionMixer( bidirectional_images: bool = False, bidirectional_blocks: bool = False, mrope_section: tuple[int, int, int] | None = None, nope: bool = False, alibi: bool = False, exclusive_self_attention: bool = False, gate_projection: Literal['query', 'own', 'head'] = 'query', qk_norm_weight: bool = True, rotary_pairs: Literal['half', 'adjacent'] = 'half', qk_norm_after_rotary: bool = False, temperature_tuning: tuple[float, float] | None = None,)Grouped-query attention with optional image- and block-bidirectional masking and M-RoPE.
Geometry, norms and kernel policy come from the decoder context. Image and
block bidirectionality, spatial rotary sections, NoPE and exclusive self
attention are this mixer’s own fields, which a hybrid names on its attention
kind ({"name": "attention", "fields": {"nope": true}}).
bidirectional_images: bool-
Whether a causal layer reads an image whole (
CausalSelfAttention.bidirectional_images). bidirectional_blocks: bool-
Whether a causal layer reads a block whole (
CausalSelfAttention.bidirectional_blocks). nope: bool-
No positional encoding: q and k enter the kernel unrotated and the logits keep their scale (lm-engine’s
position_embedding_type="nope", Granite 4.0-H). alibi: bool-
Per-head linear key-position bias (
CausalSelfAttention.alibi). exclusive_self_attention: bool-
XSA (arXiv 2603.09078, lm-engine’s
exclusive_self_attention): each head’s output loses its component along the token’s own value (exclusive_self_attention). gate_projection: Literal['query', 'own', 'head']-
Where the decoder’s output gate comes from (
CausalSelfAttention.gate_projection). qk_norm_weight: bool-
Whether the decoder’s qk norm has a weight (
CausalSelfAttention.qk_norm_weight): Muse Glimmer’s has none (modeling_muse_glimmer.py:307), nor Llama 4’s L2 norm. rotary_pairs: Literal['half', 'adjacent']-
The channels each rotary angle turns (
CausalSelfAttention.rotary_pairs): halves, or adjacent pairs, as Llama 4’s rotated layers turn them. qk_norm_after_rotary: bool-
Whether each head’s queries and keys are normed after their rotation (
CausalSelfAttention.qk_norm_after_rotary), as Hunyuan’s are. temperature_tuning: tuple[float, float] | None-
A (floor_scale, attn_scale) that scales each query by its position (
temperature_scale, arXiv 2501.19399), as Llama 4’s unrotated layers do; None scales none.
AttentionMixer.mixed_refusal
Section titled “AttentionMixer.mixed_refusal”def mixed_refusal(ctx: MixerContext) -> str | NoneAttentionMixer.build
Section titled “AttentionMixer.build”def build(ctx: MixerContext) -> Callable[..., nn.Module]QSASelfAttention
Section titled “QSASelfAttention”class QSASelfAttention(CausalSelfAttention)CausalSelfAttention under QSA (qsa), which reads no pairwise mask or key runs: either may hide a
key from one query that sees it from another, where QSA pools each key once (qsa_selection).
QSAAttentionMixer
Section titled “QSAAttentionMixer”class QSAAttentionMixer( bidirectional_images: bool = False, bidirectional_blocks: bool = False, mrope_section: tuple[int, int, int] | None = None, nope: bool = False, alibi: bool = False, exclusive_self_attention: bool = False, gate_projection: Literal['query', 'own', 'head'] = 'query', qk_norm_weight: bool = True, rotary_pairs: Literal['half', 'adjacent'] = 'half', qk_norm_after_rotary: bool = False, temperature_tuning: tuple[float, float] | None = None, indexer_n_heads: int = 4, indexer_head_dim: int = 128, indexer_budget: int = 2048, indexer_compress_ratio: int = 4,)Qwen4-Exp’s qwen_sparse_attention layer: the attention reading the keys its QSA indexer picks
(dew.nn.qsa), the indexer’s geometry by the release’s config names.
QSAAttentionMixer.mixed_refusal
Section titled “QSAAttentionMixer.mixed_refusal”def mixed_refusal(ctx: MixerContext) -> str | NoneQSAAttentionMixer.build
Section titled “QSAAttentionMixer.build”def build(ctx: MixerContext) -> Callable[..., nn.Module]