Skip to content

dew.nn.multimodal

Image and audio conditioning around the shared causal decoder.

NameSummary
MultimodalTransformerConditions the shared decoder on images and audio, using the ordinary decoder cache and head.

class source

class MultimodalTransformer(DecoderWrapper)

Conditions the shared decoder on images and audio, using the ordinary decoder cache and head.

image_indices and audio_indices give, for each token slot, the index of the soft feature that replaces it, or -1 for a text token. The towers run during conditioned forward and prefill calls; later decode steps read the cached language states and do not run the encoders again. The parameters keep the existing language_model, tower and projector names, and audio adds audio_tower and audio_projector. Gemma 3n also embeds its hard vision and audio vocabulary ranges through the embedders, and keeps placeholder ids for its per-layer inputs, as modeling_gemma3n.py does. The serving and training hooks are its language model’s (DecoderWrapper); a clone keeps the towers and every variable’s path. So is its cross-attention (CrossAttending): a call’s context entries (dew.nn.inputs.CONTEXT) go to the language model’s layers, beside the media the towers read.

in_context: bool

Whether the decoder holds a continuum level (SteppedInContext).

in_context_levels: int

The decoder’s continuum levels (SteppedInContext).

def model_flops(lengths) -> float

The language model’s work over the real tokens; the media towers’ and projectors’ work is left out. A model with an audio encoder refuses: its encoder’s work over its frames has no count here. A projector of raw audio frames alone (Gemma 4 Unified) is left out, as a vision projector is.

def parameter_flops(lengths) -> float
def cross_flops(lengths, context_lengths) -> float
def held_cross_flops(lengths, context_lengths) -> float

MultimodalTransformer.with_cross_attention

Section titled “MultimodalTransformer.with_cross_attention”
def with_cross_attention(cross: CrossBlocks | None) -> Self
def with_dex(extension: DifferentialExtension | None) -> Self
def with_scanned_layers() -> Self
def dex_exported(variables: Mapping[str, Mapping]) -> dict[str, Mapping]
def dex_sites() -> tuple[tuple[str, ...], frozenset[int]]
def logits_flops(lengths) -> float
def matrix_usage(path: tuple[str, ...], lengths, *, logits: bool = False)

The same native language scope the wrapper actually invokes without media.

def setup()
def mtp_hidden_states(
hidden,
tokens,
train: bool = False,
image_indices=None,
conditioning=None,
image_groups=None,
block_groups=None,
rotary_positions=None,
audio_indices=None,
coordinates: Coordinates | None = None,
)

Run the multi-token prediction layers over the same media embeddings and token fields as the main decoder (hidden_states), its tokens laid out by their coordinates.

def mtp_logits(hidden, tokens, **kwargs={})

Return the shared language head’s logits for each media-aware prediction depth.

def mtp_step(
hidden,
tokens,
*,
image_indices=None,
audio_indices=None,
conditioning=None,
input_embeddings=None,
**kwargs={},
)

Run one candidate prediction step with the decoder’s separate MTP cache.

It returns the step’s logits and hidden state. Pass either prepared input_embeddings or media conditioning, not both.

def stream_embeddings(tokens)

Return the decoder’s own embeddings of tokens at its stream’s width (stream_embeddings); a sampled token is always text, never media.

def init_mtp_cache(batch_size: int)
def hidden_states(
tokens,
train: bool = False,
decode: bool = False,
image_indices=None,
conditioning: Mapping[str, jax.Array] | None = None,
image_groups=None,
block_groups=None,
rotary_positions=None,
audio_indices=None,
routed_experts=None,
routed=None,
coordinates: Coordinates | None = None,
whole_blocks: bool = False,
progress: Progress | None = None,
)

Run the decoder over text with the media embeddings spliced in.

conditioning holds the towers’ payloads, and image_indices and audio_indices say which token positions each one replaces. A decode step tracks the next position in the cache collection, because a media span advances a row by more than one token. routed_experts and routed replay a routing record and go to the language model unchanged. Engines record a row for every placeholder position too, so the [B, S, layers, top_k] layout matches the text. coordinates, whole_blocks and progress are the language model’s (CausalTransformer.hidden_and_mtp_inputs): the tokens as the decoder reads them, so the cursor counts the places the layers do.

def logits(tokens, *, train: bool = False, **fields={})

Return the [B, S, vocab] fp32 logits of the whole sequence, media fused as hidden_states fuses them from fields. Without decode the call writes no cache.

def logits_from_hidden(hidden)

Return the decoder’s logits of final states hidden.

def head_bounds(states)

What bounds the decoder’s head’s curvature and slope past its level at states (SteppedInContext).

def in_context_level(variables)

The decoder’s continuum level’s parameters in variables (SteppedInContext).

MultimodalTransformer.with_in_context_level

Section titled “MultimodalTransformer.with_in_context_level”
def with_in_context_level(variables, level)

variables holding level as the decoder’s continuum level (SteppedInContext).

def continuum_states(hidden, index: int | None = None)

The decoder’s final states hidden past its continuum levels, or past level index alone (SteppedInContext).

def level_terms(hidden, parts: tuple[str, ...], index: int = 0, passed: bool = False)

What bounds a damped step of the decoder’s level index’s parts at the states it reads (SteppedInContext).

def head_logits(states)

The decoder’s head over states its continuum level already read (SteppedInContext).

def states_and_logits(tokens, **kwargs={})

Return the final hidden states and their logits from one media-aware forward pass.

MultimodalTransformer.states_and_logits_at

Section titled “MultimodalTransformer.states_and_logits_at”
def states_and_logits_at(tokens, slots, **kwargs={})

Return the final hidden states, and the logits of one slot per row.

A prefill needs logits only at the position the first sample reads. Running the head over every prompt position would take most of a long request’s transient memory, so this scores only slots.

def output_table() -> OutputTable | None

Return the decoder’s head as the matrix its final states contract, or None where none does (CausalTransformer.output_table).

def vocabulary_rows() -> jax.Array

The decoder’s head rows (CausalTransformer.vocabulary_rows).

def init_cache(batch_size: int)

Allocate the language model’s cache and the next-position counter, without running the towers.

def with_passes(
steps: int | None = None,
layers: tuple[int, int] | None = None,
*,
backprop_steps: int | None = None,
remat: bool | None = None,
) -> Self | None

MultimodalTransformer.inference_projection_groups

Section titled “MultimodalTransformer.inference_projection_groups”
def inference_projection_groups(
variables: Mapping[str, Mapping],
) -> tuple[ProjectionGroup, ...]

The language model’s (held_groups); the towers run once per prefill and keep their layout.

MultimodalTransformer.mixed_admission_refusal

Section titled “MultimodalTransformer.mixed_admission_refusal”
def mixed_admission_refusal() -> str | None

MultimodalTransformer.split_prefill_refusal

Section titled “MultimodalTransformer.split_prefill_refusal”
def split_prefill_refusal() -> str | None