dew.nn.multimodal
Image and audio conditioning around the shared causal decoder.
| Name | Summary |
|---|---|
MultimodalTransformer | Conditions the shared decoder on images and audio, using the ordinary decoder cache and head. |
MultimodalTransformer
Section titled “MultimodalTransformer”class MultimodalTransformer(DecoderWrapper)Conditions the shared decoder on images and audio, using the ordinary decoder cache and head.
image_indices and audio_indices give, for each token slot, the
index of the soft feature that replaces it, or -1 for a text token. The
towers run during conditioned forward and prefill calls; later decode
steps read the cached language states and do not run the encoders again.
The parameters keep the existing language_model, tower and projector
names, and audio adds audio_tower and audio_projector. Gemma 3n also
embeds its hard vision and audio vocabulary ranges through the embedders,
and keeps placeholder ids for its per-layer inputs, as modeling_gemma3n.py
does. The serving and training hooks are its language model’s (DecoderWrapper); a
clone keeps the towers and every variable’s path. So is its cross-attention
(CrossAttending): a call’s context entries (dew.nn.inputs.CONTEXT) go to the
language model’s layers, beside the media the towers read.
in_context: bool-
Whether the decoder holds a continuum level (
SteppedInContext). in_context_levels: int-
The decoder’s continuum levels (
SteppedInContext).
MultimodalTransformer.model_flops
Section titled “MultimodalTransformer.model_flops”def model_flops(lengths) -> floatThe language model’s work over the real tokens; the media towers’ and projectors’ work is left out. A model with an audio encoder refuses: its encoder’s work over its frames has no count here. A projector of raw audio frames alone (Gemma 4 Unified) is left out, as a vision projector is.
MultimodalTransformer.parameter_flops
Section titled “MultimodalTransformer.parameter_flops”def parameter_flops(lengths) -> floatMultimodalTransformer.cross_flops
Section titled “MultimodalTransformer.cross_flops”def cross_flops(lengths, context_lengths) -> floatMultimodalTransformer.held_cross_flops
Section titled “MultimodalTransformer.held_cross_flops”def held_cross_flops(lengths, context_lengths) -> floatMultimodalTransformer.with_cross_attention
Section titled “MultimodalTransformer.with_cross_attention”def with_cross_attention(cross: CrossBlocks | None) -> SelfMultimodalTransformer.with_dex
Section titled “MultimodalTransformer.with_dex”def with_dex(extension: DifferentialExtension | None) -> SelfMultimodalTransformer.with_scanned_layers
Section titled “MultimodalTransformer.with_scanned_layers”def with_scanned_layers() -> SelfMultimodalTransformer.dex_exported
Section titled “MultimodalTransformer.dex_exported”def dex_exported(variables: Mapping[str, Mapping]) -> dict[str, Mapping]MultimodalTransformer.dex_sites
Section titled “MultimodalTransformer.dex_sites”def dex_sites() -> tuple[tuple[str, ...], frozenset[int]]MultimodalTransformer.logits_flops
Section titled “MultimodalTransformer.logits_flops”def logits_flops(lengths) -> floatMultimodalTransformer.matrix_usage
Section titled “MultimodalTransformer.matrix_usage”def matrix_usage(path: tuple[str, ...], lengths, *, logits: bool = False)The same native language scope the wrapper actually invokes without media.
MultimodalTransformer.setup
Section titled “MultimodalTransformer.setup”def setup()MultimodalTransformer.mtp_hidden_states
Section titled “MultimodalTransformer.mtp_hidden_states”def mtp_hidden_states( hidden, tokens, train: bool = False, image_indices=None, conditioning=None, image_groups=None, block_groups=None, rotary_positions=None, audio_indices=None, coordinates: Coordinates | None = None,)Run the multi-token prediction layers over the same media embeddings and
token fields as the main decoder (hidden_states), its tokens laid out by their
coordinates.
MultimodalTransformer.mtp_logits
Section titled “MultimodalTransformer.mtp_logits”def mtp_logits(hidden, tokens, **kwargs={})Return the shared language head’s logits for each media-aware prediction depth.
MultimodalTransformer.mtp_step
Section titled “MultimodalTransformer.mtp_step”def mtp_step( hidden, tokens, *, image_indices=None, audio_indices=None, conditioning=None, input_embeddings=None, **kwargs={},)Run one candidate prediction step with the decoder’s separate MTP cache.
It returns the step’s logits and hidden state. Pass either prepared
input_embeddings or media conditioning, not both.
MultimodalTransformer.stream_embeddings
Section titled “MultimodalTransformer.stream_embeddings”def stream_embeddings(tokens)Return the decoder’s own embeddings of tokens at its stream’s width (stream_embeddings); a
sampled token is always text, never media.
MultimodalTransformer.init_mtp_cache
Section titled “MultimodalTransformer.init_mtp_cache”def init_mtp_cache(batch_size: int)MultimodalTransformer.hidden_states
Section titled “MultimodalTransformer.hidden_states”def hidden_states( tokens, train: bool = False, decode: bool = False, image_indices=None, conditioning: Mapping[str, jax.Array] | None = None, image_groups=None, block_groups=None, rotary_positions=None, audio_indices=None, routed_experts=None, routed=None, coordinates: Coordinates | None = None, whole_blocks: bool = False, progress: Progress | None = None,)Run the decoder over text with the media embeddings spliced in.
conditioning holds the towers’ payloads, and image_indices and
audio_indices say which token positions each one replaces. A
decode step tracks the next position in the cache collection,
because a media span advances a row by more than one token.
routed_experts and routed replay a routing record and go to the
language model unchanged. Engines record a row for every placeholder
position too, so the [B, S, layers, top_k] layout matches the text.
coordinates, whole_blocks and progress are the language model’s
(CausalTransformer.hidden_and_mtp_inputs): the tokens as the decoder reads them, so the
cursor counts the places the layers do.
MultimodalTransformer.logits
Section titled “MultimodalTransformer.logits”def logits(tokens, *, train: bool = False, **fields={})Return the [B, S, vocab] fp32 logits of the whole sequence, media
fused as hidden_states fuses them from fields. Without decode
the call writes no cache.
MultimodalTransformer.logits_from_hidden
Section titled “MultimodalTransformer.logits_from_hidden”def logits_from_hidden(hidden)Return the decoder’s logits of final states hidden.
MultimodalTransformer.head_bounds
Section titled “MultimodalTransformer.head_bounds”def head_bounds(states)What bounds the decoder’s head’s curvature and slope past its level at states
(SteppedInContext).
MultimodalTransformer.in_context_level
Section titled “MultimodalTransformer.in_context_level”def in_context_level(variables)The decoder’s continuum level’s parameters in variables (SteppedInContext).
MultimodalTransformer.with_in_context_level
Section titled “MultimodalTransformer.with_in_context_level”def with_in_context_level(variables, level)variables holding level as the decoder’s continuum level (SteppedInContext).
MultimodalTransformer.continuum_states
Section titled “MultimodalTransformer.continuum_states”def continuum_states(hidden, index: int | None = None)The decoder’s final states hidden past its continuum levels, or past level index alone
(SteppedInContext).
MultimodalTransformer.level_terms
Section titled “MultimodalTransformer.level_terms”def level_terms(hidden, parts: tuple[str, ...], index: int = 0, passed: bool = False)What bounds a damped step of the decoder’s level index’s parts at the states it reads
(SteppedInContext).
MultimodalTransformer.head_logits
Section titled “MultimodalTransformer.head_logits”def head_logits(states)The decoder’s head over states its continuum level already read (SteppedInContext).
MultimodalTransformer.states_and_logits
Section titled “MultimodalTransformer.states_and_logits”def states_and_logits(tokens, **kwargs={})Return the final hidden states and their logits from one media-aware forward pass.
MultimodalTransformer.states_and_logits_at
Section titled “MultimodalTransformer.states_and_logits_at”def states_and_logits_at(tokens, slots, **kwargs={})Return the final hidden states, and the logits of one slot per row.
A prefill needs logits only at the position the first sample reads.
Running the head over every prompt position would take most of a
long request’s transient memory, so this scores only slots.
MultimodalTransformer.output_table
Section titled “MultimodalTransformer.output_table”def output_table() -> OutputTable | NoneReturn the decoder’s head as the matrix its final states contract,
or None where none does (CausalTransformer.output_table).
MultimodalTransformer.vocabulary_rows
Section titled “MultimodalTransformer.vocabulary_rows”def vocabulary_rows() -> jax.ArrayThe decoder’s head rows (CausalTransformer.vocabulary_rows).
MultimodalTransformer.init_cache
Section titled “MultimodalTransformer.init_cache”def init_cache(batch_size: int)Allocate the language model’s cache and the next-position counter, without running the towers.
MultimodalTransformer.with_passes
Section titled “MultimodalTransformer.with_passes”def with_passes( steps: int | None = None, layers: tuple[int, int] | None = None, *, backprop_steps: int | None = None, remat: bool | None = None,) -> Self | NoneMultimodalTransformer.inference_projection_groups
Section titled “MultimodalTransformer.inference_projection_groups”def inference_projection_groups( variables: Mapping[str, Mapping],) -> tuple[ProjectionGroup, ...]The language model’s (held_groups); the towers run once per prefill and keep their layout.
MultimodalTransformer.mixed_admission_refusal
Section titled “MultimodalTransformer.mixed_admission_refusal”def mixed_admission_refusal() -> str | NoneMultimodalTransformer.split_prefill_refusal
Section titled “MultimodalTransformer.split_prefill_refusal”def split_prefill_refusal() -> str | None