Skip to content

dew.nn.vision

Vision towers, projectors and reference checkpoint maps.

transformers 5 ships no Flax classes, so the towers are vendored the way dew/nn/text_encoders.py vendors CLIP, in the reference layout, with each weight read from the checkpoint’s safetensors under its reference tensor name. The operation order follows transformers 5.16.1 models/siglip/modeling_siglip.py, models/llama4/modeling_llama4.py, models/gemma4/modeling_gemma4.py, models/qwen3_5/modeling_qwen3_5.py and models/muse_glimmer/modeling_muse_glimmer.py, MiMo-V2.6’s modeling_mimo_v2.py and LFM3-VL’s modeling_lfm3_vl.py.

The SigLIP trunk is patch convolution with bias, learned position embeddings with no class token, pre-norm encoder blocks and a post layer norm. Its block shares both halves with CLIP’s: the attention and the feed-forward (dew.nn.text_encoders), the MLP carrying the config’s activation. The Llama 4 trunk is MetaCLIP-style: an unfold patch embedding without bias, a class token appended after the patches, learned positions, a pre norm, full-attention blocks with a complex rotary over the patch grid, a post norm, the class token dropped, and the pixel-shuffle MLP inside the tower where the reference keeps it. The Gemma 4 trunk is patch pixels scaled to [-1, 1] through a bias-free map, summed 2D position tables, RMS-normed blocks with a 2D rotary and gated feed-forwards, and a position pooler with standardization. The Qwen 3.5, MiMo-V2.6, Muse Glimmer and LFM3-VL trunks are configs of one windowed and full vision transformer (vit): a NaViT-style patchify with the still frame repeated along time, an optional resampled position table, decoder blocks with a 2D rotary that attend over a frame, a band of keys or a window of patches, or convolve the frame’s patch grid (LFM3-VL’s), each layer in its own token order, and the merge-block projector. Each tower’s projector is a value beside it: Gemma 3’s averages each patch block, norms and maps to the decoder width, Llama 4’s maps the shuffled output to the decoder width, Gemma 4’s norms without a scale and maps, and Qwen 3.5’s is the merger. Gemma 3n uses the MobileNet-v5 encoder in dew.nn.mobilenet and its own hard/soft vision embedder. DeepSeek-V4.1’s trunk (the release’s inference/vision.py) is a biased patch map, RMS-normed blocks with the same 2D rotary as Qwen 3.5’s and a SwiGLU feed-forward, and a final norm; its projector is the aligner, which groups squares of the patch grid through an exact-GELU MLP and lays the image out as its decoder reads it.

Each family lives in its own module of this package (siglip, llama4, gemma4, qwen35, mimo, muse_glimmer, lfm3, deepseek_v41, gemma3n) over the bases in common and the shared trunk in vit, with its path map and its config and weight translators, and each family’s classes say where their tensors sit (TowerLayout, ProjectorLayout), so the kinds dew.registry names are the one list of them. The package exports the classes a caller constructs or subclasses, and reads a kind’s tensors through its class’s layout (tower_variables, projector_variables). Each name is imported from its own module when first read, so a caller of one family’s tower reads no other family’s.

NameSummary
DeepseekV41ProjectorDeepSeek-V4.1’s aligner and image span (vision.py:106-119, model.py:1228-1239).
DeepseekV41VisionDeepSeek-V4.1’s ViT geometry, under its vision_config’s names (vision.py:87-103).
Gemma3nProjectorGemma3nMultimodalEmbedder’s vision hard and soft token paths.
Gemma3nVisionGemma 3n’s vision tower: MobileNet-v5’s encoder, by its construction fields (MobileNetV5).
Gemma4ProjectorSoft tokens to text width: a scale-free RMS norm, then the map.
Gemma4VisionA Gemma 4 trunk’s geometry, under the reference’s field names.
Gemma4VisionTransformerThe Gemma 4 vision trunk, param layout of Gemma4VisionConfig.
GemmaProjectorPatch features into soft tokens: block average, norm, map to text width.
Lfm3VlProjectorLFM3-VL’s projector (Lfm3VlMultiModalProjector) is MergeProjector: a layer norm of each trunk token at projector_layer_norm_eps, its bias kept whatever projector_bias says, each merge block’s tokens joined unit by unit, and two maps, biased by projector_bias, around the projector_hidden_act activation.
Lfm3VlVisionLFM3-VL’s trunk geometry under its vision_config’s names.
Llama4ProjectorTower output to text width: one bias-free map.
Llama4VisionA Llama 4 trunk’s geometry, under the reference’s field names.
Llama4VisionTransformerThe Llama 4 vision trunk, param layout of Llama4VisionConfig.
MergeProjectorEach merge block’s features as one vector, mapped to the decoder width.
MiMoProjectorMiMo’s merger (MiMoVisionPatchMerger): a layer norm, each merge block grouped, and two maps around an exact GELU to the decoder width.
MiMoVisionMiMo ViT’s geometry under its vision_config’s names.
MuseGlimmerProjectorThe merged image tokens as the decoder reads them (MuseGlimmerModel.get_image_features): each merge block’s features channel by channel through the adapter’s two bias-free maps, each followed by the activation (MergeProjector), a bias-free map to the decoder width, and a scale-free RMS norm (perception_emb_norm).
MuseGlimmerVisionMuse Glimmer’s trunk geometry under its vision_config’s names.
ProjectorBaseA native projector module whose constructor is its record schema.
ProjectorLayoutWhere one projector kind’s tensors sit in a media checkpoint and the leaves they fill: prefix before every name (or several, whose tensors each keep the name past their own), tensors each name’s leaf path, strip a further prefix a name may carry (a merger’s), and read the projector’s own reader where it checks its tensors (placed otherwise).
Qwen35ProjectorQwen 3.5’s projector fields: the trunk width, the merge size, the merged width the text embeddings read directly.
Qwen35VisionA Qwen 3.5 trunk’s geometry, under the reference’s field names.
SiglipVisionA SigLIP trunk’s geometry, under the reference’s field names.
TowerBaseOne tower kind’s value: its fields, how it builds its module, and where its tensors sit in a media checkpoint (checkpoint, None for a tower read another way, as the audio towers are).
TowerGeometryThe shapes a conditioner has to invent to create a tower’s media leaves.
TowerLayoutWhere one tower kind’s tensors sit in a media checkpoint and the leaves they fill: prefix before every name, and path from a name past it to its leaf, rooted at its collection (params, and Gemma 4’s constants beside it).
ViTLayoutThe geometry and wiring of one family’s tower.
VisionTransformerA family’s trunk (ViTLayout): patches mapped and placed, the blocks, and the trunk sequence in merge-block order.
projector_layoutWhere projector kind kind’s tensors sit: its class’s checkpoint (dew.registry.projectors).
projector_variablesOne projector kind’s parameters, in the requested storage (projector_layout), and the layouts that write its tensors back from them.
tower_layoutWhere tower kind kind’s tensors sit: its class’s checkpoint (dew.registry.towers).
tower_variablesOne vision tower’s variables, in the requested storage (tower_layout), and the layouts that write its tensors back from them.

class source

class DeepseekV41Projector(ProjectorBase)

DeepSeek-V4.1’s aligner and image span (vision.py:106-119, model.py:1228-1239).

The patch grid is zero-padded at its bottom and right to whole downsample_ratio squares, each square’s features concatenate channel-major (unfold’s order), and a two-layer exact-GELU map takes them to the decoder width. The span the decoder reads is a learned start vector, each row of aligned features followed by a learned newline vector, and a learned end vector: rows * (columns + 1) + 2 positions.

dataclass source

class DeepseekV41Vision(
num_hidden_layers: int = 32,
hidden_size: int = 1024,
num_attention_heads: int = 16,
intermediate_size: int = 2816,
patch_size: int = 14,
rope_theta: float = 10000.0,
)

DeepSeek-V4.1’s ViT geometry, under its vision_config’s names (vision.py:87-103).

Its layout (ViTLayout): each patch_size square’s pixels, channel-major, through one biased map; blocks of full attention whose queries, keys and values come from one biased map, rotated by the grid’s rows then columns in fp32 as Qwen 3.5’s vision rope turns them (vision.py:8-21), then a SwiGLU whose gate and up maps are one, each after an RMS norm (vision.py:46-84); a final RMS norm, and the trunk returned as its [images, rows, columns, hidden_size] grid, which the aligner groups.

def layout() -> ViTLayout
def build() -> nn.Module
def geometry() -> TowerGeometry

class source

class Gemma3nProjector(ProjectorBase)

Gemma3nMultimodalEmbedder’s vision hard and soft token paths.

scales_soft: bool

Whether soft features are multiplied by sqrt(vision_width) before the embedder reads them: Gemma 3n’s image features are (modeling_gemma3n.py:1983 at transformers 5.17.0), and its audio embedder reads its encoder’s features as they are (:2192), which its norm’s epsilon tells apart.

def setup()
def soft_embeddings(features)

The reference embedder over soft features, without scales_soft’s scaling.

def embed_hard(ids)

Embed vocabulary ids in [vocab_offset, vocab_offset + vocab_size).

Pure: the host processor validates ids before device work.

def merge_hard_embeddings(token_embeddings, ids)

Replace this vocabulary range’s slots with hard embeddings, through the reference’s dummy id for every other slot.

dataclass source

class Gemma3nVision(
channel_multiplier: float = 1.0,
stem_size: int = 64,
stem_bias: bool = True,
fix_stem: bool | None = None,
in_chans: int = 3,
pad_type: str = 'same',
group_size: int | None = None,
msfa_indices: tuple[int, ...] = (-2, -1),
msfa_output_resolution: int = 16,
layer_scale_init_value: float | None = 1e-05,
drop_path_rate: float = 0.0,
)

Gemma 3n’s vision tower: MobileNet-v5’s encoder, by its construction fields (MobileNetV5).

def build() -> nn.Module
def geometry() -> TowerGeometry

class source

class Gemma4Projector(ProjectorBase)

Soft tokens to text width: a scale-free RMS norm, then the map.

The reference keeps this beside the tower (modeling_gemma4.py, Gemma4MultimodalEmbedder): the pre-projection norm carries no weight and the projection no bias.

def setup()

dataclass source

class Gemma4Vision(
hidden_size: int = 1152,
intermediate_size: int = 4304,
num_layers: int = 27,
num_heads: int = 16,
num_key_value_heads: int = 16,
patch_size: int = 16,
pooling_kernel_size: int = 3,
position_embedding_size: int = 10240,
hidden_act: str = 'gelu_pytorch_tanh',
rms_norm_eps: float = 1e-06,
rope_theta: float = 100.0,
standardize: bool = False,
use_clipped_linears: bool = False,
head_dim: int | None = None,
)

A Gemma 4 trunk’s geometry, under the reference’s field names.

head_dim: int | None

None retains hidden_size // num_heads; checkpoints may project a wider head.

def build() -> nn.Module
def geometry() -> TowerGeometry

Flax module source

class Gemma4VisionTransformer(
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

The Gemma 4 vision trunk, param layout of Gemma4VisionConfig.

The processor supplies [B, patches, patch_pixels] and (x, y) position IDs with (-1, -1) for padding. Fixed NCHW images are patchified in the same HWC order. Outputs retain padded soft-token slots so their shape remains static under JIT; callers select valid features using processor lengths.

Each layer is the decoder block with RMS norms before and after both halves (modeling_gemma4.py, Gemma4VisionEncoderLayer). Its attention has bias-free maps over grouped key heads, the queries and keys normed with a scale and the values without, its scores unscaled, and each spatial axis turning its own half of the head the NeoX way (Gemma4VisionAttention, apply_multidimensional_rope). The MLP gates the checkpoint’s GELU. With use_clipped_linears every map clips its input and output to the bounds the checkpoint stores (Gemma4ClippableLinear).

def setup()

class source

class GemmaProjector(ProjectorBase)

Patch features into soft tokens: block average, norm, map to text width.

The reference reshapes the trunk sequence into its patch grid, averages each kernel block, norms and multiplies by its (vision, text) matrix (modeling_gemma3.py, Gemma3MultiModalProjector).

def setup()

class source

class Lfm3VlProjector(MergeProjector, ProjectorBase)

LFM3-VL’s projector (Lfm3VlMultiModalProjector) is MergeProjector: a layer norm of each trunk token at projector_layer_norm_eps, its bias kept whatever projector_bias says, each merge block’s tokens joined unit by unit, and two maps, biased by projector_bias, around the projector_hidden_act activation.

dataclass source

class Lfm3VlVision(
hidden_size: int = 1024,
intermediate_size: int = 2560,
num_hidden_layers: int = 14,
num_attention_heads: int = 16,
num_key_value_heads: int = 8,
patch_size: int = 16,
temporal_patch_size: int = 2,
in_channels: int = 3,
spatial_merge_size: int = 2,
conv_kernel_size: int = 3,
norm_eps: float = 1e-05,
rope_theta: float = 1000000.0,
layer_types: tuple[str, ...] = default_layer_types(14),
)

LFM3-VL’s trunk geometry under its vision_config’s names.

Its layout (ViTLayout): a bias-free map of each patch’s channel-then-time pixels in merge-block order (patchifier), an RMS norm after it (embedding_norm), and pre-norm blocks of RMS norms at norm_eps with a bias-free SwiGLU feed-forward (w1 its gate, w3 its up map), each mixing its frame’s patches by full attention, grouped and unbiased with an RMS norm on each head’s queries and keys before the rotary turns them by the patch grid, rows then columns (full_attention), or by a gated depthwise convolution over the frame’s grid (conv, GatedGridConv); no norm after the blocks. Every RMS norm weighs its input after its cast back, as the release’s Lfm3VlRMSNorm does.

def layout() -> ViTLayout
def build() -> nn.Module
def geometry() -> TowerGeometry

class source

class Llama4Projector(ProjectorBase)

Tower output to text width: one bias-free map.

def setup()

dataclass source

class Llama4Vision(
hidden_size: int = 1408,
intermediate_size: int = 5632,
num_layers: int = 32,
num_heads: int = 16,
image_size: int = 336,
patch_size: int = 14,
num_channels: int = 3,
layer_norm_eps: float = 1e-05,
rope_theta: float = 10000.0,
pixel_shuffle_ratio: float = 0.5,
projector_input_dim: int = 4096,
projector_output_dim: int = 4096,
)

A Llama 4 trunk’s geometry, under the reference’s field names.

def layout() -> ViTLayout
def build() -> nn.Module
def geometry() -> TowerGeometry

Flax module source

class Llama4VisionTransformer(
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

The Llama 4 vision trunk, param layout of Llama4VisionConfig.

pixel_values are what the checkpoint’s image processor emits: [B, C, H, W] at image_size. The patches embed through one bias-free map, the class token rides last through the trunk and is dropped before the adapter, and what returns is the pixel-shuffled MLP output the outer projector maps to text width.

def setup()

Flax module source

class MergeProjector(
norm: bool = True,
norm_eps: float = 1e-06,
norm_bias: bool | None = None,
channel_major: bool = False,
bias: bool = True,
activation: str = 'gelu',
activation_after: bool = False,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

Each merge block’s features as one vector, mapped to the decoder width.

The trunk sequence comes in merge-block order (VisionTransformer). With norm, a layer norm first at norm_eps (Qwen 3.5’s, MiMo’s and LFM3-VL’s); bias gives both maps their biases, and the norm its own where norm_bias is unset (LFM3-VL’s norm keeps its bias either way). Each block’s merge**2 features join unit by unit, or channel by channel where channel_major (Muse Glimmer’s pixel_shuffle), and pass two maps with the checkpoint’s activation (its hidden_act name) between them, and after the second too where activation_after (Muse Glimmer’s adapter).

class source

class MiMoProjector(MergeProjector, ProjectorBase)

MiMo’s merger (MiMoVisionPatchMerger): a layer norm, each merge block grouped, and two maps around an exact GELU to the decoder width. The release stores none of its biases.

dataclass source

class MiMoVision(
depth: int = 28,
hidden_size: int = 1280,
intermediate_size: int = 4608,
num_heads: int = 32,
num_key_value_heads: int = 8,
qk_channels: int = 64,
in_channels: int = 3,
patch_size: int = 16,
spatial_merge_size: int = 2,
temporal_patch_size: int = 2,
rms_norm_eps: float = 1e-06,
fullatt_block_indexes: tuple[int, ...] = (0, 9, 18, 27),
vit_window_attn_types: tuple[int, ...] = (-1, 0, 0, 0, 0, 1, 1, 1, 1) * 3 + (-1,),
use_sink: bool = True,
visual_token_window_size: int = 64,
)

MiMo ViT’s geometry under its vision_config’s names.

Its layout (ViTLayout): a bias-free patch map and no position table; RMS-normed blocks whose grouped queries, keys and values come from one biased map, rotated by the patch grid as Qwen 3.5’s are, then a biased SwiGLU. Each layer of fullatt_block_indexes attends over its whole frame, every other one over the visual_token_window_size tokens either side of each query, plus a learned logit per head on each frame’s first key where use_sink; a layer whose vit_window_attn_types entry is 1 runs with the merge blocks taken column by column.

def layout() -> ViTLayout
def build() -> nn.Module
def geometry() -> TowerGeometry

class source

class MuseGlimmerProjector(ProjectorBase)

The merged image tokens as the decoder reads them (MuseGlimmerModel.get_image_features): each merge block’s features channel by channel through the adapter’s two bias-free maps, each followed by the activation (MergeProjector), a bias-free map to the decoder width, and a scale-free RMS norm (perception_emb_norm).

dataclass source

class MuseGlimmerVision(
num_hidden_layers: int = 50,
hidden_size: int = 1536,
intermediate_size: int = 8960,
num_attention_heads: int = 16,
hidden_act: str = 'gelu',
patch_size: int = 14,
patch_temporal: int = 2,
merge_size: int = 2,
pos_emb_height: int = 32,
pos_emb_width: int = 32,
layer_norm_eps: float = 1e-05,
rope_theta: float = 10000.0,
layer_types: tuple[str, ...] = default_layer_types(50),
)

Muse Glimmer’s trunk geometry under its vision_config’s names.

Its layout (ViTLayout): a bias-free map of each raster patch’s time-then-channel pixels, the learned pos_emb_height x pos_emb_width table resampled with half-pixel centres and zero padding (MuseGlimmerVisionPatchEmbedder), a layer norm before the blocks and one after, and CLIP-style blocks (layer norms at 1e-5, biased query, key and value maps, an ungated feed-forward) rotated by the patch grid, columns then rows, each counted from one. The tokens run window by window, pos_emb_height patches square; a window_attention layer attends within its window, a full_attention one over its frame.

def layout() -> ViTLayout
def build() -> nn.Module
def geometry() -> TowerGeometry

Flax module source

class ProjectorBase(checkpoint: ProjectorLayout | None = None)

A native projector module whose constructor is its record schema. The registry names its kind and checkpoint locates the tensors it consumes.

input_width: int | None

Declared feature width when this projector consumes raw media without an encoder.

dataclass source

class ProjectorLayout(
prefix: str | tuple[str, ...],
tensors: Mapping[str, tuple[str, ...]],
strip: str = '',
read: Callable[..., tuple[Variables, tuple[WeightLayout, ...]]] | None = None,
)

Where one projector kind’s tensors sit in a media checkpoint and the leaves they fill: prefix before every name (or several, whose tensors each keep the name past their own), tensors each name’s leaf path, strip a further prefix a name may carry (a merger’s), and read the projector’s own reader where it checks its tensors (placed otherwise).

def path(name: str) -> tuple[str, ...]

The leaf of tensor name, shared by load and source export.

def variables(
hf_tensors: Mapping[str, Stored],
param_dtype: str,
) -> tuple[Variables, tuple[WeightLayout, ...]]

The projector’s parameters, in the requested storage, and the layouts that write its tensors back from them.

class source

class Qwen35Projector(MergeProjector, ProjectorBase)

Qwen 3.5’s projector fields: the trunk width, the merge size, the merged width the text embeddings read directly. The merger (modeling_qwen3_5.py, Qwen3_5VisionPatchMerger) is MergeProjector: norm, each merge block grouped, two maps around an exact GELU.

dataclass source

class Qwen35Vision(
depth: int = 27,
hidden_size: int = 1152,
hidden_act: str = 'gelu_pytorch_tanh',
intermediate_size: int = 4304,
num_heads: int = 16,
in_channels: int = 3,
patch_size: int = 16,
spatial_merge_size: int = 2,
temporal_patch_size: int = 2,
num_position_embeddings: int = 2304,
)

A Qwen 3.5 trunk’s geometry, under the reference’s field names.

Its layout (ViTLayout): a biased patch map, the learned table resampled with aligned corners, CLIP’s layer at a 1e-6 epsilon (flax’s one-pass layer norm with its bias, the queries, keys and values in one biased map, an ungated feed-forward) and full attention over each frame, rotated by the patch grid, heights then widths (modeling_qwen3_5.py, Qwen3_5VisionRotaryEmbedding and apply_rotary_pos_emb_vision). The trunk sequence leaves for the merger, its projector.

def layout() -> ViTLayout
def build() -> nn.Module
def geometry() -> TowerGeometry

dataclass source

class SiglipVision(
hidden_size: int = 768,
intermediate_size: int = 3072,
num_layers: int = 12,
num_heads: int = 12,
image_size: int = 224,
patch_size: int = 16,
num_channels: int = 3,
hidden_act: str = 'gelu_pytorch_tanh',
layer_norm_eps: float = 1e-06,
)

A SigLIP trunk’s geometry, under the reference’s field names.

def layout() -> ViTLayout
def build() -> nn.Module
def geometry() -> TowerGeometry

class source

class TowerBase

One tower kind’s value: its fields, how it builds its module, and where its tensors sit in a media checkpoint (checkpoint, None for a tower read another way, as the audio towers are).

def build() -> nn.Module

The Flax module for this value.

def geometry() -> TowerGeometry

What an initialising input has to look like for this tower.

Nothing by default: a tower states the fields it has, and a conditioner reads them here instead of asking the value at runtime whether it carries each one.

dataclass source

class TowerGeometry(
image_size: int | None = None,
patch_size: int | None = None,
block_size: int | None = None,
channels: int | None = None,
mel_features: int | None = None,
frames_per_token: int | None = None,
position_size: int | None = None,
retains_patch_slots: bool = False,
)

The shapes a conditioner has to invent to create a tower’s media leaves.

A token-only init has no batch, so VisionConditioner and AudioConditioner build one smallest input the tower accepts. What that is differs by kind: a fixed-resolution tower states its image_size, a patch-and-pool tower states the patch and block that make one, and an audio tower states how many mel bins a frame carries. None means this tower has no such field, and the conditioner uses its own default.

frames_per_token: int | None

How many input frames an audio tower encodes into one token: its convolutions’ time strides times any frame reduction after them.

dataclass source

class TowerLayout(
prefix: str,
path: Callable[[str], tuple[str, ...] | None],
read: Reader = placed,
)

Where one tower kind’s tensors sit in a media checkpoint and the leaves they fill: prefix before every name, and path from a name past it to its leaf, rooted at its collection (params, and Gemma 4’s constants beside it). read binds them (placed, or, for a tower that runs its patch convolution as one Dense map, translate_vit_weights). Loading (tower_variables), routing a checkpoint’s names (hf_decoders) and binding an export (interop.pretrained) read it alike.

def variables(
hf_tensors: Mapping[str, Stored],
param_dtype: str,
) -> tuple[Variables, tuple[WeightLayout, ...]]

The tower’s variables from its tensors, in the requested storage, and the layouts that write each tensor it placed back from them; a tensor the tower does not compute with, as SigLIP’s pooling head, has none.

dataclass source

class ViTLayout(
depth: int,
hidden_size: int,
num_heads: int,
num_kv_heads: int,
head_dim: int,
intermediate_size: int,
activation: str | Callable[[jax.Array], jax.Array],
in_channels: int,
patch_size: int,
temporal_patch_size: int,
spatial_merge_size: int,
norm_type: Literal['rms', 'layer', 'fast_layer'],
norm_eps: float,
norm_bias: bool,
mlp_bias: bool,
packed: bool,
patch_bias: bool,
attention: tuple[Attention, ...],
order: tuple[Order, ...],
sinks: tuple[bool, ...],
positions: LearnedPositions | None = None,
special_tokens: SpecialTokens = SpecialTokens(),
patch_projection: Literal['dense', 'conv'] = 'dense',
image_size: int | None = None,
rotary: Literal['half', 'adjacent'] | None = 'half',
pre_norm: Literal['layer', 'rms'] | None = None,
post_norm: Literal['layer', 'rms'] | None = None,
post_norm_scope: Literal['sequence', 'class'] = 'sequence',
post_norm_affine: bool = True,
outer_norm_eps: float = 1e-05,
mlp_packed: bool = False,
output: Literal['sequence', 'patches', 'patch_grid'] = 'sequence',
branch_scale: float | None = None,
rope_theta: float = 10000.0,
rotary_columns_first: bool = False,
rotary_origin: int = 0,
band: int = 0,
window: int = 0,
merge_blocks: bool = True,
channels_first: bool = True,
attention_bias: bool = True,
qk_norm: bool = False,
conv_kernel: int = 3,
scale_after_cast: bool = False,
)

The geometry and wiring of one family’s tower.

attention, order and sinks name each layer’s: its keys (a frame’s every patch, band keys either side in the layer’s order, or the patches of its window-patch square) or, for conv, a gated conv_kernel-square convolution over its frame’s patch grid in the attention’s place (LFM3-VL’s, GatedGridConv, which reads each patch’s grid place as its rotary position), the order its tokens run in (the processor’s, merge blocks column by column, or window by window), and whether a learned logit per head joins the first key of each frame (MiMo’s sinks). rotary_columns_first turns the first rotary half by the column and the second by the row, both counted from rotary_origin; positions keeps a fixed grid or resizes its square table by its stated interpolation. Bilinear sampling without aligned corners gives out-of-range taps zero weight (F.grid_sample’s zero padding). merge_blocks is the processor’s patch order: merge block by merge block, else raster; channels_first its pixel order in a patch, channels then time, else time then channels. mlp_packed keeps a gated feed-forward’s gate and up maps in one (as packed the attention’s queries, keys and values). post_norm names the norm after the blocks, a layer or an RMS norm, or none. output selects the sequence, patches alone or a [B, rows, columns, F] patch grid; post_norm_scope distinguishes CLIP’s normalized class row from a normalized whole sequence. pre_norm names the norm after the patch map, as post_norm does. attention_bias biases the attention’s maps, and qk_norm norms each head’s queries and keys (an RMS norm at the layers’ epsilon) before the rotary; under scale_after_cast every RMS norm weighs its normalized input after its cast back, as Llama’s does.

Flax module source

class VisionTransformer(dtype: Dtype | None = None, precision: PrecisionLike = None)

A family’s trunk (ViTLayout): patches mapped and placed, the blocks, and the trunk sequence in merge-block order.

Packed pixels [B, patches, pixels] come with each row’s [time, height, width] patch grid, a row’s padding past its patches; attention stays inside a frame, or a window of it, and padding (segment 0) sees nothing. NCHW images are one frame each, repeated along time, packed in the processor’s order. Each layer runs in its own token order, the tokens, their rotary places and their frames gathered into it from the previous layer’s; the sequence returns to the processor’s order after the last.

def setup()
def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarray
def model_flops(lengths) -> float
def sequence_and_pool(pixel_values, grid_thw=None) -> tuple[jax.Array, jax.Array]

The selected sequence and its class row through the declared post norm.

function source

def projector_layout(kind: str) -> ProjectorLayout

Where projector kind kind’s tensors sit: its class’s checkpoint (dew.registry.projectors).

function source

def projector_variables(
kind: str,
hf_tensors: Mapping[str, Stored],
param_dtype: str,
) -> tuple[Variables, tuple[WeightLayout, ...]]

One projector kind’s parameters, in the requested storage (projector_layout), and the layouts that write its tensors back from them.

function source

def tower_layout(kind: str) -> TowerLayout

Where tower kind kind’s tensors sit: its class’s checkpoint (dew.registry.towers).

function source

def tower_variables(
kind: str,
hf_tensors: Mapping[str, Stored],
param_dtype: str,
) -> tuple[Variables, tuple[WeightLayout, ...]]

One vision tower’s variables, in the requested storage (tower_layout), and the layouts that write its tensors back from them.