dew.nn.backbones
Model backbones.
You build a model by calling its class, as in SimpleDiT(patch_size=4, ...).
A run record names the class by its import path, and dew.registry.models
holds short aliases such as simple_dit, so either and its fields rebuild
the same model.
Each backbone is imported only when it is used (dew._exports), so a module that reads one
backbone imports that backbone alone.
| Name | Summary |
|---|---|
CausalTransformer | Decoder-only transformer over token ids: [B, S] int32 -> [B, S, vocab] fp32. Documented in dew.nn.backbones.causal_transformer. |
EDM2UNet | Runs EDM2’s U-Net. Documented in dew.nn.backbones.edm2. |
Flux2Transformer | Runs Diffusers 0.40.0’s Flux2Transformer2DModel behind Dew’s model interface. Documented in dew.nn.backbones.flux2. |
FluxTransformer | Runs Diffusers 0.34.0’s FluxTransformer2DModel behind Dew’s model interface. Documented in dew.nn.backbones.flux. |
HierarchicalMMDiT | U-shaped MM-DiT: dual-stream blocks per stage with patch merging on the way down and expansion + skip fusion on the way up. |
HybridSSMAttentionDiT | A DiT that interleaves SSM blocks with attention blocks. |
Mixture | The experts that some layers route to, and how the router chooses among them. Documented in dew.nn.backbones.decoder_block. |
QwenImageTransformer | Runs QwenImage21Transformer2DModel behind Dew’s model interface. Documented in dew.nn.backbones.qwen_image. |
SD3Transformer | Runs Diffusers 0.34.0’s SD3Transformer2DModel behind Dew’s model interface. Documented in dew.nn.backbones.sd3. |
SimpleDiT | The standard DiT: a plain stack of adaLN-Zero attention blocks. |
SimpleMMDiT | SD3-style MM-DiT: a plain stack of dual-stream blocks, the image tokens rotated by their raster index (rope_for_scan). |
SimpleUDiT | A U-shaped DiT: SimpleDiT’s adaLN-Zero blocks with the first half’s outputs skipping into the second half through a dense layer over the concatenation. |
UNet2DCondition | Denoises NHWC latents conditioned on text, pooled and size features, and optional inpainting inputs. |
UNet3D | Denoises video (B, T, H, W, C) with the 2D Unet body per frame and a TemporalBlock per level. |
UNetStage | Describes one resolution level: its width, heads and transformer depth. |
UViT | Denoises patches as U-ViT does, following baofff/U-ViT’s libs/uvit_t2i.py. |
Unet | Denoises images with a convolutional UNet of residual blocks and cross-attention stages. |
VideoDiT | Factorized spatial-temporal DiT over (B, T, H, W, C) inputs. |
WanTransformer | Runs Diffusers 0.34.0’s WanTransformer3DModel for text-to-video, behind Dew’s model interface. Documented in dew.nn.backbones.wan. |
ZImageTransformer | Runs Diffusers 0.40.0’s ZImageTransformer2DModel behind Dew’s model interface. Documented in dew.nn.backbones.z_image. |
HierarchicalMMDiT
Section titled “HierarchicalMMDiT”class HierarchicalMMDiT(RematLadder, _AttentionStackOptions)U-shaped MM-DiT: dual-stream blocks per stage with patch merging on the way down and expansion + skip fusion on the way up.
Raster order only: the merge/expand grid reshapes assume row-major token order, so a hilbert scan would scramble the neighborhoods being merged.
text_keyword: str-
Every call takes the text as
textcontext, which runs as a second stream through every block.
HierarchicalMMDiT.projections
Section titled “HierarchicalMMDiT.projections”def projections(text_features: int = 0) -> dict[str, tuple[int, int]]HierarchicalMMDiT.model_flops
Section titled “HierarchicalMMDiT.model_flops”def model_flops(lengths) -> floatHierarchicalMMDiT.image_lengths
Section titled “HierarchicalMMDiT.image_lengths”def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarrayHierarchicalMMDiT.multimodal_flops
Section titled “HierarchicalMMDiT.multimodal_flops”def multimodal_flops( lengths, context_lengths, input_channels: int, text_features: int,) -> floatPer-stage streams, patch merges/expansions, skip maps and output.
HierarchicalMMDiT.stage_blocks
Section titled “HierarchicalMMDiT.stage_blocks”def stage_blocks(stage: int, prefix: str) -> listBuild one stage’s dual-stream blocks, at that stage’s width and heads.
HierarchicalMMDiT.encoder_path
Section titled “HierarchicalMMDiT.encoder_path”def encoder_path(num_stages: int)Build the encoder, fine to coarse: each stage’s blocks and its merger.
Sets encoder_blocks, one list per stage, and patch_mergers, one
between each pair of stages.
HierarchicalMMDiT.decoder_path
Section titled “HierarchicalMMDiT.decoder_path”def decoder_path(num_stages: int)Build the decoder, coarse to fine, for stages N-2 down to 0.
Sets patch_expanders, the fusion_layers that take an expanded
stage beside its skip, and decoder_blocks. All three are indexed
by decoder step, not by stage.
HierarchicalMMDiT.setup
Section titled “HierarchicalMMDiT.setup”def setup()Build the patch embedding, the two paths of stages, and the head.
cond_projs and txt_embeds carry the conditioning and the text
into each stage’s own width. encoder_path and decoder_path set
the blocks and the resampling layers between them.
HybridSSMAttentionDiT
Section titled “HybridSSMAttentionDiT”class HybridSSMAttentionDiT(SimpleDiT)A DiT that interleaves SSM blocks with attention blocks.
Each layer’s mixer comes from ssm_attention_ratio, a shorthand that
reads the same at any depth (“3:1”, “all-ssm”), or from block_pattern,
which names each layer. Setting both raises a ValueError at setup.
Everything around the layers is SimpleDiT’s.
HybridSSMAttentionDiT.layer
Section titled “HybridSSMAttentionDiT.layer”def layer(index: int, kind: str, *, counted: bool = False) -> ModulatedBlockBuild layer index’s block, with an SSM mixer or an attention mixer; counted,
unbound and without remat, for work accounting.
The two share the width, the MLP ratio and the norms. They differ in the mixer they name, the fields only that mixer reads, the remat policy, and the name a checkpoint stores them under.
HybridSSMAttentionDiT.stack
Section titled “HybridSSMAttentionDiT.stack”def stack(*, counted: bool = False) -> list[ModulatedBlock]Return the layers’ blocks, each the block that ssm_attention_ratio or block_pattern names;
counted, unbound and without remat (counted_blocks).
Raises ValueError when both are set, or when TREAD’s routes are
combined with use_2d_fusion.
HybridSSMAttentionDiT.kinds
Section titled “HybridSSMAttentionDiT.kinds”def kinds() -> list[str]SimpleDiT
Section titled “SimpleDiT”class SimpleDiT(_DiTStackOptions)The standard DiT: a plain stack of adaLN-Zero attention blocks.
adaln_silu=Falseandtext_pooling="all"give FlaxDiff 0.2’s conditioning, as inSimpleUDiT.routesis TREAD’s token routing (Krause et al. 2025) during training. Each(ratio, start, end)drawsint(tokens * ratio)tokens per example, from thedropoutstream, that skip blocksstarttoendand rejoin afterendwith the values they had on entry, as CompVis/tread’sRouterdoes; the kept tokens stay in order at their own positions. Sampling runs every token through every block.patch_bottleneckis JiT’s bottleneck patch embedding (Li and He 2025), 128 in its models.intervaladds thedurationof the predicted interval next to the time, for MeanFlow and shortcut models (Process.interval).time_scalescales the Fourier frequencies; keep it small for a model trained through a JVP in time.
block_prefixes: tuple[str, ...]-
The names the stack’s blocks are stored under,
{prefix}N, one prefix a mixer.
SimpleDiT.counted_blocks
Section titled “SimpleDiT.counted_blocks”def counted_blocks() -> list[ModulatedBlock]The layers stack builds, unbound and without lifted remat, for host-side work
accounting; a configuration stack refuses is refused here too.
SimpleDiT.model_flops
Section titled “SimpleDiT.model_flops”def model_flops(lengths) -> floatInput channels and conditioning width belong to the objective’s geometry.
SimpleDiT.image_lengths
Section titled “SimpleDiT.image_lengths”def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarrayPatch-token lengths from the actual latent/image shape, never a guessed channel count.
SimpleDiT.image_flops
Section titled “SimpleDiT.image_flops”def image_flops(lengths, input_channels: int, text_features: int | None = None) -> floatEvery matmul, including patch/time/text input maps; only lookups excluded.
SimpleDiT.setup
Section titled “SimpleDiT.setup”def setup()SimpleDiT.stack
Section titled “SimpleDiT.stack”def stack(*, counted: bool = False) -> list[ModulatedBlock]The layers between the patch embedding and the output, all attention; counted,
unbound and without remat (counted_blocks).
SimpleDiT.kinds
Section titled “SimpleDiT.kinds”def kinds() -> list[str]Each layer’s mixer, in order.
SimpleDiT.stage_sites
Section titled “SimpleDiT.stage_sites”def stage_sites(store: Variables, stages: int) -> tuple[DecoderBank, ...]StagedStacks: the blocks, where stages split them (pipelined).
SimpleDiT.layer
Section titled “SimpleDiT.layer”def layer(index: int, kind: str, *, counted: bool = False) -> ModulatedBlockLayer index’s block, of mixer kind, stored as dit_block_{index}; counted, unbound
and without remat.
SimpleDiT.pipelined
Section titled “SimpleDiT.pipelined”def pipelined(tokens, conditioning, freqs_cis, train: bool)The blocks over tokens as a pipeline over the stage axis (pipelined_blocks), each row
modulated by its own conditioning, every one rotated alike.
SimpleDiT.checked_routes
Section titled “SimpleDiT.checked_routes”def checked_routes() -> list[tuple[float, int, int]]Return routes as (ratio, start, end) tuples, after checking them.
Each ratio must be in (0, 1), and the spans must be ordered, disjoint and inside the stack.
SimpleDiT.kept_tokens
Section titled “SimpleDiT.kept_tokens”def kept_tokens(tokens: jax.Array, ratio: float, start: int) -> jax.ArrayReturn the [B, keep] indices of the tokens a route at start computes on.
As in the reference, these are the first keep of a uniform shuffle,
in sequence order.
SimpleMMDiT
Section titled “SimpleMMDiT”class SimpleMMDiT(_DiTStackOptions)SD3-style MM-DiT: a plain stack of dual-stream blocks, the image tokens
rotated by their raster index (rope_for_scan).
text_keyword: str-
Every call takes the text as
textcontext, which runs as a second stream through every block.
SimpleMMDiT.model_flops
Section titled “SimpleMMDiT.model_flops”def model_flops(lengths) -> floatSimpleMMDiT.image_lengths
Section titled “SimpleMMDiT.image_lengths”def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarraySimpleMMDiT.multimodal_flops
Section titled “SimpleMMDiT.multimodal_flops”def multimodal_flops( lengths, context_lengths, input_channels: int, text_features: int,) -> floatAll input/output maps, pooled/time conditioning and joint blocks.
SimpleMMDiT.projections
Section titled “SimpleMMDiT.projections”def projections(text_features: int = 0) -> dict[str, tuple[int, int]]SimpleMMDiT.setup
Section titled “SimpleMMDiT.setup”def setup()SimpleUDiT
Section titled “SimpleUDiT”class SimpleUDiT(RematLadder, _TransformerOptions)A U-shaped DiT: SimpleDiT’s adaLN-Zero blocks with the first half’s
outputs skipping into the second half through a dense layer over the
concatenation. Position comes from RoPE over the sequence index, so a
hilbert scan carries the rotation of its curve index and no 2D signal.
adaln_silu False and text_pooling “all” are FlaxDiff 0.2’s U-DiT: its
blocks project the conditioning vector without a SiLU and it averages the
text over every position. dew.interop.flaxdiff loads its checkpoints
with them.
SimpleUDiT.projections
Section titled “SimpleUDiT.projections”def projections(input_channels: int = 0) -> dict[str, tuple[int, int]]SimpleUDiT.model_flops
Section titled “SimpleUDiT.model_flops”def model_flops(lengths) -> floatSimpleUDiT.image_lengths
Section titled “SimpleUDiT.image_lengths”def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarraySimpleUDiT.image_flops
Section titled “SimpleUDiT.image_flops”def image_flops(lengths, input_channels: int, text_features: int | None = None) -> floatPatch/time/pooled maps, modulated blocks and the up-path skip maps.
SimpleUDiT.setup
Section titled “SimpleUDiT.setup”def setup()UNet2DCondition
Section titled “UNet2DCondition”class UNet2DCondition( in_channels: int = 4, out_channels: int = 4, blocks_per_level: int = 2, linear_projection: bool = False, additional_time_features: int = 0, middle_attention: bool = True, frequency_shift: float = 0, cosine_first: bool = True, dropout: float = 0.0, dtype: Dtype = jnp.float32, precision: PrecisionLike = None, attention_impl: AttentionImpl = 'auto', norm_groups: int = 32, norm_epsilon: float = 1e-05, attention_norm_epsilon: float = 1e-05, approximate_gelu: bool = True,)Denoises NHWC latents conditioned on text, pooled and size features, and optional inpainting inputs.
The stages set each level’s spatial width, attention heads and transformer depth. The encoder saves each residual output and each downsample, and the decoder uses those skips in reverse order, with one extra residual block per level.
text_keyword: str-
Every call takes the text as
conditioning, which every attention stage cross-attends to.
UNet2DCondition.model_flops
Section titled “UNet2DCondition.model_flops”def model_flops(lengths) -> floatUNet2DCondition.condition_lengths
Section titled “UNet2DCondition.condition_lengths”def condition_lengths(lengths: np.ndarray, spans: np.ndarray) -> np.ndarrayUNet2DCondition.conditioned_spatial_flops
Section titled “UNet2DCondition.conditioned_spatial_flops”def conditioned_spatial_flops( sample_shape: tuple[int, ...], rows: int, condition: ConditionWork,) -> floatUNet2DCondition.spatial_flops
Section titled “UNet2DCondition.spatial_flops”def spatial_flops( sample_shape: tuple[int, ...], rows: int, context_lengths=None, text_features: int | None = None, *, additional_features: int | None = None,) -> floatThe residual/attention stage walk, including every resizer and skip width.
UNet2DCondition.stage_sites
Section titled “UNet2DCondition.stage_sites”def stage_sites(store: Variables, stages: int) -> tuple[DecoderBank, ...]StagedStacks: each level’s transformers, where stages split them, found by walking the
levels’ shapes (spatial_walk) as the FLOP count does.
UNet2DCondition.spatial_walk
Section titled “UNet2DCondition.spatial_walk”def spatial_walk(x, time, context, *, train=False, operations)UNet3D
Section titled “UNet3D”class UNet3D(Unet)Denoises video (B, T, H, W, C) with the 2D Unet body per frame and a TemporalBlock per level.
There is a TemporalBlock at every resolution level. The spatial
parameter paths are identical to Unet’s, so 2D checkpoints inflate
directly.
UNet3D.spatial_flops
Section titled “UNet3D.spatial_flops”def spatial_flops( sample_shape: tuple[int, ...], rows: int, context_lengths=None, text_features: int | None = None,) -> floatThe frame body and its temporal hook share the forward’s ordered walk.
UNet3D.temporal_block
Section titled “UNet3D.temporal_block”def temporal_block(name, **scope={})UNetStage
Section titled “UNetStage”class UNetStage( features: int, heads: int, depth: int = 1, cross_attention: bool = True, cross_only: bool = False,)Describes one resolution level: its width, heads and transformer depth.
With cross_attention, the level has a spatial attention block, and
without it the level has none. cross_only drops the self-attention
inside that block.
class UViT(_TransformerOptions)Denoises patches as U-ViT does, following baofff/U-ViT’s libs/uvit_t2i.py.
U-ViT is from Bao et al. (2023). A time token and, when there is text, one token per text state come before the patches: [time, text…, patches].
- The time token is the sinusoidal embedding [cos, sin] of the time
multiplied by
time_scale, passed through a d -> 4d -> d SiLU MLP whenmlp_time_embedis set. U-ViT reads timesteps on a 0 to 999 scale, while Dew’s processes give a time in [0, 1]. - A dense layer projects the text states, and a
patch_sizeconvolution embeds the patches. - A learned position table covers every token: one row for the time,
text_tokensrows for the text, and rows for the patches of animage_sizeimage, each part using its rows from the first.
Every block is a plain pre-norm transformer block, x + attention(LN(x))
then x + MLP(LN(x)), with q, k and v projections without bias, an output
projection with bias, and an MLP mlp_ratio wide on exact GELU.
num_layers / 2 blocks go down, one sits in the middle and num_layers / 2
go up; each up block first projects [x, skip] back to the width. A final
LN and dense layer read the patches, which are unpatchified and, with
conv, refined by a 3x3 convolution. The text attends unmasked, padding
included, as U-ViT’s CLIP states do. A Hilbert scan_order gathers the
embedded patches along the curve (dew.nn.dit.scan_ordered), and the
position rows follow that order.
UViT.projections
Section titled “UViT.projections”def projections( input_channels: int = 0, text_features: int = 0,) -> dict[str, tuple[int, int]]UViT.model_flops
Section titled “UViT.model_flops”def model_flops(lengths) -> floatUViT.image_lengths
Section titled “UViT.image_lengths”def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarrayUViT.multimodal_flops
Section titled “UViT.multimodal_flops”def multimodal_flops( lengths, context_lengths, input_channels: int, text_features: int,) -> floatThe time token, U-shaped skips, all input maps and final image convolution.
UViT.image_flops
Section titled “UViT.image_flops”def image_flops(lengths, input_channels: int, text_features: int | None = None) -> floatUViT.setup
Section titled “UViT.setup”def setup()class Unet( output_channels: int = 3, emb_features: int = 64 * 4, feature_depths: Sequence[int] = (64, 128, 256, 512), attention_configs: Sequence[Stage | None] = (Stage(heads=8), Stage(heads=8), Stage(heads=8), Stage(heads=8)), num_res_blocks: int = 2, num_middle_res_blocks: int = 1, activation: Callable = jax.nn.swish, norm_groups: int = 8, dtype: Dtype | None = None, precision: PrecisionLike = None, attention_impl: AttentionImpl = 'auto',)Denoises images with a convolutional UNet of residual blocks and cross-attention stages.
Without text, the attention stages self-attend (TransformerBlock’s
context defaults to its input). Cross-attention reads the whole text
sequence, so the mask that the DiT family pools with is not used here.
It is FlaxDiff’s Unet except for one width. The decoder upsamples level
i into feature_depths[-i - 2], the width of the level it enters, where
FlaxDiff’s upsamples into feature_depths[-i]. The two agree for two
levels or for equal depths.
attention_configs: Sequence[Stage | None]-
The attention of each resolution stage, one entry per feature depth. None is a stage with no attention.
Unet.condition_lengths
Section titled “Unet.condition_lengths”def condition_lengths(lengths: np.ndarray, spans: np.ndarray) -> np.ndarrayUnet.model_flops
Section titled “Unet.model_flops”def model_flops(lengths) -> floatUnet.spatial_flops
Section titled “Unet.spatial_flops”def spatial_flops( sample_shape: tuple[int, ...], rows: int, context_lengths=None, text_features: int | None = None,) -> floatCount the same residual/attention/skip walk the forward executes.
Unet.setup
Section titled “Unet.setup”def setup()Unet.time_embedding
Section titled “Unet.time_embedding”def time_embedding(temb)VideoDiT
Section titled “VideoDiT”class VideoDiT(_DiTStackOptions)Factorized spatial-temporal DiT over (B, T, H, W, C) inputs.
VideoDiT.model_flops
Section titled “VideoDiT.model_flops”def model_flops(lengths) -> floatVideoDiT.video_lengths
Section titled “VideoDiT.video_lengths”def video_lengths( sample_shape: tuple[int, ...], rows: int,) -> tuple[np.ndarray, np.ndarray]VideoDiT.spatiotemporal_flops
Section titled “VideoDiT.spatiotemporal_flops”def spatiotemporal_flops( frames, patches, input_channels: int, text_features: int | None = None,) -> floatFactorized squares and the modulation copies computed for each block row.
VideoDiT.setup
Section titled “VideoDiT.setup”def setup()