Skip to content

dew.nn.backbones

Model backbones.

You build a model by calling its class, as in SimpleDiT(patch_size=4, ...). A run record names the class by its import path, and dew.registry.models holds short aliases such as simple_dit, so either and its fields rebuild the same model.

Each backbone is imported only when it is used (dew._exports), so a module that reads one backbone imports that backbone alone.

NameSummary
CausalTransformerDecoder-only transformer over token ids: [B, S] int32 -> [B, S, vocab] fp32. Documented in dew.nn.backbones.causal_transformer.
EDM2UNetRuns EDM2’s U-Net. Documented in dew.nn.backbones.edm2.
Flux2TransformerRuns Diffusers 0.40.0’s Flux2Transformer2DModel behind Dew’s model interface. Documented in dew.nn.backbones.flux2.
FluxTransformerRuns Diffusers 0.34.0’s FluxTransformer2DModel behind Dew’s model interface. Documented in dew.nn.backbones.flux.
HierarchicalMMDiTU-shaped MM-DiT: dual-stream blocks per stage with patch merging on the way down and expansion + skip fusion on the way up.
HybridSSMAttentionDiTA DiT that interleaves SSM blocks with attention blocks.
MixtureThe experts that some layers route to, and how the router chooses among them. Documented in dew.nn.backbones.decoder_block.
QwenImageTransformerRuns QwenImage21Transformer2DModel behind Dew’s model interface. Documented in dew.nn.backbones.qwen_image.
SD3TransformerRuns Diffusers 0.34.0’s SD3Transformer2DModel behind Dew’s model interface. Documented in dew.nn.backbones.sd3.
SimpleDiTThe standard DiT: a plain stack of adaLN-Zero attention blocks.
SimpleMMDiTSD3-style MM-DiT: a plain stack of dual-stream blocks, the image tokens rotated by their raster index (rope_for_scan).
SimpleUDiTA U-shaped DiT: SimpleDiT’s adaLN-Zero blocks with the first half’s outputs skipping into the second half through a dense layer over the concatenation.
UNet2DConditionDenoises NHWC latents conditioned on text, pooled and size features, and optional inpainting inputs.
UNet3DDenoises video (B, T, H, W, C) with the 2D Unet body per frame and a TemporalBlock per level.
UNetStageDescribes one resolution level: its width, heads and transformer depth.
UViTDenoises patches as U-ViT does, following baofff/U-ViT’s libs/uvit_t2i.py.
UnetDenoises images with a convolutional UNet of residual blocks and cross-attention stages.
VideoDiTFactorized spatial-temporal DiT over (B, T, H, W, C) inputs.
WanTransformerRuns Diffusers 0.34.0’s WanTransformer3DModel for text-to-video, behind Dew’s model interface. Documented in dew.nn.backbones.wan.
ZImageTransformerRuns Diffusers 0.40.0’s ZImageTransformer2DModel behind Dew’s model interface. Documented in dew.nn.backbones.z_image.

class source

class HierarchicalMMDiT(RematLadder, _AttentionStackOptions)

U-shaped MM-DiT: dual-stream blocks per stage with patch merging on the way down and expansion + skip fusion on the way up.

Raster order only: the merge/expand grid reshapes assume row-major token order, so a hilbert scan would scramble the neighborhoods being merged.

text_keyword: str

Every call takes the text as textcontext, which runs as a second stream through every block.

def projections(text_features: int = 0) -> dict[str, tuple[int, int]]
def model_flops(lengths) -> float
def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarray
def multimodal_flops(
lengths,
context_lengths,
input_channels: int,
text_features: int,
) -> float

Per-stage streams, patch merges/expansions, skip maps and output.

def stage_blocks(stage: int, prefix: str) -> list

Build one stage’s dual-stream blocks, at that stage’s width and heads.

def encoder_path(num_stages: int)

Build the encoder, fine to coarse: each stage’s blocks and its merger.

Sets encoder_blocks, one list per stage, and patch_mergers, one between each pair of stages.

def decoder_path(num_stages: int)

Build the decoder, coarse to fine, for stages N-2 down to 0.

Sets patch_expanders, the fusion_layers that take an expanded stage beside its skip, and decoder_blocks. All three are indexed by decoder step, not by stage.

def setup()

Build the patch embedding, the two paths of stages, and the head.

cond_projs and txt_embeds carry the conditioning and the text into each stage’s own width. encoder_path and decoder_path set the blocks and the resampling layers between them.

class source

class HybridSSMAttentionDiT(SimpleDiT)

A DiT that interleaves SSM blocks with attention blocks.

Each layer’s mixer comes from ssm_attention_ratio, a shorthand that reads the same at any depth (“3:1”, “all-ssm”), or from block_pattern, which names each layer. Setting both raises a ValueError at setup. Everything around the layers is SimpleDiT’s.

def layer(index: int, kind: str, *, counted: bool = False) -> ModulatedBlock

Build layer index’s block, with an SSM mixer or an attention mixer; counted, unbound and without remat, for work accounting.

The two share the width, the MLP ratio and the norms. They differ in the mixer they name, the fields only that mixer reads, the remat policy, and the name a checkpoint stores them under.

def stack(*, counted: bool = False) -> list[ModulatedBlock]

Return the layers’ blocks, each the block that ssm_attention_ratio or block_pattern names; counted, unbound and without remat (counted_blocks).

Raises ValueError when both are set, or when TREAD’s routes are combined with use_2d_fusion.

def kinds() -> list[str]

class source

class SimpleDiT(_DiTStackOptions)

The standard DiT: a plain stack of adaLN-Zero attention blocks.

  • adaln_silu=False and text_pooling="all" give FlaxDiff 0.2’s conditioning, as in SimpleUDiT.
  • routes is TREAD’s token routing (Krause et al. 2025) during training. Each (ratio, start, end) draws int(tokens * ratio) tokens per example, from the dropout stream, that skip blocks start to end and rejoin after end with the values they had on entry, as CompVis/tread’s Router does; the kept tokens stay in order at their own positions. Sampling runs every token through every block.
  • patch_bottleneck is JiT’s bottleneck patch embedding (Li and He 2025), 128 in its models.
  • interval adds the duration of the predicted interval next to the time, for MeanFlow and shortcut models (Process.interval).
  • time_scale scales the Fourier frequencies; keep it small for a model trained through a JVP in time.
block_prefixes: tuple[str, ...]

The names the stack’s blocks are stored under, {prefix}N, one prefix a mixer.

def counted_blocks() -> list[ModulatedBlock]

The layers stack builds, unbound and without lifted remat, for host-side work accounting; a configuration stack refuses is refused here too.

def model_flops(lengths) -> float

Input channels and conditioning width belong to the objective’s geometry.

def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarray

Patch-token lengths from the actual latent/image shape, never a guessed channel count.

def image_flops(lengths, input_channels: int, text_features: int | None = None) -> float

Every matmul, including patch/time/text input maps; only lookups excluded.

def setup()
def stack(*, counted: bool = False) -> list[ModulatedBlock]

The layers between the patch embedding and the output, all attention; counted, unbound and without remat (counted_blocks).

def kinds() -> list[str]

Each layer’s mixer, in order.

def stage_sites(store: Variables, stages: int) -> tuple[DecoderBank, ...]

StagedStacks: the blocks, where stages split them (pipelined).

def layer(index: int, kind: str, *, counted: bool = False) -> ModulatedBlock

Layer index’s block, of mixer kind, stored as dit_block_{index}; counted, unbound and without remat.

def pipelined(tokens, conditioning, freqs_cis, train: bool)

The blocks over tokens as a pipeline over the stage axis (pipelined_blocks), each row modulated by its own conditioning, every one rotated alike.

def checked_routes() -> list[tuple[float, int, int]]

Return routes as (ratio, start, end) tuples, after checking them.

Each ratio must be in (0, 1), and the spans must be ordered, disjoint and inside the stack.

def kept_tokens(tokens: jax.Array, ratio: float, start: int) -> jax.Array

Return the [B, keep] indices of the tokens a route at start computes on.

As in the reference, these are the first keep of a uniform shuffle, in sequence order.

class source

class SimpleMMDiT(_DiTStackOptions)

SD3-style MM-DiT: a plain stack of dual-stream blocks, the image tokens rotated by their raster index (rope_for_scan).

text_keyword: str

Every call takes the text as textcontext, which runs as a second stream through every block.

def model_flops(lengths) -> float
def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarray
def multimodal_flops(
lengths,
context_lengths,
input_channels: int,
text_features: int,
) -> float

All input/output maps, pooled/time conditioning and joint blocks.

def projections(text_features: int = 0) -> dict[str, tuple[int, int]]
def setup()

class source

class SimpleUDiT(RematLadder, _TransformerOptions)

A U-shaped DiT: SimpleDiT’s adaLN-Zero blocks with the first half’s outputs skipping into the second half through a dense layer over the concatenation. Position comes from RoPE over the sequence index, so a hilbert scan carries the rotation of its curve index and no 2D signal.

adaln_silu False and text_pooling “all” are FlaxDiff 0.2’s U-DiT: its blocks project the conditioning vector without a SiLU and it averages the text over every position. dew.interop.flaxdiff loads its checkpoints with them.

def projections(input_channels: int = 0) -> dict[str, tuple[int, int]]
def model_flops(lengths) -> float
def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarray
def image_flops(lengths, input_channels: int, text_features: int | None = None) -> float

Patch/time/pooled maps, modulated blocks and the up-path skip maps.

def setup()

Flax module source

class UNet2DCondition(
in_channels: int = 4,
out_channels: int = 4,
blocks_per_level: int = 2,
linear_projection: bool = False,
additional_time_features: int = 0,
middle_attention: bool = True,
frequency_shift: float = 0,
cosine_first: bool = True,
dropout: float = 0.0,
dtype: Dtype = jnp.float32,
precision: PrecisionLike = None,
attention_impl: AttentionImpl = 'auto',
norm_groups: int = 32,
norm_epsilon: float = 1e-05,
attention_norm_epsilon: float = 1e-05,
approximate_gelu: bool = True,
)

Denoises NHWC latents conditioned on text, pooled and size features, and optional inpainting inputs.

The stages set each level’s spatial width, attention heads and transformer depth. The encoder saves each residual output and each downsample, and the decoder uses those skips in reverse order, with one extra residual block per level.

text_keyword: str

Every call takes the text as conditioning, which every attention stage cross-attends to.

def model_flops(lengths) -> float
def condition_lengths(lengths: np.ndarray, spans: np.ndarray) -> np.ndarray
def conditioned_spatial_flops(
sample_shape: tuple[int, ...],
rows: int,
condition: ConditionWork,
) -> float
def spatial_flops(
sample_shape: tuple[int, ...],
rows: int,
context_lengths=None,
text_features: int | None = None,
*,
additional_features: int | None = None,
) -> float

The residual/attention stage walk, including every resizer and skip width.

def stage_sites(store: Variables, stages: int) -> tuple[DecoderBank, ...]

StagedStacks: each level’s transformers, where stages split them, found by walking the levels’ shapes (spatial_walk) as the FLOP count does.

def spatial_walk(x, time, context, *, train=False, operations)

class source

class UNet3D(Unet)

Denoises video (B, T, H, W, C) with the 2D Unet body per frame and a TemporalBlock per level.

There is a TemporalBlock at every resolution level. The spatial parameter paths are identical to Unet’s, so 2D checkpoints inflate directly.

def spatial_flops(
sample_shape: tuple[int, ...],
rows: int,
context_lengths=None,
text_features: int | None = None,
) -> float

The frame body and its temporal hook share the forward’s ordered walk.

def temporal_block(name, **scope={})

dataclass source

class UNetStage(
features: int,
heads: int,
depth: int = 1,
cross_attention: bool = True,
cross_only: bool = False,
)

Describes one resolution level: its width, heads and transformer depth.

With cross_attention, the level has a spatial attention block, and without it the level has none. cross_only drops the self-attention inside that block.

class source

class UViT(_TransformerOptions)

Denoises patches as U-ViT does, following baofff/U-ViT’s libs/uvit_t2i.py.

U-ViT is from Bao et al. (2023). A time token and, when there is text, one token per text state come before the patches: [time, text…, patches].

  • The time token is the sinusoidal embedding [cos, sin] of the time multiplied by time_scale, passed through a d -> 4d -> d SiLU MLP when mlp_time_embed is set. U-ViT reads timesteps on a 0 to 999 scale, while Dew’s processes give a time in [0, 1].
  • A dense layer projects the text states, and a patch_size convolution embeds the patches.
  • A learned position table covers every token: one row for the time, text_tokens rows for the text, and rows for the patches of an image_size image, each part using its rows from the first.

Every block is a plain pre-norm transformer block, x + attention(LN(x)) then x + MLP(LN(x)), with q, k and v projections without bias, an output projection with bias, and an MLP mlp_ratio wide on exact GELU. num_layers / 2 blocks go down, one sits in the middle and num_layers / 2 go up; each up block first projects [x, skip] back to the width. A final LN and dense layer read the patches, which are unpatchified and, with conv, refined by a 3x3 convolution. The text attends unmasked, padding included, as U-ViT’s CLIP states do. A Hilbert scan_order gathers the embedded patches along the curve (dew.nn.dit.scan_ordered), and the position rows follow that order.

def projections(
input_channels: int = 0,
text_features: int = 0,
) -> dict[str, tuple[int, int]]
def model_flops(lengths) -> float
def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarray
def multimodal_flops(
lengths,
context_lengths,
input_channels: int,
text_features: int,
) -> float

The time token, U-shaped skips, all input maps and final image convolution.

def image_flops(lengths, input_channels: int, text_features: int | None = None) -> float
def setup()

Flax module source

class Unet(
output_channels: int = 3,
emb_features: int = 64 * 4,
feature_depths: Sequence[int] = (64, 128, 256, 512),
attention_configs: Sequence[Stage | None] = (Stage(heads=8), Stage(heads=8), Stage(heads=8), Stage(heads=8)),
num_res_blocks: int = 2,
num_middle_res_blocks: int = 1,
activation: Callable = jax.nn.swish,
norm_groups: int = 8,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
attention_impl: AttentionImpl = 'auto',
)

Denoises images with a convolutional UNet of residual blocks and cross-attention stages.

Without text, the attention stages self-attend (TransformerBlock’s context defaults to its input). Cross-attention reads the whole text sequence, so the mask that the DiT family pools with is not used here.

It is FlaxDiff’s Unet except for one width. The decoder upsamples level i into feature_depths[-i - 2], the width of the level it enters, where FlaxDiff’s upsamples into feature_depths[-i]. The two agree for two levels or for equal depths.

attention_configs: Sequence[Stage | None]

The attention of each resolution stage, one entry per feature depth. None is a stage with no attention.

def condition_lengths(lengths: np.ndarray, spans: np.ndarray) -> np.ndarray
def model_flops(lengths) -> float
def spatial_flops(
sample_shape: tuple[int, ...],
rows: int,
context_lengths=None,
text_features: int | None = None,
) -> float

Count the same residual/attention/skip walk the forward executes.

def setup()
def time_embedding(temb)

class source

class VideoDiT(_DiTStackOptions)

Factorized spatial-temporal DiT over (B, T, H, W, C) inputs.

def model_flops(lengths) -> float
def video_lengths(
sample_shape: tuple[int, ...],
rows: int,
) -> tuple[np.ndarray, np.ndarray]
def spatiotemporal_flops(
frames,
patches,
input_channels: int,
text_features: int | None = None,
) -> float

Factorized squares and the modulation copies computed for each block row.

def setup()