Skip to content

dew.nn.backbones.wan

Wan 2.1’s video transformer, as Diffusers 0.34.0’s WanTransformer3DModel runs it for text-to-video.

A strided 3-D convolution cuts the clip’s latent into 1x2x2 patches, one token each, in frame, row, column order. Every block normalizes its residual stream and modulates it by the time, self-attends with a three-axis rotary table over (frame, row, column), cross-attends to the text, and runs the tanh-GELU feed-forward. A block’s six modulation pieces are its own learned table plus one projection of the time embedding that all blocks share. Queries and keys are RMS-normalized across all heads, before they are split into heads.

The source keeps its norms, its modulation tables and its time embedder in fp32 (_keep_in_fp32_modules) and adds the gated residuals in fp32. A bfloat16 model here does the same at at_least_fp32 of its dtype, and rounds the stream back after each sum. The source rotates queries and keys in float64; Dew rotates them at at_least_fp32 of their dtype, by cosines and sines computed in float64 (dew.nn.rope.axis_tables) and rounded to float32.

NameSummary
WanBlockRuns WanTransformerBlock: modulated self-attention, cross-attention to the text, and a feed-forward.
WanTransformerRuns Diffusers 0.34.0’s WanTransformer3DModel for text-to-video, behind Dew’s model interface.

Flax module source

class WanBlock(
qk_norm: bool = True,
cross_attn_norm: bool = True,
epsilon: float = 1e-06,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
attention_impl: AttentionImpl = 'auto',
)

Runs WanTransformerBlock: modulated self-attention, cross-attention to the text, and a feed-forward.

The cross-attention reads the text through its own affine norm (cross_attn_norm), and the feed-forward is modulated like the self-attention. modulation is the model’s shared projection of the time, [B, 6, features], and the block adds its scale_shift_table to it.

def model_flops(lengths) -> float
def multisequence_flops(lengths, context_lengths) -> float

Self-attention, cross-attention maps/products and the ungated FFN.

Flax module source

class WanTransformer(
patch_size: Sequence[int] = (1, 2, 2),
num_attention_heads: int = 40,
attention_head_dim: int = 128,
in_channels: int = 16,
out_channels: int = 16,
text_dim: int = 4096,
freq_dim: int = 256,
ffn_dim: int = 13824,
num_layers: int = 40,
cross_attn_norm: bool = True,
qk_norm: str | None = 'rms_norm_across_heads',
eps: float = 1e-06,
rope_max_seq_len: int = 1024,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
attention_impl: AttentionImpl = 'auto',
)

Runs Diffusers 0.34.0’s WanTransformer3DModel for text-to-video, behind Dew’s model interface.

__call__ takes a clip’s latent [B, F, H, W, C], the model time the schedule supplies, and a DenoisingCondition, and returns the flow [B, F, H, W, out_channels]. The model time is the sigma times the training count, which is the timestep the source’s pipeline passes. The condition’s context is the UMT5 states [B, L, text_dim], and the cross-attention reads every one of them, as the source’s does. The fields are the config’s; qk_norm takes the value the source names, rms_norm_across_heads, or None.

text_keyword: str

Every call takes the text as conditioning, which every block cross-attends to.

def condition_lengths(lengths: np.ndarray, spans: np.ndarray) -> np.ndarray
def model_flops(lengths) -> float
def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarray
def multisequence_flops(
lengths,
context_lengths,
*,
input_channels: int,
text_features: int,
pooled_features: int | None = None,
) -> float

Every video/text/time projection and the shared sixfold time modulation.