dew.nn.backbones.wan
Wan 2.1’s video transformer, as Diffusers 0.34.0’s WanTransformer3DModel runs it for text-to-video.
A strided 3-D convolution cuts the clip’s latent into 1x2x2 patches, one token each, in frame, row, column order. Every block normalizes its residual stream and modulates it by the time, self-attends with a three-axis rotary table over (frame, row, column), cross-attends to the text, and runs the tanh-GELU feed-forward. A block’s six modulation pieces are its own learned table plus one projection of the time embedding that all blocks share. Queries and keys are RMS-normalized across all heads, before they are split into heads.
The source keeps its norms, its modulation tables and its time embedder in
fp32 (_keep_in_fp32_modules) and adds the gated residuals in fp32. A
bfloat16 model here does the same at at_least_fp32 of its dtype, and
rounds the stream back after each sum. The source rotates queries and keys
in float64; Dew rotates them at at_least_fp32 of their dtype, by
cosines and sines computed in float64 (dew.nn.rope.axis_tables) and
rounded to float32.
| Name | Summary |
|---|---|
WanBlock | Runs WanTransformerBlock: modulated self-attention, cross-attention to the text, and a feed-forward. |
WanTransformer | Runs Diffusers 0.34.0’s WanTransformer3DModel for text-to-video, behind Dew’s model interface. |
WanBlock
Section titled “WanBlock”class WanBlock( qk_norm: bool = True, cross_attn_norm: bool = True, epsilon: float = 1e-06, dtype: Dtype | None = None, precision: PrecisionLike = None, attention_impl: AttentionImpl = 'auto',)Runs WanTransformerBlock: modulated self-attention, cross-attention to the text, and a feed-forward.
The cross-attention reads the text through its own affine norm
(cross_attn_norm), and the feed-forward is modulated like the
self-attention. modulation is the model’s shared projection of the
time, [B, 6, features], and the block adds its scale_shift_table to
it.
WanBlock.model_flops
Section titled “WanBlock.model_flops”def model_flops(lengths) -> floatWanBlock.multisequence_flops
Section titled “WanBlock.multisequence_flops”def multisequence_flops(lengths, context_lengths) -> floatSelf-attention, cross-attention maps/products and the ungated FFN.
WanTransformer
Section titled “WanTransformer”class WanTransformer( patch_size: Sequence[int] = (1, 2, 2), num_attention_heads: int = 40, attention_head_dim: int = 128, in_channels: int = 16, out_channels: int = 16, text_dim: int = 4096, freq_dim: int = 256, ffn_dim: int = 13824, num_layers: int = 40, cross_attn_norm: bool = True, qk_norm: str | None = 'rms_norm_across_heads', eps: float = 1e-06, rope_max_seq_len: int = 1024, dtype: Dtype | None = None, precision: PrecisionLike = None, attention_impl: AttentionImpl = 'auto',)Runs Diffusers 0.34.0’s WanTransformer3DModel for text-to-video, behind Dew’s model interface.
__call__ takes a clip’s latent [B, F, H, W, C], the model time the
schedule supplies, and a DenoisingCondition, and returns the flow
[B, F, H, W, out_channels]. The model time is the sigma times the
training count, which is the timestep the source’s pipeline passes. The
condition’s context is the UMT5 states [B, L, text_dim], and the
cross-attention reads every one of them, as the source’s does. The
fields are the config’s; qk_norm takes the value the source names,
rms_norm_across_heads, or None.
text_keyword: str-
Every call takes the text as
conditioning, which every block cross-attends to.
WanTransformer.condition_lengths
Section titled “WanTransformer.condition_lengths”def condition_lengths(lengths: np.ndarray, spans: np.ndarray) -> np.ndarrayWanTransformer.model_flops
Section titled “WanTransformer.model_flops”def model_flops(lengths) -> floatWanTransformer.image_lengths
Section titled “WanTransformer.image_lengths”def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarrayWanTransformer.multisequence_flops
Section titled “WanTransformer.multisequence_flops”def multisequence_flops( lengths, context_lengths, *, input_channels: int, text_features: int, pooled_features: int | None = None,) -> floatEvery video/text/time projection and the shared sixfold time modulation.