dew.nn.backbones.flux2
FLUX.2’s transformer, as Diffusers 0.40.0’s Flux2Transformer2DModel runs it.
Like Flux, it runs double-stream blocks, where the image and the text meet only in attention, and then single-stream blocks. It differs from Flux in these ways:
- No projection has a bias, and the feed-forwards are SwiGLU.
- The time (and guidance) embedding gives one modulation per call: one set for each of the double blocks’ two streams, and one for the single blocks.
- A single block computes its qkv and its feed-forward input with one map, and maps its attention output and gated state back with one map.
- The rotary table has four axes (time, row, column, token) of 32 channels at theta 2000. A text token sits at (0, 0, 0, i) and an image position at (0, row, column, 0).
- The text is the Mistral-3 encoder’s stacked hidden states, with no pooled vector.
The latent is the pipeline’s 32 VAE channels folded 2x2 into 128, which
dew.nn.autoencoders.flux2 produces.
| Name | Summary |
|---|---|
Flux2SingleBlock | Runs one Flux2SingleTransformerBlock over the joined sequence. |
Flux2Transformer | Runs Diffusers 0.40.0’s Flux2Transformer2DModel behind Dew’s model interface. |
flux2_positions | Return the position ids Flux2Pipeline lays out, text first. |
Flux2SingleBlock
Section titled “Flux2SingleBlock”class Flux2SingleBlock( epsilon: float = 1e-06, dtype: Dtype | None = None, precision: PrecisionLike = None, attention_impl: AttentionImpl = 'auto',)Runs one Flux2SingleTransformerBlock over the joined sequence.
One map gives the queries, keys, values and SwiGLU input. Per-head RMS norms and the rotary apply to the queries and keys. One map back takes the attention output together with the gated hidden state.
Flux2SingleBlock.projections
Section titled “Flux2SingleBlock.projections”def projections() -> dict[str, tuple[int, int]]Flux2SingleBlock.model_flops
Section titled “Flux2SingleBlock.model_flops”def model_flops(lengths) -> floatThe fused QKV/gate/up map and joined output; modulation is shared.
Flux2Transformer
Section titled “Flux2Transformer”class Flux2Transformer( in_channels: int = 128, out_channels: int = 128, num_layers: int = 8, num_single_layers: int = 48, heads: int = 48, head_dim: int = 128, joint_attention_dim: int = 15360, timestep_guidance_channels: int = 256, mlp_ratio: float = 3.0, axes_dims_rope: Sequence[int] = (32, 32, 32, 32), rope_theta: float = 2000.0, eps: float = 1e-06, guidance_embeds: bool = True, dtype: Dtype | None = None, precision: PrecisionLike = None, attention_impl: AttentionImpl = 'auto',)Runs Diffusers 0.40.0’s Flux2Transformer2DModel behind Dew’s model interface.
__call__ takes the pipeline’s NHWC latent (the VAE’s channels folded
2x2), the model time the schedule supplies, and a DenoisingCondition.
The model time is the sigma times the training count; the source reaches
the same product by dividing its timestep by a thousand and multiplying
it back. The condition’s context is the stacked encoder states, and its
guidance is the distilled guidance that a guidance-embedded checkpoint
reads.
Flux2Transformer.image_lengths
Section titled “Flux2Transformer.image_lengths”def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarrayFlux2Transformer.multisequence_flops
Section titled “Flux2Transformer.multisequence_flops”def multisequence_flops( lengths, context_lengths, *, input_channels: int, text_features: int, pooled_features: int | None = None,) -> floatActual input widths and one shared modulation, not one per block.
flux2_positions
Section titled “flux2_positions”def flux2_positions(rows: int, columns: int, text: int) -> np.ndarrayReturn the position ids Flux2Pipeline lays out, text first.
Text token i sits at (0, 0, 0, i), followed by the latent grid in row-major order at (0, row, column, 0).