Skip to content

dew.nn.backbones.flux2

FLUX.2’s transformer, as Diffusers 0.40.0’s Flux2Transformer2DModel runs it.

Like Flux, it runs double-stream blocks, where the image and the text meet only in attention, and then single-stream blocks. It differs from Flux in these ways:

  • No projection has a bias, and the feed-forwards are SwiGLU.
  • The time (and guidance) embedding gives one modulation per call: one set for each of the double blocks’ two streams, and one for the single blocks.
  • A single block computes its qkv and its feed-forward input with one map, and maps its attention output and gated state back with one map.
  • The rotary table has four axes (time, row, column, token) of 32 channels at theta 2000. A text token sits at (0, 0, 0, i) and an image position at (0, row, column, 0).
  • The text is the Mistral-3 encoder’s stacked hidden states, with no pooled vector.

The latent is the pipeline’s 32 VAE channels folded 2x2 into 128, which dew.nn.autoencoders.flux2 produces.

NameSummary
Flux2SingleBlockRuns one Flux2SingleTransformerBlock over the joined sequence.
Flux2TransformerRuns Diffusers 0.40.0’s Flux2Transformer2DModel behind Dew’s model interface.
flux2_positionsReturn the position ids Flux2Pipeline lays out, text first.

Flax module source

class Flux2SingleBlock(
epsilon: float = 1e-06,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
attention_impl: AttentionImpl = 'auto',
)

Runs one Flux2SingleTransformerBlock over the joined sequence.

One map gives the queries, keys, values and SwiGLU input. Per-head RMS norms and the rotary apply to the queries and keys. One map back takes the attention output together with the gated hidden state.

def projections() -> dict[str, tuple[int, int]]
def model_flops(lengths) -> float

The fused QKV/gate/up map and joined output; modulation is shared.

Flax module source

class Flux2Transformer(
in_channels: int = 128,
out_channels: int = 128,
num_layers: int = 8,
num_single_layers: int = 48,
heads: int = 48,
head_dim: int = 128,
joint_attention_dim: int = 15360,
timestep_guidance_channels: int = 256,
mlp_ratio: float = 3.0,
axes_dims_rope: Sequence[int] = (32, 32, 32, 32),
rope_theta: float = 2000.0,
eps: float = 1e-06,
guidance_embeds: bool = True,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
attention_impl: AttentionImpl = 'auto',
)

Runs Diffusers 0.40.0’s Flux2Transformer2DModel behind Dew’s model interface.

__call__ takes the pipeline’s NHWC latent (the VAE’s channels folded 2x2), the model time the schedule supplies, and a DenoisingCondition. The model time is the sigma times the training count; the source reaches the same product by dividing its timestep by a thousand and multiplying it back. The condition’s context is the stacked encoder states, and its guidance is the distilled guidance that a guidance-embedded checkpoint reads.

def image_lengths(sample_shape: tuple[int, ...], rows: int) -> np.ndarray
def multisequence_flops(
lengths,
context_lengths,
*,
input_channels: int,
text_features: int,
pooled_features: int | None = None,
) -> float

Actual input widths and one shared modulation, not one per block.

function source

def flux2_positions(rows: int, columns: int, text: int) -> np.ndarray

Return the position ids Flux2Pipeline lays out, text first.

Text token i sits at (0, 0, 0, i), followed by the latent grid in row-major order at (0, row, column, 0).