Skip to content

dew.nn.backbones.joint

The blocks shared by the published MM-DiT family: SD3, Flux, FLUX.2, Z-Image and Qwen-Image.

Each of these models is a stack of modulated residual blocks whose attention joins an image stream and a text stream, as Diffusers 0.34.0 to 0.40.0 run them. This module holds the shared parts: the adaLN modulation, the layer norm without affine parameters, one joint attention, the double-stream block that SD3, Flux and FLUX.2 run, the feed-forwards and the embedders. Each family’s own module keeps what only that family does.

NameSummary
DoubleStreamBlockRuns one double-stream block of SD3, Flux, FLUX.2 or Dew’s own MM-DiT.
JointAttentionRuns the source’s Attention over an image stream and an optional context.
embeddingApply TimestepEmbedding or PixArtAlphaTextProjection to x.
guided_timeReturn the time’s embed, plus the guidance’s when the checkpoint embeds its distilled guidance.
layer_normReturn the source’s LayerNorm(elementwise_affine=False).
modulateReturn the source’s norm(x) * (1 + scale) + shift for a normed x, broadcast over tokens.

Flax module source

class DoubleStreamBlock(
context_first: bool = False,
qk_norm: bool = True,
bias: bool = True,
epsilon: float = 1e-06,
qk_epsilon: float | None = None,
mlp_hidden: int | None = None,
swiglu: bool = False,
shared_modulation: bool = False,
zero_modulation: bool = False,
context_pre_only: bool = False,
dual_attention: bool = False,
rotary_pairs: Literal['half', 'adjacent'] = 'adjacent',
dropout_rate: float = 0.0,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
attention_impl: AttentionImpl = 'auto',
force_fp32_for_softmax: bool = True,
)

Runs one double-stream block of SD3, Flux, FLUX.2 or Dew’s own MM-DiT.

It matches SD3’s JointTransformerBlock, Flux’s FluxTransformerBlock and FLUX.2’s Flux2TransformerBlock. The image and the context each modulate and norm their own residual stream. They meet in one JointAttention, and each adds its gated attention output back to its stream. Then each runs its own modulated feed-forward and adds the gated output. A block’s modulation is its own projection of the conditioning vector (norm1, norm1_context): six pieces per stream, a shift, scale and gate before the attention and again before the feed-forward. dropout_rate drops each projected attention and feed-forward output while train, before its gate.

context_first: bool

Whether the context comes first in the joined attention, as in Flux and FLUX.2.

bias: bool

Whether the projections have biases. FLUX.2’s have none.

qk_epsilon: float | None

The query and key norms’ epsilon, None for epsilon. Dew’s MM-DiT norms its streams at its stack’s epsilon and its queries and keys at 1e-6.

mlp_hidden: int | None

The feed-forward’s hidden width, four times the width for None.

swiglu: bool

Whether the feed-forward is FLUX.2’s SwiGLU rather than the tanh GELU.

shared_modulation: bool

Whether the conditioning is the model’s shared modulation, as in FLUX.2: an (image, context) pair of six pieces each, which every block reads.

zero_modulation: bool

Whether each modulation starts at zero (Modulation.zero_init).

context_pre_only: bool

Whether this is SD3’s last block, which reads the context and returns it unchanged. Its context norm then takes only a scale and a shift, and it has no context output projection or feed-forward.

dual_attention: bool

Whether the block runs SD3.5’s second self-attention over the same normalized input, modulated by three more image pieces.

rotary_pairs: Literal['half', 'adjacent']

The channels each angle turns (JointAttention.rotary_pairs).

def model_flops(lengths) -> float
def multisequence_flops(lengths, context_lengths) -> float

The declared attention and MLPs, plus each stream’s per-row modulation.

Flax module source

class JointAttention(
bias: bool = True,
qk_norm: bool = True,
epsilon: float = 1e-06,
context_first: bool = False,
project: bool = True,
context_out: bool = True,
rotary_pairs: Literal['half', 'adjacent'] = 'adjacent',
dtype: Dtype | None = None,
precision: PrecisionLike = None,
attention_impl: AttentionImpl = 'auto',
force_fp32_for_softmax: bool = True,
)

Runs the source’s Attention over an image stream and an optional context.

Each stream projects its own queries, keys and values; the context uses the add_* projections. Per-head RMS norms normalize the queries and keys, the rotation is applied to them, and one softmax attends over the joined sequence. The result is split back into the two streams, and each goes through its own output projection. SD3’s JointAttnProcessor2_0 puts the image first in the joined sequence, and Flux’s processor puts the context first. Without a context, it is self-attention over the image. The rotation is the cosines and sines of every joined token, one angle per channel pair, [B or 1, S, D // 2], or of the image’s tokens alone, which turn before the join and leave the context as it is (Dew’s MM-DiT); lengths is the number of real keys in each row.

bias: bool

Whether every projection has a bias. FLUX.2’s, Z-Image’s and Qwen-Image’s have none.

qk_norm: bool

Whether the queries and keys are RMS-normed. SD3 without a qk_norm config leaves them as they are.

context_first: bool

Whether the context’s tokens come first in the joined sequence, as in Flux.

project: bool

Whether the attention applies its own output projection. Flux’s single-stream block projects the attention output itself, together with its feed-forward, so it takes the output unprojected.

context_out: bool

Whether the context’s output is projected and returned. SD3’s last block reads no context back, so it has no to_add_out.

rotary_pairs: Literal['half', 'adjacent']

The channels each angle turns (dew.nn.rope.rotate): the published families’ adjacent pairs, or the rotate-half halves of Dew’s own MM-DiT.

def model_flops(lengths) -> float
def stream_flops(
lengths,
context_lengths,
image_features: int,
context_features: int | None = None,
) -> float

Both streams’ Q/K/V maps and joined QK/AV, with optional output maps.

def attend(query, key, value, lengths)

Return one softmax attention over the joined sequence, masking each row’s keys past lengths.

function source

def embedding(
owner: nn.Module,
x,
features: int,
name: str,
*,
bias: bool = True,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
activation: Callable[[jax.Array], jax.Array] = nn.silu,
sharded: bool = True,
)

Apply TimestepEmbedding or PixArtAlphaTextProjection to x.

Both are a linear layer, an activation and a second linear layer. The activation is SiLU, or Wan’s tanh GELU. The two layers are stored flat as {name}_linear_1 and {name}_linear_2 in the calling module, each sharding its output, the model’s width, where sharded; a model that names no axes of them (the UNet’s) leaves them to the shape heuristic.

function source

def guided_time(
embed: Callable[[jax.Array, str], jax.Array],
time,
guidance,
*,
guidance_embeds: bool,
)

Return the time’s embed, plus the guidance’s when the checkpoint embeds its distilled guidance.

The guidance is multiplied by a thousand before embedding, as the source does. A guidance-embedded checkpoint requires guidance, and any other checkpoint raises ValueError when guidance is given.

function source

def layer_norm(dtype, epsilon: float = 1e-06)

Return the source’s LayerNorm(elementwise_affine=False).

Its eps is 1e-6, the value SD3 and Flux fix, unless a config gives another.

function source

def modulate(x, shift, scale)

Return the source’s norm(x) * (1 + scale) + shift for a normed x, broadcast over tokens.