Skip to content

dew.objectives.diffusion.alignment

Representation alignment: REPA and iREPA.

REPA (Yu et al. 2025, “Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think”) adds a term to the denoising loss: the negative cosine similarity between a frozen encoder’s patch features of the clean image and a projection of the denoiser’s hidden tokens at one layer, averaged over tokens and examples. iREPA (Singh et al. 2026, “What matters for Representation Alignment: Global Information or Spatial Structure?”) projects with one convolution over the token grid where REPA uses an MLP, and z-scores the encoder’s features over space. The official code is sihyun-yu/REPA’s loss.py and models/sit.py, and End2End-Diffusion/iREPA’s projectors.py and spatial_zscore.

NameSummary
ALIGNMENTThe projector’s key in params, beside the model’s own modules.
REPRESENTATIONThe variables collection that holds the frozen encoder’s weights, as encoders hold theirs.
AlignmentAligns the model’s hidden tokens at layer with a frozen encoder’s features.
ProjectorProjects hidden tokens to features with REPA’s build_mlp or iREPA’s convolution.
spatial_zscoreReturn iREPA’s spatial normalization of [B, N, D] features.

attribute source

ALIGNMENT = 'alignment_projector'

The projector’s key in params, beside the model’s own modules.

attribute source

REPRESENTATION = 'representation'

The variables collection that holds the frozen encoder’s weights, as encoders hold theirs.

dataclass source

class Alignment(
layer: str = 'dit_block_7',
encoder: nn.Module | None = None,
source: str | None = 'facebook/dinov2-base',
weight: float = 0.5,
projector: Literal['mlp', 'conv'] = 'mlp',
width: int = 2048,
kernel_size: int = 3,
spatial_norm: float | None = None,
resolution: int | None = 224,
mean: tuple[float, float, float] = IMAGENET_MEAN,
std: tuple[float, float, float] = IMAGENET_STD,
)

Aligns the model’s hidden tokens at layer with a frozen encoder’s features.

encoder is a Flax module that maps pixels [B, S, S, 3] to patch features [B, N, D] or [B, h, w, D], any module a record names, as a model is named. Its weights are the starting tree’s REPRESENTATION when it holds them, else source’s: a transformers Dinov2Model checkpoint (repo, repo@revision or a directory), by default REPA’s DINOv2-B/14, which a run’s record pins to a commit. With encoder None the encoder is the source’s own, read at resolution pixels. The pixels are resized to resolution (None keeps the data’s size) and normalized by mean and std. layer names the submodule of the model whose output is aligned (dit_block_7, REPA’s depth 8 on simple_dit). That output must be a [B, N, width] token sequence in raster order over the same grid as the encoder’s patches.

weight is REPA’s proj_coeff. projector="mlp" with width is REPA’s projector. projector="conv" with kernel_size and a spatial_norm gamma (0.6 in iREPA’s training script) is iREPA’s, which z-scores each feature over the tokens after subtracting gamma times its spatial mean. The pixel statistics default to ImageNet’s, which DINOv2 uses; REPA’s DINOv2 reads 224 pixels of a 256 image. The loss adds weight / 2 times the alignment term, because Dew’s L2 halves the denoising error that REPA adds the term to.

def network(held: Variables | None) -> tuple[Alignment, Variables]

This alignment with its encoder, and the encoder’s weights: held, a starting tree’s REPRESENTATION, when given, else source’s.

def targets(variables: Variables, images: jax.Array) -> jax.Array

Return the encoder’s [B, N, D] features of images in [-1, 1], with no gradient.

Under iREPA (spatial_norm set) the features are spatially normalized.

def module(features: int) -> Projector
def init(key, width: int, count: int, features: int) -> Variables

Return the projector’s variables for count hidden tokens of width.

def loss(
projector: Variables,
hidden: jax.Array,
targets: jax.Array,
batch: Batch,
) -> jax.Array

Return REPA’s projection loss, the mean of -cos(target, projection) over tokens and examples.

The examples are batch’s rows (Objective.row_mean).

Raises ValueError when the model’s token count differs from the encoder’s patch count.

def captures(module: nn.Module, method: str) -> bool

Return whether capture_intermediates should keep this call: True for layer’s output.

Flax module source

class Projector(width: int = 2048, kernel_size: int = 3)

Projects hidden tokens to features with REPA’s build_mlp or iREPA’s convolution.

build_mlp has two hidden SiLU layers of width. iREPA’s projector is one same-padded convolution over the token grid, which must be square.

function source

def spatial_zscore(features: jax.Array, gamma: float) -> jax.Array

Return iREPA’s spatial normalization of [B, N, D] features.

Gamma times each feature’s mean over the tokens is subtracted, and the result is divided by the feature’s standard deviation over the tokens plus 1e-6. The standard deviation is torch’s unbiased Tensor.std, as in the reference.