Skip to content

dew.nn.mp

EDM2’s magnitude-preserving layers.

The layers follow Karras et al. 2024, “Analyzing and Improving the Training Dynamics of Diffusion Models”, section 3 and appendix B. Every layer keeps the expected magnitude of its activations at one. The weights are normalized in the forward pass and scaled by the fan-in (Eq. 47), the nonlinearity is divided by its expected output magnitude (Eq. 81), and a sum or a concatenation weights its operands so the result has unit magnitude (Eqs. 88, 103). The layers are NVlabs/edm2’s training/networks_edm2.py, in Flax’s channels-last layout.

Forced weight normalization (Eq. 66) also keeps each stored weight at unit magnitude, so an update’s relative size, and with it the effective learning rate, does not decay as the weights grow. The paper applies it in place in the training forward pass. Here it is forced_weight_normalization, an optimizer step that renormalizes every mp_kernel after its update; OptimConfig.build appends it when OptimConfig.forced_weight_normalization is set.

NameSummary
MP_KERNELThe parameter name of a magnitude-preserving weight.
MPConvApplies a convolution whose weight is normalized per output channel and scaled by the fan-in.
MPFourierComputes Fourier features of a scalar at unit magnitude.
UncertaintyComputes EDM2’s learned uncertainty u(sigma), one log-variance per example.
forced_weight_normalizationReturn an optax transformation that keeps every mp_kernel at unit magnitude.
mp_catReturn the channel concatenation of a and b, weighted by t and scaled to unit magnitude.
mp_siluReturn SiLU divided by its expected output magnitude on a unit normal input.
mp_sumReturn the interpolation (1 - t) a + t b, scaled to unit magnitude.
normalizeReturn x scaled to unit root-mean-square magnitude over axes.

attribute source

MP_KERNEL = 'mp_kernel'

The parameter name of a magnitude-preserving weight. Forced weight normalization keeps these weights at unit magnitude.

Flax module source

class MPConv(kernel_size: tuple[int, ...] = ())

Applies a convolution whose weight is normalized per output channel and scaled by the fan-in.

An empty kernel_size makes it a dense layer. This is Eq. 47 of the EDM2 paper. The weight is mp_kernel, with the spatial axes first, then the input and output channels. A convolution pads to keep the size, as the reference’s odd kernels do, and runs as Conv’s does (dew.nn.conv.convolution).

def spatial_shape(shape: tuple[int, ...]) -> tuple[int, ...]
def model_flops(lengths, input_features: int) -> float

Dense or convolution products; weight normalization is pointwise work.

Flax module source

class MPFourier(bandwidth: float = 1.0)

Computes Fourier features of a scalar at unit magnitude.

The random frequencies and phases are drawn once and held in the constants collection. This is Eq. 75 of the EDM2 paper.

Flax module source

class Uncertainty(channels: int = 128)

Computes EDM2’s learned uncertainty u(sigma), one log-variance per example.

Magnitude-preserving Fourier features of the model time go through one magnitude-preserving dense layer. This is Eq. 21 of the EDM2 paper, and the logvar head of the reference’s Precond.

function source

def forced_weight_normalization() -> optax.GradientTransformation

Return an optax transformation that keeps every mp_kernel at unit magnitude.

It changes each mp_kernel update so that the kernel has unit magnitude per output channel once the update is applied (Eq. 66 of the EDM2 paper). Every other update passes through unchanged. The transformation needs the parameters and raises ValueError without them.

function source

def mp_cat(a: jax.Array, b: jax.Array, t: float = 0.5) -> jax.Array

Return the channel concatenation of a and b, weighted by t and scaled to unit magnitude.

This is Eq. 103 of the EDM2 paper.

function source

def mp_silu(x: jax.Array) -> jax.Array

Return SiLU divided by its expected output magnitude on a unit normal input.

This is Eq. 81 of the EDM2 paper.

function source

def mp_sum(a: jax.Array, b: jax.Array, t: float = 0.5) -> jax.Array

Return the interpolation (1 - t) a + t b, scaled to unit magnitude.

This is Eq. 88 of the EDM2 paper.

function source

def normalize(
x: jax.Array,
axes: Sequence[int] | None = None,
eps: float = 0.0001,
*,
compensated: bool = False,
) -> jax.Array

Return x scaled to unit root-mean-square magnitude over axes.

axes defaults to all but the last, which normalizes each output channel of a kernel, or each feature vector. compensated sums the squares with compensated_sum, as a kernel’s initialization does (drawn_kernel).