Skip to content

dew.objectives.diffusion.adversarial

Adversarial diffusion distillation: LADD, with ADD’s distillation term.

LADD (Sauer et al. 2024, “Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation”) trains a few-step student against a projected discriminator built on the frozen teacher. The student’s clean prediction and a real sample are both noised again, and the teacher reads each. Independent heads on the teacher’s token sequences after chosen blocks score them with the hinge loss. The heads are StyleGAN-T’s (Sauer et al. 2023, autonomousvision/stylegan-t networks/discriminator.py DiscHead), with LADD’s change from 1D to 2D convolutions over the token grid, and they are conditioned by projection on the noise level and the pooled text. ADD (Sauer et al. 2023, “Adversarial Diffusion Distillation”) adds the R1 penalty on each head’s input, with gamma 1e-5, and a distillation term that pulls the student’s prediction toward the teacher’s denoising of it, with lambda 2.5 and the exponential weighting alpha_t on the squared L2 distance. Neither paper publishes its training code. The heads follow StyleGAN-T’s, which tools/stylegan_t_reference.py runs, and the losses follow the papers’ equations.

NameSummary
AdversarialDistillationObjectiveTrains LADD, with ADD’s R1 penalty and its distillation term.

class source

class AdversarialDistillationObjective(
model: nn.Module,
process: Process | Preset,
inputs: InputSpec,
*,
feature_layers: tuple[str, ...],
student_times: tuple[float, ...] = (1.0, 0.75, 0.5, 0.25),
renoise_times: tuple[float, float] = (1.0, 1.0),
distillation_weight: float = 2.5,
r1_weight: float = 1e-05,
cmap_dim: int = 64,
kernel_size: tuple[int, int] = (9, 9),
time_features: int = 256,
steps: int = 2,
**kwargs={},
)

Trains LADD, with ADD’s R1 penalty and its distillation term.

The student starts from the teacher’s weights (FlowDistillationObjective), and a Head reads the teacher’s token grid after each layer in feature_layers, at least one. The student’s time is drawn from student_times, and the renoising level from LADD’s logit-normal with renoise_times (mean 1, std 1, the high-noise setting the paper uses for images).

The discriminator and the student train in the same step, each through its own loss with the other stopped. StyleGAN-T, whose heads these are, alternates a generator step and a discriminator step, and neither LADD nor ADD says which it does. The heads’ spectral norms take one power iteration per step. The real pass iterates and keeps its u; the held, fake and R1 passes iterate from that u and keep nothing, while StyleGAN-T keeps the u of each training pass.

The discriminator adds r1_weight (ADD’s gamma, 1e-5) times R1 on its real inputs. The student adds distillation_weight (ADD’s lambda, 2.5) times alpha_t ||x_0 - sg(teacher’s x_0 of the renoised x_0)||^2, summed over the sample. LADD itself drops that term when it trains on synthetic data, and on CIFAR-10 at 32 pixels the term at 2.5 dominated and the student did better without it. Sampling runs Consistency.

AdversarialDistillationObjective.network_variables

Section titled “AdversarialDistillationObjective.network_variables”
def network_variables(key: jax.Array, state: Variables) -> Variables

The heads, drawn on the teacher’s token grids.

def loss(variables, batch, step: Step)