dew.objectives.diffusion.adversarial
Adversarial diffusion distillation: LADD, with ADD’s distillation term.
LADD (Sauer et al. 2024, “Fast High-Resolution Image Synthesis with Latent
Adversarial Diffusion Distillation”) trains a few-step student against a
projected discriminator built on the frozen teacher. The student’s clean
prediction and a real sample are both noised again, and the teacher reads
each. Independent heads on the teacher’s token sequences after chosen
blocks score them with the hinge loss. The heads are StyleGAN-T’s (Sauer et
al. 2023, autonomousvision/stylegan-t networks/discriminator.py
DiscHead), with LADD’s change from 1D to 2D convolutions over the token
grid, and they are conditioned by projection on the noise level and the
pooled text. ADD (Sauer et al. 2023, “Adversarial Diffusion Distillation”)
adds the R1 penalty on each head’s input, with gamma 1e-5, and a
distillation term that pulls the student’s prediction toward the teacher’s
denoising of it, with lambda 2.5 and the exponential weighting alpha_t on
the squared L2 distance. Neither paper publishes its training code. The
heads follow StyleGAN-T’s, which tools/stylegan_t_reference.py runs, and
the losses follow the papers’ equations.
| Name | Summary |
|---|---|
AdversarialDistillationObjective | Trains LADD, with ADD’s R1 penalty and its distillation term. |
AdversarialDistillationObjective
Section titled “AdversarialDistillationObjective”class AdversarialDistillationObjective( model: nn.Module, process: Process | Preset, inputs: InputSpec, *, feature_layers: tuple[str, ...], student_times: tuple[float, ...] = (1.0, 0.75, 0.5, 0.25), renoise_times: tuple[float, float] = (1.0, 1.0), distillation_weight: float = 2.5, r1_weight: float = 1e-05, cmap_dim: int = 64, kernel_size: tuple[int, int] = (9, 9), time_features: int = 256, steps: int = 2, **kwargs={},)Trains LADD, with ADD’s R1 penalty and its distillation term.
The student starts from the teacher’s weights (FlowDistillationObjective),
and a Head reads the teacher’s token grid after each layer in
feature_layers, at least one. The
student’s time is drawn from student_times, and the renoising level
from LADD’s logit-normal with renoise_times (mean 1, std 1, the
high-noise setting the paper uses for images).
The discriminator and the student train in the same step, each through
its own loss with the other stopped. StyleGAN-T, whose heads these are,
alternates a generator step and a discriminator step, and neither LADD
nor ADD says which it does. The heads’ spectral norms take one power
iteration per step. The real pass iterates and keeps its u; the held,
fake and R1 passes iterate from that u and keep nothing, while
StyleGAN-T keeps the u of each training pass.
The discriminator adds r1_weight (ADD’s gamma, 1e-5) times R1 on its
real inputs. The student adds distillation_weight (ADD’s lambda, 2.5)
times alpha_t ||x_0 - sg(teacher’s x_0 of the renoised x_0)||^2, summed
over the sample. LADD itself drops that term when it trains on synthetic
data, and on CIFAR-10 at 32 pixels the term at 2.5 dominated and the
student did better without it. Sampling runs Consistency.
AdversarialDistillationObjective.network_variables
Section titled “AdversarialDistillationObjective.network_variables”def network_variables(key: jax.Array, state: Variables) -> VariablesThe heads, drawn on the teacher’s token grids.
AdversarialDistillationObjective.loss
Section titled “AdversarialDistillationObjective.loss”def loss(variables, batch, step: Step)