Samplers and schedules
A trained diffusion model only tells us, for a noisy image, what it thinks the clean image is. It’s the sampler that turns that into a picture: it starts from pure noise and takes a series of steps down the noise levels. In notebook 01 we wrote DDIM, Euler, DDPM and Heun by hand and talked about how each of them removes noise. Here we take the flower model we trained in notebook 02. We compare seven built-in samplers at a few step counts, all from the same starting noise, and time each one.
That’s the whole story in two dimensions. From the same starting noise, Euler’s three steps stop short of the data, and Heun’s, which ask the model twice each, land on it.
The model loads from the Hub, where notebook 02’s run is saved as dewml/flowers-dit-35m.
Install Dependencies
Section titled “Install Dependencies”This time we only need Dew and its cuda13 extra.
%pip install -q "dewml[cuda13] @ git+https://github.com/AshishKumar4/dew@fcc2bf6f5141d7971f9ad7886aa8592a015e2fdc"install log
Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Building wheel for dewml (pyproject.toml) ... done
Settings
Section titled “Settings”STEP_COUNTS are the numbers of noise levels each sampler uses, and SAMPLES is how many images each run draws.
IMAGE_SIZE = 128CHECKPOINT = "dewml/flowers-dit-35m"STEP_COUNTS = [10, 20, 40]SAMPLES = 4SEED = 0import time
import dew # before JAX's backend opens, so its numerical settings applyimport jaximport jax.numpy as jnpimport numpy as np
from dew import figuresfrom dew.artifacts import uint8_pixels
print(jax.devices())[CudaDevice(id=0)]
Loading the trained model
Section titled “Loading the trained model”Remember from notebook 02: a checkpoint holds arrays, not code. Next to the arrays, though, each checkpoint records how to rebuild the model, its noise process and its sampler, and dew.pipeline reads that record back with the weights. So we don’t repeat notebook 02’s settings here. Of the weights it reads the averaged (EMA) copy the run kept, the one notebook 02 sampled from.
task = dew.pipeline(CHECKPOINT)model, process, params = task.model, task.process, task.variablesprint(f"{sum(x.size for x in jax.tree.leaves(params)) / 1e6:.1f}M parameters")stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
35.5M parameters
The same starting noise for every sampler
Section titled “The same starting noise for every sampler”To compare samplers fairly, everything except the sampler has to stay fixed. So we draw the starting noise once and hand the same array to every run. Its spread is about 80, the highest noise level of the EDM schedule. The ancestral samplers (DDPM and Euler ancestral) also add fresh noise at every step; sample derives that noise from the key we pass, and we pass the same key every time.
x_T = process.noise(jax.random.key(SEED), (SAMPLES, IMAGE_SIZE, IMAGE_SIZE, 3))print("x_T spread:", float(jnp.std(x_T)))x_T spread: 79.90058898925781
The samplers
Section titled “The samplers”If any of these feel unfamiliar, notebook 01 explains DDIM, Euler, DDPM and Heun in a lot more detail.
- DDPM follows the reverse Markov chain of the original DDPM paper by Ho et al. (2020) and adds fresh noise at every step. It needs many small steps.
- DDIM, from the DDIM paper by Song et al. (2020), is the deterministic version: jump to the next noise level along the direction the model points to.
- Euler takes one Euler step of the probability flow ODE per noise level. On this schedule it is the same update as DDIM, just written differently; this is the algebraic equivalence I left as an exercise in notebook 01.
- Euler ancestral takes an Euler step to a slightly lower level and then adds back some fresh noise, so it solves the reverse SDE instead of the ODE.
- Heun takes an Euler step, asks the model again at the end of it, and averages the two slopes. That costs two model calls per step.
- RK4 is the classical fourth-order Runge-Kutta method: four model calls per step.
- MultiStepDPM reuses the model’s answers from the last three steps to correct the current one, at one model call per step.
Each is a small value in dew.sampling. As in notebook 02, process.denoiser(model, params, {}) wraps the model and its weights into the function a sampler calls. Then sample(denoise, x_T, steps, solver=..., key=...) runs any of the samplers over it in one loop.
from dew.sampling import DDIM, DDPM, RK4, Euler, EulerAncestral, Heun, MultiStepDPM, sample
SOLVERS = { "DDPM": DDPM(), "DDIM": DDIM(), "Euler": Euler(), "Euler ancestral": EulerAncestral(), "Heun": Heun(), "RK4": RK4(), "MultiStepDPM": MultiStepDPM(),}To time the samplers fairly, we compile each one once with jax.jit and time only a second call, otherwise we would mostly be timing XLA’s compiler rather than the sampler. The step count and the solver are static arguments, so every combination compiles its own loop. The weights are a normal argument, which keeps them out of the compiled program.
import functools
@functools.partial(jax.jit, static_argnames=("steps", "solver"))def run(params, x_T, key, steps, solver): denoise = process.denoiser(model, params, {}) return sample(denoise, x_T, steps, solver=solver, key=key)
images, seconds = {}, {}for steps in STEP_COUNTS: for name, solver in SOLVERS.items(): run(params, x_T, jax.random.key(SEED), steps, solver).block_until_ready() start = time.perf_counter() out = run(params, x_T, jax.random.key(SEED), steps, solver).block_until_ready() seconds[name, steps] = time.perf_counter() - start images[name, steps] = np.asarray(out)The grid flips through the three step counts. Each row is a sampler and each column one starting noise, so images in the same column started from the same point.
frames = [uint8_pixels(np.concatenate([images[name, steps] for name in SOLVERS])) for steps in STEP_COUNTS]figures.images(frames, columns=SAMPLES, size=72, rows=list(SOLVERS), captions=[f"{steps} steps" for steps in STEP_COUNTS])At 10 steps, DDPM and Euler ancestral leave soft, almost featureless blobs, while the deterministic samplers already draw the same four flowers. Heun and RK4 are sharper than Euler at that step count: their petals have edges where Euler’s are still blurred. By 40 steps the deterministic samplers all look alike. The ancestral samplers have taken different paths from that shared starting noise, and at each step count they land somewhere a little different.
Time per run
Section titled “Time per run”The timings broadly follow the number of model calls. At 40 steps RK4 takes 412 ms against Euler’s 117 ms, three and a half times as long, and Heun takes 230 ms, about twice as long. At 10 steps the corresponding times are 106, 38 and 72 ms. Each entry is one timed call after a warm-up call, not a repeated benchmark.
print(f"{'sampler':<16}" + "".join(f"{steps:>8} steps" for steps in STEP_COUNTS))for name in SOLVERS: print(f"{name:<16}" + "".join(f"{seconds[name, steps] * 1000:>11.0f} ms" for steps in STEP_COUNTS))sampler 10 steps 20 steps 40 steps DDPM 39 ms 75 ms 140 ms DDIM 39 ms 66 ms 125 ms Euler 38 ms 73 ms 117 ms Euler ancestral 39 ms 75 ms 138 ms Heun 72 ms 124 ms 230 ms RK4 106 ms 204 ms 412 ms MultiStepDPM 39 ms 77 ms 142 ms
A fast sampler that lands somewhere else isn’t a bargain, though. So let’s put each deterministic sampler’s time next to how far it lands from the answer. For the answer we take a 200-step Heun solve from the same noise. The ancestral samplers sit this one out: they add noise on purpose, so they aren’t aiming for that answer.
fine = np.asarray(run(params, x_T, jax.random.key(SEED), 200, Heun()))deterministic = ["Euler", "Heun", "RK4", "MultiStepDPM"]for name in deterministic: print(f"{name:<14}" + " ".join( f"{steps} steps: {seconds[name, steps] * 1000:5.0f} ms, " f"{np.mean(np.abs(images[name, steps] - fine)):.4f}" for steps in STEP_COUNTS))figures.points({name: [(seconds[name, steps] * 1000, np.mean(np.abs(images[name, steps] - fine))) for steps in STEP_COUNTS] for name in deterministic}, x_label="milliseconds per run", y_label="distance from the 200-step solve", log_x=True, log_y=True)Euler 10 steps: 38 ms, 0.1155 20 steps: 73 ms, 0.0674 40 steps: 117 ms, 0.0359 Heun 10 steps: 72 ms, 0.0429 20 steps: 124 ms, 0.0149 40 steps: 230 ms, 0.0071 RK4 10 steps: 106 ms, 0.0115 20 steps: 204 ms, 0.0118 40 steps: 412 ms, 0.0047 MultiStepDPM 10 steps: 39 ms, 0.0916 20 steps: 77 ms, 0.0365 40 steps: 142 ms, 0.0110
Heun at 10 steps lands about as close as Euler at 40, in less time: 0.043 in 72 ms against 0.036 in 117 ms. MultiStepDPM costs about what Euler costs and lands three times closer at 40 steps, 0.011 against 0.036. RK4 is closest of all at 10 steps, 0.012 in 106 ms, though doubling its steps doesn’t help. So a sampler that asks the model more than once per step, or reuses its earlier answers, buys a lot more accuracy than the extra milliseconds would suggest.
How close the samplers get to each other
Section titled “How close the samplers get to each other”The deterministic samplers all solve the same ODE, so with enough steps they should land on nearly the same images. Below is the mean absolute pixel difference between each sampler’s images and Euler’s at 40 steps, on the [-1, 1] scale. The ancestral samplers add their own noise along the way, so they end up at different flowers.
The last line compares the two ancestral samplers with each other: with the same key they give almost the same images, so on this schedule their updates nearly coincide. That is expected: on this schedule, where the signal rate is 1, Dew’s DDPM step (with its default variance="small") and Euler ancestral both work out to , so in exact arithmetic they are the same update. It is DDIM with (Song, Meng and Ermon, 2021), one particular discretization of the reverse diffusion SDE.
reference = images["Euler", STEP_COUNTS[-1]]for name in SOLVERS: difference = np.mean(np.abs(images[name, STEP_COUNTS[-1]] - reference)) print(f"{name:<16} {difference:.4f}")print("DDPM against Euler ancestral:", np.mean(np.abs(images["DDPM", STEP_COUNTS[-1]] - images["Euler ancestral", STEP_COUNTS[-1]])))DDPM 0.3803 DDIM 0.0000 Euler 0.0000 Euler ancestral 0.3804 Heun 0.0381 RK4 0.0372 MultiStepDPM 0.0269 DDPM against Euler ancestral: 0.00064495887
Presets pair a training schedule with a sampling schedule
Section titled “Presets pair a training schedule with a sampling schedule”A sampler uses whatever noise levels the process gives it, and those have to match the model: the sampling rates, the model’s time and the prediction transform all have to be the ones it was trained with. That is why Dew keeps both in one Process. EDM trains on log-normal noise levels and samples on the Karras grid. The cosine preset trains and samples one discrete schedule with v-prediction, the parameterization from the paper “Progressive Distillation for Fast Sampling of Diffusion Models” by Salimans and Ho (2022). The flow preset is rectified flow, from the paper “Flow Straight and Fast” by Liu et al. (2022), on both sides.
Sampling a model with different rates, time or prediction transform than it trained with gives the wrong result. The noise levels it trains on can differ, though: EDM and Karras train on different ones and sample the same way.
Here are the noise levels each preset’s sampler steps through on a grid of 21 levels, leaving out the two ends, where the flow preset reaches pure noise and pure signal. The presets write their noise levels differently, so the chart puts them on one axis as the signal-to-noise ratio , on a log scale.
from dew.diffusion.presets import EDM, Cosine, Flow, Karras
for name, preset in (("edm", EDM(regime="pixel")), ("karras", Karras()), ("cosine", Cosine()), ("flow", Flow())): built = preset() print(f"{name:<7} train: {type(built.schedule).__name__:<24} " f"sample: {type(built.sampler_schedule).__name__:<24} " f"predicts: {type(built.prediction).__name__}")
grids = {}for name, preset in (("edm, karras", EDM(regime="pixel")), ("cosine", Cosine()), ("flow", Flow())): built = preset() # The flow preset's two ends are pure noise and pure signal, infinitely far # apart on this scale. alphas, sigmas = built.sampler_schedule.rates(built.times(21)[1:-1]) grids[name] = np.log10(np.asarray(alphas) ** 2 / np.asarray(sigmas) ** 2)figures.curve(grids, x_label="sampling step", y_label="log₁₀ α² / σ²")edm train: EDMNoiseScheduler sample: KarrasVENoiseScheduler predicts: KarrasPredictionTransform karras train: KarrasVENoiseScheduler sample: KarrasVENoiseScheduler predicts: KarrasPredictionTransform cosine train: CosineNoiseScheduler sample: CosineNoiseScheduler predicts: VPredictionTransform flow train: FlowMatchingScheduler sample: FlowMatchingScheduler predicts: FlowMatchPredictionTransform
EDM’s Karras grid covers far more ground than the other two: from a signal-to-noise ratio of about to , eight orders of magnitude, where cosine covers about four and flow five. Cosine and flow pass a ratio of 1 halfway through their steps; EDM is still at about there, so it packs its steps more tightly at the noisy end.
Where to go next
Section titled “Where to go next”Notebook 03 adds a text caption and classifier-free guidance on top of the very same sample call.