Train a diffusion model with Dew
This is part 2 of the series, and so I would strongly recommend going through Notebook 01 first. We carry on from the ideas we built there, the forward and reverse diffusion processes, noise schedules and samplers, and I will only recap them briefly here.
In the previous notebook, we built a diffusion model completely by hand: the noise schedule, the loss, the network and the samplers, every piece written from scratch so that we could see exactly what each one does. That is honestly the best way to understand how these models work, but you wouldn’t want to write all of it again every time you train a model. And that’s where Dew comes in. Each of those pieces already exists in Dew, so this time we can focus on the training run itself.
That’s one training step, and training is this step repeated a few tens of thousands of times.
In this notebook, we shall train a small diffusion transformer on Oxford Flowers at 128x128, look at how the loss behaves, generate flowers from pure noise, and then reload the trained weights from the checkpoint.
Install Dependencies
Section titled “Install Dependencies”This installs Dew from GitHub with two of its extras: cuda13, which brings JAX’s CUDA 13 plugin along with the CUDA libraries it needs, and streaming, which brings Hugging Face datasets to download the flower images.
%pip install -q "dewml[cuda13,streaming] @ git+https://github.com/AshishKumar4/dew@71b43f2dc070cf289e1fbaf6177e484215d4ee1a" --extra-index-url https://pypi.dewml.dev/simpleinstall log
Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 604.4/604.4 kB 2.7 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.7/77.7 MB 23.1 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 532.5/532.5 kB 40.1 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 206.6/206.6 kB 18.0 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 91.4/91.4 kB 8.3 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.3/3.3 MB 87.7 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.9/53.9 MB 28.0 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 81.9 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 216.9/216.9 kB 20.2 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 137.7/137.7 kB 13.2 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 102.6/102.6 kB 9.2 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 57.6 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 134.2/134.2 MB 15.9 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.9/89.9 MB 19.5 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 56.6/56.6 kB 5.5 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 168.1/168.1 kB 16.5 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 32.1/32.1 MB 42.2 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.7/135.7 kB 13.2 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 87.5/87.5 kB 8.7 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.0/16.0 MB 52.4 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 109.2/109.2 kB 10.2 MB/s eta 0:00:00 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.9/115.9 kB 11.3 MB/s eta 0:00:00 Building wheel for dewml (pyproject.toml) ... done ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. cudf-cu13 26.6.0 requires pyarrow<24,>=19.0.0, but you have pyarrow 26.0.0 which is incompatible.
Settings
Section titled “Settings”IMAGE_SIZE = 128BATCH_SIZE = 64STEPS = 30000STAGES = 6LEARNING_RATE = 2e-4SAMPLE_STEPS = 40RUN_DIR = "runs/02-diffusion"SEED = 0import jaximport jax.numpy as jnpimport numpy as np
from dew import figuresfrom dew.training import prepare_process
prepare_process() # Dew's setup for a training process, before the first JAX callprint(jax.devices())[CudaDevice(id=0)]
Some Important Utils
Section titled “Some Important Utils”A small helper to draw a batch of images in a grid. Dew keeps images as uint8 in [0, 255] when they come out of the data pipeline, but the model works in [-1, 1], so the helper accepts both.
from dew.artifacts import uint8_pixels
def show_images(images, columns=4): images = np.asarray(images) if images.dtype != np.uint8: images = uint8_pixels(images) return figures.images([images], columns=columns, size=440 / columns - 4)Data pipeline
Section titled “Data pipeline”We use the Hub copy pranked03/flowers-blip-captions, which carries a caption for each Oxford Flowers photo. We shuffle the photos again every epoch and keep going past the end, and Dew’s image preset resizes each photo to the size we want, flips and colour-jitters it so the model sees a little variety, and groups the photos into batches. A photo that doesn’t decode stops the run rather than being skipped, so we always know how many photos an epoch holds. train(DataPartition()) then gives us a stream of batches for this process.
read="memory" reads the whole split into memory. The very first run has to download the split from the Hub and prepare it, and provision=True is what allows that; later runs just reuse the prepared copy. And Loading(workers=0, threads=16) does all the reading right here in this process, with 16 threads, instead of in separate worker processes.
from dew.data import Hub, Loading, presetsfrom dew.data.compile import compile as compile_datafrom dew.data.dataset import DataPartition
flowers = Hub("pranked03/flowers-blip-captions").shuffle(seed=0).repeat()data = compile_data(flowers.pipe(presets.images, IMAGE_SIZE, batch=BATCH_SIZE, augment=True, rejects=False), read="memory", provision=True, loading=Loading(workers=0, threads=16, read_buffer=64))
print(data.records, "images,", data.steps_per_epoch, "steps per epoch")stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. 16:23:00 WARNING dew.data: preparing hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train (264.3 MiB) into /root/.cache/huggingface/datasets/pranked03___flowers-blip-captions/default/0.0.0/b45984ffbef8596a8b6ac2ac8959f2f8998049bb
6552 images, 102 steps per epoch
batch = next(iter(data.train(DataPartition())))print(batch["image"].shape, batch["image"].dtype)show_images(batch["image"][:16])(64, 128, 128, 3) uint8
The Noise Process
Section titled “The Noise Process”A diffusion model learns to undo noise. During training, we take a clean image , pick a noise level, and make a noisy image
where:
- is the noisy image at time ,
- is the clean image,
- and are the signal and noise rates at time ,
- is Gaussian noise.
The model sees and the noise level and has to recover the clean image. Sampling then starts from pure noise at the highest noise level and removes it a little at a time, which is the reverse diffusion process we integrated with our samplers in notebook 01.
In notebook 01, we wrote the noise schedule, the model’s parameterization and the loss as separate pieces. Dew calls this combination of a noise schedule, what the network predicts and how the loss is weighted a Process, and it ships a few of them as presets (EDM, Karras, Cosine, Flow and Sqrt). We will use the EDM preset, which follows the paper “Elucidating the Design Space of Diffusion-Based Generative Models” by Karras et al. (2022).
Unlike the cosine schedule of notebook 01, which was variance preserving, this one is variance exploding: just stays at 1, and goes from 0.002 up to 80. Training draws its noise levels from a log-normal distribution, and sampling uses the paper’s grid, which spends more of its steps at low noise. The network also doesn’t predict the noise directly like our epsilon model did. The preset wraps it in the paper’s preconditioning, which scales the network’s input and mixes its output with the noisy image depending on .
EDM(regime="pixel") is the configuration, and calling it builds the Process. The regime picks the noise levels training draws: Karras et al.’s for images in pixel space, as here, or the higher, narrower ones of EDM2 for a model that denoises an autoencoder’s latents.
from dew.diffusion.presets import EDM
process = EDM(regime="pixel")()schedule = process.sampler_schedule
times = process.times(8)alphas, sigmas = schedule.rates(times)print("sampling times:", np.round(np.asarray(times), 3))print("alpha:", np.round(np.asarray(alphas), 3))print("sigma:", np.round(np.asarray(sigmas), 3))sampling times: [1. 0.857 0.714 0.571 0.429 0.286 0.143 0. ] alpha: [1. 1. 1. 1. 1. 1. 1. 1.] sigma: [8.0000e+01 3.4992e+01 1.3699e+01 4.6370e+00 1.2870e+00 2.6700e-01 3.5000e-02 2.0000e-03]
Here is one flower at each of those eight noise levels, from the most noise on the left to the least on the right. The images are clipped to [-1, 1] for display, so the leftmost ones look like plain static. And they really are: at the noise has a standard deviation of 80 while the pixels lie in [-1, 1], so there is basically nothing of the flower left to see.
image = batch["image"][0].astype(np.float32) / 127.5 - 1noise = jax.random.normal(jax.random.key(SEED), image.shape)noisy = [alpha * image + sigma * noise for alpha, sigma in zip(alphas, sigmas)]show_images(np.clip(np.stack(noisy), -1, 1), columns=8)Modeling
Section titled “Modeling”In notebook 01, we built a UNet for our denoising model. This time we use a small diffusion transformer (DiT), the architecture from the paper “Scalable Diffusion Models with Transformers” by Peebles and Xie. It is a plain stack of transformer blocks conditioned on the noise level with adaLN-Zero. It cuts the 128x128 image into 4x4 patches, which gives us 1,024 tokens, runs them through twelve transformer layers, and returns one prediction per pixel. SimpleDiT from dew.nn.backbones is that transformer.
InputSpec(Field("image", (128, 128, 3))) tells the objective which batch field to generate and what shape it has. Note that this model has no conditions: it learns what flowers look like in general, but we can’t ask it for a particular one. We will get to that in notebook 03.
from dew import Field, InputSpecfrom dew.nn.backbones import SimpleDiT
model = SimpleDiT( patch_size=4, emb_features=384, num_layers=12, num_heads=6, output_channels=3, dtype=jnp.bfloat16, attention_impl="auto",)inputs = InputSpec(Field("image", (IMAGE_SIZE, IMAGE_SIZE, 3)))The Objective and the Trainer
Section titled “The Objective and the Trainer”DiffusionObjective computes the diffusion loss, the one we wrote by hand in notebook 01: it draws a noise level for each image, makes the noisy image, runs the model and weights the error. It also asks the trainer to keep an exponential moving average (EMA) of the weights. The EMA copy changes slowly and usually gives cleaner samples than the live weights, which is why we sample with it later. solver, guidance and steps only say how the objective samples when it previews images; they don’t change training at all.
Trainer does the rest: it computes the gradients, applies the AdamW optimizer, updates the EMA, and writes checkpoints to RUN_DIR. LocalTracker records every logged number in a JSON file, which we will plot after training.
from dew import Checkpoints, LocalTracker, Trainerfrom dew.config import OptimConfigfrom dew.objectives.diffusion import DiffusionObjectivefrom dew.sampling import EulerAncestral
objective = DiffusionObjective( model, process, inputs, ema_decay=0.999, solver=EulerAncestral(), guidance=None, steps=SAMPLE_STEPS,)tracker = LocalTracker(f"{RUN_DIR}/tracking")trainer = Trainer( objective, OptimConfig(learning_rate=LEARNING_RATE), key=jax.random.key(SEED), checkpoints=Checkpoints(RUN_DIR), tracker=tracker,)
variables = jax.eval_shape(objective.init, jax.random.key(0))n_params = sum(x.size for x in jax.tree_util.tree_leaves(variables["params"]))print(f"{n_params / 1e6:.1f}M parameters")35.5M parameters
Training
Section titled “Training”fit trains until the step counter reaches STEPS, prints the loss every log_every steps, and writes a checkpoint at the end. We call it once for each of STAGES stages, each time with a later end, and keep the averaged weights after every stage. Each call carries on from the checkpoint the one before it wrote:
If RUN_DIR already holds a checkpoint, fit resumes from it, and the loop skips the stages already done.
done = Checkpoints(RUN_DIR).latest or 0snapshots = []for stage in range(1, STAGES + 1): end = STEPS * stage // STAGES if end < done: continue # an earlier run of this cell trained past this stage state = trainer.fit(data, steps=end, log_every=2000, checkpoint_every=STEPS) snapshots.append((int(state.step), jax.device_get(state.averaged)))Training SimpleDiT from step 0 to 5000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute step 2000/5000 loss 0.06594 data_wait_ms 0.03913 grad_norm 0.08774 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 15.58 step_time_ms 116.7 samples_per_sec 548.3 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:05:46 left step 4000/5000 loss 0.06934 data_wait_ms 0.03810 grad_norm 0.05747 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 15.58 step_time_ms 116.7 samples_per_sec 548.2 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:01:57 left step 5000/5000 loss 0.07218 data_wait_ms 0.03980 grad_norm 0.05083 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 15.58 step_time_ms 116.7 samples_per_sec 548.3 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 Trained 5000 steps in 0:10:40: first step after 52.41 s, then 8.5 step/s 91.5% of the wall time in steps, final loss 0.07218 Resumed from step 5000 in /content/runs/02-diffusion Training SimpleDiT from step 5000 to 10000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute step 6000/10000 loss 0.06449 data_wait_ms 0.03747 grad_norm 0.04855 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.10 step_time_ms 117.0 samples_per_sec 547.1 mfu 33.1% hlo_flops_util 33.1% hfu 33.1% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:07:47 left step 8000/10000 loss 0.07277 data_wait_ms 0.03687 grad_norm 0.04408 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.10 step_time_ms 116.7 samples_per_sec 548.2 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:03:53 left step 10000/10000 loss 0.07595 data_wait_ms 0.03689 grad_norm 0.04232 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.7 samples_per_sec 548.3 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 WARNING:absl:Waiting for previous save to complete took 2.275977 seconds. If this number is high, consider checkpointing less frequently. Trained 5000 steps in 0:10:13: first step after 27.25 s, then 8.5 step/s 95.5% of the wall time in steps, final loss 0.07595 Resumed from step 10000 in /content/runs/02-diffusion Training SimpleDiT from step 10000 to 15000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute step 12000/15000 loss 0.06293 data_wait_ms 0.03695 grad_norm 0.03926 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.8 samples_per_sec 547.9 mfu 33.1% hlo_flops_util 33.1% hfu 33.1% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:05:50 left step 14000/15000 loss 0.06328 data_wait_ms 0.03735 grad_norm 0.03839 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.7 samples_per_sec 548.5 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:01:58 left step 15000/15000 loss 0.05546 data_wait_ms 0.03948 grad_norm 0.03752 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.7 samples_per_sec 548.5 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 WARNING:absl:Waiting for previous save to complete took 2.393647 seconds. If this number is high, consider checkpointing less frequently. Trained 5000 steps in 0:10:13: first step after 27.08 s, then 8.5 step/s 95.5% of the wall time in steps, final loss 0.05546 Resumed from step 15000 in /content/runs/02-diffusion Training SimpleDiT from step 15000 to 20000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute step 16000/20000 loss 0.07462 data_wait_ms 0.03753 grad_norm 0.03523 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.9 samples_per_sec 547.4 mfu 33.1% hlo_flops_util 33.1% hfu 33.1% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:07:53 left step 18000/20000 loss 0.06143 data_wait_ms 0.03911 grad_norm 0.03630 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.7 samples_per_sec 548.6 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:03:53 left step 20000/20000 loss 0.07199 data_wait_ms 0.03837 grad_norm 0.03461 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.7 samples_per_sec 548.6 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 WARNING:absl:Waiting for previous save to complete took 2.469201 seconds. If this number is high, consider checkpointing less frequently. Trained 5000 steps in 0:10:15: first step after 28.85 s, then 8.5 step/s 95.2% of the wall time in steps, final loss 0.07199 Resumed from step 20000 in /content/runs/02-diffusion Training SimpleDiT from step 20000 to 25000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute step 22000/25000 loss 0.06882 data_wait_ms 0.03734 grad_norm 0.03436 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.7 samples_per_sec 548.3 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:05:55 left step 24000/25000 loss 0.06065 data_wait_ms 0.03622 grad_norm 0.03366 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.6 samples_per_sec 548.8 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:01:57 left step 25000/25000 loss 0.07334 data_wait_ms 0.03719 grad_norm 0.03216 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.6 samples_per_sec 548.7 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 WARNING:absl:Waiting for previous save to complete took 2.411053 seconds. If this number is high, consider checkpointing less frequently. Trained 5000 steps in 0:10:16: first step after 29.96 s, then 8.5 step/s 95.1% of the wall time in steps, final loss 0.07334 Resumed from step 25000 in /content/runs/02-diffusion Training SimpleDiT from step 25000 to 30000: 35.5M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute step 26000/30000 loss 0.06060 data_wait_ms 0.03639 grad_norm 0.03320 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 117.0 samples_per_sec 546.9 mfu 33.1% hlo_flops_util 33.1% hfu 33.1% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:07:47 left step 28000/30000 loss 0.05745 data_wait_ms 0.03873 grad_norm 0.03192 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.8 samples_per_sec 548.2 mfu 33.2% hlo_flops_util 33.2% hfu 33.2% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:03:50 left step 30000/30000 loss 0.06163 data_wait_ms 0.03767 grad_norm 0.03184 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.11 step_time_ms 116.8 samples_per_sec 548.1 mfu 33.2% hlo_flops_util 33.1% hfu 33.1% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 WARNING:absl:Waiting for previous save to complete took 2.813380 seconds. If this number is high, consider checkpointing less frequently. Trained 5000 steps in 0:10:16: first step after 29.53 s, then 8.5 step/s 95.1% of the wall time in steps, final loss 0.06163
The tracker wrote one line per log step to scalars.jsonl, here every 2,000 steps and at the end of each stage. By step 2,000, the first point we logged, the loss is already down to about 0.07, and after that it jumps around between about 0.055 and 0.076 without falling much further. Don’t worry, that is expected: every batch draws new noise levels, and some noise levels are much harder to denoise than others. The model keeps improving long after the loss flattens out; the noise hides it.
import json
rows = [json.loads(line) for line in open(f"{RUN_DIR}/tracking/scalars.jsonl")]rows = [row for row in rows if "train/loss" in row["scalars"]]figures.curve({"loss": [row["scalars"]["train/loss"] for row in rows]}, x=[row["step"] for row in rows], x_label="step", y_label="loss")Generate Samples
Section titled “Generate Samples”To generate an image, we run the noise process backwards, just like we did with our samplers in notebook 01. process.noise draws the starting point, pure noise with the spread of the highest noise level. process.denoiser wraps the model and its weights into the function a sampler calls: given a noisy image and its time, it returns the model’s guess of the clean image and of the noise. sample then runs the solver over SAMPLE_STEPS noise levels in one compiled loop.
The solver is EulerAncestral, the Euler-A sampler we talked about in notebook 01: Euler’s method on the reverse diffusion SDE, so a bit of fresh noise goes back in at every step.
state.averaged is the weights tree with the EMA weights in place of the live ones, and objective.model_variables keeps only the model’s own weights.
from dew.sampling import sample
def generate(params, count, key): denoise = process.denoiser(model, objective.model_variables(params), {}) x_T = process.noise(key, (count, IMAGE_SIZE, IMAGE_SIZE, 3)) return sample(denoise, x_T, SAMPLE_STEPS, solver=EulerAncestral(), key=key)
images = generate(state.averaged, 32, jax.random.key(1))show_images(images[:16])Most of these are clearly flowers, with petals, a centre and leaves behind them, but they are still soft and painterly, and one or two are more blob than flower.
Watching it learn
Section titled “Watching it learn”Remember the stages? Here is the same starting noise, sampled with the averaged weights we kept after each one. Every frame starts from exactly the same and draws the same fresh noise along the way, so whatever changes from frame to frame is the model.
frames = [uint8_pixels(generate(params, 8, jax.random.key(1))) for _, params in snapshots]figures.images(frames, captions=[f"step {step:,}" for step, _ in snapshots], columns=4, size=96)By step 5,000 every image already holds a flower-shaped patch of the right colours on green. What the later stages add is detail. The pink flower at the top left gets separate petals and a yellow centre by step 10,000, and the one at the bottom right turns from pink to red over the last three stages. The third in the top row and the first in the bottom row never quite come into focus.
Reloading the checkpoint
Section titled “Reloading the checkpoint”The checkpoint in RUN_DIR holds the arrays: the weights, the EMA, the optimizer state and the step counters. It does not hold the code. So to reload, we build a new Trainer over the same objective and call place(), which restores the latest checkpoint. Sampling with the same key gives the same images, which shows that the weights came back unchanged.
restored, _, _ = Trainer( objective, OptimConfig(learning_rate=LEARNING_RATE).build(STEPS), key=jax.random.key(SEED), checkpoints=Checkpoints(RUN_DIR),).place()print("restored step:", int(restored.step))
again = generate(restored.averaged, 32, jax.random.key(1))print("largest difference from the samples above:", float(jnp.max(jnp.abs(again - images))))Resumed from step 30000 in /content/runs/02-diffusion restored step: 30000 largest difference from the samples above: 0.0
Where to go next
Section titled “Where to go next”Notebook 03 trains on the captions as well, so that you can ask for a flower in words. Notebook 04 loads this notebook’s checkpoint and compares Dew’s samplers on it. And if you would rather train from the command line, recipes/diffusion/train.py runs the same kind of training.