Text to image with classifier-free guidance
This is part 3 of the series, and it builds directly on notebook 02, so I would recommend going through that one first.
The model we trained in notebook 02 learned what flowers look like, but there was no way to tell it which flower to draw. You simply got whatever flower the starting noise happened to lead to. But what we usually want is to describe the image in words and get that image. So in this notebook, we give the model a caption with every image, so that at the end we can type “a red rose” and get something red and rose-like.
Install Dependencies
Section titled “Install Dependencies”Same as notebook 02: Dew from GitHub with its cuda13 and streaming extras.
%pip install -q "dewml[cuda13,streaming] @ git+https://github.com/AshishKumar4/dew@71b43f2dc070cf289e1fbaf6177e484215d4ee1a" --extra-index-url https://pypi.dewml.dev/simpleinstall log
Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done
Settings
Section titled “Settings”IMAGE_SIZE = 128BATCH_SIZE = 64STEPS = 30000LEARNING_RATE = 2e-4UNCONDITIONAL_PROB = 0.12SAMPLE_STEPS = 40PROMPTS = ["a red rose", "a yellow sunflower", "a white daisy", "a purple iris"]GUIDANCE_SCALES = [1.0, 3.0, 6.0]RUN_DIR = "runs/03-text-to-image"SEED = 0import jaximport numpy as np
from dew import figuresfrom dew.artifacts import uint8_pixelsfrom dew.training import prepare_process
prepare_process() # Dew's setup for a training process, before the first JAX callprint(jax.devices())[CudaDevice(id=0)]
def show_images(images, titles, columns=4): # Each row's captions, then the grid. for row in range(0, len(titles), columns): print(" | ".join(titles[row:row + columns])) return figures.images([np.asarray(images)], columns=columns, size=440 / columns - 4)Turning text into numbers
Section titled “Turning text into numbers”A neural network can’t read words, so a text encoder first turns each caption into a sequence of vectors. We use the text half of CLIP (ViT-L/14), from the paper “Learning Transferable Visual Models From Natural Language Supervision” by Radford et al. (2021). CLIP was trained to match captions with images, so its vectors already say something about what a caption looks like, which is exactly what we want here. We keep CLIP frozen; only the diffusion model trains.
The InputSpec now has a condition as well as the image. Condition(encoder, field="text", unconditional="") basically says: read tokens from the batch field text, encode them with CLIP, and use the empty caption as “no caption”. The key textcontext is the keyword the model receives the encoded caption under.
from dew import Condition, Field, InputSpecfrom dew.inputs import CLIPText
encoder = CLIPText.from_pretrained("openai/clip-vit-large-patch14")inputs = InputSpec( sample=Field("image", (IMAGE_SIZE, IMAGE_SIZE, 3)), conditions={"textcontext": Condition(encoder, field="text", unconditional="")},)print("context length:", encoder.context)print(inputs.tokenize(["a red rose"])["text"].keys())stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
context length: 77 dict_keys(['input_ids', 'attention_mask'])
Captioned data
Section titled “Captioned data”The data is the same Hugging Face copy of Oxford Flowers as in notebook 02, and every photo carries a caption written by an image captioning model (BLIP). We build the same chain through the images preset, and passing host=(EncodeCaptions(inputs.tokenize),) to compile turns each caption into CLIP tokens while the batch is being built, so the batches carry token arrays under text instead of strings.
from dew.data import Hub, Loading, presetsfrom dew.data.compile import compile as compile_datafrom dew.data.dataset import DataPartitionfrom dew.data.images import EncodeCaptions
flowers = Hub("pranked03/flowers-blip-captions").shuffle(seed=0).repeat()data = compile_data(flowers.pipe(presets.images, IMAGE_SIZE, batch=BATCH_SIZE, augment=True, rejects=False), read="memory", provision=True, loading=Loading(workers=0, threads=16, read_buffer=64), host=(EncodeCaptions(inputs.tokenize),))
stream = data.train(DataPartition())try: batch = next(stream)finally: stream.close()captions = encoder.captions(batch["text"])show_images(batch["image"][:8], captions[:8])a pelargonium in the garden | pink azaleas in the garden | a red and yellow columbine in a garden | a wild pansy in the garden a yellow sunflower in the sky | a passion flower in a pot on a planter | a white windflower in the grass | a red red ginger in the garden
The model and the objective
Section titled “The model and the objective”The model is the same DiT from notebook 02, except that now it takes the caption vectors through the textcontext keyword and mixes them into every layer.
DiffusionObjective gets two new arguments. unconditional_prob makes it swap the caption for the empty one on 12% of training rows, and guidance=CFG(3.0) is how its previews sample. We keep the CLIP weights in the training state as frozen parameters, so they are saved in the checkpoint but never updated.
import jax.numpy as jnpfrom dew import Checkpoints, LocalTracker, Trainerfrom dew.config import OptimConfigfrom dew.diffusion.presets import EDMfrom dew.nn.backbones import SimpleDiTfrom dew.objectives.diffusion import DiffusionObjectivefrom dew.sampling import CFG, EulerAncestral
preset = EDM(regime="pixel")model = SimpleDiT( patch_size=4, emb_features=384, num_layers=12, num_heads=6, output_channels=3, dtype=jnp.bfloat16, attention_impl="auto",)objective = DiffusionObjective( model, preset, inputs, unconditional_prob=UNCONDITIONAL_PROB, ema_decay=0.999, solver=EulerAncestral(), guidance=CFG(3.0), steps=SAMPLE_STEPS,)trainer = Trainer( objective, OptimConfig(learning_rate=LEARNING_RATE), key=jax.random.key(SEED), checkpoints=Checkpoints(RUN_DIR), tracker=LocalTracker(f"{RUN_DIR}/tracking"),)Training
Section titled “Training”fit trains until STEPS, prints the loss every 2,000 steps and writes a checkpoint at the end, and then we plot the loss from the tracker.
state = trainer.fit(data, steps=STEPS, log_every=2000, checkpoint_every=STEPS)Training SimpleDiT from step 0 to 30000: 35.8M parameters, on 1 × NVIDIA A100-SXM4-40GB, batch 64, float32 parameters, bfloat16 compute step 2000/30000 loss 0.06584 data_wait_ms 0.03893 grad_norm 0.09647 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.49 step_time_ms 132.6 samples_per_sec 482.6 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 1:01:51 left step 4000/30000 loss 0.06888 data_wait_ms 0.03927 grad_norm 0.06400 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.50 step_time_ms 132.6 samples_per_sec 482.8 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:58:25 left step 6000/30000 loss 0.06451 data_wait_ms 0.03874 grad_norm 0.05582 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.53 step_time_ms 132.5 samples_per_sec 483.0 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:53:00 left step 8000/30000 loss 0.07280 data_wait_ms 0.03918 grad_norm 0.05082 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.53 step_time_ms 132.5 samples_per_sec 482.9 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:48:37 left step 10000/30000 loss 0.07597 data_wait_ms 0.03939 grad_norm 0.04776 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.53 step_time_ms 132.5 samples_per_sec 482.8 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:44:10 left step 12000/30000 loss 0.06271 data_wait_ms 0.03939 grad_norm 0.04445 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.53 step_time_ms 132.6 samples_per_sec 482.8 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:38:55 left step 14000/30000 loss 0.06348 data_wait_ms 0.03958 grad_norm 0.04270 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.53 step_time_ms 132.6 samples_per_sec 482.8 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:35:21 left step 16000/30000 loss 0.07443 data_wait_ms 0.03920 grad_norm 0.04067 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.54 step_time_ms 132.5 samples_per_sec 482.8 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:31:33 left step 18000/30000 loss 0.06137 data_wait_ms 0.03897 grad_norm 0.04020 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.54 step_time_ms 132.5 samples_per_sec 482.9 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:26:30 left step 20000/30000 loss 0.07190 data_wait_ms 0.03934 grad_norm 0.03898 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.54 step_time_ms 132.6 samples_per_sec 482.8 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:22:32 left step 22000/30000 loss 0.06853 data_wait_ms 0.03947 grad_norm 0.03862 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.54 step_time_ms 132.6 samples_per_sec 482.8 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:17:40 left step 24000/30000 loss 0.06056 data_wait_ms 0.03947 grad_norm 0.03657 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.54 step_time_ms 132.5 samples_per_sec 482.8 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:13:16 left step 26000/30000 loss 0.06012 data_wait_ms 0.03952 grad_norm 0.03679 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.54 step_time_ms 132.6 samples_per_sec 482.8 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:08:50 left step 28000/30000 loss 0.05736 data_wait_ms 0.03922 grad_norm 0.03549 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.54 step_time_ms 132.6 samples_per_sec 482.8 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 0:04:25 left step 30000/30000 loss 0.06149 data_wait_ms 0.03932 grad_norm 0.03530 compiles 1.000 cache_hits 0 cache_misses 0 peak_memory_gib 16.54 step_time_ms 132.5 samples_per_sec 482.9 mfu 29.5% hlo_flops_util 31.3% hfu 31.3% accepted 100.0% learning_rate 2.000e-04 accepted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 attempted/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 6,552 chunks_per_record/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 truncated/hf:pranked03/flowers-blip-captions@b45984ffbef8596a8b6ac2ac8959f2f8998049bb:None:train 0 Trained 30000 steps in 1:07:30: first step after 62.35 s, then 7.5 step/s 98.2% of the wall time in steps, final loss 0.06149
import json
rows = [json.loads(line) for line in open(f"{RUN_DIR}/tracking/scalars.jsonl")]rows = [row for row in rows if "train/loss" in row["scalars"]]print("loss:", ", ".join(f"{row['scalars']['train/loss']:.4f} at {row['step']}" for row in rows))figures.curve({"loss": [row["scalars"]["train/loss"] for row in rows]}, x=[row["step"] for row in rows], x_label="step", y_label="loss")loss: 0.0658 at 2000, 0.0689 at 4000, 0.0645 at 6000, 0.0728 at 8000, 0.0760 at 10000, 0.0627 at 12000, 0.0635 at 14000, 0.0744 at 16000, 0.0614 at 18000, 0.0719 at 20000, 0.0685 at 22000, 0.0606 at 24000, 0.0601 at 26000, 0.0574 at 28000, 0.0615 at 30000
The loss looks almost exactly like notebook 02’s: about 0.066 at step 2,000, and then jumping between about 0.057 and 0.076 for the rest of the run. The loss can’t show what the caption adds; the samples below can.
Classifier-free guidance
Section titled “Classifier-free guidance”Because some captions were blanked during training, one model has actually learned two things: how to denoise a flower given its caption, and how to denoise a flower with no caption at all. At sampling time, we ask it both questions at every step and push the answer away from the uncaptioned one:
where:
- is the model’s prediction given the caption,
- is its prediction given the empty caption,
- is the guidance scale.
With this is just the plain captioned prediction. The difference is basically the direction the caption pulls the image in, and the scale says how far we follow it. Larger scales follow the caption harder and give up some variety in return.
This is classifier-free guidance, from the paper “Classifier-Free Diffusion Guidance” by Ho and Salimans (2022), and CFG(scale) in Dew computes it.
But why “classifier-free”? Well, the guidance that came before it, in the paper Diffusion Models beat GANs on image synthesis, needed a separate classifier trained on noisy images, whose gradient pushed the sample towards the class we wanted. Here the same model plays both roles, so there is no classifier at all.
objective.pipeline(state) packages the trained model, the process, the text encoder and the EMA weights into a TextToImage pipeline. Calling it with a list of prompts returns the images.
pipe = objective.pipeline(state)
rows = []for scale in GUIDANCE_SCALES: out = pipe(PROMPTS * 4, steps=SAMPLE_STEPS, guidance=CFG(scale), solver=EulerAncestral(), key=1) rows.append(np.asarray(out.images))Each row below is one prompt, drawn from four different starting noises, and the picture steps through the guidance scales, 1, then 3, then 6.
# The pipeline returns the prompts in turn; group each prompt's images into a row.frames = [uint8_pixels(images.reshape(4, len(PROMPTS), *images.shape[1:]).swapaxes(0, 1) .reshape(images.shape)) for images in rows]figures.images(frames, columns=4, size=96, rows=PROMPTS, captions=[f"guidance {scale:g}" for scale in GUIDANCE_SCALES])At scale 1 the samples only loosely follow the prompts: one “yellow sunflower” is purple and another orange, one “white daisy” is peach, and the first “purple iris” is a green blur. At scale 3 every image has the colour its prompt asks for, and the sunflowers and daisies have the right shape too, dark or yellow centres and all. The irises are purple but look more like petunias. Scale 6 pushes further: the colours get more saturated, most of the sunflowers’ centres darker, and the four images in each row look more and more alike. The second sunflower loses its centre altogether in flat yellow. That’s what guidance gives up in return: variety, and some of the detail.
Where to go next
Section titled “Where to go next”recipes/diffusion/train.py runs this setup from the command line and writes run.json next to the checkpoints, so that TextToImage.from_run(directory) can rebuild the pipeline later. For larger images, StableDiffusionVAE in dew.nn.autoencoders.sd_vae lets the model denoise small latents instead of pixels, which is the idea behind the paper “High-Resolution Image Synthesis with Latent Diffusion Models” by Rombach et al. (2022). Notebook 04 compares the samplers.