Basics of Diffusion based generative models from Scratch
Please bear with me at times and read through the entire notebook thoroughly. I have tried to explain the concepts in the simplest way possible but some concepts are inherently complex and require a bit of mathematical understanding. I have tried to simplify the math as much as possible and have provided links to the original papers for further reading.
Diffusion and Score Based Models
Section titled “Diffusion and Score Based Models”Introduction
Section titled “Introduction”In recent years, several methods have arisen for generative modeling of data such as Variational Autoencoders (VAEs), Generative Adversarial Networks (GANs), Autoregressive models, and Normalizing Flows just to name a few. But none really broke the internet quite like the way diffusion models did. Reason being, none could generate images with the same level of quality and diversity as diffusion models. GANs did come close but they were really painful to train and were really unstable. Autoregressive models were too slow. Normalizing Flows were too computationally expensive. VAEs were too blurry.
The Idea
Section titled “The Idea”The idea was that trying to train a model to generate an image in a single step was too hard, so why not instead let it model it in multiple steps? There were two approaches that were proposed to do this: Score based models and diffusion models. But in both, the idea was that our data could be thought of as a special complicated distribution that we could sample from, and that we could start from a simple distribution such as a normal distribution and find a way to transform samples from this simple distribution into those from the data distribution (or as close as we can get).
Therefore, it’s a problem of going from a random high entropy point in the input space to a low entropy point where the scale of entropy is defined as a continuous mapping from the data distribution to the simple normal distribution.
Thus we would start from a random sample from normal distribution aka some random Gaussian noise, and iteratively try to find our way to gradually transform it into some sample that looks like it came from the data distribution. We do this in some defined time steps . Thus we define a process of starting from a random noise sample at time and gradually removing the noise and finding some useful data at each time step until we reach where we have ideally no noise and only data left. (Time runs backwards here because the forward, noising process goes from data at to noise at ; the rest of this notebook uses that convention.)
How do we do this?
Section titled “How do we do this?”Score Based Models
Section titled “Score Based Models”The score based models tackle this by trying to estimate the gradient of the log-likelihood of the data distribution (don’t get scared, just bear it for now) at each time step. This gradient basically points us in the direction of the data distribution from the current sample, and we can then take a step in that direction to remove some noise in kind of like a gradient descent style process. This process is repeated until we reach the final time step where we have ideally no noise left. This is the basic idea behind score based models. The noising and denoising processes are defined by stochastic differential equations (SDEs). We will learn about these formulations in later sections.
Diffusion Models
Section titled “Diffusion Models”The way diffusion models work is by simply training a model that, given an intermediate noised sample, tries to denoise the sample a bit at each discrete time step in an image-to-image manner rather than explicitly learning to estimate the gradient. The time step itself can also be provided to the model to assist it in the process. It has been shown that both score based and diffusion principles are equivalent, they only really differ in the way their objective loss functions are defined.
In the original DDPM paper, diffusion processes were defined as a Markov chain where in forward time, we would go from a sample with some amount of noise to a sample with more noise, and the reverse of this in reverse time, in discrete time steps . This has been cited as the only major difference between score based models and diffusion models but if you think about it, they are basically the same thing. In discrete time steps, we have Markov chains of states of DDPM, in continuous time steps (i.e., infinitesimal time steps), we have stochastic differential equations (SDEs) of score based models.
And thus, more recent papers use the more generalized approach of formulating the processes as SDEs and ODEs as finding solutions i.e., integrating differential equations is a very well studied field with several solvers available.
But we need to train the model as well right? and what exactly is this diffusion process? Well it’s pretty intuitive from what we have understood so far: We can simply take samples from our actual data distribution, and construct a set of samples with gradually increasing noise levels by adding Gaussian noise at each time step, until we basically have a set of samples that are just noise. This gradual noising process is very similar to the process of diffusion from thermodynamics where particles move from low entropy to high entropy, hence the name. Score based models just stumbled upon the same idea independently.
The Diffusion Process
Section titled “The Diffusion Process”The diffusion process (the forward diffusion process) can be described as a stochastic differential equation (SDE):
where:
- is the infinitesimal change in the data sample at time ,
- is the drift term, which represents the deterministic part of the process,
- is the diffusion term, which represents the stochastic part of the process,
- is the infinitesimal increment of a Wiener process (or standard Brownian motion).
Don’t get overwhelmed by the math, it’s just a fancy way of saying that the data sample at time is a function of the data sample at time and some noise. The drift term is the deterministic part of the process that tells us how the data sample changes over time, and the diffusion term is the stochastic part that tells us how the noise affects the data sample. We would soon look at how we can model these terms in the form of a defined noise schedule.
Given the above forward-time diffusion process SDE, we can also formulate the reverse-time diffusion process SDE (as found by Anderson (1982)):
where:
- is the infinitesimal change in the data sample at time ,
- is the drift term from the forward process,
- is the diffusion term from the forward process,
- is the gradient of the log-density of the data at time ,
- is the infinitesimal increment of a reverse-time Wiener process (or standard Brownian motion).
Again, it’s nothing fancy and quite easy to understand when we will later try implementing these.
We will thus implement our forward diffusion process via a Noise schedule and our reverse diffusion process via a Sampler, which is essentially a SDE (or ODE) solver that finds a solution to the above differential equation.
An important thing to note is that every reverse-time diffusion SDE has a corresponding probability flow ODE with the same marginal densities. This essentially means that we can find solutions to go from a noised sample to a denoised sample in a deterministic manner following an ODE as well. This is the difference between ancestral (like DDPM, Euler-A) and non ancestral samplers (like DDIM, Euler etc). Ancestral samplers solve the SDE, while non-ancestral samplers solve the ODE. The Probability Flow ODE is given by:
where:
- is the time derivative of the data sample ,
- is the drift term from the forward process,
- is the diffusion term from the forward process,
- is the gradient of the log-density of the data at time .
This ODE provides a deterministic trajectory that samples from the same distribution as the original stochastic process.
If we try to model this term directly as a neural network , we get what is known as a score based model. But if we try to think of it as a kind of noise term , we get a diffusion model. Both of these are equivalent but just for brevity, we shall discuss what the original DDPM paper did.
DDPM’s original formulation for Diffusion
Section titled “DDPM’s original formulation for Diffusion”The DDPM paper defined the forward diffusion process as a Markov chain of conditional distributions :
where:
- is the data sample at time ,
- is the data sample at time ,
- is the noise level at time ,
- is the identity matrix.
And the reverse diffusion process was defined as:
where:
- is the data sample at time ,
- is the data sample at time ,
- is the mean of the reverse distribution,
- is the covariance of the reverse distribution.
The noise level was defined as a function of time and the number of time steps and the was basically our neural network that we would train to denoise the sample at time , with the weights being .
But remember that this is just a discretized version of the diffusion differential equations we discussed above, and latest research papers use the more generalized version formulated as the SDEs and ODEs rather than the Markov chain. But just for the sake of accuracy, we shall implement DDPM and DDIM with this Markov chain formulation.
These are basically defined via noise schedules in Dew and thus using a discretized noise schedule vs a continuous one is what differentiates DDPM/DDIM from the latest techniques (and also score based methods but along with the loss function which is just slightly different).
You can read more about the generalization of the diffusion process in the paper Score based generative modeling through stochastic differential equations
The Intuition
Section titled “The Intuition”Now that we have formulated all the maths in the previous sections, let’s try to build the intuition of what we are trying to do. As we discussed briefly in the The Idea section, every image (or data sample) can be thought of as a point in a high dimensional space. For example, a 64x64 image is basically an array of 64x64x3=12288 pixels. If you treat this array as a vector, it’s a vector in a 12288 dimensional space. Now, it’s hard for our monkey brains to imagine anything beyond a 4D space, let alone a 12288D one, but bear with me. A noisy image is thus also a similar point in this same high dimensional space.
Now, in the forward diffusion process, all we are doing is trying to go from the point of low entropy i.e., the data sample (image) to a point of high entropy i.e., a complete noise sample . Every point we encounter along the trajectory we take is our intermediate noisy image . Here is a simple visualization of the process:
The denoising diffusion model that we would train (or the score based model, depending on the formulation) is what guides us along the trajectory of the reverse diffusion process. It shows the way (the direction) from any noisy image towards a cleaner image sample. The samplers then take steps along this direction to reach the final denoised image. In the above visualization, the green curve does not truly represent the reverse path though because it can be that our denoising model takes us on a different path towards a different image in the vicinity of the starting noise, but it would be a clean image of low entropy nonetheless.
Install Dependencies
Section titled “Install Dependencies”You must install a proper version of Dew and JAX, and set up the GPU for this notebook to work. Go through the official Dew installation steps
The notebook only works for JAX version 0.11.2
%pip install -q "dewml[cuda13,streaming] @ git+https://github.com/AshishKumar4/dew@71b43f2dc070cf289e1fbaf6177e484215d4ee1a" --extra-index-url https://pypi.dewml.dev/simpleinstall log
Installing build dependencies ... done Getting requirements to build wheel ... done Preparing metadata (pyproject.toml) ... done Building wheel for dewml (pyproject.toml) ... done
Imports
Section titled “Imports”import tqdmfrom flax import linen as nnimport jaxfrom typing import Dict, Callable, Sequence, Any, Unionfrom dataclasses import fieldimport jax.numpy as jnpfrom dew.data import Hub, Loading, presetsfrom dew.data.compile import compile as compile_datafrom dew.data.dataset import DataPartition
import matplotlib.pyplot as pltfrom flax.training import train_state # Useful dataclass to keep train stateimport optaxfrom flax import struct # Flax dataclassesimport timeimport osfrom datetime import datetimeimport orbax.checkpoint as ocpimport functoolsimport numpy as npSome Important Utils
Section titled “Some Important Utils”normalizeImage = lambda x: (x - 127.5) / 127.5denormalizeImage = lambda x: (x + 1.0) * 127.5
def plotImages(imgs, fig_size=(8, 8), dpi=100): fig = plt.figure(figsize=fig_size, dpi=dpi) imglen = imgs.shape[0] for i in range(imglen): plt.subplot(fig_size[0], fig_size[1], i + 1) plt.imshow(np.clip(np.asarray(denormalizeImage(imgs[i, :, :, :])), 0, 255).astype(np.uint8)) plt.axis("off") plt.show()
class RandomClass(): def __init__(self, rng: jax.random.PRNGKey): self.rng = rng
def get_random_key(self): self.rng, subkey = jax.random.split(self.rng) return subkey
def get_sigmas(self, steps): return jnp.tan(self.theta_min + steps * (self.theta_max - self.theta_min)) / self.kappa
def reset_random_key(self): self.rng = jax.random.PRNGKey(42)
class MarkovState(struct.PyTreeNode): pass
class RandomMarkovState(MarkovState): rng: jax.random.PRNGKey
def get_random_key(self): rng, subkey = jax.random.split(self.rng) return RandomMarkovState(rng), subkeyData pipeline
Section titled “Data pipeline”This notebook uses Oxford Flowers records from the Hugging Face Hub. An Images recipe selects the image size and augmentation; load compiles it after explicitly preparing the Hub Arrow split.
def get_dataset(data_name="oxford_flowers102", batch_size=64, image_scale=256): hub_names = {"oxford_flowers102": "pranked03/flowers-blip-captions"} images = Hub(hub_names.get(data_name, data_name)).shuffle(seed=0).repeat() data = compile_data(images.pipe(presets.images, image_scale, batch=batch_size, rejects=False), read="memory", provision=True, loading=Loading(workers=0, threads=16, read_buffer=64))
def batches(): stream = data.train(DataPartition()) try: for batch in stream: image = batch["image"].astype(np.float32) / 127.5 - 1.0 yield np.clip(image, -1.0, 1.0) finally: stream.close()
return batches(), data.recordsNoise Schedulers
Section titled “Noise Schedulers”A Noise schedule governs how noise is added in the forward diffusion steps. Basically, given a time step , it returns the signal rate and noise rate with which to scale the initial data sample and Gaussian noise as given by the equation:
where is the data sample at time , is the initial data sample, and is the Gaussian noise, and and are the signal and noise rates at time respectively.
In variance preserving diffusion, the noise schedule is such that the variance of the data sample remains constant across time steps, as long as the data itself has unit variance. This basically means the following:
The idea is that with increasing time step , the signal rate decreases and the noise rate increases, decreasing the of the initial data sample and increasing the amount of noise, slowly diffusing the data sample smoothly to the target normal distribution.
Of course there are many ways to schedule the noise, and the constraint of variance preserving isn’t the only way to go about it either. There are Variance exploding and variance preserving schedules as well. In this notebook, we will be looking at the variance preserving noise schedules only.
class NoiseScheduler(): def __init__(self, timesteps, dtype=jnp.float32, clip_min=-1.0, clip_max=1.0, *args, **kwargs): self.max_timesteps = timesteps self.dtype = dtype self.clip_min = clip_min self.clip_max = clip_max if type(timesteps) == int and timesteps > 1: timestep_generator = lambda rng, batch_size, max_timesteps = timesteps: jax.random.randint(rng, (batch_size,), 0, max_timesteps) else: timestep_generator = lambda rng, batch_size, max_timesteps = timesteps: jax.random.uniform(rng, (batch_size,), minval=0, maxval=max_timesteps) self.timestep_generator = timestep_generator
def generate_timesteps(self, batch_size, state:RandomMarkovState) -> tuple[jnp.ndarray, RandomMarkovState]: state, rng = state.get_random_key() timesteps = self.timestep_generator(rng, batch_size, self.max_timesteps) return timesteps, state
def get_weights(self, steps): raise NotImplementedError
def reshape_rates(self, rates:tuple[jnp.ndarray, jnp.ndarray], shape=(-1, 1, 1, 1)) -> tuple[jnp.ndarray, jnp.ndarray]: signal_rates, noise_rates = rates signal_rates = jnp.reshape(signal_rates, shape) noise_rates = jnp.reshape(noise_rates, shape) return signal_rates, noise_rates
def get_rates(self, steps, shape=(-1, 1, 1, 1)) -> tuple[jnp.ndarray, jnp.ndarray]: raise NotImplementedError
def add_noise(self, images, noise, steps) -> jnp.ndarray: signal_rates, noise_rates = self.get_rates(steps) return signal_rates * images + noise_rates * noise
def remove_all_noise(self, noisy_images, noise, steps, clip_denoised=True, rates=None): signal_rates, noise_rates = self.get_rates(steps) x_0 = (noisy_images - noise * noise_rates) / signal_rates return x_0
def transform_inputs(self, x, steps): return x, steps
def get_max_variance(self): alpha_n, sigma_n = self.get_rates(self.max_timesteps) variance = jnp.sqrt(alpha_n ** 2 + sigma_n ** 2) return varianceCosine Noise Scheduler
Section titled “Cosine Noise Scheduler”Cosine Schedule is one of the most widely used noise schedules. It’s a Variance Preserving noise schedule and can either be parameterized directly as the functions and defined directly as
and ,
which is simple to understand as , which satisfies the variance preserving constraint, or in terms of functions depending on a single time dependent function as defined in the original DDPM paper and many more places.
Why cosine schedule? The original DDPM paper used a linear beta schedule, but cosine schedule was shown to be much more effective at training the denoising model, introduced in the paper Improved Denoising Diffusion Probabilistic Models
You see, as described before, In the original DDPM Paper, the forward diffusion step is defined as
where is the forward diffusion step. It’s the conditional distribution of the data sample given the previous data sample , and the equation states that is normally distributed with mean and variance .
In human speak, the forward diffusion is phrased as:
where is the data sample at time , is the previous data sample, and is the Gaussian noise at time . Notice that is phrased in terms of instead of the initial data sample as we did in the ‘Noise Schedulers’ section.
To convert this formulation to the type we discussed above
,
we can find that our signal rate and noise rate can be defined as
Note: The thing is, the maths in DDPM paper use the symbol for a different thing, as an intermediate to denote the value , so just be careful with the notation. We use to denote the signal rate in this notebook.
We shall thus define the Cosine Noise Scheduler twice: Once in terms of the parameter aka the way it’s formulated in the DDPM paper, and then in a more simplified way directly in terms of and .
Cosine Noise Scheduler in terms of
Section titled “Cosine Noise Scheduler in terms of β(t)\beta(t)β(t)”def cosine_beta_schedule(timesteps, start_angle=0.008, end_angle=0.999): ts = np.linspace(0, 1, timesteps + 1, dtype=np.float64) alphas_bar = np.cos((ts + start_angle) / (1 + start_angle) * np.pi /2) ** 2 alphas_bar = alphas_bar/alphas_bar[0] betas = 1 - (alphas_bar[1:] / alphas_bar[:-1]) return np.clip(betas, 0, end_angle)
class CosineDiscreteNoiseSchedule(NoiseScheduler): """ Variance Preserving Noise Scheduler signal_rate**2 + noise_rate**2 = 1 """ def __init__(self, timesteps, beta_start=0.008, beta_end=0.999, *args, **kwargs): super().__init__(timesteps, *args, **kwargs) betas = cosine_beta_schedule(timesteps, beta_start, beta_end) self.betas = jnp.array(betas) alphas = jnp.cumprod(jnp.sqrt(1 - betas)) self.alphas = alphas # This is not the same alpha as mentioned in the original paper! self.sigmas = jnp.sqrt(1 - alphas ** 2)
def get_rates(self, steps, shape=(-1, 1, 1, 1)): steps = jnp.astype(steps, jnp.int32) alpha = self.alphas[steps] sigma = self.sigmas[steps] alpha = jnp.reshape(alpha, shape) sigma = jnp.reshape(sigma, shape) return alpha, sigma
def get_weights(self, steps, shape=(-1, 1, 1, 1)): alpha, sigma = self.get_rates(steps, shape) return 1 / (1 + ((alpha**2) / (sigma**2)))Cosine Noise Scheduler in terms of and
Section titled “Cosine Noise Scheduler in terms of α(t)\alpha(t)α(t) and σ(t)\sigma(t)σ(t)”class ContinuousNoiseScheduler(NoiseScheduler): """ General Continuous Noise Scheduler """ def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs)
class CosineContinuousNoiseScheduler(ContinuousNoiseScheduler): def get_rates(self, steps, shape=(-1, 1, 1, 1)) -> tuple[jnp.ndarray, jnp.ndarray]: steps = jnp.float32(steps) signal_rates = jnp.cos((jnp.pi * steps) / (2 * self.max_timesteps)) noise_rates = jnp.sin((jnp.pi * steps) / (2 * self.max_timesteps)) return self.reshape_rates((signal_rates, noise_rates), shape=shape)
def get_weights(self, steps): alpha, sigma = self.get_rates(steps, shape=()) return 1 / (1 + (alpha ** 2 / sigma ** 2))Let’s play with our Noise Schedulers
Section titled “Let’s play with our Noise Schedulers”Visualization of and with linear time steps
Section titled “Visualization of α(t)\alpha(t)α(t) and σ(t)\sigma(t)σ(t) with linear time steps ttt”to prove that both are equivalent.
maxT = 1000timesteps = jnp.linspace(0, maxT, 1000)get_variance = lambda rates: rates[0] ** 2 + rates[1] ** 2"""Lets first plot the signal and noise rates for the discrete cosine noise schedule"""print("Discrete Cosine Noise Schedule")schedule = CosineDiscreteNoiseSchedule(maxT)plt.plot([schedule.get_rates(i, shape=()) for i in timesteps], label=["signal", "noise"])plt.plot([get_variance(schedule.get_rates(i, shape=())) for i in timesteps], label="variance")plt.plot([schedule.get_weights(i, shape=()) for i in timesteps], label="weights")plt.show()
print("Continuous Cosine Noise Schedule")schedule = CosineContinuousNoiseScheduler(maxT)plt.plot([schedule.get_rates(i, shape=()) for i in timesteps], label=["signal", "noise"])plt.plot([get_variance(schedule.get_rates(i, shape=())) for i in timesteps], label="variance")plt.plot([schedule.get_weights(i) for i in timesteps], label="weights")plt.show()Discrete Cosine Noise Schedule

Continuous Cosine Noise Schedule

Now let’s test the noise scheduler on some simple images
Section titled “Now let’s test the noise scheduler on some simple images”We shall first sample some images from our dataset, then apply some noise to them using the noise scheduler, and then visualize the images at different time steps to see how the noise is added, and then how it is removed.
# Visualize adding some noise to some sample images and then removing itdata, _ = get_dataset("oxford_flowers102", batch_size=8, image_scale=64)images = next(iter(data))plotImages(images)noise_level = 500noise_level_max = 1000schedule = CosineContinuousNoiseScheduler(noise_level_max)
alpha_t, sigma_t = schedule.get_rates(noise_level)print(f"Rates: alpha={alpha_t}, sigma={sigma_t}")noise = jax.random.normal(jax.random.PRNGKey(4), shape=images.shape, dtype=jnp.float64)noisy_images = schedule.add_noise(images, noise, noise_level)print("Noisy Images")plotImages(noisy_images)print("Reconstructed Images")reconstructed_images = schedule.remove_all_noise(noisy_images, noise, noise_level)plotImages(reconstructed_images)stderr
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads. WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.

Rates: alpha=[[[[0.70710677]]]], sigma=[[[[0.7071068]]]] /tmp/ipykernel_21774/668185268.py:11: UserWarning: Explicitly requested dtype float64 is not available, and will be truncated to dtype float32. To enable more dtypes, set the jax_enable_x64 configuration option or the JAX_ENABLE_X64 shell environment variable. See https://github.com/jax-ml/jax#current-gotchas for more. noise = jax.random.normal(jax.random.PRNGKey(4), shape=images.shape, dtype=jnp.float64) Noisy Images

Reconstructed Images

Modeling
Section titled “Modeling”Neural network comes into play for learning to estimate the gradient in the reverse diffusion process as described above:
Score based models estimate the gradient directly as
where is the score model, and are the parameters of the score model. But we can also model it in terms of a neural network that predicts the actual noise at time as well
where is the denoising model, and are the parameters of the denoising model. The minus sign comes from the forward process: since , the score of is .
But why predict noise? Well we could also predict given a noisy sample, the original sample. That works too, but the first few original diffusion papers used the epsilon formulation and they argued that it was easier for a model to learn to predict the noise in an image rather than the original image itself. And anyway, once we have the noise, we can just subtract it from the noisy image and undo the scaling to get the original image, .
In this notebook, we will be using this denoising model for solving the reverse diffusion process but there are other parameterizations as well that work really well. We will look into them in some other examples.
Denoising Model
Section titled “Denoising Model”We will be using a simple UNet architecture for our denoising model. The UNet architecture is a popular architecture for image segmentation tasks, but it proves to be quite useful for denoising tasks as well. The architecture is quite simple and consists of an encoder and a decoder connected by skip connections. The encoder downsamples the input image and extracts features, while the decoder upsamples the features and reconstructs the image. The skip connections help in preserving spatial information and gradients.
We will also add some self attention blocks at each level of the UNet to help the model focus on finding clues in the image that are important for denoising.
The encoder and decoder blocks will be made of several resolution levels, and on each level will be made of several residual blocks, an attention block and a downsampling or upsampling block (for encoder and decoder respectively). There would also be a middle block of a single level of the same design but without downsampling or upsampling.
As our model takes as input the time step as well, we will feed the model an additional input of the time step which would be added to each residual block after being embedded in a certain way and transformed and projected by few linear layers. The time step would be embedded in a way similar to how positional encodings are added in transformers.
We may use some separable convolutions in our model to reduce the number of parameters and thus the memory required. If these models do not compile on your machine, try switching all blocks to separable convs and then further reducing the resolution wise feature level sizes.
The basic building blocks
Section titled “The basic building blocks”# Kernel initializer to usedef kernel_init(scale): scale = max(scale, 1e-10) return nn.initializers.variance_scaling(scale=scale, mode="fan_avg", distribution="truncated_normal")
class NormalAttention(nn.Module): """ Simple implementation of the normal attention. """ query_dim: int heads: int = 4 dim_head: int = 64 dtype: Any = jnp.float32 precision: Any = jax.lax.Precision.HIGHEST use_bias: bool = True kernel_init: Callable = lambda : kernel_init(1.0)
def setup(self): inner_dim = self.dim_head * self.heads dense = functools.partial( nn.DenseGeneral, features=[self.heads, self.dim_head], axis=-1, precision=self.precision, use_bias=self.use_bias, kernel_init=self.kernel_init(), dtype=self.dtype ) self.query = dense(name="to_q") self.key = dense(name="to_k") self.value = dense(name="to_v")
self.proj_attn = nn.DenseGeneral( self.query_dim, axis=(-2, -1), precision=self.precision, use_bias=self.use_bias, dtype=self.dtype, name="to_out_0", kernel_init=self.kernel_init() # kernel_init=jax.nn.initializers.xavier_uniform() )
@nn.compact def __call__(self, x, context=None): # x has shape [B, H, W, C]; attention runs over all H * W positions, so flatten them orig_x_shape = x.shape if len(x.shape) == 4: x = x.reshape((x.shape[0], x.shape[1] * x.shape[2], x.shape[3])) context = x if context is None else context if len(context.shape) == 4: context = context.reshape((context.shape[0], context.shape[1] * context.shape[2], context.shape[3])) query = self.query(x) key = self.key(context) value = self.value(context)
hidden_states = nn.dot_product_attention( query, key, value, dtype=self.dtype, broadcast_dropout=False, dropout_rng=None, precision=self.precision ) proj = self.proj_attn(hidden_states) return proj.reshape(orig_x_shape)
class AttentionBlock(nn.Module): heads: int = 4 dim_head: int = 32 use_linear_attention: bool = True dtype: Any = jnp.float32 precision: Any = jax.lax.Precision.HIGH use_projection: bool = False
@nn.compact def __call__(self, x): inner_dim = self.heads * self.dim_head B, H, W, C = x.shape normed_x = nn.RMSNorm(epsilon=1e-5, dtype=self.dtype)(x) projected_x = NormalAttention( query_dim=C, heads=self.heads, dim_head=self.dim_head, name=f'Attention', precision=self.precision, use_bias=False, )(normed_x) out = x + projected_x return out
class TimeEmbedding(nn.Module): features:int nax_positions:int=10000
def setup(self): half_dim = self.features // 2 emb = jnp.log(self.nax_positions) / (half_dim - 1) emb = jnp.exp(-emb * jnp.arange(half_dim, dtype=jnp.float32)) self.embeddings = emb
def __call__(self, x): x = jax.lax.convert_element_type(x, jnp.float32) emb = x[:, None] * self.embeddings[None, :] emb = jnp.concatenate([jnp.sin(emb), jnp.cos(emb)], axis=-1) return emb
class TimeProjection(nn.Module): features:int activation:Callable=jax.nn.gelu
@nn.compact def __call__(self, x): x = nn.DenseGeneral(self.features, kernel_init=kernel_init(1.0))(x) x = self.activation(x) x = nn.DenseGeneral(self.features, kernel_init=kernel_init(1.0))(x) x = self.activation(x) return x
class SeparableConv(nn.Module): features:int kernel_size:tuple=(3, 3) strides:tuple=(1, 1) use_bias:bool=False kernel_init:Callable=kernel_init(1.0) padding:str="SAME"
@nn.compact def __call__(self, x): in_features = x.shape[-1] depthwise = nn.Conv( features=in_features, kernel_size=self.kernel_size, strides=self.strides, kernel_init=self.kernel_init, feature_group_count=in_features, use_bias=self.use_bias, padding=self.padding )(x) pointwise = nn.Conv( features=self.features, kernel_size=(1, 1), strides=(1, 1), kernel_init=self.kernel_init, use_bias=self.use_bias )(depthwise) return pointwise
class ConvLayer(nn.Module): conv_type:str features:int kernel_size:tuple=(3, 3) strides:tuple=(1, 1) kernel_init:Callable=kernel_init(1.0)
def setup(self): if self.conv_type == "conv": self.conv = nn.Conv( features=self.features, kernel_size=self.kernel_size, strides=self.strides, kernel_init=self.kernel_init, ) elif self.conv_type == "separable": self.conv = SeparableConv( features=self.features, kernel_size=self.kernel_size, strides=self.strides, kernel_init=self.kernel_init, )
def __call__(self, x): return self.conv(x)
class Upsample(nn.Module): features:int scale:int activation:Callable=jax.nn.swish
@nn.compact def __call__(self, x, residual=None): out = x # out = PixelShuffle(scale=self.scale)(out) B, H, W, C = x.shape out = jax.image.resize(x, (B, H * self.scale, W * self.scale, C), method="nearest") out = ConvLayer( "conv", features=self.features, kernel_size=(3, 3), strides=(1, 1), )(out) if residual is not None: out = jnp.concatenate([out, residual], axis=-1) return out
class Downsample(nn.Module): features:int scale:int activation:Callable=jax.nn.swish
@nn.compact def __call__(self, x, residual=None): out = ConvLayer( "conv", features=self.features, kernel_size=(3, 3), strides=(2, 2) )(x) if residual is not None: if residual.shape[1] > out.shape[1]: residual = nn.avg_pool(residual, window_shape=(2, 2), strides=(2, 2), padding="SAME") out = jnp.concatenate([out, residual], axis=-1) return out
class ResidualBlock(nn.Module): conv_type:str features:int kernel_size:tuple=(3, 3) strides:tuple=(1, 1) padding:str="SAME" activation:Callable=jax.nn.swish direction:str=None res:int=2 norm_groups:int=8 kernel_init:Callable=kernel_init(1.0)
@nn.compact def __call__(self, x:jax.Array, temb:jax.Array, extra_features:jax.Array=None): residual = x out = nn.GroupNorm(self.norm_groups)(x) out = self.activation(out)
out = ConvLayer( self.conv_type, features=self.features, kernel_size=self.kernel_size, strides=self.strides, kernel_init=self.kernel_init, name="conv1" )(out)
temb = nn.DenseGeneral(features=self.features, name="temb_projection")(temb) temb = jnp.expand_dims(jnp.expand_dims(temb, 1), 1) out = out + temb
out = nn.GroupNorm(self.norm_groups)(out) out = self.activation(out)
out = ConvLayer( self.conv_type, features=self.features, kernel_size=self.kernel_size, strides=self.strides, kernel_init=self.kernel_init, name="conv2" )(out)
if residual.shape != out.shape: residual = ConvLayer( self.conv_type, features=self.features, kernel_size=(1, 1), strides=1, kernel_init=self.kernel_init, name="residual_conv" )(residual) out = out + residual
out = jnp.concatenate([out, extra_features], axis=-1) if extra_features is not None else out
return outMain UNet Model
Section titled “Main UNet Model”class Unet(nn.Module): emb_features:int=64*4, feature_depths:list=[64, 128, 256, 512], attention_configs:list=[{"heads":8}, {"heads":8}, {"heads":8}, {"heads":8}], num_res_blocks:int=2, num_middle_res_blocks:int=1, activation:Callable = jax.nn.swish norm_groups:int=8 major_conv_type:str="conv" mid_conv_type:str="conv"
@nn.compact def __call__(self, x, temb): temb = TimeEmbedding(features=self.emb_features)(temb) temb = TimeProjection(features=self.emb_features)(temb) feature_depths = self.feature_depths attention_configs = self.attention_configs
conv_type = "conv" up_conv_type = down_conv_type = self.major_conv_type middle_conv_type = self.mid_conv_type
x = ConvLayer( conv_type, features=self.feature_depths[0], kernel_size=(3, 3), strides=(1, 1), kernel_init=kernel_init(1.0) )(x) downs = [x]
# Downscaling blocks for i, (dim_out, attention_config) in enumerate(zip(feature_depths, attention_configs)): dim_in = x.shape[-1] # dim_in = dim_out for j in range(self.num_res_blocks): x = ResidualBlock( down_conv_type, name=f"down_{i}_residual_{j}", features=dim_in, kernel_init=kernel_init(1.0), kernel_size=(3, 3), strides=(1, 1), activation=self.activation, norm_groups=self.norm_groups )(x, temb) if attention_config is not None and j == self.num_res_blocks - 1: # Apply attention only on the last block x = AttentionBlock(heads=attention_config['heads'], dim_head=dim_in // attention_config['heads'], name=f"down_{i}_attention_{j}")(x) downs.append(x) if i != len(feature_depths) - 1: x = Downsample( features=dim_out, scale=2, activation=self.activation, name=f"down_{i}_downsample" )(x)
# Middle Blocks middle_dim_out = self.feature_depths[-1] middle_attention = self.attention_configs[-1] for j in range(self.num_middle_res_blocks): x = ResidualBlock( middle_conv_type, name=f"middle_res1_{j}", features=middle_dim_out, kernel_init=kernel_init(1.0), kernel_size=(3, 3), strides=(1, 1), activation=self.activation, norm_groups=self.norm_groups )(x, temb) if middle_attention is not None and j == self.num_middle_res_blocks - 1: # Apply attention only on the last block x = AttentionBlock(heads=attention_config['heads'], dim_head=middle_dim_out // attention_config['heads'], use_linear_attention=False, name=f"middle_attention_{j}")(x) x = ResidualBlock( middle_conv_type, name=f"middle_res2_{j}", features=middle_dim_out, kernel_init=kernel_init(1.0), kernel_size=(3, 3), strides=(1, 1), activation=self.activation, norm_groups=self.norm_groups )(x, temb)
# Upscaling Blocks for i, (dim_out, attention_config) in enumerate(zip(reversed(feature_depths), reversed(attention_configs))): for j in range(self.num_res_blocks): x = jnp.concatenate([x, downs.pop()], axis=-1) kernel_size = (3, 3) x = ResidualBlock( up_conv_type, name=f"up_{i}_residual_{j}", features=dim_out, kernel_init=kernel_init(1.0), kernel_size=kernel_size, strides=(1, 1), activation=self.activation, norm_groups=self.norm_groups )(x, temb) if attention_config is not None and j == self.num_res_blocks - 1: # Apply attention only on the last block x = AttentionBlock(heads=attention_config['heads'], dim_head=dim_out // attention_config['heads'], name=f"up_{i}_attention_{j}")(x) if i != len(feature_depths) - 1: x = Upsample( features=feature_depths[-i], scale=2, activation=self.activation, name=f"up_{i}_upsample" )(x)
# x = nn.GroupNorm(8)(x) x = ConvLayer( conv_type, features=self.feature_depths[0], kernel_size=(3, 3), strides=(1, 1), kernel_init=kernel_init(0.0) )(x)
x = jnp.concatenate([x, downs.pop()], axis=-1)
x = ResidualBlock( conv_type, name="final_residual", features=self.feature_depths[0], kernel_init=kernel_init(1.0), kernel_size=(3,3), strides=(1, 1), activation=self.activation, norm_groups=self.norm_groups )(x, temb)
x = nn.GroupNorm(self.norm_groups)(x) x = self.activation(x)
noise_out = ConvLayer( conv_type, features=3, kernel_size=(3, 3), strides=(1, 1), # activation=jax.nn.mish kernel_init=kernel_init(0.0) )(x) return noise_out#, attentionsValidate the model
Section titled “Validate the model”The output hasn’t been rendered as it is too large to be rendered in the notebook.
model = Unet(emb_features=256, feature_depths=[64, 128, 256, 512], attention_configs=[None, {"heads": 8}, {"heads": 8}, {"heads": 8}], num_res_blocks=2, num_middle_res_blocks=1, major_conv_type="conv" )inp = jnp.ones((1, 64, 64, 3))temb = jnp.ones((1,))print(model.tabulate(jax.random.key(0), inp, temb, console_kwargs={"width": 200, "force_jupyter": True, }))Training utils
Section titled “Training utils”We shall define some important utilities for training our model. These are as follows:
-
Diffusion Prediction Transform: Often the model’s input and outputs need to be transformed to parameterize the model in the way we intend to. Although we are using the epsilon (denoising) formulation, using this general class to wrap the model can allow us to easily switch to other parameterizations as well.
-
TrainState: This class will help us keep track of the training state of our model. It will keep track of the optimizer, the learning rate schedule, the model, the loss function, and the parameters of the model.
-
DiffusionTrainer: This class will help us train our model. It will take in the train state and the training data and has the training loop defined in it. It would also periodically save the model parameters to disk
class DiffusionPredictionTransform(): def pred_transform(self, x_t, preds, rates) -> jnp.ndarray: return preds
def __call__(self, x_t, preds, current_step, noise_schedule:NoiseScheduler) -> Union[jnp.ndarray, jnp.ndarray]: rates = noise_schedule.get_rates(current_step) preds = self.pred_transform(x_t, preds, rates) x_0, epsilon = self.backward_diffusion(x_t, preds, rates) return x_0, epsilon
def forward_diffusion(self, x_0, epsilon, rates: tuple[jnp.ndarray, jnp.ndarray]) -> Union[jnp.ndarray, jnp.ndarray, jnp.ndarray]: signal_rate, noise_rate = rates x_t = signal_rate * x_0 + noise_rate * epsilon expected_output = self.get_target(x_0, epsilon, (signal_rate, noise_rate)) c_in = self.get_input_scale((signal_rate, noise_rate)) return x_t, c_in, expected_output
def backward_diffusion(self, x_t, preds, rates: tuple[jnp.ndarray, jnp.ndarray]) -> Union[jnp.ndarray, jnp.ndarray]: raise NotImplementedError
def get_target(self, x_0, epsilon, rates) ->jnp.ndarray: return x_0
def get_input_scale(self, rates: tuple[jnp.ndarray, jnp.ndarray]) -> jnp.ndarray: return 1
class EpsilonPredictionTransform(DiffusionPredictionTransform): def backward_diffusion(self, x_t, preds, rates: tuple[jnp.ndarray, jnp.ndarray]) -> Union[jnp.ndarray, jnp.ndarray]: # preds is the predicted noise epsilon = preds signal_rates, noise_rates = rates x_0 = (x_t - epsilon * noise_rates) / signal_rates return x_0, epsilon
def get_target(self, x_0, epsilon, rates) ->jnp.ndarray: return epsilonclass ModelState(): model: nn.Module params: dict noise_schedule: NoiseScheduler model_output_transform: DiffusionPredictionTransform
# Define the TrainState with EMA parametersclass TrainState(train_state.TrainState): rngs: jax.random.PRNGKey ema_params: dict
def get_random_key(self): rngs, subkey = jax.random.split(self.rngs) return self.replace(rngs=rngs), subkey
def apply_ema(self, decay: float=0.999): new_ema_params = jax.tree_util.tree_map( lambda ema, param: decay * ema + (1 - decay) * param, self.ema_params, self.params, ) return self.replace(ema_params=new_ema_params)
class DiffusionTrainer: state : TrainState best_state : TrainState best_loss : float model : nn.Module noise_schedule : NoiseScheduler model_output_transform:DiffusionPredictionTransform ema_decay:float = 0.999
def __init__(self, model:nn.Module, optimizer: optax.GradientTransformation, noise_schedule:NoiseScheduler, rngs:jax.random.PRNGKey, image_size:int=64, unconditional_prob:float=0.2, train_state:TrainState=None, name:str="Diffusion", load_from_checkpoint:bool=False, checkpoint_suffix:str="", param_transforms:Callable=None, model_output_transform:DiffusionPredictionTransform=EpsilonPredictionTransform(), loss_fn=optax.l2_loss ): self.model = model self.noise_schedule = noise_schedule self.name = name self.model_output_transform = model_output_transform self.loss_fn = loss_fn self.image_size = image_size self.unconditional_prob = unconditional_prob
options = ocp.CheckpointManagerOptions(max_to_keep=4, create=True) self.checkpointer = ocp.CheckpointManager(self.checkpoint_path() + checkpoint_suffix, options=options)
if load_from_checkpoint: latest_step, old_state, old_best_state = self.load() else: latest_step, old_state, old_best_state = 0, None, None
self.latest_step = latest_step
if train_state == None: self.init_state(optimizer, rngs, existing_state=old_state, existing_best_state=old_best_state, model=model, param_transforms=param_transforms) else: self.state = train_state self.best_state = train_state self.best_loss = 1e9
def init_state(self, optimizer: optax.GradientTransformation, rngs:jax.random.PRNGKey, existing_state:dict=None, existing_best_state:dict=None, model:nn.Module=None, param_transforms:Callable=None, ): inp = jnp.ones((1, self.image_size, self.image_size, 3)) temb = jnp.ones((1,)) rngs, subkey = jax.random.split(rngs)
if existing_state == None: params = model.init(subkey, inp, temb) existing_state = {"params":params, "ema_params":params}
if param_transforms is not None: params = param_transforms(params) self.best_loss = 1e9 self.state = TrainState.create( apply_fn=model.apply, params=existing_state['params'], ema_params=existing_state['ema_params'], tx=optimizer, rngs=rngs, ) if existing_best_state is not None: self.best_state = self.state.replace(params=existing_best_state['params'], ema_params=existing_best_state['ema_params']) else: self.best_state = self.state
def checkpoint_path(self): experiment_name = self.name path = os.path.join(os.path.abspath('./checkpoints'), experiment_name) if not os.path.exists(path): os.makedirs(path) return path
def load(self): step = self.checkpointer.latest_step() print("Loading model from checkpoint", step) ckpt = self.checkpointer.restore(step, args=ocp.args.StandardRestore()) state = ckpt['state'] best_state = ckpt['best_state'] # Convert the state to a TrainState self.best_loss = ckpt['best_loss'] print(f"Loaded model from checkpoint at step {step}", ckpt['best_loss']) return step, state, best_state
def save(self, epoch=0): print(f"Saving model at epoch {epoch}") # The arrays of both states: their functions and optimizer are rebuilt from code. ckpt = { 'state': {'params': self.state.params, 'ema_params': self.state.ema_params}, 'best_state': {'params': self.best_state.params, 'ema_params': self.best_state.ema_params}, 'best_loss': float(self.best_loss) } try: self.checkpointer.save(epoch, args=ocp.args.StandardSave(ckpt), force=True) self.checkpointer.wait_until_finished() except Exception as e: print(f"Error saving model: {e}")
def summary(self): inp = jnp.ones((1, self.image_size, self.image_size, 3)) temb = jnp.ones((1,)) print(self.model.tabulate(jax.random.key(0), inp, temb, console_kwargs={"width": 200, "force_jupyter":True, }))
def _define_train_step(self): noise_schedule = self.noise_schedule model = self.model model_output_transform = self.model_output_transform loss_fn = self.loss_fn @jax.jit def train_step(state:TrainState, batch): """Train for a single step.""" images = batch noise_level, state = noise_schedule.generate_timesteps(images.shape[0], state) state, rngs = state.get_random_key() noise:jax.Array = jax.random.normal(rngs, shape=images.shape) rates = noise_schedule.get_rates(noise_level) noisy_images, c_in, expected_output = model_output_transform.forward_diffusion(images, noise, rates) def model_loss(params): preds = model.apply(params, *noise_schedule.transform_inputs(noisy_images*c_in, noise_level)) preds = model_output_transform.pred_transform(noisy_images, preds, rates) nloss = loss_fn(preds, expected_output) # nloss *= noise_schedule.get_weights(noise_level) nloss = jnp.mean(nloss) loss = nloss return loss loss, grads = jax.value_and_grad(model_loss)(state.params) state = state.apply_gradients(grads=grads) state = state.apply_ema(self.ema_decay) return state, loss return train_step
def fit(self, data, steps_per_epoch, epochs): data = iter(data) train_step = self._define_train_step() state = self.state for epoch in range(epochs): current_epoch = self.latest_step + epoch + 1 print(f"\nEpoch {current_epoch}/{epochs}") start_time = time.time() epoch_loss = 0 with tqdm.tqdm(total=steps_per_epoch, desc=f'\t\tEpoch {current_epoch}', ncols=100, unit='step') as pbar: for i in range(steps_per_epoch): batch = next(data) state, loss = train_step(state, batch) epoch_loss += loss if i % 100 == 0: pbar.set_postfix(loss=f'{loss:.4f}') pbar.update(100) end_time = time.time() self.state = state total_time = end_time - start_time avg_time_per_step = total_time / steps_per_epoch avg_loss = epoch_loss / steps_per_epoch if avg_loss < self.best_loss: self.best_loss = avg_loss self.best_state = state self.save(current_epoch) print(f"\n\tEpoch {current_epoch} completed. Avg Loss: {avg_loss}, Time: {total_time:.2f}s, Best Loss: {self.best_loss}") self.save(epochs) return self.stateTraining
Section titled “Training”We shall first define the schedules we would wanna use, and then the model, some params, the optimizer and the trainer, and then start training the model.
cosine_schedule = CosineDiscreteNoiseSchedule(1000)experiment_name = "{name}_{date}".format( name="Diffusion_SDE_VE", date=datetime.now().strftime("%Y-%m-%d_%H:%M:%S"))# experiment_name = 'Diffusion_SDE_VE_2024-07-25_00:07:09'print("Experiment_Name:", experiment_name)
# Suggested configurations: one Colab GPU trains this in about an hour.BATCH_SIZE=64IMAGE_SIZE=64total_epochs = 120learning_rate = 2e-4feature_depths = [64, 128, 256, 512]NUM_SAMPLES = 64
# The tutorial test sets DEW_TUTORIAL_SMOKE=1 to run every cell in minutes on a CPU.if os.environ.get("DEW_TUTORIAL_SMOKE") == "1": BATCH_SIZE, total_epochs, feature_depths, NUM_SAMPLES = 8, 1, [16, 32, 32, 64], 4
unet = Unet(emb_features=256, feature_depths=feature_depths, attention_configs=[None, {"heads": 8}, {"heads": 8}, {"heads": 8}], num_res_blocks=2, num_middle_res_blocks=1, major_conv_type="conv", mid_conv_type="conv")
data, datalen = get_dataset("oxford_flowers102", batch_size=BATCH_SIZE, image_scale=IMAGE_SIZE)batches = datalen // BATCH_SIZEif os.environ.get("DEW_TUTORIAL_SMOKE") == "1": batches = 2
steps_per_epoch = batches
solver = optax.adam(learning_rate)
trainer = DiffusionTrainer(unet, optimizer=solver, noise_schedule=cosine_schedule, rngs=jax.random.PRNGKey(4), name=experiment_name, model_output_transform=EpsilonPredictionTransform(), # load_from_checkpoint=True, )Experiment_Name: Diffusion_SDE_VE_2026-10-19_17:33:33
trainer.summary()data, datalen = get_dataset("oxford_flowers102", batch_size=BATCH_SIZE, image_scale=IMAGE_SIZE)final_state = trainer.fit(data, batches, epochs=1 if os.environ.get("DEW_TUTORIAL_SMOKE") == "1" else 5)Epoch 1/5 Epoch 1: 200step [01:50, 1.81step/s, loss=0.0794] Saving model at epoch 1 Epoch 1 completed. Avg Loss: 0.2397734671831131, Time: 110.45s, Best Loss: 0.2397734671831131 Epoch 2/5 Epoch 2: 200step [00:12, 16.29step/s, loss=0.0634] WARNING:absl:Waiting for previous save to complete took 3.249436 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 2 Epoch 2 completed. Avg Loss: 0.060364965349435806, Time: 12.28s, Best Loss: 0.060364965349435806 Epoch 3/5 Epoch 3: 200step [00:12, 16.30step/s, loss=0.0483] WARNING:absl:Waiting for previous save to complete took 2.612351 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 3 Epoch 3 completed. Avg Loss: 0.051985371857881546, Time: 12.27s, Best Loss: 0.051985371857881546 Epoch 4/5 Epoch 4: 200step [00:12, 16.29step/s, loss=0.0524] WARNING:absl:Waiting for previous save to complete took 2.718664 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 4 Epoch 4 completed. Avg Loss: 0.04800354689359665, Time: 12.28s, Best Loss: 0.04800354689359665 Epoch 5/5 Epoch 5: 200step [00:12, 16.32step/s, loss=0.0339] WARNING:absl:Waiting for previous save to complete took 2.732236 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 5 WARNING:absl:Waiting for previous save to complete took 3.224547 seconds. If this number is high, consider checkpointing less frequently. Epoch 5 completed. Avg Loss: 0.04493911191821098, Time: 12.26s, Best Loss: 0.04493911191821098 Saving model at epoch 5 Error saving model: Checkpoint for step 5 already exists.
data, datalen = get_dataset("oxford_flowers102", batch_size=BATCH_SIZE, image_scale=IMAGE_SIZE)final_state = trainer.fit(data, batches, epochs=total_epochs)Epoch 1/120 Epoch 1: 200step [00:38, 5.15step/s, loss=0.0453] Saving model at epoch 1 Epoch 1 completed. Avg Loss: 0.04283605515956879, Time: 38.82s, Best Loss: 0.04283605515956879 Epoch 2/120 Epoch 2: 200step [00:12, 16.28step/s, loss=0.0360] WARNING:absl:Waiting for previous save to complete took 2.885580 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 2 Epoch 2 completed. Avg Loss: 0.040446873754262924, Time: 12.29s, Best Loss: 0.040446873754262924 Epoch 3/120 Epoch 3: 200step [00:12, 16.29step/s, loss=0.0299] WARNING:absl:Waiting for previous save to complete took 3.379653 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 3 Epoch 3 completed. Avg Loss: 0.03996783494949341, Time: 12.28s, Best Loss: 0.03996783494949341 Epoch 4/120 Epoch 4: 200step [00:12, 16.30step/s, loss=0.0394] Epoch 4 completed. Avg Loss: 0.04061711207032204, Time: 12.27s, Best Loss: 0.03996783494949341 Epoch 5/120 Epoch 5: 200step [00:12, 16.32step/s, loss=0.0511] WARNING:absl:Waiting for previous save to complete took 2.843708 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 5 Error saving model: Checkpoint for step 5 already exists. Epoch 5 completed. Avg Loss: 0.03862394019961357, Time: 12.26s, Best Loss: 0.03862394019961357 Epoch 6/120 Epoch 6: 200step [00:12, 15.91step/s, loss=0.0377] Saving model at epoch 6 Epoch 6 completed. Avg Loss: 0.036652106791734695, Time: 12.58s, Best Loss: 0.036652106791734695 Epoch 7/120 Epoch 7: 200step [00:12, 16.30step/s, loss=0.0383] WARNING:absl:Waiting for previous save to complete took 2.883675 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 7 Epoch 7 completed. Avg Loss: 0.03642510250210762, Time: 12.27s, Best Loss: 0.03642510250210762 Epoch 8/120 Epoch 8: 200step [00:12, 16.30step/s, loss=0.0327] Epoch 8 completed. Avg Loss: 0.03754964470863342, Time: 12.27s, Best Loss: 0.03642510250210762 Epoch 9/120 Epoch 9: 200step [00:12, 16.29step/s, loss=0.0225] Epoch 9 completed. Avg Loss: 0.03665987774729729, Time: 12.28s, Best Loss: 0.03642510250210762 Epoch 10/120 Epoch 10: 200step [00:12, 16.33step/s, loss=0.0390] WARNING:absl:Waiting for previous save to complete took 3.438683 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 10 Epoch 10 completed. Avg Loss: 0.03598170727491379, Time: 12.25s, Best Loss: 0.03598170727491379 Epoch 11/120 Epoch 11: 200step [00:12, 15.91step/s, loss=0.0339] WARNING:absl:Waiting for previous save to complete took 2.868574 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 11 Epoch 11 completed. Avg Loss: 0.0348195880651474, Time: 12.57s, Best Loss: 0.0348195880651474 Epoch 12/120 Epoch 12: 200step [00:12, 16.32step/s, loss=0.0484] Epoch 12 completed. Avg Loss: 0.035138051956892014, Time: 12.26s, Best Loss: 0.0348195880651474 Epoch 13/120 Epoch 13: 200step [00:12, 15.91step/s, loss=0.0457] Epoch 13 completed. Avg Loss: 0.03546351566910744, Time: 12.57s, Best Loss: 0.0348195880651474 Epoch 14/120 Epoch 14: 200step [00:12, 16.33step/s, loss=0.0254] WARNING:absl:Waiting for previous save to complete took 2.849257 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 14 Epoch 14 completed. Avg Loss: 0.03477298095822334, Time: 12.25s, Best Loss: 0.03477298095822334 Epoch 15/120 Epoch 15: 200step [00:12, 16.30step/s, loss=0.0313] Epoch 15 completed. Avg Loss: 0.0355941466987133, Time: 12.27s, Best Loss: 0.03477298095822334 Epoch 16/120 Epoch 16: 200step [00:12, 16.32step/s, loss=0.0520] Epoch 16 completed. Avg Loss: 0.03478928282856941, Time: 12.26s, Best Loss: 0.03477298095822334 Epoch 17/120 Epoch 17: 200step [00:12, 15.93step/s, loss=0.0415] WARNING:absl:Waiting for previous save to complete took 2.869037 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 17 Epoch 17 completed. Avg Loss: 0.03367600962519646, Time: 12.56s, Best Loss: 0.03367600962519646 Epoch 18/120 Epoch 18: 200step [00:12, 16.28step/s, loss=0.0279] Epoch 18 completed. Avg Loss: 0.033991605043411255, Time: 12.29s, Best Loss: 0.03367600962519646 Epoch 19/120 Epoch 19: 200step [00:12, 16.32step/s, loss=0.0452] Epoch 19 completed. Avg Loss: 0.034821972250938416, Time: 12.26s, Best Loss: 0.03367600962519646 Epoch 20/120 Epoch 20: 200step [00:12, 15.90step/s, loss=0.0356] WARNING:absl:Waiting for previous save to complete took 2.794168 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 20 Epoch 20 completed. Avg Loss: 0.032871171832084656, Time: 12.58s, Best Loss: 0.032871171832084656 Epoch 21/120 Epoch 21: 200step [00:12, 16.24step/s, loss=0.0258] Epoch 21 completed. Avg Loss: 0.033281359821558, Time: 12.31s, Best Loss: 0.032871171832084656 Epoch 22/120 Epoch 22: 200step [00:12, 16.30step/s, loss=0.0365] Epoch 22 completed. Avg Loss: 0.034462735056877136, Time: 12.27s, Best Loss: 0.032871171832084656 Epoch 23/120 Epoch 23: 200step [00:12, 15.89step/s, loss=0.0422] WARNING:absl:Waiting for previous save to complete took 2.897143 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 23 Epoch 23 completed. Avg Loss: 0.03253646939992905, Time: 12.60s, Best Loss: 0.03253646939992905 Epoch 24/120 Epoch 24: 200step [00:12, 16.28step/s, loss=0.0308] Epoch 24 completed. Avg Loss: 0.032832466065883636, Time: 12.28s, Best Loss: 0.03253646939992905 Epoch 25/120 Epoch 25: 200step [00:12, 16.31step/s, loss=0.0271] Epoch 25 completed. Avg Loss: 0.03391920030117035, Time: 12.26s, Best Loss: 0.03253646939992905 Epoch 26/120 Epoch 26: 200step [00:12, 15.92step/s, loss=0.0321] Epoch 26 completed. Avg Loss: 0.03339046612381935, Time: 12.57s, Best Loss: 0.03253646939992905 Epoch 27/120 Epoch 27: 200step [00:12, 16.31step/s, loss=0.0435] WARNING:absl:Waiting for previous save to complete took 2.869307 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 27 Epoch 27 completed. Avg Loss: 0.032055046409368515, Time: 12.27s, Best Loss: 0.032055046409368515 Epoch 28/120 Epoch 28: 200step [00:12, 16.25step/s, loss=0.0312] Epoch 28 completed. Avg Loss: 0.03287725895643234, Time: 12.31s, Best Loss: 0.032055046409368515 Epoch 29/120 Epoch 29: 200step [00:12, 15.87step/s, loss=0.0358] Epoch 29 completed. Avg Loss: 0.033635228872299194, Time: 12.61s, Best Loss: 0.032055046409368515 Epoch 30/120 Epoch 30: 200step [00:12, 16.29step/s, loss=0.0350] Epoch 30 completed. Avg Loss: 0.032835863530635834, Time: 12.28s, Best Loss: 0.032055046409368515 Epoch 31/120 Epoch 31: 200step [00:12, 16.29step/s, loss=0.0295] Epoch 31 completed. Avg Loss: 0.03212837502360344, Time: 12.28s, Best Loss: 0.032055046409368515 Epoch 32/120 Epoch 32: 200step [00:12, 16.31step/s, loss=0.0285] Epoch 32 completed. Avg Loss: 0.032902028411626816, Time: 12.27s, Best Loss: 0.032055046409368515 Epoch 33/120 Epoch 33: 200step [00:12, 16.28step/s, loss=0.0415] Epoch 33 completed. Avg Loss: 0.03381903097033501, Time: 12.29s, Best Loss: 0.032055046409368515 Epoch 34/120 Epoch 34: 200step [00:12, 15.88step/s, loss=0.0311] WARNING:absl:Waiting for previous save to complete took 2.778992 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 34 Epoch 34 completed. Avg Loss: 0.03135020658373833, Time: 12.60s, Best Loss: 0.03135020658373833 Epoch 35/120 Epoch 35: 200step [00:12, 16.28step/s, loss=0.0293] Epoch 35 completed. Avg Loss: 0.03201644867658615, Time: 12.28s, Best Loss: 0.03135020658373833 Epoch 36/120 Epoch 36: 200step [00:12, 16.31step/s, loss=0.0374] Epoch 36 completed. Avg Loss: 0.03190924972295761, Time: 12.27s, Best Loss: 0.03135020658373833 Epoch 37/120 Epoch 37: 200step [00:12, 15.88step/s, loss=0.0333] Epoch 37 completed. Avg Loss: 0.03147611767053604, Time: 12.60s, Best Loss: 0.03135020658373833 Epoch 38/120 Epoch 38: 200step [00:12, 16.27step/s, loss=0.0260] Epoch 38 completed. Avg Loss: 0.03216132894158363, Time: 12.30s, Best Loss: 0.03135020658373833 Epoch 39/120 Epoch 39: 200step [00:12, 16.31step/s, loss=0.0377] Epoch 39 completed. Avg Loss: 0.03176102414727211, Time: 12.27s, Best Loss: 0.03135020658373833 Epoch 40/120 Epoch 40: 200step [00:12, 16.31step/s, loss=0.0251] WARNING:absl:Waiting for previous save to complete took 2.799737 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 40 Epoch 40 completed. Avg Loss: 0.03129110112786293, Time: 12.27s, Best Loss: 0.03129110112786293 Epoch 41/120 Epoch 41: 200step [00:12, 16.30step/s, loss=0.0256] Epoch 41 completed. Avg Loss: 0.03203636035323143, Time: 12.27s, Best Loss: 0.03129110112786293 Epoch 42/120 Epoch 42: 200step [00:12, 16.32step/s, loss=0.0277] Epoch 42 completed. Avg Loss: 0.03162436559796333, Time: 12.26s, Best Loss: 0.03129110112786293 Epoch 43/120 Epoch 43: 200step [00:12, 16.31step/s, loss=0.0396] Epoch 43 completed. Avg Loss: 0.0335795022547245, Time: 12.27s, Best Loss: 0.03129110112786293 Epoch 44/120 Epoch 44: 200step [00:12, 16.32step/s, loss=0.0189] Epoch 44 completed. Avg Loss: 0.03157155588269234, Time: 12.26s, Best Loss: 0.03129110112786293 Epoch 45/120 Epoch 45: 200step [00:12, 16.29step/s, loss=0.0271] Epoch 45 completed. Avg Loss: 0.03188423439860344, Time: 12.28s, Best Loss: 0.03129110112786293 Epoch 46/120 Epoch 46: 200step [00:12, 15.88step/s, loss=0.0294] Epoch 46 completed. Avg Loss: 0.03185481205582619, Time: 12.60s, Best Loss: 0.03129110112786293 Epoch 47/120 Epoch 47: 200step [00:12, 16.28step/s, loss=0.0338] WARNING:absl:Waiting for previous save to complete took 3.290947 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 47 Epoch 47 completed. Avg Loss: 0.031134815886616707, Time: 12.29s, Best Loss: 0.031134815886616707 Epoch 48/120 Epoch 48: 200step [00:12, 16.27step/s, loss=0.0261] Epoch 48 completed. Avg Loss: 0.03155133128166199, Time: 12.30s, Best Loss: 0.031134815886616707 Epoch 49/120 Epoch 49: 200step [00:12, 15.86step/s, loss=0.0282] Epoch 49 completed. Avg Loss: 0.03240012750029564, Time: 12.61s, Best Loss: 0.031134815886616707 Epoch 50/120 Epoch 50: 200step [00:12, 16.26step/s, loss=0.0401] Epoch 50 completed. Avg Loss: 0.03146661818027496, Time: 12.30s, Best Loss: 0.031134815886616707 Epoch 51/120 Epoch 51: 200step [00:12, 16.28step/s, loss=0.0220] Epoch 51 completed. Avg Loss: 0.031914498656988144, Time: 12.29s, Best Loss: 0.031134815886616707 Epoch 52/120 Epoch 52: 200step [00:12, 16.30step/s, loss=0.0354] WARNING:absl:Waiting for previous save to complete took 2.733636 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 52 Epoch 52 completed. Avg Loss: 0.031110454350709915, Time: 12.28s, Best Loss: 0.031110454350709915 Epoch 53/120 Epoch 53: 200step [00:12, 16.26step/s, loss=0.0269] WARNING:absl:Waiting for previous save to complete took 3.296209 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 53 Epoch 53 completed. Avg Loss: 0.031079385429620743, Time: 12.30s, Best Loss: 0.031079385429620743 Epoch 54/120 Epoch 54: 200step [00:12, 16.28step/s, loss=0.0374] WARNING:absl:Waiting for previous save to complete took 3.010356 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 54 Epoch 54 completed. Avg Loss: 0.030486641451716423, Time: 12.29s, Best Loss: 0.030486641451716423 Epoch 55/120 Epoch 55: 200step [00:12, 16.26step/s, loss=0.0325] Epoch 55 completed. Avg Loss: 0.03057207353413105, Time: 12.30s, Best Loss: 0.030486641451716423 Epoch 56/120 Epoch 56: 200step [00:12, 16.31step/s, loss=0.0366] Epoch 56 completed. Avg Loss: 0.03183101490139961, Time: 12.27s, Best Loss: 0.030486641451716423 Epoch 57/120 Epoch 57: 200step [00:12, 16.30step/s, loss=0.0208] WARNING:absl:Waiting for previous save to complete took 3.250292 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 57 Epoch 57 completed. Avg Loss: 0.030082179233431816, Time: 12.28s, Best Loss: 0.030082179233431816 Epoch 58/120 Epoch 58: 200step [00:12, 16.27step/s, loss=0.0367] Epoch 58 completed. Avg Loss: 0.031851910054683685, Time: 12.29s, Best Loss: 0.030082179233431816 Epoch 59/120 Epoch 59: 200step [00:12, 16.29step/s, loss=0.0232] Epoch 59 completed. Avg Loss: 0.03114568255841732, Time: 12.28s, Best Loss: 0.030082179233431816 Epoch 60/120 Epoch 60: 200step [00:12, 16.30step/s, loss=0.0254] Epoch 60 completed. Avg Loss: 0.03024885058403015, Time: 12.28s, Best Loss: 0.030082179233431816 Epoch 61/120 Epoch 61: 200step [00:12, 16.30step/s, loss=0.0414] Epoch 61 completed. Avg Loss: 0.031271129846572876, Time: 12.28s, Best Loss: 0.030082179233431816 Epoch 62/120 Epoch 62: 200step [00:12, 16.30step/s, loss=0.0225] Epoch 62 completed. Avg Loss: 0.031310074031353, Time: 12.28s, Best Loss: 0.030082179233431816 Epoch 63/120 Epoch 63: 200step [00:12, 15.87step/s, loss=0.0260] Epoch 63 completed. Avg Loss: 0.031340740621089935, Time: 12.61s, Best Loss: 0.030082179233431816 Epoch 64/120 Epoch 64: 200step [00:12, 16.32step/s, loss=0.0230] Epoch 64 completed. Avg Loss: 0.03230246528983116, Time: 12.26s, Best Loss: 0.030082179233431816 Epoch 65/120 Epoch 65: 200step [00:12, 16.29step/s, loss=0.0306] Epoch 65 completed. Avg Loss: 0.031478844583034515, Time: 12.28s, Best Loss: 0.030082179233431816 Epoch 66/120 Epoch 66: 200step [00:12, 16.32step/s, loss=0.0298] Epoch 66 completed. Avg Loss: 0.030865676701068878, Time: 12.26s, Best Loss: 0.030082179233431816 Epoch 67/120 Epoch 67: 200step [00:12, 16.30step/s, loss=0.0317] Epoch 67 completed. Avg Loss: 0.03038589097559452, Time: 12.28s, Best Loss: 0.030082179233431816 Epoch 68/120 Epoch 68: 200step [00:12, 15.90step/s, loss=0.0408] Epoch 68 completed. Avg Loss: 0.030374344438314438, Time: 12.59s, Best Loss: 0.030082179233431816 Epoch 69/120 Epoch 69: 200step [00:12, 16.30step/s, loss=0.0245] WARNING:absl:Waiting for previous save to complete took 3.108387 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 69 Epoch 69 completed. Avg Loss: 0.029681654646992683, Time: 12.28s, Best Loss: 0.029681654646992683 Epoch 70/120 Epoch 70: 200step [00:12, 16.27step/s, loss=0.0284] Epoch 70 completed. Avg Loss: 0.03086642362177372, Time: 12.29s, Best Loss: 0.029681654646992683 Epoch 71/120 Epoch 71: 200step [00:12, 15.88step/s, loss=0.0325] Epoch 71 completed. Avg Loss: 0.03074629418551922, Time: 12.60s, Best Loss: 0.029681654646992683 Epoch 72/120 Epoch 72: 200step [00:12, 16.30step/s, loss=0.0255] Epoch 72 completed. Avg Loss: 0.030853355303406715, Time: 12.28s, Best Loss: 0.029681654646992683 Epoch 73/120 Epoch 73: 200step [00:12, 16.30step/s, loss=0.0346] Epoch 73 completed. Avg Loss: 0.031319018453359604, Time: 12.27s, Best Loss: 0.029681654646992683 Epoch 74/120 Epoch 74: 200step [00:12, 16.31step/s, loss=0.0254] Epoch 74 completed. Avg Loss: 0.03094436787068844, Time: 12.27s, Best Loss: 0.029681654646992683 Epoch 75/120 Epoch 75: 200step [00:12, 16.28step/s, loss=0.0339] Epoch 75 completed. Avg Loss: 0.030656905844807625, Time: 12.29s, Best Loss: 0.029681654646992683 Epoch 76/120 Epoch 76: 200step [00:12, 16.28step/s, loss=0.0314] Epoch 76 completed. Avg Loss: 0.030391914770007133, Time: 12.29s, Best Loss: 0.029681654646992683 Epoch 77/120 Epoch 77: 200step [00:12, 15.87step/s, loss=0.0301] Epoch 77 completed. Avg Loss: 0.031235026195645332, Time: 12.61s, Best Loss: 0.029681654646992683 Epoch 78/120 Epoch 78: 200step [00:12, 16.30step/s, loss=0.0331] Epoch 78 completed. Avg Loss: 0.031034797430038452, Time: 12.28s, Best Loss: 0.029681654646992683 Epoch 79/120 Epoch 79: 200step [00:12, 16.30step/s, loss=0.0324] Epoch 79 completed. Avg Loss: 0.030315423384308815, Time: 12.28s, Best Loss: 0.029681654646992683 Epoch 80/120 Epoch 80: 200step [00:12, 16.31step/s, loss=0.0293] WARNING:absl:Waiting for previous save to complete took 2.849926 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 80 Epoch 80 completed. Avg Loss: 0.029599107801914215, Time: 12.27s, Best Loss: 0.029599107801914215 Epoch 81/120 Epoch 81: 200step [00:12, 16.27step/s, loss=0.0284] Epoch 81 completed. Avg Loss: 0.03032945655286312, Time: 12.30s, Best Loss: 0.029599107801914215 Epoch 82/120 Epoch 82: 200step [00:12, 16.30step/s, loss=0.0289] Epoch 82 completed. Avg Loss: 0.029963843524456024, Time: 12.27s, Best Loss: 0.029599107801914215 Epoch 83/120 Epoch 83: 200step [00:12, 16.29step/s, loss=0.0281] Epoch 83 completed. Avg Loss: 0.030780570581555367, Time: 12.28s, Best Loss: 0.029599107801914215 Epoch 84/120 Epoch 84: 200step [00:12, 16.30step/s, loss=0.0259] WARNING:absl:Waiting for previous save to complete took 3.308355 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 84 Epoch 84 completed. Avg Loss: 0.02939843200147152, Time: 12.28s, Best Loss: 0.02939843200147152 Epoch 85/120 Epoch 85: 200step [00:12, 16.26step/s, loss=0.0374] Epoch 85 completed. Avg Loss: 0.030705202370882034, Time: 12.30s, Best Loss: 0.02939843200147152 Epoch 86/120 Epoch 86: 200step [00:12, 16.31step/s, loss=0.0260] Epoch 86 completed. Avg Loss: 0.030660895630717278, Time: 12.27s, Best Loss: 0.02939843200147152 Epoch 87/120 Epoch 87: 200step [00:12, 16.31step/s, loss=0.0372] Epoch 87 completed. Avg Loss: 0.030043411999940872, Time: 12.27s, Best Loss: 0.02939843200147152 Epoch 88/120 Epoch 88: 200step [00:12, 16.30step/s, loss=0.0342] Epoch 88 completed. Avg Loss: 0.030957866460084915, Time: 12.28s, Best Loss: 0.02939843200147152 Epoch 89/120 Epoch 89: 200step [00:12, 16.30step/s, loss=0.0295] Epoch 89 completed. Avg Loss: 0.030108027160167694, Time: 12.28s, Best Loss: 0.02939843200147152 Epoch 90/120 Epoch 90: 200step [00:12, 15.89step/s, loss=0.0197] Epoch 90 completed. Avg Loss: 0.030451424419879913, Time: 12.59s, Best Loss: 0.02939843200147152 Epoch 91/120 Epoch 91: 200step [00:12, 16.30step/s, loss=0.0269] Epoch 91 completed. Avg Loss: 0.029446490108966827, Time: 12.27s, Best Loss: 0.02939843200147152 Epoch 92/120 Epoch 92: 200step [00:12, 16.31step/s, loss=0.0406] Epoch 92 completed. Avg Loss: 0.030974069610238075, Time: 12.26s, Best Loss: 0.02939843200147152 Epoch 93/120 Epoch 93: 200step [00:12, 16.29step/s, loss=0.0311] WARNING:absl:Waiting for previous save to complete took 3.362116 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 93 Epoch 93 completed. Avg Loss: 0.02936515398323536, Time: 12.28s, Best Loss: 0.02936515398323536 Epoch 94/120 Epoch 94: 200step [00:12, 16.25step/s, loss=0.0308] WARNING:absl:Waiting for previous save to complete took 2.867023 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 94 Epoch 94 completed. Avg Loss: 0.02915726602077484, Time: 12.31s, Best Loss: 0.02915726602077484 Epoch 95/120 Epoch 95: 200step [00:12, 16.27step/s, loss=0.0213] Epoch 95 completed. Avg Loss: 0.030109360814094543, Time: 12.29s, Best Loss: 0.02915726602077484 Epoch 96/120 Epoch 96: 200step [00:12, 16.31step/s, loss=0.0376] Epoch 96 completed. Avg Loss: 0.02940916083753109, Time: 12.27s, Best Loss: 0.02915726602077484 Epoch 97/120 Epoch 97: 200step [00:12, 15.90step/s, loss=0.0328] Epoch 97 completed. Avg Loss: 0.030528564006090164, Time: 12.58s, Best Loss: 0.02915726602077484 Epoch 98/120 Epoch 98: 200step [00:12, 16.30step/s, loss=0.0322] Epoch 98 completed. Avg Loss: 0.030596928671002388, Time: 12.27s, Best Loss: 0.02915726602077484 Epoch 99/120 Epoch 99: 200step [00:12, 16.31step/s, loss=0.0249] Epoch 99 completed. Avg Loss: 0.029173335060477257, Time: 12.27s, Best Loss: 0.02915726602077484 Epoch 100/120 Epoch 100: 200step [00:12, 16.30step/s, loss=0.0319] WARNING:absl:Waiting for previous save to complete took 2.834158 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 100 Epoch 100 completed. Avg Loss: 0.029037456959486008, Time: 12.28s, Best Loss: 0.029037456959486008 Epoch 101/120 Epoch 101: 200step [00:12, 16.30step/s, loss=0.0392] Epoch 101 completed. Avg Loss: 0.030368611216545105, Time: 12.27s, Best Loss: 0.029037456959486008 Epoch 102/120 Epoch 102: 200step [00:12, 16.31step/s, loss=0.0303] Epoch 102 completed. Avg Loss: 0.029508575797080994, Time: 12.26s, Best Loss: 0.029037456959486008 Epoch 103/120 Epoch 103: 200step [00:12, 16.29step/s, loss=0.0293] WARNING:absl:Waiting for previous save to complete took 3.248264 seconds. If this number is high, consider checkpointing less frequently. Saving model at epoch 103 Epoch 103 completed. Avg Loss: 0.02812921069562435, Time: 12.28s, Best Loss: 0.02812921069562435 Epoch 104/120 Epoch 104: 200step [00:12, 15.86step/s, loss=0.0322] Epoch 104 completed. Avg Loss: 0.029146283864974976, Time: 12.61s, Best Loss: 0.02812921069562435 Epoch 105/120 Epoch 105: 200step [00:12, 16.29step/s, loss=0.0405] Epoch 105 completed. Avg Loss: 0.029454702511429787, Time: 12.28s, Best Loss: 0.02812921069562435 Epoch 106/120 Epoch 106: 200step [00:12, 16.31step/s, loss=0.0309] Epoch 106 completed. Avg Loss: 0.02946491166949272, Time: 12.27s, Best Loss: 0.02812921069562435 Epoch 107/120 Epoch 107: 200step [00:12, 16.29step/s, loss=0.0417] Epoch 107 completed. Avg Loss: 0.03041270188987255, Time: 12.28s, Best Loss: 0.02812921069562435 Epoch 108/120 Epoch 108: 200step [00:12, 16.31step/s, loss=0.0270] Epoch 108 completed. Avg Loss: 0.028621835634112358, Time: 12.27s, Best Loss: 0.02812921069562435 Epoch 109/120 Epoch 109: 200step [00:12, 15.88step/s, loss=0.0219] Epoch 109 completed. Avg Loss: 0.029894324019551277, Time: 12.60s, Best Loss: 0.02812921069562435 Epoch 110/120 Epoch 110: 200step [00:12, 16.29step/s, loss=0.0244] Epoch 110 completed. Avg Loss: 0.029655728489160538, Time: 12.29s, Best Loss: 0.02812921069562435 Epoch 111/120 Epoch 111: 200step [00:12, 16.27step/s, loss=0.0290] Epoch 111 completed. Avg Loss: 0.030832564458251, Time: 12.30s, Best Loss: 0.02812921069562435 Epoch 112/120 Epoch 112: 200step [00:12, 16.28step/s, loss=0.0305] Epoch 112 completed. Avg Loss: 0.030523793771862984, Time: 12.29s, Best Loss: 0.02812921069562435 Epoch 113/120 Epoch 113: 200step [00:12, 16.29step/s, loss=0.0231] Epoch 113 completed. Avg Loss: 0.02961576171219349, Time: 12.28s, Best Loss: 0.02812921069562435 Epoch 114/120 Epoch 114: 200step [00:12, 15.79step/s, loss=0.0271] Epoch 114 completed. Avg Loss: 0.028563303872942924, Time: 12.68s, Best Loss: 0.02812921069562435 Epoch 115/120 Epoch 115: 200step [00:12, 16.29step/s, loss=0.0267] Epoch 115 completed. Avg Loss: 0.029559249058365822, Time: 12.29s, Best Loss: 0.02812921069562435 Epoch 116/120 Epoch 116: 200step [00:12, 16.30step/s, loss=0.0240] Epoch 116 completed. Avg Loss: 0.02985670045018196, Time: 12.28s, Best Loss: 0.02812921069562435 Epoch 117/120 Epoch 117: 200step [00:12, 16.31step/s, loss=0.0273] Epoch 117 completed. Avg Loss: 0.029099639505147934, Time: 12.27s, Best Loss: 0.02812921069562435 Epoch 118/120 Epoch 118: 200step [00:12, 16.31step/s, loss=0.0221] Epoch 118 completed. Avg Loss: 0.028886837884783745, Time: 12.27s, Best Loss: 0.02812921069562435 Epoch 119/120 Epoch 119: 200step [00:12, 16.29step/s, loss=0.0243] Epoch 119 completed. Avg Loss: 0.029581556096673012, Time: 12.28s, Best Loss: 0.02812921069562435 Epoch 120/120 Epoch 120: 200step [00:12, 15.91step/s, loss=0.0310] WARNING:absl:Waiting for previous save to complete took 2.844717 seconds. If this number is high, consider checkpointing less frequently. Epoch 120 completed. Avg Loss: 0.029415404424071312, Time: 12.58s, Best Loss: 0.02812921069562435 Saving model at epoch 120
Generate Samples
Section titled “Generate Samples”The Intuition of reverse diffusion process
Section titled “The Intuition of reverse diffusion process”As described before, the process of diffusion is basically traveling from a point in the input space of low entropy to a point of high entropy (forward time diffusion) or vice versa (the reverse diffusion).
Let’s try to understand the reverse diffusion process as forward diffusion is easy to understand: Things go from hot to cold, from ordered to unordered, things diffuse out from order to chaos aka noise. We have talked about it in the introduction to this notebook.
Every image or noise aka sample is basically a point in a high dimensional input space (to be precise, in a 64 x 64 x 3 dimensions space!). In the start, to generate a new image, you randomly start from some random point in the input space (as the initial sample image you start with for generating your images are random Gaussian noise). Now, as the point you started with was random, it’s most probable that it’s not a good image resembling your dataset. After all, high dimensional spaces are REALLY FREAKING HUGE!
So, we need to find our way to some useful point in this input space which actually resembles some meaningful image (which there would be for sure as by definition, in an infinite space, everything exists!). But randomly trying to stumble your way to it would be really hard and time taking (like really really really time taking, black holes would die before you find a good image). So, we need to find a way to guide us. And that’s what those pesky differential equations are for! They describe a path way back from noise to signal aka data.
The idea is simple. We have a neural network model that, provided a noisy image, can try to predict the denoised image. This means that given the current point in the input space, the model can learn to predict some point in its vicinity that is more likely to be a good viable image. But of course neural networks are imperfect. What GANs and VAEs etc failed to realize was this very fact that you won’t get a perfect image in one go. We should treat it as a rough guess. What we then do is, take a very small step in the direction of the predicted image, and then predict again. This time the idea is that the model’s prediction would be closer to some meaningful image. We keep doing this until we reach a point where the model’s prediction is very close to the actual image. This is the reverse diffusion process.
Now, this process of taking a small step in the direction of the predicted image, which honestly is indeed gradient descent but in input space, can be thought in the way that you actually went to the predicted point, and then came back again near to our original point but a bit closer to the predicted point, i.e., you traveled all the way to the prediction and then came mostly back but not quite completely. This is the same as saying you took a small step in that direction but this idea is actually exactly how the simplest technique for generating images aka sampling works: The DDIM (Denoising Diffusion Implicit Model). Samplers of this type are broadly defined as ordinary differential equation (ODE) solvers as our reverse diffusion process is defined as an ODE (reminder, a differential equation is just a fancy way of talking about how things change with respect to some other thing like time).
Now that we think of it this way, there is also another thing that we can do, when we get to the predicted point, we had actually a choice that we could either come back precisely in the direction of the original point, or we could come back in the rough direction (i.e., mostly the direction but with some random uncertainty). What this does is, it allows the model to try to find that perfect data point from multiple other directions rather than the same direction, allowing for even better error minimization and thus better quality (at least mostly, although it’s painfully slow, in fact the slowest method for sampling). This is the idea behind the DDPM (Denoising Diffusion Probabilistic Model). These samplers are defined as stochastic differential equation (SDE) solvers as our reverse diffusion process is defined as an SDE.
Sampling from the model
Section titled “Sampling from the model”Sampling or generating images is basically the reverse diffusion process. It can either be thought of as a Markov chain of conditional distributions if we are using the DDPM’s discrete time step formulation, or as numerically integrating a stochastic differential equation (SDE) if we are talking about general continuous time step formulations. But even in the case of the SDE, we actually would be integrating the SDE over discrete time steps of course, because we can’t really integrate over infinite time steps in a computer, we simply do not have infinite time.
Although we would like to follow the formulations of DDPM paper even further, for the sake of simplicity and ease of understanding, we shall use the differential equation formulations in this notebook for generating samples. But keep in mind that both describe the same reverse process: DDPM’s Markov chain is one particular discretization of the reverse-time SDE (Song et al., 2021, Score-Based Generative Modeling through Stochastic Differential Equations). With a finite number of steps, different discretizations don’t give exactly the same images, but they approach the same process as the steps get smaller.
As discussed previously, the reverse diffusion process can be thought of as a stochastic differential equation (SDE):
where:
- is the infinitesimal change in the data sample at time ,
- is the drift term from the forward process,
- is the diffusion term from the forward process,
- is the gradient of the log-density of the data at time ,
- is the infinitesimal increment of a reverse-time Wiener process (or standard Brownian motion).
or a corresponding probability flow ordinary differential equation (ODE):
where:
- is the time derivative of the data sample ,
- is the drift term from the forward process,
- is the diffusion term from the forward process,
- is the gradient of the log-density of the data at time .
We model the term as a neural network. Our current formulation of ‘denoising’ or noise predicting (epsilon predicting) model is
Thus we have a detailed mathematical description of our reverse diffusion process. But having a differential equation is just having a description of the process, to actually get something out of it like what happens at the end of the process, we need to solve the differential equation. And by solving, we mean to integrate it over a range of time steps . Doing this shall get us to a final generated sample given an initial starting sample.
These integrators are defined in Dew as Solver classes.
We shall now first define a general abstract DiffusionSampler class, which we can later extend to implement various samplers.
def clip_images(images, clip_min=-1, clip_max=1): return jnp.clip(images, clip_min, clip_max)
class DiffusionSampler(): model:nn.Module noise_schedule:NoiseScheduler params:dict model_output_transform:DiffusionPredictionTransform
def __init__(self, model:nn.Module, params:dict, noise_schedule:NoiseScheduler, model_output_transform:DiffusionPredictionTransform=EpsilonPredictionTransform()): self.model = model self.noise_schedule = noise_schedule self.params = params self.model_output_transform = model_output_transform
@jax.jit def sample_model(x_t, t): rates = self.noise_schedule.get_rates(t) c_in = self.model_output_transform.get_input_scale(rates) model_output = self.model.apply(self.params, *self.noise_schedule.transform_inputs(x_t * c_in, t)) x_0, eps = self.model_output_transform(x_t, model_output, t, self.noise_schedule) # print(f"model output {jnp.mean(model_output)}, x_0 {jnp.mean(x_0)}, eps {jnp.mean(eps)}") return x_0, eps, model_output
self.sample_model = sample_model
# Used to sample from the diffusion model def sample_step(self, current_samples:jnp.ndarray, current_step, next_step=None, state:MarkovState=None) -> tuple[jnp.ndarray, MarkovState]: # First clip the noisy images step_ones = jnp.ones((current_samples.shape[0], ), dtype=jnp.int32) current_step = step_ones * current_step next_step = step_ones * next_step pred_images, pred_noise, _ = self.sample_model(current_samples, current_step) # plotImages(pred_images) pred_images = clip_images(pred_images) new_samples, state = self.take_next_step(current_samples=current_samples, reconstructed_samples=pred_images, pred_noise=pred_noise, current_step=current_step, next_step=next_step, state=state) return new_samples, state
def take_next_step(self, current_samples, reconstructed_samples, pred_noise, current_step, state:RandomMarkovState, next_step=1) -> tuple[jnp.ndarray, RandomMarkovState]: # estimate the p(x_{t-1} | x_t, x_0). # pred_images is x_0, noisy_images is x_t, steps is t return NotImplementedError
def scale_steps(self, steps): scale_factor = self.noise_schedule.max_timesteps / 1000 return steps * scale_factor
def get_steps(self, start_step, end_step, diffusion_steps): step_range = start_step - end_step if diffusion_steps is None or diffusion_steps == 0: diffusion_steps = start_step - end_step diffusion_steps = min(diffusion_steps, step_range) steps = jnp.linspace(end_step, start_step, diffusion_steps, dtype=jnp.int16)[::-1] return steps
def get_initial_samples(self, num_images, rngs:jax.random.PRNGKey, start_step, image_size=64): start_step = self.scale_steps(start_step) alpha_n, sigma_n = self.noise_schedule.get_rates(start_step) variance = jnp.sqrt(alpha_n ** 2 + sigma_n ** 2) return jax.random.normal(rngs, (num_images, image_size, image_size, 3)) * variance
def generate_images(self, num_images=16, diffusion_steps=1000, start_step:int = None, end_step:int = 0, steps_override=None, priors=None, rngstate:RandomMarkovState=RandomMarkovState(jax.random.PRNGKey(42))) -> jnp.ndarray: if priors is None: rngstate, newrngs = rngstate.get_random_key() samples = self.get_initial_samples(num_images, newrngs, start_step) else: print("Using priors") samples = priors
@jax.jit def sample_step(state:RandomMarkovState, samples, current_step, next_step): samples, state = self.sample_step(current_samples=samples, current_step=current_step, state=state, next_step=next_step) return samples, state
if start_step is None: start_step = self.noise_schedule.max_timesteps
if steps_override is not None: steps = steps_override else: steps = self.get_steps(start_step, end_step, diffusion_steps)
# print("Sampling steps", steps) for i in tqdm.tqdm(range(0, len(steps))): current_step = self.scale_steps(steps[i]) next_step = self.scale_steps(steps[i+1] if i+1 < len(steps) else 0) if i != len(steps) - 1: # print("normal step") samples, rngstate = sample_step(rngstate, samples, current_step, next_step) else: # print("last step") step_ones = jnp.ones((num_images, ), dtype=jnp.int32) samples, _, _ = self.sample_model(samples, current_step * step_ones) samples = clip_images(samples) return samplesDDIM Sampler
Section titled “DDIM Sampler”The Denoising Diffusion Implicit Model (DDIM) Sampler implements the idea we discussed above, that we shall, starting from the current noisy point in the input space, get to the predicted point by the model, and then come almost all the way back in the precise direction of our original image. This ‘almost all the way’ is defined by the magnitude of how noise rates change in our noise schedule over time. This is the simplest and the fastest way to sample from the model and can generate good resolution images within a fraction of the time steps that going through each time step would take. e.g., if the total time steps we were talking about were 1000, this method would take only 100 time steps to generate a good image. But the quality won’t be as good as DDPM or SDE solvers of course.
where:
- is the data sample at time aka the next predicted sample,
- is the initial data sample as predicted by our model from the current sample ,
- is the noise our model predicts from the current sample . DDIM reuses this predicted noise instead of drawing fresh noise, which is why it is deterministic.
class DDIMSampler(DiffusionSampler): def take_next_step(self, current_samples, reconstructed_samples, pred_noise, current_step, state:RandomMarkovState, next_step=None) -> tuple[jnp.ndarray, RandomMarkovState]: next_signal_rate, next_noise_rate = self.noise_schedule.get_rates(next_step) return reconstructed_samples * next_signal_rate + pred_noise * next_noise_rate, statesampler = DDIMSampler(trainer.model, trainer.best_state.ema_params, cosine_schedule, model_output_transform=EpsilonPredictionTransform())samples = sampler.generate_images(num_images=NUM_SAMPLES, diffusion_steps=100, start_step=1000, end_step=0, priors=None)plotImages(samples, dpi=150)stderr
100%|██████████| 100/100 [00:36<00:00, 2.77it/s]

Euler Method
Section titled “Euler Method”Now that we have looked at the simplest possible sampler according to our intuition, let’s now look at solving, aka integrating, the ODEs and SDEs we talked about.
The Euler Method is the simplest ODE/SDE solver to implement and thus understand. To explain what the Euler method (and thus a solver) is, we shall first look at how to integrate a very simple and familiar differential equation, the differential equation of an object in constant acceleration.
When an object is in constant acceleration, it means that its speed is increasing at a constant rate. We were taught in our high school physics classes that the speed of an object at any time is given by the equation:
where:
- is the speed of the object at time ,
- is the initial speed of the object,
- is the acceleration of the object,
- is the time.
In terms of position, the position of the object at any time is given by the equation:
eqn 1
where:
- is the position of the object at time ,
- is the initial position of the object.
We can see that an object under constant acceleration would have a quadratic position-time graph, which would look like a parabola.
Well, the above equations are actually integrals. What acceleration really is is the rate of change of speed which in itself is rate of change of position. So, we can also say that for an object under constant acceleration, its rate of change of speed is constant:
Which basically means that its rate of change of position (i.e., speed) is increasing at a constant rate:
Thus we can have a differential equation for the position of the object as:
This differential equation captures the essence of an object under constant acceleration and describes how it will behave over time. This is what differential equations are for, to describe how a system will evolve over time or some other variable. This is a very simple differential equation and I bet you can solve i.e., integrate it using high school calculus to get the equation 1. But differential equations are not always as simple to integrate as this one. For more complex systems, we need to use numerical methods to solve them.
And that’s where the Euler method comes in. Taking our accelerating body example, if we want to know the actual position of the object w.r.t time , our differential equation tells us that it’s some value changing over time in some very small time steps (infinitesimal in reality but that’s not possible in reality so let’s stick with ‘very small’). Well, the integral is just a sum of all these very small changes over time i.e., the area under the curve. So why not just do exactly that? That’s what the Euler method does.
To illustrate, let’s just define our differential equation for the position of the object under constant acceleration as:
# Simple differential equation for increasing speed with constant rate (constant acceleration)# dX = X0 + a * t; the integrator applies the step size dtdef dX(dt: float, X0: float, t: float, a: float = 1, **params) -> float: return X0 + t * a
# We can plot the function to see how dX changes with timet = jnp.linspace(0, 10, 1000)X0 = 0dt = 1a = 1plt.plot(t, dX(dt, X0, t, a))# We should get a straight line with slope a to signify that the speed is increasing linearly[<matplotlib.lines.Line2D at 0x7a9bc07bc2d0>]

# Now let us integrate this linear function to get the positiondef eulerIntegrate(dX: Callable, X0: float, dt: float, max_t:float = 1000, **params): steps = int(max_t / dt) X = [] t = [] t0 = 0 Xval = X0 tVal = t0 for i in jnp.linspace(0, max_t, steps): t.append(tVal) X.append(Xval) Xval += dX(X0=X0, dt=dt, X=Xval, t=tVal, **params) * dt tVal += dt return jnp.array(t), jnp.array(X)
def plotDiff(dX: Callable, X0: float, dt: float, max_t:float = 1000, **params): # Integrate the system t, X = eulerIntegrate(dX, X0, dt, max_t, **params) plt.plot(t, X, linestyle='dashdot')
plotDiff(dX=dX, X0=X0, dt=dt, a=a, max_t=10)# We should get a parabolic curve to signify that the position is increasing quadratically
# Now let us plot the actual X(t) = X0 + V0 * t + 0.5 * a * t^2def X(t: float, X0: float, V0: float, a: float) -> float: return X0 + V0 * t + 0.5 * a * t ** 2
# We can plot the function to see how X changes with timeplt.plot(t, X(t, X0, 0, a), linestyle='solid')[<matplotlib.lines.Line2D at 0x7a9c8417bc50>]

From the above demonstration, we can see that the Euler method does give us an approximation of the actual solution to the differential equation. But the approximation is not perfect because the timesteps are not infinitesimally small. There are other more advanced numerical solvers that can integrate even better than this with the same imperfect timesteps as well such as Heun’s method, Runge-Kutta methods, etc. But the Euler method is the simplest and easiest to understand and implement.
The Euler Sampler
Section titled “The Euler Sampler”The Euler Sampler is basically our SDE/ODE solver that integrates the reverse diffusion process over given time steps and implements our DiffusionSampler class. In this notebook, we shall define the EulerSampler to be a solver of the corresponding probability flow ODE of our SDE diffusion process. This basically means that there would be no new randomness introduced in the integration process and is as simple as it can get i.e., just like the Euler method for the constant acceleration example. EulerSampler is thus our implementation of the Euler method for solving the reverse diffusion probability flow ODE. If we also add fresh noise at every step, the way DDPM’s ancestral sampling does, we get what is called the Euler-Ancestral or Euler-A sampler. The idea is the same, but there is some stochasticity aka noise in that process.
This sampler is actually also equivalent to the DDIM technique: written out, its update is exactly DDIM’s. Only the clipping of the predicted image can make the two differ slightly, because DDIM uses the model’s own noise prediction while this sampler uses the noise implied by the clipped image. That’s no accident. Song, Meng and Ermon (2021, Denoising Diffusion Implicit Models, Section 4.3) show that DDIM’s update is Euler’s method applied to the probability flow ODE written in against , and Karras et al. (2022, Elucidating the Design Space of Diffusion-Based Generative Models) describe DDIM the same way, as Euler’s method with their and . Euler’s method in other variables, say plain against , gives a different sampler. As an exercise, you can try to find the algebraic equivalence between the two.
class EulerSampler(DiffusionSampler): # Basically a DDIM Sampler but parameterized as an ODE def take_next_step(self, current_samples, reconstructed_samples, pred_noise, current_step, state:RandomMarkovState, next_step=None) -> tuple[jnp.ndarray, RandomMarkovState]: current_alpha, current_sigma = self.noise_schedule.get_rates(current_step) next_alpha, next_sigma = self.noise_schedule.get_rates(next_step)
dt = next_sigma - current_sigma # Actually its not dt at all, but rather dsigma
# We get the following X0 coefficient from simple algebra over trying to find the differential equation for # our forward processes formulated as X_t = alpha * X_0 + sigma * epsilon. You can do the algebra bit yourself as # an exercise. x_0_coeff = (current_alpha * next_sigma - next_alpha * current_sigma) / (dt)
dx = (current_samples - x_0_coeff * reconstructed_samples) / current_sigma
next_samples = current_samples + dx * dt # The Euler Integration Step return next_samples, statesampler = EulerSampler(trainer.model, trainer.state.ema_params, cosine_schedule, model_output_transform=EpsilonPredictionTransform())samples = sampler.generate_images(num_images=NUM_SAMPLES, diffusion_steps=100, start_step=1000, end_step=0, priors=None)plotImages(samples, dpi=150)stderr
100%|██████████| 100/100 [00:20<00:00, 4.79it/s]

Please ignore the overexposure of the images for now, should not have occurred, will fix soon.
The (Denoising Diffusion Probabilistic Model) DDPM Sampler
Section titled “The (Denoising Diffusion Probabilistic Model) DDPM Sampler”Above, just for sake of simplicity and to have the essence of the concept, we defined the DDIMSampler EulerSampler, but now that we have seen that our model works and how to sample from it, let’s talk about the DDPM sampler as presented in the original DDPM paper. In the introduction to this notebook, we discussed that the reverse diffusion process is defined as a Markov chain of conditional distributions as follows:
where:
- is the data sample at time ,
- is the data sample at time ,
- is the mean of the reverse distribution,
- is the covariance of the reverse distribution.
Therefore, if we want to generate images following the DDPM technique (one particular discretization of the reverse diffusion SDE, Song et al., 2021, Score-Based Generative Modeling through Stochastic Differential Equations; it is also what the Euler-Ancestral sampler computes when it adds the full amount of fresh noise, see the fun fact below), we need to follow the above Markov chain of conditional distributions. This is what our DDPM Sampler does.
The Implementation
Section titled “The Implementation”The term i.e., the mean of the conditional distribution is the term parameterized by our model (with weights ), but it’s not just the model. To be precise, In the DDPM paper, this term is defined as:
where:
- is the data sample at time ,
- is defined as , This is not the same as our signal rate that we use, so be careful with the notation,
- is defined as ,
- is the parameter of the noise schedule at time which we discussed in the Cosine Noise Scheduler in terms of section,
- is the denoising model (i.e., model trained in epsilon prediction model), and are the parameters of the denoising model.
Don’t forget, the way forward diffusion is defined in DDPM is:
Thus, the signal term in DDPM is and the noise term is . Remember this to understand our DDPM sampler.
The term i.e., the covariance of the conditional distribution is defined as:
Ho et al. (Denoising Diffusion Probabilistic Models, 2020) try two choices for : , and
where:
- is the standard deviation of the new noise added at time ,
- is the parameter of the noise schedule at time which we discussed above.
Our DDPM samplers below use .
We shall implement the DDPM Sampler in two variations: The exact formulation we discussed as DDPMSampler, and an algebraically equivalent but more simplified version as SimpleDDPMSampler. You can try to find the algebraic equivalence between the two as an exercise.
As the DDPM paper uses sigma as the standard deviation of the new noise but we use it as the noise rate aka standard deviation of all the noise in general, we shall use the term instead:
As we do not have access to the parameter of the noise schedule, we can calculate it back from the current and next signal rates (or noise rates for that matter) as follows:
Fun Fact, DDIM and DDPM are two ends of one family of samplers (Song, Meng and Ermon, 2021). DDIM’s generalized update has a knob for how much fresh noise to add: with it is exactly DDPM with , and with it adds no fresh noise and reuses the model’s predicted noise instead, which is the DDIM we used above. This fits the picture from before: DDIM is Euler’s method on the probability flow ODE, and DDPM’s ancestral step is a discretization of the reverse SDE of the same process. You can try to find the algebraic equivalence between the two as an exercise.
class DDPMSampler(DiffusionSampler): def take_next_step(self, current_samples, reconstructed_samples, pred_noise, current_step, state:RandomMarkovState, next_step=None) -> tuple[jnp.ndarray, RandomMarkovState]: state, rng = state.get_random_key() new_noise = jax.random.normal(rng, reconstructed_samples.shape, dtype=jnp.float32) current_signal_rate, current_noise_rate = self.noise_schedule.get_rates(current_step) next_signal_rate, next_noise_rate = self.noise_schedule.get_rates(next_step)
noise_ratio_squared = (next_noise_rate ** 2) / (current_noise_rate ** 2) signal_ratio_squared = (current_signal_rate ** 2) / (next_signal_rate ** 2)
betas = (1 - signal_ratio_squared) gamma = jnp.sqrt(noise_ratio_squared * betas)
something_small = 1e-6 # To prevent division by zero
next_samples = (1 / (jnp.sqrt(1 - betas) + something_small)) * (current_samples - betas * (pred_noise / current_noise_rate)) + new_noise * gamma return next_samples, statesampler = DDPMSampler(trainer.model, trainer.state.ema_params, cosine_schedule, model_output_transform=EpsilonPredictionTransform())samples = sampler.generate_images(num_images=NUM_SAMPLES, diffusion_steps=998, start_step=998, end_step=0, priors=None)plotImages(samples, dpi=150)stderr
100%|██████████| 998/998 [00:54<00:00, 18.46it/s]

An interesting thing about the SimpleDDPMSampler is that, unlike DDPMSampler where we need to take all the time steps to generate a sample, we can actually generate a sample in a fraction of the time steps as well, just like DDIMSampler. Don’t ask me why; it’s just something I realized after I implemented the simplified equations I derived on the backside of some bills.
class SimpleDDPMSampler(DiffusionSampler): def take_next_step(self, current_samples, reconstructed_samples, pred_noise, current_step, state:RandomMarkovState, next_step=None) -> tuple[jnp.ndarray, RandomMarkovState]: state, rng = state.get_random_key() noise = jax.random.normal(rng, reconstructed_samples.shape, dtype=jnp.float32)
# Compute noise rates and signal rates only once current_signal_rate, current_noise_rate = self.noise_schedule.get_rates(current_step) next_signal_rate, next_noise_rate = self.noise_schedule.get_rates(next_step)
pred_noise_coeff = ((next_noise_rate ** 2) * current_signal_rate) / (current_noise_rate * next_signal_rate)
noise_ratio_squared = (next_noise_rate ** 2) / (current_noise_rate ** 2) signal_ratio_squared = (current_signal_rate ** 2) / (next_signal_rate ** 2) gamma = jnp.sqrt(noise_ratio_squared * (1 - signal_ratio_squared))
next_samples = next_signal_rate * reconstructed_samples + pred_noise_coeff * pred_noise + noise * gamma return next_samples, statesampler = SimpleDDPMSampler(trainer.model, trainer.state.ema_params, cosine_schedule, model_output_transform=EpsilonPredictionTransform())samples = sampler.generate_images(num_images=NUM_SAMPLES, diffusion_steps=100, start_step=1000, end_step=0, priors=None)plotImages(samples, dpi=150)stderr
100%|██████████| 100/100 [00:21<00:00, 4.76it/s]

The Heun Sampler for ODEs
Section titled “The Heun Sampler for ODEs”We have already looked at the Euler Sampler above. Even with the simple case of integrating for acceleration, we saw that the Euler method was not perfect and the approximation was not very accurate. I said then that we could use more advanced methods to increase the accuracy, well Heun is one such method. Heun’s method is a second order solver, and below we use it to solve the ODE probability flow of the reverse diffusion process. If you look at the code and try to compare it with Euler Sampler, you would realize the similarities. It’s basically taking an Euler step, then making a new prediction on that step, then taking the average of the two predictions as the final output of the overall step. Thus Heun would actually do 2 model predictions on every step, but in reality it justifies itself with the faster convergence and quality. Heun’s method can generate samples in 6-10 steps as compared to ~50 steps of Euler (20 model evals vs 50).
class HeunSampler(DiffusionSampler): def take_next_step(self, current_samples, reconstructed_samples, pred_noise, current_step, state:RandomMarkovState, next_step=None) -> tuple[jnp.ndarray, RandomMarkovState]: # Get the noise and signal rates for the current and next steps current_alpha, current_sigma = self.noise_schedule.get_rates(current_step) next_alpha, next_sigma = self.noise_schedule.get_rates(next_step)
dt = next_sigma - current_sigma x_0_coeff = (current_alpha * next_sigma - next_alpha * current_sigma) / dt
dx_0 = (current_samples - x_0_coeff * reconstructed_samples) / current_sigma next_samples_0 = current_samples + dx_0 * dt
# Recompute x_0 and eps at the first estimate to refine the derivative state, rng = state.get_random_key() c_in = self.model_output_transform.get_input_scale((next_alpha, next_sigma)) model_output = self.model.apply(self.params, *self.noise_schedule.transform_inputs(next_samples_0*c_in, next_step)) estimated_x_0, _ = self.model_output_transform(next_samples_0, model_output, next_step, self.noise_schedule) estimated_x_0 = clip_images(estimated_x_0) # Estimate the refined derivative using the midpoint (Heun's method) dx_1 = (next_samples_0 - x_0_coeff * estimated_x_0) / next_sigma # Compute the final next samples by averaging the initial and refined derivatives final_next_samples = current_samples + 0.5 * (dx_0 + dx_1) * dt
return final_next_samples, state
def generate_images(self, num_images=16, diffusion_steps=1000, start_step: int = None, *args, **kwargs): return super().generate_images(num_images=num_images, diffusion_steps=diffusion_steps, start_step=start_step, *args, **kwargs)sampler = HeunSampler(trainer.model, trainer.state.ema_params, cosine_schedule, model_output_transform=EpsilonPredictionTransform())samples = sampler.generate_images(num_images=NUM_SAMPLES, diffusion_steps=20, start_step=1000, end_step=0, priors=None)plotImages(samples, dpi=150)stderr
100%|██████████| 20/20 [00:24<00:00, 1.20s/it]
