Skip to content

dew.diffusion.process

The diffusion process a model is trained and sampled with, and the denoiser that runs a model on it.

NameSummary
DenoiserDenoises with one model, its parameters and its conditions.
DenoisingConditionText conditioning in the form the published model families read it.
ProcessPairs a noise schedule with what the model predicts on it and how the loss is weighted.
aligned_conditionsReturn unconditional with each row’s own model inputs taken from conditions.

dataclass source

class Denoiser(
process: Process,
model: nn.Module,
params: Variables,
conditions: dict[str, Conditioning],
unconditional: dict[str, Conditioning] | None,
schedule: NoiseScheduler,
opened: tuple[str, ...] = (),
)

Denoises with one model, its parameters and its conditions.

A call computes the model’s raw output at (x_t, t) and then the process’s conversion of it into (x_0, epsilon). The two steps are separate methods because a conversion is not always linear in the output. Dynamic thresholding and sample clipping limit x_0, and a consistency boundary computes a function of it, so combining two raw outputs after conversion gives a different result from combining them before. Guidance therefore calls raw_both, combines the raw outputs and converts once, in the same order a source pipeline runs its scheduler.

opened: tuple[str, ...]

The collections its model’s calls open and return beside their outputs: what a watched step reads of the forward that trains (raw_observed). A guidance’s prediction on an opened denoiser returns its pair and these (observed), through any transform of its own, as the Guidance protocol asks; none leaves every call’s output alone.

schedule: NoiseScheduler

The schedule t is on: the process’s sampling schedule for a solver’s grid, its training schedule for a training objective’s draw.

def convert(x_t, t, output) -> tuple[jax.Array, jax.Array]

Return (x_0, epsilon) recovered from a raw model output at (x_t, t).

def predict(
x_t,
t,
) -> tuple[jax.Array, jax.Array] | tuple[tuple[jax.Array, jax.Array], Variables | None]

(x_0, epsilon) at (x_t, t) as an unguided walk predicts it, observed where the denoiser is opened (observed).

def observed(
pair: tuple[jax.Array, jax.Array],
sown: Variables | None,
) -> tuple[jax.Array, jax.Array] | tuple[tuple[jax.Array, jax.Array], Variables | None]

A prediction’s return: pair, and beside it what its model’s calls opened (opened), where the denoiser opens any; the shape every guidance’s prediction returns.

def spanning(t, t_next) -> Denoiser

Return this denoiser over the interval from t to t_next.

Every call of the returned denoiser passes the interval’s length in model time to the model as the duration condition.

def raw(x_t, t) -> jax.Array

Return the model’s raw output at (x_t, t) under the conditions.

def raw_observed(x_t, t) -> tuple[jax.Array, Variables | None]

The model’s raw output at (x_t, t) under the conditions, and what the call opened (opened), None where it opens none.

def raw_both(x_t, t) -> tuple[jax.Array, jax.Array]

Return the conditional and the unconditional raw outputs.

def raw_both_observed(x_t, t) -> tuple[tuple[jax.Array, jax.Array], Variables | None]

Return the conditional and the unconditional raw outputs, and what their call opened.

Both come from one model call over the doubled batch. Raises ValueError when the denoiser has no unconditional conditions.

dataclass source

class DenoisingCondition(
pooled: jax.Array | None = None,
time_ids: jax.Array | None = None,
guidance: jax.Array | None = None,
mask: jax.Array | None = None,
)

Text conditioning in the form the published model families read it.

  • context holds the token states.
  • pooled is a pooled vector.
  • time_ids are the size and crop ids that the XL towers add.
  • guidance is the distilled guidance value, which a guidance-embedded transformer takes as a model input instead of running two guided branches.
  • mask is the [B, tokens] mask of the real token states in a right-padded context, read by a family that excludes padded keys.
def aligned(given: DenoisingCondition) -> DenoisingCondition

Return this conditioning with given’s distilled guidance value.

The guidance value belongs to the row and not to its caption, so a dropped or unconditional caption keeps the guidance scale the checkpoint samples at.

dataclass source

class Process(
schedule: NoiseScheduler,
prediction: PredictionTransform,
weighting: Weighting = ScheduleWeighting(),
sampling: NoiseScheduler | None = None,
interval: bool = False,
)

Pairs a noise schedule with what the model predicts on it and how the loss is weighted.

sampling is the schedule that inference integrates when it differs from the training schedule. For example, EDM trains on log-normal sigmas and samples on the Karras grid. None means sampling uses the training schedule.

interval is whether the model predicts over an interval of time and not at an instant, as MeanFlow’s average velocity and a shortcut model’s step do. Such a model reads the interval’s length in model time as the duration condition. At each step, sample gives the denoiser the interval to the next grid point (Denoiser.spanning), so a single-evaluation solver such as Euler covers the whole interval in one step. A zero duration gives the instantaneous prediction.

def weight(t) -> jax.Array
def rates(t, *, like: jax.Array) -> tuple[jax.Array, jax.Array]

Return the sampling schedule’s (alpha, sigma) at t, shaped to broadcast against like.

like is a [B, ...] state.

def variances(t, *, like: jax.Array) -> tuple[jax.Array, jax.Array]

Return the sampling schedule’s (alpha^2, sigma^2) at t (NoiseScheduler.variances), shaped to broadcast against like.

def times(steps: int) -> jax.Array

Return the descending grid of steps times, from T to 0, that a solver steps through.

A tabulated schedule cannot take more steps than it has entries, so steps is capped at T there.

def noise(key, shape) -> jax.Array

Draw an array of shape from the sampling schedule’s Gaussian prior.

By default the prior’s standard deviation is that of the marginal at T for unit-variance data. A schedule may declare a different scale through prior_scale. The draw is float32’s whatever the default dtype, as the samplers’ innovations are, so a float64 walk from a key starts from the float32 walk’s numbers.

def denoiser(
model,
params,
conditions: Mapping[str, Conditioning],
unconditional: Mapping[str, Conditioning] | None = None,
*,
schedule: NoiseScheduler | None = None,
opened: Sequence[str] = (),
) -> Denoiser

Return the denoiser of model under params with the given conditions.

The denoiser maps (x_t, t) to (x_0, epsilon) with t on schedule, the sampling schedule unless a training objective passes its training schedule for the t it drew. unconditional has the same keys with the unconditional values, and classifier-free guidance interpolates against it. opened names the collections its model’s calls return beside their outputs (Denoiser.opened), a watched step’s (dew.objectives.base.Step.sown).

function source

def aligned_conditions(
conditions: Mapping[str, Conditioning],
unconditional: Mapping[str, Conditioning],
) -> dict[str, Conditioning]

Return unconditional with each row’s own model inputs taken from conditions.

A keyword must name a conditioning record on both sides or on neither. Array keywords, such as the spatial ones an inpainting source adds, pass through unchanged. A keyword that is a record on one side only means the caller paired two different conditionings, so it raises ValueError.