Skip to content

dew.objectives.diffusion

Diffusion objectives and their distillations, each imported from its module when first used, so importing one of the package’s modules, or a name from one, reaches only what that module imports.

NameSummary
VALIDATION_SAMPLES
AdversarialDistillationObjectiveTrains LADD, with ADD’s R1 penalty and its distillation term. Documented in dew.objectives.diffusion.adversarial.
AlignmentAligns the model’s hidden tokens at layer with a frozen encoder’s features. Documented in dew.objectives.diffusion.alignment.
BlockDiffusionObjectiveFine-tunes DiffusionGemma on clean text rows split into a prompt and canvases.
ConsistencyDistillationObjectiveTrains rCM: sCM distillation of a flow teacher, regularized by DMD2. Documented in dew.objectives.diffusion.consistency.
DiffusionObjectiveTrains a denoising diffusion model: draw a noise level, corrupt, predict, and weight the loss.
EndToEndREPA-E’s end-to-end tuning of the autoencoder. Documented in dew.objectives.diffusion.end_to_end.
GuidanceDistillationObjectiveDistills a teacher’s classifier-free guidance into this model. Documented in dew.objectives.diffusion.guidance_distillation.
MaskedDiffusionObjectiveTrains a masked diffusion model on the MDLM negative ELBO.
MeanFlowObjectiveTrains MeanFlow on an interval process (presets.MeanFlow). Documented in dew.objectives.diffusion.few_step.
ShortcutObjectiveTrains a shortcut model on an interval process (presets.Shortcut). Documented in dew.objectives.diffusion.few_step.

attribute source

VALIDATION_SAMPLES = 4

class source

class BlockDiffusionObjective(
model: DiffusionGemma | Source,
*,
prompt_length: int,
num_canvases: int = 1,
canvas_size: int | None = None,
variables: Variables | None | Omitted = OMITTED,
pad_token_id: int = 0,
self_cond_prob: float = 0.5,
safety_epsilon: float = 0.0001,
stop_gradient_from_denoiser_to_encoder: bool = False,
encoder_loss_weight: float = 1.0,
decoder_loss_weight: float = 1.0,
ema_decay: float | None = None,
head_chunks: int = 4,
head_tile: tuple[int, int] | Literal['whole', 'tiled'] | None = None,
processor: Processor | None | Omitted = OMITTED,
eos_token_ids: Sequence[int] | Omitted = OMITTED,
)

Fine-tunes DiffusionGemma on clean text rows split into a prompt and canvases.

A row has prompt_length + canvas_size * num_canvases tokens. text holds token arrays or ModelInputs, and any media in it conditions only the clean encoder. When the batch supplies attention validity, that decides which cache slots are occupied; otherwise the prompt_mask and canvas_mask a canvas row carries decide (dew.data.steps.chat.ResponseCanvases), and the pad ID marks a bare row’s context. The optional canvas_mask and encoder_target_mask select the text targets, and media placeholders are never labels. By default an encoder target requires adjacent valid slots, as Google’s SequenceTargetShift does. Every response token is corrupted, but only one valid canvas, chosen uniformly, contributes the diffusion cross entropy.

model may be the loaded SFT source, and variables and processor override its own (Objective.bind_model), as eos_token_ids, the ids its task’s responses stop at, overrides the source’s (ended).

Both cross entropies score the final states through the head by the one policy every token objective takes (dew.objectives.lm.head, head_chunks and head_tile as an LM’s): its whole logits kept where they fit, and tiled where the fit ladder moves it, whose backward then holds no vocabulary-sized fp32 logits, unless an adapter on the head leaves no matrix to tile. The one place a full row of logits always exists is the first denoising pass, whose logits condition the second pass and receive no gradient.

A step whose optimizer clips by the attention logit maxima (Step.qk_stats, a QK-Clip) gets the maxima of the attention its passes run through the trained projections, merged by module path (_observed_maxima): a module several passes observe keeps their elementwise maximum, and one only the encoder reaches keeps its own (Aux.qk_stats).

def data(tokenizer=None)
def task_settings() -> BlockSettings

The sequence length, canvas budget, block process, pad id, end-of-sequence ids, and tokenizer or source processor: the task’s own.

def bank_sites(store: Variables) -> tuple[DecoderBank, ...]

The shared text stack, as the training model declares it over store.

def held_variables() -> Variables | None

The SFT source this objective starts from, its split kept, read as the trained model reads it (trainable_variables); or None for a fresh init.

A split tree’s scalars go under frozen beside the rest of the base, so an adapter’s run moves its factors alone, as the source’s frozen scalars did not move either. The move rearranges the tree and copies no array.

def program_key() -> tuple[ProgramModule, ...]

The model the losses run, sized to the SFT sequence, trained, with its head.

def substitute(modules: Sequence[nn.Module]) -> None
def fresh_variables(key: jax.Array, held: Variables | None) -> Variables
def loss(variables: Variables, batch: Batch, step: Step)
def reduce_loss(stats: BlockSFTStatistics)

class source

class DiffusionObjective(
model: nn.Module | PipelineSource,
process: Process | Preset | None = None,
inputs: InputSpec | None = None,
*,
autoencoder: AutoEncoder | None | Omitted = OMITTED,
unconditional_prob: float = 0.12,
ema_decay: float | optax.Schedule | None = 0.999,
solver: Solver = _DEFAULT_SOLVER,
guidance: Guidance | None = _DEFAULT_GUIDANCE,
steps: int | None = None,
variables: Variables | None | Omitted = OMITTED,
uncertainty: int | None = None,
alignment: Alignment | None = None,
end_to_end: EndToEnd | None = None,
grid: SourceGrid | None | Omitted = OMITTED,
)

Trains a denoising diffusion model: draw a noise level, corrupt, predict, and weight the loss.

final_denoise: bool

Whether a sample ends on a closing denoise: on the process’s own times, and not on a pipeline’s grid, whose solver ends the walk itself (PretrainedPipeline.text_to_image).

latent_shape: tuple[int, ...]

The per-example shape the model denoises.

That is the sample field’s shape, or its latent’s shape when an autoencoder sits in front of the model.

def data(tokenizer=None) -> ObjectiveData

Each batch’s captions tokenized for the conditions, on the host, as the step reads them, and kept beside as host strings, which a metric or a reward reads with its own tokenizer (dew.eval.images.CLIPScore), under CAPTIONS; and the fields other conditions read (a clip’s sound) encoded.

def real_lengths(batch: Batch) -> np.ndarray

The denoiser’s real patch-token lengths from the objective’s latent shape.

def model_flops(model, batch: Batch) -> float

Assemble the denoiser’s patch/time/text projections and its layers.

def training_flops(model, batch: Batch) -> float

The denoiser trains; the frozen condition encoder executes only forward.

def task_settings() -> ImageSettings

The process, input encoders, autoencoder and sampling convention.

def build_task(
variables: Variables,
*,
processor: Processor | None | Omitted = OMITTED,
) -> TextToImage

Return the model over variables’ published weights as a TextToImage task.

The task samples the same way this objective’s evaluation does. Its conditions are encoded by the run’s own towers, so it takes no processor.

def towers() -> tuple[Held, ...]

The modules program_key names, the conditions’ encoders a step encodes its captions with, the autoencoder it encodes its images with and an alignment’s frozen representation encoder.

def encoder_params() -> dict
def encode(encoders, tokens: dict | None = None) -> dict

Encode the conditions with the given parameters.

A condition whose tokens are omitted uses its configured unconditional input.

def blank_conditions(params, like: dict) -> dict

The configured unconditional conditions, encoded by params’ own towers and cast to the conditional branch’s dtypes in like: the one function a task over these weights encodes its blank with too, so a step and the task read the same branch.

def held_variables(*, beside: Variables | None = None) -> Variables

Return every array init starts from instead of drawing: the starting tree, or the frozen towers.

A text tower and a VAE are released weights of hundreds of megabytes, which a trace without arguments would compile into the state executable as constants. The loss’s frozen networks ride beside a starting tree without them. This is one mapping, so an objective that starts from more than the towers extends both this and init. A part beside holds, a caller’s own, is taken as given and not built: LPIPS’s published weights are not fetched for a caller that brings its own.

def optimizer(tx: optax.GradientTransformation) -> optax.GradientTransformation

Return tx, or under end_to_end one copy of it per network.

REPA-E updates the model (with its alignment projector), the autoencoder and the discriminator with three optimizers, each clipping its own gradient, so each gets its own copy of tx with its own clip, moments and update count.

def tower_axes() -> dict[tuple[str, ...], LogicalAxes]

Each frozen encoder’s table under the encoders collection at its keyword.

def denoiser_axes(encoders) -> dict[tuple[str, ...], LogicalAxes]

The axes the denoiser draws each parameter in over what encoders’ parameters, arrays or their shapes, encode, traced for shapes alone (drawn_axes): the table a pipeline’s load places its denoiser by.

def fresh_variables(key: jax.Array, held: Variables | None) -> Variables

Draw the denoiser beside the held frozen towers.

def complete_variables(key: jax.Array, tree: Variables) -> Variables

Add the loss’s own heads the tree lacks: EDM2’s uncertainty head, REPA-E’s tuned autoencoder, statistics and discriminator, and REPA’s projector. A split is kept: the optimizer moves what it leaves in params.

def model_variables(variables: Variables) -> Variables

Return the model’s own collections (model_part).

def encoded_conditions(params, batch) -> dict

Encode each condition’s own batch field with the tree’s frozen towers.

def clean_samples(variables, batch, key, step: Step | None = None) -> jax.Array

Return the batch’s samples in [-1, 1], or under an autoencoder their latents drawn with key: in a step, each example’s posterior draw its own (Step.sample).

def denoiser(params, given, unconditional, process: Process | None = None)

Build process’s denoiser, the training process’s by default, over the model’s own collections.

The unconditional branch is passed only when this objective is guided; without guidance the solver never evaluates it.

def loss(variables, batch, step: Step)
def published_autoencoder(variables: Variables) -> tuple[AutoEncoder | None, Variables]

Return the autoencoder a task over variables decodes with, and its weights under autoencoder.

That is the frozen autoencoder, or under end_to_end the tuned one, with its latents normalized by the running statistics.

def evaluate(params, batch, step: Step)

Generate one sample for every real row, without decoding for display.

def preview(params, batch, step: Step, *, scored=None)

Draw a separate small batch for display.

It draws up to VALIDATION_SAMPLES rows on every process and gathers them to the host, and process zero decodes the captions; the other processes return None.

class source

class MaskedDiffusionObjective(
model: nn.Module | Source,
process: DiscreteProcess,
seq_len: int,
*,
head_chunks: int = 4,
head_tile: tuple[int, int] | Literal['whole', 'tiled'] | None = None,
balance: ExpertBalance | None = None,
ema_decay: float | None = 0.999,
solver: Unmask | Confident | None = None,
steps: int | None = None,
block: int | None | Omitted = OMITTED,
samples: int = 4,
decode: Callable[[Sequence[int]], str] | None = None,
variables: Variables | None | Omitted = OMITTED,
processor: Processor | None | Omitted = OMITTED,
eos_token_ids: Sequence[int] | Omitted = OMITTED,
)

Trains a masked diffusion model on the MDLM negative ELBO.

A batch holds [B, seq_len] token ids under batch["text"], and packed windows also their text_coordinates (dew.batch.Coordinates). head_chunks and head_tile set how the head is scored, as an LM’s (LMObjective). balance trains the routers of a mixture as an LM’s does (ExpertBalance, dew.objectives.routing).

def data(tokenizer=None)
def task_settings() -> MaskedSettings

The row length, response budget, process, solver, sampling steps, end-of-sequence ids, and tokenizer or source processor: the task’s own.

def fresh_variables(key: jax.Array, held: Variables | None) -> Variables
def row_tokens(batch: Batch) -> int | None

The tokens of each row, every one of them a slot the corruption may mask.

def loss(variables, batch, step: Step)
def reduce_loss(stats: Ratio | RoutedStatistics) -> tuple[jax.Array, jax.Array]
def apply_effects(variables: Variables, effects: Variables) -> Variables
def preview(params, batch, step: Step, *, scored=None)

Generate samples rows on every process, then decode them on process zero.

The other processes return None. Without decode, the artifact holds the ids alone.