dew.objectives.diffusion
Diffusion objectives and their distillations, each imported from its module when first used, so importing one of the package’s modules, or a name from one, reaches only what that module imports.
| Name | Summary |
|---|---|
VALIDATION_SAMPLES | |
AdversarialDistillationObjective | Trains LADD, with ADD’s R1 penalty and its distillation term. Documented in dew.objectives.diffusion.adversarial. |
Alignment | Aligns the model’s hidden tokens at layer with a frozen encoder’s features. Documented in dew.objectives.diffusion.alignment. |
BlockDiffusionObjective | Fine-tunes DiffusionGemma on clean text rows split into a prompt and canvases. |
ConsistencyDistillationObjective | Trains rCM: sCM distillation of a flow teacher, regularized by DMD2. Documented in dew.objectives.diffusion.consistency. |
DiffusionObjective | Trains a denoising diffusion model: draw a noise level, corrupt, predict, and weight the loss. |
EndToEnd | REPA-E’s end-to-end tuning of the autoencoder. Documented in dew.objectives.diffusion.end_to_end. |
GuidanceDistillationObjective | Distills a teacher’s classifier-free guidance into this model. Documented in dew.objectives.diffusion.guidance_distillation. |
MaskedDiffusionObjective | Trains a masked diffusion model on the MDLM negative ELBO. |
MeanFlowObjective | Trains MeanFlow on an interval process (presets.MeanFlow). Documented in dew.objectives.diffusion.few_step. |
ShortcutObjective | Trains a shortcut model on an interval process (presets.Shortcut). Documented in dew.objectives.diffusion.few_step. |
VALIDATION_SAMPLES
Section titled “VALIDATION_SAMPLES”VALIDATION_SAMPLES = 4BlockDiffusionObjective
Section titled “BlockDiffusionObjective”class BlockDiffusionObjective( model: DiffusionGemma | Source, *, prompt_length: int, num_canvases: int = 1, canvas_size: int | None = None, variables: Variables | None | Omitted = OMITTED, pad_token_id: int = 0, self_cond_prob: float = 0.5, safety_epsilon: float = 0.0001, stop_gradient_from_denoiser_to_encoder: bool = False, encoder_loss_weight: float = 1.0, decoder_loss_weight: float = 1.0, ema_decay: float | None = None, head_chunks: int = 4, head_tile: tuple[int, int] | Literal['whole', 'tiled'] | None = None, processor: Processor | None | Omitted = OMITTED, eos_token_ids: Sequence[int] | Omitted = OMITTED,)Fine-tunes DiffusionGemma on clean text rows split into a prompt and canvases.
A row has prompt_length + canvas_size * num_canvases tokens. text
holds token arrays or ModelInputs, and any media in it conditions only
the clean encoder. When the batch supplies attention validity, that
decides which cache slots are occupied; otherwise the prompt_mask and
canvas_mask a canvas row carries decide (dew.data.steps.chat.ResponseCanvases),
and the pad ID marks a bare row’s context. The optional canvas_mask and
encoder_target_mask select the text targets, and media placeholders
are never labels. By default an encoder target requires adjacent valid
slots, as Google’s SequenceTargetShift does. Every response token is
corrupted, but only one valid canvas, chosen uniformly, contributes the
diffusion cross entropy.
model may be the loaded SFT source, and variables and processor
override its own (Objective.bind_model), as eos_token_ids, the ids its
task’s responses stop at, overrides the source’s (ended).
Both cross entropies score the final states through the head by the one
policy every token objective takes (dew.objectives.lm.head, head_chunks
and head_tile as an LM’s): its whole logits kept where they fit, and
tiled where the fit ladder moves it, whose backward then holds no
vocabulary-sized fp32 logits, unless an adapter on the head leaves no
matrix to tile. The one place a full row of logits always exists is the
first denoising pass, whose logits condition the second pass and receive
no gradient.
A step whose optimizer clips by the attention logit maxima (Step.qk_stats, a QK-Clip) gets the
maxima of the attention its passes run through the trained projections, merged by module path
(_observed_maxima): a module several passes observe keeps their elementwise maximum, and one only
the encoder reaches keeps its own (Aux.qk_stats).
BlockDiffusionObjective.data
Section titled “BlockDiffusionObjective.data”def data(tokenizer=None)BlockDiffusionObjective.task_settings
Section titled “BlockDiffusionObjective.task_settings”def task_settings() -> BlockSettingsThe sequence length, canvas budget, block process, pad id, end-of-sequence ids, and tokenizer or source processor: the task’s own.
BlockDiffusionObjective.bank_sites
Section titled “BlockDiffusionObjective.bank_sites”def bank_sites(store: Variables) -> tuple[DecoderBank, ...]The shared text stack, as the training model declares it over store.
BlockDiffusionObjective.held_variables
Section titled “BlockDiffusionObjective.held_variables”def held_variables() -> Variables | NoneThe SFT source this objective starts from, its split kept, read as
the trained model reads it (trainable_variables); or None for a
fresh init.
A split tree’s scalars go under frozen beside the rest of the base,
so an adapter’s run moves its factors alone, as the source’s frozen
scalars did not move either. The move rearranges the tree and copies
no array.
BlockDiffusionObjective.program_key
Section titled “BlockDiffusionObjective.program_key”def program_key() -> tuple[ProgramModule, ...]The model the losses run, sized to the SFT sequence, trained, with its head.
BlockDiffusionObjective.substitute
Section titled “BlockDiffusionObjective.substitute”def substitute(modules: Sequence[nn.Module]) -> NoneBlockDiffusionObjective.fresh_variables
Section titled “BlockDiffusionObjective.fresh_variables”def fresh_variables(key: jax.Array, held: Variables | None) -> VariablesBlockDiffusionObjective.loss
Section titled “BlockDiffusionObjective.loss”def loss(variables: Variables, batch: Batch, step: Step)BlockDiffusionObjective.reduce_loss
Section titled “BlockDiffusionObjective.reduce_loss”def reduce_loss(stats: BlockSFTStatistics)DiffusionObjective
Section titled “DiffusionObjective”class DiffusionObjective( model: nn.Module | PipelineSource, process: Process | Preset | None = None, inputs: InputSpec | None = None, *, autoencoder: AutoEncoder | None | Omitted = OMITTED, unconditional_prob: float = 0.12, ema_decay: float | optax.Schedule | None = 0.999, solver: Solver = _DEFAULT_SOLVER, guidance: Guidance | None = _DEFAULT_GUIDANCE, steps: int | None = None, variables: Variables | None | Omitted = OMITTED, uncertainty: int | None = None, alignment: Alignment | None = None, end_to_end: EndToEnd | None = None, grid: SourceGrid | None | Omitted = OMITTED,)Trains a denoising diffusion model: draw a noise level, corrupt, predict, and weight the loss.
final_denoise: bool-
Whether a sample ends on a closing denoise: on the process’s own times, and not on a pipeline’s grid, whose solver ends the walk itself (
PretrainedPipeline.text_to_image). latent_shape: tuple[int, ...]-
The per-example shape the model denoises.
That is the sample field’s shape, or its latent’s shape when an autoencoder sits in front of the model.
DiffusionObjective.data
Section titled “DiffusionObjective.data”def data(tokenizer=None) -> ObjectiveDataEach batch’s captions tokenized for the conditions, on the host, as
the step reads them, and kept beside as host strings, which a metric
or a reward reads with its own tokenizer (dew.eval.images.CLIPScore),
under CAPTIONS; and the fields other conditions read (a clip’s sound)
encoded.
DiffusionObjective.real_lengths
Section titled “DiffusionObjective.real_lengths”def real_lengths(batch: Batch) -> np.ndarrayThe denoiser’s real patch-token lengths from the objective’s latent shape.
DiffusionObjective.model_flops
Section titled “DiffusionObjective.model_flops”def model_flops(model, batch: Batch) -> floatAssemble the denoiser’s patch/time/text projections and its layers.
DiffusionObjective.training_flops
Section titled “DiffusionObjective.training_flops”def training_flops(model, batch: Batch) -> floatThe denoiser trains; the frozen condition encoder executes only forward.
DiffusionObjective.task_settings
Section titled “DiffusionObjective.task_settings”def task_settings() -> ImageSettingsThe process, input encoders, autoencoder and sampling convention.
DiffusionObjective.build_task
Section titled “DiffusionObjective.build_task”def build_task( variables: Variables, *, processor: Processor | None | Omitted = OMITTED,) -> TextToImageReturn the model over variables’ published weights as a TextToImage task.
The task samples the same way this objective’s evaluation does. Its conditions are encoded by the run’s own towers, so it takes no processor.
DiffusionObjective.towers
Section titled “DiffusionObjective.towers”def towers() -> tuple[Held, ...]The modules program_key names, the conditions’ encoders a step encodes its captions with,
the autoencoder it encodes its images with and an alignment’s frozen representation encoder.
DiffusionObjective.encoder_params
Section titled “DiffusionObjective.encoder_params”def encoder_params() -> dictDiffusionObjective.encode
Section titled “DiffusionObjective.encode”def encode(encoders, tokens: dict | None = None) -> dictEncode the conditions with the given parameters.
A condition whose tokens are omitted uses its configured unconditional input.
DiffusionObjective.blank_conditions
Section titled “DiffusionObjective.blank_conditions”def blank_conditions(params, like: dict) -> dictThe configured unconditional conditions, encoded by params’ own towers and cast to the
conditional branch’s dtypes in like: the one function a task over these weights encodes
its blank with too, so a step and the task read the same branch.
DiffusionObjective.held_variables
Section titled “DiffusionObjective.held_variables”def held_variables(*, beside: Variables | None = None) -> VariablesReturn every array init starts from instead of drawing: the starting tree, or the frozen towers.
A text tower and a VAE are released weights of hundreds of megabytes, which a
trace without arguments would compile into the state executable as
constants. The loss’s frozen networks ride beside a starting tree without them.
This is one mapping, so an objective that starts from more than the towers
extends both this and init. A part beside holds, a caller’s own, is
taken as given and not built: LPIPS’s published weights are not fetched for
a caller that brings its own.
DiffusionObjective.optimizer
Section titled “DiffusionObjective.optimizer”def optimizer(tx: optax.GradientTransformation) -> optax.GradientTransformationReturn tx, or under end_to_end one copy of it per network.
REPA-E updates the model (with its alignment projector), the autoencoder and
the discriminator with three optimizers, each clipping its own gradient, so
each gets its own copy of tx with its own clip, moments and update count.
DiffusionObjective.tower_axes
Section titled “DiffusionObjective.tower_axes”def tower_axes() -> dict[tuple[str, ...], LogicalAxes]Each frozen encoder’s table under the encoders collection at its keyword.
DiffusionObjective.denoiser_axes
Section titled “DiffusionObjective.denoiser_axes”def denoiser_axes(encoders) -> dict[tuple[str, ...], LogicalAxes]The axes the denoiser draws each parameter in over what encoders’ parameters, arrays or their
shapes, encode, traced for shapes alone (drawn_axes): the table a pipeline’s load places its
denoiser by.
DiffusionObjective.fresh_variables
Section titled “DiffusionObjective.fresh_variables”def fresh_variables(key: jax.Array, held: Variables | None) -> VariablesDraw the denoiser beside the held frozen towers.
DiffusionObjective.complete_variables
Section titled “DiffusionObjective.complete_variables”def complete_variables(key: jax.Array, tree: Variables) -> VariablesAdd the loss’s own heads the tree lacks: EDM2’s uncertainty head,
REPA-E’s tuned autoencoder, statistics and discriminator, and REPA’s
projector. A split is kept: the optimizer moves what it leaves in params.
DiffusionObjective.model_variables
Section titled “DiffusionObjective.model_variables”def model_variables(variables: Variables) -> VariablesReturn the model’s own collections (model_part).
DiffusionObjective.encoded_conditions
Section titled “DiffusionObjective.encoded_conditions”def encoded_conditions(params, batch) -> dictEncode each condition’s own batch field with the tree’s frozen towers.
DiffusionObjective.clean_samples
Section titled “DiffusionObjective.clean_samples”def clean_samples(variables, batch, key, step: Step | None = None) -> jax.ArrayReturn the batch’s samples in [-1, 1], or under an autoencoder their latents drawn with key:
in a step, each example’s posterior draw its own (Step.sample).
DiffusionObjective.denoiser
Section titled “DiffusionObjective.denoiser”def denoiser(params, given, unconditional, process: Process | None = None)Build process’s denoiser, the training process’s by default, over the model’s own
collections.
The unconditional branch is passed only when this objective is guided; without guidance the solver never evaluates it.
DiffusionObjective.loss
Section titled “DiffusionObjective.loss”def loss(variables, batch, step: Step)DiffusionObjective.published_autoencoder
Section titled “DiffusionObjective.published_autoencoder”def published_autoencoder(variables: Variables) -> tuple[AutoEncoder | None, Variables]Return the autoencoder a task over variables decodes with, and its weights under autoencoder.
That is the frozen autoencoder, or under end_to_end the tuned one, with its
latents normalized by the running statistics.
DiffusionObjective.evaluate
Section titled “DiffusionObjective.evaluate”def evaluate(params, batch, step: Step)Generate one sample for every real row, without decoding for display.
DiffusionObjective.preview
Section titled “DiffusionObjective.preview”def preview(params, batch, step: Step, *, scored=None)Draw a separate small batch for display.
It draws up to VALIDATION_SAMPLES rows on every process and gathers them to
the host, and process zero decodes the captions; the other processes return
None.
MaskedDiffusionObjective
Section titled “MaskedDiffusionObjective”class MaskedDiffusionObjective( model: nn.Module | Source, process: DiscreteProcess, seq_len: int, *, head_chunks: int = 4, head_tile: tuple[int, int] | Literal['whole', 'tiled'] | None = None, balance: ExpertBalance | None = None, ema_decay: float | None = 0.999, solver: Unmask | Confident | None = None, steps: int | None = None, block: int | None | Omitted = OMITTED, samples: int = 4, decode: Callable[[Sequence[int]], str] | None = None, variables: Variables | None | Omitted = OMITTED, processor: Processor | None | Omitted = OMITTED, eos_token_ids: Sequence[int] | Omitted = OMITTED,)Trains a masked diffusion model on the MDLM negative ELBO.
A batch holds [B, seq_len] token ids under batch["text"], and packed
windows also their text_coordinates (dew.batch.Coordinates).
head_chunks and head_tile set how the head is scored, as an LM’s
(LMObjective). balance trains the routers of a mixture as an LM’s does
(ExpertBalance, dew.objectives.routing).
MaskedDiffusionObjective.data
Section titled “MaskedDiffusionObjective.data”def data(tokenizer=None)MaskedDiffusionObjective.task_settings
Section titled “MaskedDiffusionObjective.task_settings”def task_settings() -> MaskedSettingsThe row length, response budget, process, solver, sampling steps, end-of-sequence ids, and tokenizer or source processor: the task’s own.
MaskedDiffusionObjective.fresh_variables
Section titled “MaskedDiffusionObjective.fresh_variables”def fresh_variables(key: jax.Array, held: Variables | None) -> VariablesMaskedDiffusionObjective.row_tokens
Section titled “MaskedDiffusionObjective.row_tokens”def row_tokens(batch: Batch) -> int | NoneThe tokens of each row, every one of them a slot the corruption may mask.
MaskedDiffusionObjective.loss
Section titled “MaskedDiffusionObjective.loss”def loss(variables, batch, step: Step)MaskedDiffusionObjective.reduce_loss
Section titled “MaskedDiffusionObjective.reduce_loss”def reduce_loss(stats: Ratio | RoutedStatistics) -> tuple[jax.Array, jax.Array]MaskedDiffusionObjective.apply_effects
Section titled “MaskedDiffusionObjective.apply_effects”def apply_effects(variables: Variables, effects: Variables) -> VariablesMaskedDiffusionObjective.preview
Section titled “MaskedDiffusionObjective.preview”def preview(params, batch, step: Step, *, scored=None)Generate samples rows on every process, then decode them on process zero.
The other processes return None. Without decode, the artifact holds
the ids alone.