Skip to content

dew.config.diffusion

The diffusion run as one typed record, and the one function that builds it.

DiffusionRunConfig is what a diffusion recipe parses from its command line and writes as run.json next to the checkpoints. build() is the single function that turns it into a DiffusionObjective: the recipe trains the objective it returns, and TextToImage.from_run samples from the objective it returns for the same file.

NameSummary
DiffusionRunConfigA diffusion run’s configuration: the shared run fields plus the diffusion objective’s own settings.
FlowRolloutConfigA run’s flow RL rollout: groups samples of each prompt over steps time points, each scored by the image metric reward names, an alias or an import path, against its own prompt, higher being better (clip_score).

dataclass source

class DiffusionRunConfig(
model: ModelConfig = (lambda: ModelConfig('unet', dict(DEFAULT_MODEL_CONFIG)))(),
data: Annotated[Dataset | None, tyro.conf.Suppress] = None,
validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress] = None,
data_seed: int = 0,
source: SourceOptions = SourceOptions(),
optim: OptimConfig = OptimConfig(),
trainer: TrainerConfig = TrainerConfig(),
objective: ObjectiveConfig = (lambda: ObjectiveConfig('diffusion', {'solver': to_record(EulerAncestral(), Solver)}))(),
lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None = None,
val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None = None,
val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])] = dict(),
val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])] = (),
preset: PresetSpec | None = EDM(),
text: TextCondition | None = TextCondition(),
audio: AudioCondition | None = None,
autoencoder: PretrainedAutoencoder | None = None,
pretrained: str | None = None,
teacher_run: str | None = None,
rollout: Annotated[FlowRolloutConfig, tyro.conf.subcommand('rollout')] | None = None,
)

A diffusion run’s configuration: the shared run fields plus the diffusion objective’s own settings.

preset: PresetSpec | None

The convention the model is trained and sampled with.

None uses the one the pretrained pipeline’s scheduler reads, which a preset may restate.

objective: ObjectiveConfig

The objective and its arguments: the denoising loss (diffusion) with its EMA, condition dropout and validation sampling (--objective.ema-decay, --objective.guidance, --objective.steps), or a loss of its own in its place, as --objective mean_flow --objective.omega 2 or --objective rcm --teacher-run runs/teacher. Each objective refuses a preset or a guidance its loss cannot train or sample with.

text: TextCondition | None

The text condition, passed under its encoder’s keyword (ConditionEncoder.keyword).

None trains unconditionally, or on audio.

audio: AudioCondition | None

The audio condition, passed under its encoder’s keyword in place of text.

It needs text to be None and a chain of clips that hold their audio (dew.data.presets.clips(sample_rate=)).

autoencoder: PretrainedAutoencoder | None

The autoencoder for latent diffusion; None trains in pixel space.

pretrained: str | None

A published diffusion pipeline to fine-tune, or a saved diffusion run to start from.

It is a Hub repo, repo@revision or a local directory in the diffusers layout, or a saved Dew run’s directory, which loads as its own sampling task at the step --source.step selects (load_diffusion_source). The checkpoint decides the model, its text conditioning and its autoencoder, so --model holds only the precision settings and text and autoencoder stay unset.

teacher_run: str | None

The run directory a distillation’s teacher trained in. Its published weights are the frozen teacher’s (teacher_variables=), unless the starting tree holds the teacher; guidance distillation also rebuilds the teacher’s objective from the run’s record (teacher=). Only a distillation takes one.

rollout: Annotated[FlowRolloutConfig, tyro.conf.subcommand('rollout')] | None

The rollout a Flow-GRPO run trains on: its reward metric, group size, time points and trained transitions, --rollout.reward clip_score --rollout.groups 4. A Flow-GRPO run without one takes FlowRolloutConfig(), which its record then states; another objective trains on its data and takes none.

context: TextCondition | AudioCondition | None

Return the condition the model reads, text or audio, if any.

def derived_preset() -> Preset | None

The run’s preset as it builds its process, what it leaves to the run derived from the run as it stands, so an override of what it derives from (the images’ size, data.images.size, or the autoencoder) derives it afresh, where the run records the preset as given: EDM’s sigma draw the space’s, pixels without an autoencoder and latents with one, and a Flow resolution shift’s token count the data’s images’ (ResolutionShift.at). A regime, sigmas or token count the preset states wins, a model whose grid is not the 16-pixel one stating its own.

def sample_field() -> Field

Return the batch field the model generates, at the resolution the data comes in (Dataset.facts): a clip’s video, else an image. Raises ValueError for a chain of neither, or one whose size varies.

def mask_field(
sample: Field,
wanted: Field,
chain: Dataset | None = None,
name: str = 'the chain',
) -> Field

The batch field an inpainting pipeline reads its mask from (wanted, the pipeline’s own): the chain’s wanted.key (the run’s data, or chain, a validation split name) at the samples’ height and width, one channel, as dew.data.images.RandomMask draws one or a dataset’s own column decodes one. A chain that carries none is refused, naming RandomMask: an inpainting pipeline paints the region its mask marks, and a plain image chain marks none.

def model_fields(autoencoder: AutoEncoder | None) -> dict[str, Configured]

Return the fields the run derives for the model, which ModelConfig.build takes over the model’s arguments.

These are the channels the model denoises when the architecture takes them as output_channels. The published families name theirs as their sources do, in model.fields. On the model those build, an IntervalModel embeds the duration under an interval process, and MeanFlow, whose loss differentiates in time, turns a TimeScaled model’s time features at SMOOTH_TIME_SCALE unless model.fields names a scale.

def build(*, variables: Variables | None = None) -> DiffusionObjective

Build the objective, with each component built around its parameters.

Supplied variables are the authoritative saved snapshot. The encoders and the VAE read only configuration and tokenizer metadata, and use their subtrees of the supplied variables without loading source weights or casting storage.

A lora goes on the denoiser that pretrained loads, or, from scratch, on a fresh draw of it from the run’s key. The objective then trains the adapter’s factors and any loss head of its own.

def build_run() -> Prepared

The objective build makes, with the run’s Hub sources pinned (pinned), on its data, whose captions the objective’s conditions tokenize. val_metrics score each validation, and Flow-GRPO samples the batches it trains on (rollout).

def pinned() -> DiffusionRunConfig

Return this run with its Hub sources pinned to the commits they resolve to now: an imported adapter’s (RunConfig.pinned), pretrained and the alignment’s encoder, so the record names the weights the run started from.

def default_metrics(*, validates: bool) -> tuple[MetricConfig, ...]

CLIP for an image run whose data validates, and nothing otherwise.

def metric(config: MetricConfig) -> Metric

config built, loading its own weights: a video run scores a VideoGrid against its video field, which a PSNR or SSIM that states no field of its own reads there, and an image-only metric raises a ValueError naming it here, before the trainer starts.

dataclass source

class FlowRolloutConfig(
reward: str = 'clip_score',
groups: int = 4,
steps: int = 11,
train_steps: int | None = None,
)

A run’s flow RL rollout: groups samples of each prompt over steps time points, each scored by the image metric reward names, an alias or an import path, against its own prompt, higher being better (clip_score). Flow-GRPO trains the first train_steps transitions of each trajectory (all of them when None) and DiffusionNFT each clean sample at the first train_steps times of its sampler’s grid (the released 9 when None). build makes the objective’s rollout, a FlowRollout or an NFTRollout.

def build(
objective: FlowGRPOObjective | DiffusionNFTObjective,
) -> FlowRollout | NFTRollout

The rollout over objective, scoring each sample with the metric’s per-sample measure.