Skip to content

dew.inputs

What a generative objective is fed: the sample field and its conditions (dew.inputs.spec), the condition encoders (dew.inputs.encoders, their contract dew.inputs.condition) and a diffusion model’s conditioner (dew.inputs.diffusion), each imported from its module when first used, so importing one of them, or a name from one, reaches only what that module imports.

NameSummary
AudioConditionAudio conditioning on each clip, through an audio tower.
CLIPTextEncodes text with a CLIP text tower and its checkpoint’s tokenizer. Documented in dew.inputs.encoders.
CharTableEncodes text as a table lookup: one id per character, one fixed random vector per id. Documented in dew.inputs.encoders.
ConditionNames one conditioning input: its encoder, the batch field holding its tokens, and the raw datum for the unconditional branch.
ConditionEncoderTurns one modality’s raw data into a conditioning value. Documented in dew.inputs.condition.
DiffusionConditionerEncodes text into the conditioning a published latent diffusion checkpoint reads.
FieldA batch field and its per-example shape, such as Field("image", (128, 128, 3)).
HFAudioEncodes audio as a transformers model’s last hidden state, using the checkpoint’s feature extractor. Documented in dew.inputs.encoders.
InputSpecNames the sample field and the conditions, keyed by the model keyword each is passed under.
T5TextEncodes text with a T5 encoder tower and its checkpoint’s tokenizer. Documented in dew.inputs.encoders.
TextConditionText conditioning: which registered encoder reads the batch’s tokens, and from which checkpoint.
encode_conditionsEach of encoders (InputSpec.encoders) over the tokens under its keyword, with the weights under its keyword: the conditions a model is passed, by keyword.
unit_rangeConvert pixels in [0, 255] to [-1, 1].

dataclass source

class AudioCondition(
checkpoint: str = 'facebook/wav2vec2-base-960h',
revision: str | None = None,
dtype: DtypeName | None = None,
param_dtype: DtypeName = 'float32',
)

Audio conditioning on each clip, through an audio tower.

checkpoint names the hf_audio tower, whose feature extractor encodes the waveform in the batch’s audio field on the host. The dataset’s clip length sets the waveform length used to encode every clip and the silent unconditional input, so the two always agree.

revision: str | None

The checkpoint’s git revision; None reads its default branch, as it stands when the run starts.

dtype: DtypeName | None

The tower’s compute dtype; None follows the model’s compute dtype.

param_dtype: DtypeName

The storage dtype when loading source weights; supplied params keep theirs.

def build(
seconds: float,
*,
params: Variables | None = None,
dtype: DtypeName | None = None,
) -> Condition

Build the tower over supplied params for waveforms of seconds (the clips’ audio_seconds, Dataset.facts), without loading source weights or casting storage.

dataclass source

class Condition(
encoder: ConditionEncoder,
field: str = 'text',
unconditional: str | float | Mapping[str, object] = '',
)

Names one conditioning input: its encoder, the batch field holding its tokens, and the raw datum for the unconditional branch.

def bound(record: Mapping[str, object], *, params: Variables | None = None) -> Condition

The condition a run recorded (dew.registry.to_record), its encoder rebuilt around params where given (ConditionEncoder.bound) and its other fields read as the registry reads them.

dataclass source

class DiffusionConditioner(
towers: tuple[CLIPTextTransformer, ...],
tokenizers: tuple[CLIPTokenizer, ...],
names: tuple[str, ...],
params: Variables,
checkpoint: str,
height: int,
width: int,
context_width: int,
composition: Composition = 'clip',
aesthetics: bool = False,
t5: T5Segment | None = None,
guidance: float | None = None,
param_dtype: str = 'float32',
*,
projections: Mapping[str, int] = dict(),
tokenizer_files: tuple[PinnedTokenizer, ...] = (),
)

Encodes text into the conditioning a published latent diffusion checkpoint reads.

This one encoder handles every family’s composition (clip, clip_pooled, sd3 or flux). The composition decides which towers run, which of their states the model reads, and how the pooled vector is built. The towers are the native CLIP and T5 towers, called the way their own source pipelines call them. The SD3 and Flux pipelines pass their T5 ids with no attention mask. T5EncoderTransformer called without a mask does the same, so no generic T5 default changes for these families.

projections: Mapping[str, int]

Each CLIP tower’s projection width, read from its config: the width its text_projection kernel maps a pooled state to, where the composition reads it (projected, drawn).

tokenizer_files: tuple[PinnedTokenizer, ...]

The files of the CLIP tokenizers, each pinned as it was loaded (dew.data.text.pin_tokenizer), which dependencies records with the T5 segment’s: the ids a caption tokenizes to are theirs.

context_width: int

The width of the token sequence the denoiser reads: the UNet’s cross_attention_dim, the joint transformers’ joint_attention_dim. The SD3 composition pads its CLIP states out to this width, and the zero segment its pipeline writes for an absent third encoder has this width too.

projected: tuple[str, ...]

The CLIP towers whose pooled state the composition projects (text_projection): the last of an XL pair, each of SD3’s, none of another’s.

stacked: bool

Whether the CLIP ids share one array with a tower axis.

Every family except plain Stable Diffusion stacks them. The XL refiner stacks its single tower too, because its pipeline still writes a tower’s row and not a bare batch.

def drawn() -> Mapping

Each CLIP tower drawn under its name’s text_model beside the projection the composition reads, and the T5 segment under its own name, over a row of token ids, as encode reads them (ConditionEncoder.drawn).

def parameter_axes() -> dict[tuple[str, ...], tuple[str | None, ...]]

Each CLIP tower’s draw under its name’s text_model, and the T5 segment’s under its own name, over a row of token ids (ConditionEncoder.parameter_axes).

def dependencies() -> Mapping[str, object]

The encoder’s fields and the hashes of the tokenizer files it reads, so an exact resume refuses a caption’s ids from other files.

def conditioning_flops(tokens) -> ConditionWork

Count each native tower and pooled projection using the pipeline’s masks.

These CLIP/T5 calls are unmasked: their EOS/zero slots are visible model inputs, unlike masked batch padding. Their output sequences are concatenated for SD3 and only T5 is read by Flux.

def from_pretrained(
checkpoint: str,
*,
dtype: str | None = 'bfloat16',
param_dtype: str = 'float32',
revision: str | None = None,
attention_impl: AttentionImpl = 'auto',
params: Variables | None = None,
mesh: MeshSpec | Mesh | None = None,
layout: Layout | None = None,
height: int | None = None,
width: int | None = None,
)

The pipeline’s conditioning; height and width replace the image size its time ids state, which is the size the pipeline renders unless a load gave another.

def tokenize(texts: Sequence[str | Mapping[str, object]])

Tokenize one row per item, sending each text slot to the tower whose source pipeline reads it.

An item is a prompt string or a mapping of slots, and a missing second or third slot repeats text. text goes to the first CLIP tower and second to the second one. The T5 tower reads its own slot, which is third when the family has two CLIP towers beside it and second when it has one.

def time_ids(count, dtype)

The original size, zero crop, and target size or aesthetic score per row.

def encode(params, tokens) -> DenoisingCondition
def captions(tokens)
def recorded_fields()
def save_assets(destination: Path) -> None

Write into destination the tokenizer files an exported directory keeps beside the weights.

dataclass source

class Field(key: str, shape: tuple[int, ...], variable_axes: tuple[int, ...] = ())

A batch field and its per-example shape, such as Field("image", (128, 128, 3)).

def of(key: str, batch: Batch) -> Field

The field key at the per-example shape batch holds under it, a ModelInputs’s its token rows’, read from shapes alone.

def held(batch: Batch) -> tuple[int, ...]

The per-example shape batch holds under this field (of).

dataclass source

class InputSpec(
sample: Field,
conditions: Mapping[str, Condition] = dict(),
mask: Field | None = None,
)

Names the sample field and the conditions, keyed by the model keyword each is passed under.

For example, conditions={"textcontext": Condition(...)}. Two conditions cannot share a batch field.

A captioning dataset passes its text to tokenize. Every condition that reads captions tokenizes them under its own field, so the encoder a run names decides the ids and the context length, and the dataset only carries the words. A condition on another modality reads the field its dataset writes; audio conditioning, for example, reads a clip’s audio.

mask: Field | None

A binary image mask for explicit masked-image latent conditioning.

encoders: tuple[tuple[str, ConditionEncoder], ...]

Each condition’s encoder by the keyword it is passed under, a hashable value a compiled encoding takes as a static argument (encode_conditions).

def tokenize(captions: Sequence[str]) -> dict[str, Mapping[str, np.ndarray]]

Return the batch fields this run’s caption conditions read, tokenized from captions.

The result is empty for a run with no caption conditions, so the captions stop at the loader and no string array reaches a device.

def dependencies() -> Mapping[str, object]

What tokenize and encode_fields make of a batch depends on: each condition’s field and its encoder’s own (ConditionEncoder.dependencies).

def encode_fields(batch: Batch) -> dict[str, Mapping[str, np.ndarray]]

Return the batch fields this run’s other conditions read, each encoded from the rows its field holds: a clip’s audio waveforms become the audio model’s features, as tokenize makes ids of captions, so the dataset carries the sound and the run’s encoder featurizes it.

def check(batch: Batch, *, whole: bool = True) -> None

Raise ValueError if batch is missing a field this spec names or has a wrong shape.

The sample and the mask must have their declared per-example shapes. A ModelInputs field’s shape is taken from its token rows. A batch that need not be whole, a validation split’s that an objective may score from fewer fields (a rollout’s prompts alone), is held to the shapes of the fields it carries, but for an inpainting spec’s mask and the sample it masks, which every draw and loss reads (ledger 2900).

def bound(
record: Mapping[str, object],
*,
params: Mapping[str, Variables] | None = None,
) -> InputSpec

The spec a run recorded (dew.registry.to_record), each condition’s encoder rebuilt around its parameters, params by the keyword it is passed under (Condition.bound); without params each encoder loads its own weights. The other fields are read as the registry reads them.

dataclass source

class TextCondition(
encoder: str = 'clip_text',
checkpoint: str = DEFAULT_MODEL,
dtype: DtypeName | None = None,
param_dtype: DtypeName = 'float32',
field: str = 'text',
unconditional: str = '',
max_length: int | None = None,
revision: str | None = None,
)

Text conditioning: which registered encoder reads the batch’s tokens, and from which checkpoint.

dtype: DtypeName | None

The encoder’s compute dtype; None follows the model’s compute dtype.

param_dtype: DtypeName

The storage dtype when loading source weights; supplied params keep theirs.

field: str

The batch field holding the tokenized text.

unconditional: str

The prompt the unconditional branch is encoded from.

max_length: int | None

The token length every prompt is padded to.

None keeps the encoder’s own default, which for CLIP is the checkpoint’s context length.

revision: str | None

The checkpoint’s git revision.

With it, a rerun conditions on the weights the run named, even after the branch has moved on.

def build(
*,
params: Variables | None = None,
dtype: DtypeName | None = None,
) -> Condition

Build the encoder over supplied params, without loading source weights or casting storage.

dtype is the run’s own compute dtype, which an unset self.dtype follows: the tower runs next to the model it conditions in every step, so a checkpoint stored in float32 is no reason to run it in float32.

function source

def encode_conditions(
encoders: Sequence[tuple[str, ConditionEncoder]],
weights: Mapping[str, Variables],
tokens: Mapping[str, object],
) -> dict

Each of encoders (InputSpec.encoders) over the tokens under its keyword, with the weights under its keyword: the conditions a model is passed, by keyword. The one encoding a training step (DiffusionObjective.encode) and a sampling task (TextToImage) run.

function source

def unit_range(pixels: jax.typing.ArrayLike) -> jax.Array

Convert pixels in [0, 255] to [-1, 1].

uint8 and float32 pixels give float32, and a float64 run’s float64 pixels stay float64.