dew.inputs

What a generative objective is fed: the sample field and its conditions.

InputSpec names the batch field the model learns to generate and, keyed by the model’s own keyword arguments, the conditions it is given. A Condition is an encoder, the batch field it reads, and the raw datum that stands for “no condition”, which classifier-free guidance and conditioning dropout substitute. Nothing here runs a model: the spec is a description, and the objective does the encoding.

Image and video batches arrive as uint8 pixels in [0, 255], the way the data workers write them. unit_range is the one conversion to the [-1, 1] range every diffusion loss, sample, artifact and image metric lives in; dew.artifacts.uint8_pixels is the one conversion back.

Name	Summary
CLIPText	The CLIP text tower, vendored in dew.nn.text_encoders, with the checkpoint’s tokenizer. Documented in dew.inputs.encoders.
CharTable	Encodes text as a table lookup: one id per character, one fixed random vector per id. Documented in dew.inputs.encoders.
Condition	Names one conditioning input: its encoder, the batch field holding its tokens, and the raw datum for the unconditional branch.
ConditionEncoder	Carries one modality from raw data to a conditioning value. Documented in dew.inputs.encoders.
DiffusionConditioner	The text conditioning of a published latent diffusion checkpoint.
Field	A batch field and its per-example shape: Field("image", (128, 128, 3)).
InputSpec	Names the sample field and the conditions, keyed by the model keyword each is passed under: {"textcontext": Condition(...)}.
T5Text	The T5 encoder tower, vendored in dew.nn.text_encoders, with the checkpoint’s tokenizer. Documented in dew.inputs.encoders.
pixel_field	The batch field carrying one image per row for a vision tower.
rebuild	The named encoder rebuilt from its JSON fields. Documented in dew.inputs.encoders.
unit_range	uint8 pixels in [0, 255] as float32 in [-1, 1].

Condition

dataclass source

class Condition(
    encoder: ConditionEncoder,
    field: str = 'text',
    unconditional: str | float | Mapping[str, object] = '',
)

Names one conditioning input: its encoder, the batch field holding its tokens, and the raw datum for the unconditional branch.

Condition.to_json

def to_json() -> dict

Condition.from_json

def from_json(record: Mapping, *, params: Variables | None = None) -> Condition

DiffusionConditioner

dataclass source

class DiffusionConditioner(
    towers: tuple[CLIPTextTransformer, ...],
    tokenizers: tuple[CLIPTokenizer, ...],
    names: tuple[str, ...],
    params: Variables,
    checkpoint: str,
    height: int,
    width: int,
    context_width: int,
    composition: Composition = 'clip',
    aesthetics: bool = False,
    t5: T5Segment | None = None,
    guidance: float | None = None,
    param_dtype: str = 'float32',
)

The text conditioning of a published latent diffusion checkpoint.

One encoder owns every family’s composition: which towers run, which of their states the model reads, and how the pooled vector is built. The towers themselves are the native CLIP and T5 towers, called the way their own source pipelines call them. The SD3 and Flux pipelines pass their T5 ids with no attention mask, which is what T5EncoderTransformer does with none, and no generic T5 default changes for it.

context_width: int

The width of the token sequence the denoiser reads: the UNet’s cross_attention_dim, the joint transformers’ joint_attention_dim. The SD3 composition pads its CLIP states out to it and writes the zero segment its pipeline substitutes for an absent third encoder at it.

stacked: bool

Whether the CLIP ids ride one array with a tower axis.

Every family but plain Stable Diffusion does. The XL refiner carries a single tower that way too, since its pipeline still writes a tower’s row rather than a bare batch.

DiffusionConditioner.from_pretrained

def from_pretrained(
    checkpoint: str,
    *,
    dtype: str | None = 'bfloat16',
    param_dtype: str = 'float32',
    revision: str | None = None,
    attention_impl: str = 'auto',
    params: Variables | None = None,
)

DiffusionConditioner.tokenize

def tokenize(texts: Sequence[str | Mapping[str, object]])

One row per item, with each text slot routed to the tower whose source pipeline reads it: text to the first CLIP tower, second to the second one, and the T5 tower’s own slot, which is third where a family has two CLIP towers beside it and second where it has one.

DiffusionConditioner.time_ids

def time_ids(count, dtype)

SDXL’s micro-conditioning: the original size, no crop, then the target size or the refiner’s aesthetic score.

DiffusionConditioner.encode

def encode(params, tokens) -> DenoisingCondition

DiffusionConditioner.captions

def captions(tokens)

DiffusionConditioner.to_json

def to_json()

DiffusionConditioner.save_assets

def save_assets(destination: Path) -> None

Write the tokenizer files an exported directory carries beside the weights.

Field

dataclass source

class Field(key: str, shape: tuple[int, ...])

A batch field and its per-example shape: Field("image", (128, 128, 3)).

InputSpec

dataclass source

class InputSpec(
    sample: Field,
    conditions: Mapping[str, Condition] = dict(),
    mask: Field | None = None,
)

Names the sample field and the conditions, keyed by the model keyword each is passed under: {"textcontext": Condition(...)}.

tokenize is what a captioning dataset hands its text to. Every condition tokenizes the batch’s captions under its own field, so the encoder a run names decides the ids and the context length while the dataset carries the words alone.

mask: Field | None

A binary image mask for explicit masked-image latent conditioning.

InputSpec.tokenize

def tokenize(captions: Sequence[str]) -> dict[str, Mapping[str, np.ndarray]]

The batch fields this run’s conditions read out of captions.

Empty for a run that conditions on nothing, so the captions stop at the loader and no string array reaches a device.

InputSpec.to_json

def to_json() -> dict

InputSpec.from_json

def from_json(
    record: Mapping,
    *,
    params: Mapping[str, Variables] | None = None,
) -> InputSpec

Rebuilds the spec around supplied condition parameters, or loads each encoder’s own weights when none are given.

pixel_field

function source

def pixel_field(height: int, width: int, channels: int = 3) -> Field

The batch field carrying one image per row for a vision tower.

It is float32 [channels, height, width], as the checkpoint’s processor emitted it, and rides beside the decoder’s token field.

unit_range

function source

def unit_range(pixels: jax.typing.ArrayLike) -> jax.Array

uint8 pixels in [0, 255] as float32 in [-1, 1].