Skip to content

dew.inputs.encoders

Encoders that turn raw conditioning data into the value a model keyword takes.

An encoder tokenizes on the host, in the data workers or before a sampling call. It encodes on device, as a pure function of explicit parameters. The parameters are a leaf of the objective’s tree, and the trainer’s layout places them like any other leaf, so a frozen tower’s weights reach the compiled step as arguments, not as constants compiled into it.

ConditionEncoder.bound(record) rebuilds an encoder from a run’s record, the registry’s {"class": ..., "fields": ...} whose fields recorded_fields wrote.

NameSummary
CLIPTextEncodes text with a CLIP text tower and its checkpoint’s tokenizer.
CharTableEncodes text as a table lookup: one id per character, one fixed random vector per id.
ConditionEncoderTurns one modality’s raw data into a conditioning value. Documented in dew.inputs.condition.
HFAudioEncodes audio as a transformers model’s last hidden state, using the checkpoint’s feature extractor.
T5TextEncodes text with a T5 encoder tower and its checkpoint’s tokenizer.

dataclass source

class CLIPText(
checkpoint: str,
transformer: CLIPTextTransformer,
params: Variables,
tokenizer: PreTrainedTokenizerBase,
tokenizer_files: PinnedTokenizer,
dtype: Dtype | None = None,
revision: str | None = None,
param_dtype: str = 'float32',
)

Encodes text with a CLIP text tower and its checkpoint’s tokenizer.

The tower is the one vendored in dew.nn.text_encoders. Prompts are padded to the checkpoint’s own context length, which the tokenizer reports. encode returns the tower’s last hidden state and the attention mask as a TextContext, so a model can pool over the real tokens only.

def from_pretrained(
checkpoint: str = DEFAULT_MODEL,
*,
dtype=None,
revision: str | None = None,
param_dtype: str = 'float32',
params: Variables | None = None,
) -> CLIPText

dataclass source

class CharTable(
params: Variables,
tokens: int = 8,
features: int = 16,
vocab: int = 130,
seed: int = 0,
dtype: Dtype | None = None,
param_dtype: str = 'float32',
)

Encodes text as a table lookup: one id per character, one fixed random vector per id.

It costs nothing and downloads nothing, so tests, benchmarks and smoke runs use it as their text encoder. Its output has the form of a real encoder’s, a TextContext with a mask, so a model that takes CLIP’s output takes this one unchanged.

def from_pretrained(
checkpoint: str = 'char_table',
*,
dtype=None,
tokens: int = 8,
features: int = 16,
vocab: int = 130,
seed: int = 0,
param_dtype: str = 'float32',
params: Variables | None = None,
)

Return an encoder with the table seed draws, or with the table params already holds.

The table has no checkpoint: checkpoint names it as char_table, and any other name is refused rather than read as a random table.

def tokenize(texts: Sequence[str]) -> dict[str, np.ndarray]
def conditioning_flops(tokens) -> ConditionWork
def encode(params, tokens) -> TextContext
def captions(tokens) -> tuple[str, ...]
def recorded_fields() -> dict

dataclass source

class HFAudio(
checkpoint: str,
seconds: float,
module: JittableModule,
params: Variables,
extractor: FeatureExtractor,
files: Mapping[str, object],
dtype: Dtype | None = None,
param_dtype: str = 'float32',
revision: str | None = None,
)

Encodes audio as a transformers model’s last hidden state, using the checkpoint’s feature extractor.

The model is called with one array, the extractor’s first model input (model_input_names[0]), and torchax lowers transformers’ PyTorch forward to JAX. A checkpoint works when AutoModel builds it, when its forward (or, for an encoder-decoder, its encoder’s forward) takes that one input and returns last_hidden_state, and when torchax lowers every op it runs. wav2vec2 (input_values) and Whisper’s encoder (input_features) are the tested ones. Weight-norm parametrizations are folded into plain weights, which compute the same result for a frozen tower. The states come back as a TextContext in which every position is real, so a model that cross-attends to text attends to them unchanged.

Every waveform is cut or zero-padded to seconds, so every clip has the same length, including the constant the unconditional branch is encoded from, and the checkpoint’s feature extractor (extractor) reads them at its own sampling rate and padding, as Whisper’s pads to 30 seconds. A run’s batches carry the waveform under the condition’s field (a clip’s audio), which the objective encodes through tokenize on the host (InputSpec.encode_fields), so this encoder reads no captions.

revision: str | None

The checkpoint’s git revision, recorded so a rebuild reads the same weights and extractor.

files: Mapping[str, object]

The checkpoint’s configuration and feature-extractor files the tower and extractor were read from, hashed (dew.data.text.pin_files), which its identity holds (dependencies).

input_name: str

The name of the extractor’s array that the model reads.

samples: int

The number of samples every waveform is cut or padded to.

def from_pretrained(
checkpoint: str = 'facebook/wav2vec2-base-960h',
*,
seconds: float = 1.0,
dtype=None,
param_dtype: str = 'float32',
revision: str | None = None,
params: Variables | None = None,
) -> HFAudio
def waveform(row: AudioRow) -> np.ndarray

Return one row as a float32 waveform samples long.

A constant fills the whole clip, and a mapping holds the waveform or the constant under "audio". Text raises ValueError.

def tokenize(texts: Sequence[AudioRow]) -> dict[str, np.ndarray]
def encode(params, tokens) -> TextContext
def recorded_fields() -> dict
def dependencies() -> Mapping[str, object]

The encoder’s fields and the hashes of the configuration and extractor files it reads, so an extractor that moved under an unpinned checkpoint name changes the identity too.

dataclass source

class T5Text(
checkpoint: str,
transformer: T5EncoderTransformer,
params: Variables,
tokenizer: PreTrainedTokenizerBase,
tokenizer_files: PinnedTokenizer,
dtype: Dtype | None = None,
revision: str | None = None,
param_dtype: str = 'float32',
max_length: int = 256,
)

Encodes text with a T5 encoder tower and its checkpoint’s tokenizer.

The tower is the one vendored in dew.nn.text_encoders. It is the text half of an SD3.5/Flux-class run, whose MMDiT conditions on T5-XXL’s last hidden states. Prompts are padded to max_length (256 by default), which the run’s record stores. encode returns the tower’s last hidden state and the attention mask as a TextContext.

def from_pretrained(
checkpoint: str = DEFAULT_T5_MODEL,
*,
dtype=None,
revision: str | None = None,
max_length: int = 256,
param_dtype: str = 'float32',
params: Variables | None = None,
) -> T5Text