dew.inputs.encoders
Encoders that turn raw conditioning data into the value a model keyword takes.
An encoder tokenizes on the host, in the data workers or before a sampling call. It encodes on device, as a pure function of explicit parameters. The parameters are a leaf of the objective’s tree, and the trainer’s layout places them like any other leaf, so a frozen tower’s weights reach the compiled step as arguments, not as constants compiled into it.
ConditionEncoder.bound(record) rebuilds an encoder from a run’s record, the
registry’s {"class": ..., "fields": ...} whose fields recorded_fields wrote.
| Name | Summary |
|---|---|
CLIPText | Encodes text with a CLIP text tower and its checkpoint’s tokenizer. |
CharTable | Encodes text as a table lookup: one id per character, one fixed random vector per id. |
ConditionEncoder | Turns one modality’s raw data into a conditioning value. Documented in dew.inputs.condition. |
HFAudio | Encodes audio as a transformers model’s last hidden state, using the checkpoint’s feature extractor. |
T5Text | Encodes text with a T5 encoder tower and its checkpoint’s tokenizer. |
CLIPText
Section titled “CLIPText”class CLIPText( checkpoint: str, transformer: CLIPTextTransformer, params: Variables, tokenizer: PreTrainedTokenizerBase, tokenizer_files: PinnedTokenizer, dtype: Dtype | None = None, revision: str | None = None, param_dtype: str = 'float32',)Encodes text with a CLIP text tower and its checkpoint’s tokenizer.
The tower is the one vendored in dew.nn.text_encoders. Prompts are
padded to the checkpoint’s own context length, which the tokenizer
reports. encode returns the tower’s last hidden state and the
attention mask as a TextContext, so a model can pool over the real
tokens only.
CLIPText.from_pretrained
Section titled “CLIPText.from_pretrained”def from_pretrained( checkpoint: str = DEFAULT_MODEL, *, dtype=None, revision: str | None = None, param_dtype: str = 'float32', params: Variables | None = None,) -> CLIPTextCharTable
Section titled “CharTable”class CharTable( params: Variables, tokens: int = 8, features: int = 16, vocab: int = 130, seed: int = 0, dtype: Dtype | None = None, param_dtype: str = 'float32',)Encodes text as a table lookup: one id per character, one fixed random vector per id.
It costs nothing and downloads nothing, so tests, benchmarks and smoke
runs use it as their text encoder. Its output has the form of a real
encoder’s, a TextContext with a mask, so a model that takes CLIP’s
output takes this one unchanged.
CharTable.from_pretrained
Section titled “CharTable.from_pretrained”def from_pretrained( checkpoint: str = 'char_table', *, dtype=None, tokens: int = 8, features: int = 16, vocab: int = 130, seed: int = 0, param_dtype: str = 'float32', params: Variables | None = None,)Return an encoder with the table seed draws, or with the table params already holds.
The table has no checkpoint: checkpoint names it as char_table, and any other name is
refused rather than read as a random table.
CharTable.tokenize
Section titled “CharTable.tokenize”def tokenize(texts: Sequence[str]) -> dict[str, np.ndarray]CharTable.conditioning_flops
Section titled “CharTable.conditioning_flops”def conditioning_flops(tokens) -> ConditionWorkCharTable.encode
Section titled “CharTable.encode”def encode(params, tokens) -> TextContextCharTable.captions
Section titled “CharTable.captions”def captions(tokens) -> tuple[str, ...]CharTable.recorded_fields
Section titled “CharTable.recorded_fields”def recorded_fields() -> dictHFAudio
Section titled “HFAudio”class HFAudio( checkpoint: str, seconds: float, module: JittableModule, params: Variables, extractor: FeatureExtractor, files: Mapping[str, object], dtype: Dtype | None = None, param_dtype: str = 'float32', revision: str | None = None,)Encodes audio as a transformers model’s last hidden state, using the checkpoint’s feature extractor.
The model is called with one array, the extractor’s first model input
(model_input_names[0]), and torchax lowers transformers’ PyTorch
forward to JAX. A checkpoint works when AutoModel builds it, when its
forward (or, for an encoder-decoder, its encoder’s forward) takes that
one input and returns last_hidden_state, and when torchax lowers every
op it runs. wav2vec2 (input_values) and Whisper’s encoder
(input_features) are the tested ones. Weight-norm parametrizations are
folded into plain weights, which compute the same result for a frozen
tower. The states come back as a TextContext in which every position
is real, so a model that cross-attends to text attends to them
unchanged.
Every waveform is cut or zero-padded to seconds, so every clip has the
same length, including the constant the unconditional branch is encoded
from, and the checkpoint’s feature extractor (extractor) reads them at
its own sampling rate and padding, as Whisper’s pads to 30 seconds. A
run’s batches carry the waveform under the condition’s field (a
clip’s audio), which the objective encodes through tokenize on the host
(InputSpec.encode_fields), so this encoder reads no captions.
revision: str | None-
The checkpoint’s git revision, recorded so a rebuild reads the same weights and extractor.
files: Mapping[str, object]-
The checkpoint’s configuration and feature-extractor files the tower and
extractorwere read from, hashed (dew.data.text.pin_files), which its identity holds (dependencies). input_name: str-
The name of the extractor’s array that the model reads.
samples: int-
The number of samples every waveform is cut or padded to.
HFAudio.from_pretrained
Section titled “HFAudio.from_pretrained”def from_pretrained( checkpoint: str = 'facebook/wav2vec2-base-960h', *, seconds: float = 1.0, dtype=None, param_dtype: str = 'float32', revision: str | None = None, params: Variables | None = None,) -> HFAudioHFAudio.waveform
Section titled “HFAudio.waveform”def waveform(row: AudioRow) -> np.ndarrayReturn one row as a float32 waveform samples long.
A constant fills the whole clip, and a mapping holds the waveform or
the constant under "audio". Text raises ValueError.
HFAudio.tokenize
Section titled “HFAudio.tokenize”def tokenize(texts: Sequence[AudioRow]) -> dict[str, np.ndarray]HFAudio.encode
Section titled “HFAudio.encode”def encode(params, tokens) -> TextContextHFAudio.recorded_fields
Section titled “HFAudio.recorded_fields”def recorded_fields() -> dictHFAudio.dependencies
Section titled “HFAudio.dependencies”def dependencies() -> Mapping[str, object]The encoder’s fields and the hashes of the configuration and extractor files it reads, so an extractor that moved under an unpinned checkpoint name changes the identity too.
T5Text
Section titled “T5Text”class T5Text( checkpoint: str, transformer: T5EncoderTransformer, params: Variables, tokenizer: PreTrainedTokenizerBase, tokenizer_files: PinnedTokenizer, dtype: Dtype | None = None, revision: str | None = None, param_dtype: str = 'float32', max_length: int = 256,)Encodes text with a T5 encoder tower and its checkpoint’s tokenizer.
The tower is the one vendored in dew.nn.text_encoders. It is the text
half of an SD3.5/Flux-class run, whose MMDiT conditions on T5-XXL’s last
hidden states. Prompts are padded to max_length (256 by default), which
the run’s record stores. encode returns the tower’s last hidden state
and the attention mask as a TextContext.
T5Text.from_pretrained
Section titled “T5Text.from_pretrained”def from_pretrained( checkpoint: str = DEFAULT_T5_MODEL, *, dtype=None, revision: str | None = None, max_length: int = 256, param_dtype: str = 'float32', params: Variables | None = None,) -> T5Text