Skip to content

dew.data.presets

Dataset functions that prepare one kind of record for a model, from the public steps, ending in batch.

Each is applied with pipe, the same preset for training and validation: flowers.pipe(presets.images, 64, batch=16, augment=True). Augmentation is off unless asked for. Fields a preset does not read pass through, stacked, except where it packs several records into one row.

  • images(ds, size, batch=) reads image, bytes or uint8 [H, W, 3], and writes image, uint8 [B, size, size, 3];
  • lm(ds, tokenizer, seq_len=, batch=) reads text, a string, and writes text, int32 [B, seq_len + 1], and its windows’ documents as text_coordinates (dew.batch.Coordinates);
  • sft(ds, tokenizer, seq_len=, batch=) reads messages, chat turns, and writes the lm fields and text_roles;
  • clips(ds, seconds, frames=, fps=, size=, batch=) reads mp4, media bytes or a path, and writes video, uint8 [B, frames, size, size, 3], and with sample_rate=, audio, float32 [B, samples], and audio_valid, bool [B, samples].
NameSummary
imagesImages in field decoded and resized to size squares, in batches of batch.
lmDocuments in field tokenized, ended by the tokenizer’s EOS where it has one (eos=True always, False never), cut into pieces that keep every next-token target, and packed into rows of seq_len + 1 tokens (the last a target) by pack, first-fit with bins open rows, best-fit, or concatenated, in batches of batch rows (records and host as images takes them).
token_windowsA token store’s stream of tokens (TokenFiles) cut into windows of seq_len inputs and their targets, stride apart (seq_len where None), read by index: each window a record of the store’s, whose rows and valid targets it states, in batches of batch (batch=None, records and host as images takes them).
sftConversations in messages rendered through the tokenizer’s chat template, each token’s speaker kept beside it for an assistant-only loss, and packed into rows of seq_len + 1 tokens by pack, as lm packs, in batches of batch rows (batch=None, records and host as images takes them).
promptsPrompt rows (verl’s columns: “prompt”, “data_source”, “ground_truth”, …) rendered to max_prompt_len left-padded ids, keeping a longer prompt’s tail, or dropping it (long="drop"), with their reward columns padded to reward_bytes, in batches of batch (batch=None, records and host as images takes them).
preferencesPreference pairs of token ids, chosen and rejected, each laid out in seq_len tokens with its completion’s mask, padded with pad_id (preferences.PreferencePair), in batches of batch (batch=None, records, host and cache as images takes them).
canvasConversations in messages rendered through the tokenizer’s chat template, then each assistant turn one row of a block-diffusion objective (ResponseCanvases): what comes before it in prompt_length tokens, padded with pad_id, and its response in canvases canvases of canvas_size, the tokenizer’s EOS filling the rest of its last canvas and pad_id the canvases after it; a longer prompt or response keeps its first tokens.
clipsWindows of seconds drawn from the media in mp4, each record’s own on every read and resume (video.PickClip), as frames frames at fps fitted into size squares, in batches of batch.
videoA listing of media files (Files(url, format="files"), cut to those whose name ends in one of extensions), or rows with a path or video_path, each read as clips reads its media, with its caption, caption where it states none.

function source

def images(
dataset: Dataset,
size: int,
*,
batch: int | None,
augment: bool | image_steps.DeviceAugment | RandomStep | tuple[RandomStep, ...] = False,
crop_scale: tuple[float, float] = (1.0, 1.0),
field: str = 'image',
staging_size: int | None = None,
draft: bool = False,
rejects: bool = True,
caption: image_steps.Caption | None = None,
cache: str | Cached | None = None,
records: int | None = None,
host: tuple[HostStep, ...] = (),
) -> Pipeline

Images in field decoded and resized to size squares, in batches of batch.

  • augment=True crops each image of a batch to crop_scale of its area, flips and colour-jitters it on the accelerator, by a key its record draws (images.AugmentationKey), so a resumed run augments as the uninterrupted one would; an images.DeviceAugment(size, ...) value sets those three itself. A random step, or a tuple of them, runs on the host right after the resize and the caption instead.
  • staging_size resizes to that size instead, for an augmentation that crops to size: augment=True and a DeviceAugment do, and host steps return size squares, as images.RandomCrop(size) and images.RandomResizedCrop(size) do. With no augmentation it is not used. draft decodes a JPEG at a reduced scale no smaller than that size (images.DecodeImage(at_least=)).
  • With rejects=False, an image that does not decode stops the run instead of being dropped and counted.
  • caption (images.caption(...)) writes each record’s caption after the resize; its class names, where it names none, are the source’s (class_names.label in its metadata), and with templates= it draws one per record.
  • cache (a URL, Cache or published Generation) holds the decoded and resized images and a fixed caption, each input’s own whatever order reads it: a shuffle or repeat written before the preset reads the cache as it reads the inputs. Drawn captions and augmentation run after it.
  • records keeps that many prepared records before the batch; host steps then read each batch on the host, before any step on the accelerator. batch=None leaves the records unbatched, for corpora mixed or phased and batched after, which no host step or accelerator augmentation can then read.

function source

def lm(
dataset: Dataset,
tokenizer: str | SavedTokenizer | None,
*,
seq_len: int,
batch: int | None,
field: str = 'text',
eos: bool | None = None,
pack: Literal['first-fit', 'best-fit', 'concat'] | None = PACK_POLICY,
bins: int = PACK_BINS,
stride: int | None = None,
cache: str | Cached | None = None,
records: int | None = None,
host: tuple[HostStep, ...] = (),
) -> Pipeline

Documents in field tokenized, ended by the tokenizer’s EOS where it has one (eos=True always, False never), cut into pieces that keep every next-token target, and packed into rows of seq_len + 1 tokens (the last a target) by pack, first-fit with bins open rows, best-fit, or concatenated, in batches of batch rows (records and host as images takes them). pack=None joins the documents into one stream of tokens instead and cuts it into windows stride apart (windows), which carry no document coordinates. With tokenizer=None the documents are token ids already, under “text”, such as a token store’s, whose windows a store reads by index (TokenFiles(url).pipe(presets.lm, None, seq_len=512, batch=8, pack=None)); a store packed must state where its documents end. cache holds the tokenized documents, as images holds its images: packed first-fit or best-fit, cut into the pieces a row holds, so the cache is seq_len’s alone; concatenated or cut into windows, whole. A generation published from the preset’s own unbatched first-fit rows (publish) reads its packing plan in place of the pack, the rows ordered after the preset. batch=None leaves the rows unbatched, for corpora each laid out alone and then mixed or phased (phases), batched after.

function source

def token_windows(
dataset: Dataset,
*,
seq_len: int,
batch: int | None,
stride: int | None = None,
records: int | None = None,
host: tuple[HostStep, ...] = (),
) -> Pipeline

A token store’s stream of tokens (TokenFiles) cut into windows of seq_len inputs and their targets, stride apart (seq_len where None), read by index: each window a record of the store’s, whose rows and valid targets it states, in batches of batch (batch=None, records and host as images takes them). A shuffle, repeat or take written before it orders and selects the windows. A store holds one split: validation tokens are a store of their own, its TokenFiles(url) through the preset as well.

function source

def sft(
dataset: Dataset,
tokenizer: str | SavedTokenizer,
*,
seq_len: int,
batch: int | None,
messages: str = 'messages',
long: Literal['chunk', 'truncate', 'drop'] = 'chunk',
pack: Literal['first-fit', 'best-fit', 'concat'] = PACK_POLICY,
bins: int = PACK_BINS,
cache: str | Cached | None = None,
records: int | None = None,
host: tuple[HostStep, ...] = (),
) -> Pipeline

Conversations in messages rendered through the tokenizer’s chat template, each token’s speaker kept beside it for an assistant-only loss, and packed into rows of seq_len + 1 tokens by pack, as lm packs, in batches of batch rows (batch=None, records and host as images takes them). A conversation longer than a row is cut into pieces that keep every turn (long="chunk"), cut at its end ("truncate"), or dropped and counted ("drop"). cache holds the rendered conversations so cut or fitted, as lm holds its documents, so the cache is that seq_len’s and long’s alone.

function source

def prompts(
dataset: Dataset,
tokenizer: str | SavedTokenizer,
*,
max_prompt_len: int,
batch: int | None,
pad_id: int = PROMPT_PAD,
long: Literal['drop', 'truncate'] = PROMPT_LONG,
thinking: bool | None = None,
reward_bytes: int | None = None,
cache: str | Cached | None = None,
records: int | None = None,
host: tuple[HostStep, ...] = (),
) -> Pipeline

Prompt rows (verl’s columns: “prompt”, “data_source”, “ground_truth”, …) rendered to max_prompt_len left-padded ids, keeping a longer prompt’s tail, or dropping it (long="drop"), with their reward columns padded to reward_bytes, in batches of batch (batch=None, records and host as images takes them). Without reward_bytes, a memory read measures it over each store, the widest reward string it holds, so every row fits whole; another read needs it given, or stated by the store’s metadata. cache holds the rendered prompts, as images holds its images.

function source

def preferences(
dataset: Dataset,
*,
seq_len: int,
batch: int | None,
pad_id: int = 0,
cache: str | Cached | None = None,
records: int | None = None,
host: tuple[HostStep, ...] = (),
) -> Pipeline

Preference pairs of token ids, chosen and rejected, each laid out in seq_len tokens with its completion’s mask, padded with pad_id (preferences.PreferencePair), in batches of batch (batch=None, records, host and cache as images takes them). No tokenizer: a pair of raw text is tokenized by a step before it.

function source

def canvas(
dataset: Dataset,
tokenizer: str | SavedTokenizer,
*,
prompt_length: int,
canvas_size: int,
batch: int | None,
canvases: int = 1,
pad_id: int = 0,
messages: str = 'messages',
cache: str | Cached | None = None,
records: int | None = None,
host: tuple[HostStep, ...] = (),
) -> Pipeline

Conversations in messages rendered through the tokenizer’s chat template, then each assistant turn one row of a block-diffusion objective (ResponseCanvases): what comes before it in prompt_length tokens, padded with pad_id, and its response in canvases canvases of canvas_size, the tokenizer’s EOS filling the rest of its last canvas and pad_id the canvases after it; a longer prompt or response keeps its first tokens. Never packed, in batches of batch (batch=None, records, host and cache as images takes them). A tokenizer with no EOS cannot fill a canvas, and is refused.

function source

def clips(
dataset: Dataset,
seconds: float,
*,
frames: int,
fps: float,
size: int,
batch: int | None,
sample_rate: int | None = None,
margin: float = 0.0,
context: float = 0.0,
missing: Literal['silence', 'reject'] = 'silence',
rejects: bool = True,
records: int | None = None,
host: tuple[HostStep, ...] = (),
) -> Pipeline

Windows of seconds drawn from the media in mp4, each record’s own on every read and resume (video.PickClip), as frames frames at fps fitted into size squares, in batches of batch. With sample_rate, the window’s sound beside them, mono at that rate, filled with silence marked invalid where the media has none (audio.DecodeAudio), or with missing="reject" dropped. Media shorter than the window, or that does not decode, is counted and dropped, or with rejects=False stops the run. A window is drawn margin seconds clear of the media’s ends, and its sound is context seconds wider on each side. The media and its window are read here and leave the record. batch=None, records and host as images takes them.

function source

def video(
dataset: Dataset,
seconds: float,
*,
frames: int,
fps: float,
size: int,
batch: int | None,
caption: str = '',
extensions: tuple[str, ...] = ('.mp4', '.avi', '.mov', '.webm'),
sample_rate: int | None = None,
margin: float = 0.0,
context: float = 0.0,
missing: Literal['silence', 'reject'] = 'silence',
rejects: bool = True,
records: int | None = None,
host: tuple[HostStep, ...] = (),
) -> Pipeline

A listing of media files (Files(url, format="files"), cut to those whose name ends in one of extensions), or rows with a path or video_path, each read as clips reads its media, with its caption, caption where it states none. The other settings are clips’.