dew.data.presets
Dataset functions that prepare one kind of record for a model, from the public steps, ending in batch.
Each is applied with pipe, the same preset for training and validation:
flowers.pipe(presets.images, 64, batch=16, augment=True). Augmentation is
off unless asked for. Fields a preset does not read pass through, stacked,
except where it packs several records into one row.
images(ds, size, batch=)readsimage, bytes or uint8 [H, W, 3], and writesimage, uint8 [B, size, size, 3];lm(ds, tokenizer, seq_len=, batch=)readstext, a string, and writestext, int32 [B, seq_len + 1], and its windows’ documents astext_coordinates(dew.batch.Coordinates);sft(ds, tokenizer, seq_len=, batch=)readsmessages, chat turns, and writes thelmfields andtext_roles;clips(ds, seconds, frames=, fps=, size=, batch=)readsmp4, media bytes or a path, and writesvideo, uint8 [B, frames, size, size, 3], and withsample_rate=,audio, float32 [B, samples], andaudio_valid, bool [B, samples].
| Name | Summary |
|---|---|
images | Images in field decoded and resized to size squares, in batches of batch. |
lm | Documents in field tokenized, ended by the tokenizer’s EOS where it has one (eos=True always, False never), cut into pieces that keep every next-token target, and packed into rows of seq_len + 1 tokens (the last a target) by pack, first-fit with bins open rows, best-fit, or concatenated, in batches of batch rows (records and host as images takes them). |
token_windows | A token store’s stream of tokens (TokenFiles) cut into windows of seq_len inputs and their targets, stride apart (seq_len where None), read by index: each window a record of the store’s, whose rows and valid targets it states, in batches of batch (batch=None, records and host as images takes them). |
sft | Conversations in messages rendered through the tokenizer’s chat template, each token’s speaker kept beside it for an assistant-only loss, and packed into rows of seq_len + 1 tokens by pack, as lm packs, in batches of batch rows (batch=None, records and host as images takes them). |
prompts | Prompt rows (verl’s columns: “prompt”, “data_source”, “ground_truth”, …) rendered to max_prompt_len left-padded ids, keeping a longer prompt’s tail, or dropping it (long="drop"), with their reward columns padded to reward_bytes, in batches of batch (batch=None, records and host as images takes them). |
preferences | Preference pairs of token ids, chosen and rejected, each laid out in seq_len tokens with its completion’s mask, padded with pad_id (preferences.PreferencePair), in batches of batch (batch=None, records, host and cache as images takes them). |
canvas | Conversations in messages rendered through the tokenizer’s chat template, then each assistant turn one row of a block-diffusion objective (ResponseCanvases): what comes before it in prompt_length tokens, padded with pad_id, and its response in canvases canvases of canvas_size, the tokenizer’s EOS filling the rest of its last canvas and pad_id the canvases after it; a longer prompt or response keeps its first tokens. |
clips | Windows of seconds drawn from the media in mp4, each record’s own on every read and resume (video.PickClip), as frames frames at fps fitted into size squares, in batches of batch. |
video | A listing of media files (Files(url, format="files"), cut to those whose name ends in one of extensions), or rows with a path or video_path, each read as clips reads its media, with its caption, caption where it states none. |
images
Section titled “images”def images( dataset: Dataset, size: int, *, batch: int | None, augment: bool | image_steps.DeviceAugment | RandomStep | tuple[RandomStep, ...] = False, crop_scale: tuple[float, float] = (1.0, 1.0), field: str = 'image', staging_size: int | None = None, draft: bool = False, rejects: bool = True, caption: image_steps.Caption | None = None, cache: str | Cached | None = None, records: int | None = None, host: tuple[HostStep, ...] = (),) -> PipelineImages in field decoded and resized to size squares, in batches of batch.
augment=Truecrops each image of a batch tocrop_scaleof its area, flips and colour-jitters it on the accelerator, by a key its record draws (images.AugmentationKey), so a resumed run augments as the uninterrupted one would; animages.DeviceAugment(size, ...)value sets those three itself. A random step, or a tuple of them, runs on the host right after the resize and the caption instead.staging_sizeresizes to that size instead, for an augmentation that crops tosize:augment=Trueand aDeviceAugmentdo, and host steps returnsizesquares, asimages.RandomCrop(size)andimages.RandomResizedCrop(size)do. With no augmentation it is not used.draftdecodes a JPEG at a reduced scale no smaller than that size (images.DecodeImage(at_least=)).- With
rejects=False, an image that does not decode stops the run instead of being dropped and counted. caption(images.caption(...)) writes each record’s caption after the resize; its class names, where it names none, are the source’s (class_names.labelin its metadata), and withtemplates=it draws one per record.cache(a URL,Cacheor publishedGeneration) holds the decoded and resized images and a fixed caption, each input’s own whatever order reads it: a shuffle or repeat written before the preset reads the cache as it reads the inputs. Drawn captions and augmentation run after it.recordskeeps that many prepared records before the batch;hoststeps then read each batch on the host, before any step on the accelerator.batch=Noneleaves the records unbatched, for corpora mixed or phased and batched after, which no host step or accelerator augmentation can then read.
def lm( dataset: Dataset, tokenizer: str | SavedTokenizer | None, *, seq_len: int, batch: int | None, field: str = 'text', eos: bool | None = None, pack: Literal['first-fit', 'best-fit', 'concat'] | None = PACK_POLICY, bins: int = PACK_BINS, stride: int | None = None, cache: str | Cached | None = None, records: int | None = None, host: tuple[HostStep, ...] = (),) -> PipelineDocuments in field tokenized, ended by the tokenizer’s EOS where it has one (eos=True
always, False never), cut into pieces that keep every next-token target, and packed into
rows of seq_len + 1 tokens (the last a target) by pack, first-fit with bins open rows,
best-fit, or concatenated, in batches of batch rows (records and host as images takes
them). pack=None joins the documents into one stream of tokens instead and cuts it into
windows stride apart (windows), which carry no document coordinates. With
tokenizer=None the documents are token ids already, under “text”, such as a token store’s,
whose windows a store reads by index (TokenFiles(url).pipe(presets.lm, None, seq_len=512, batch=8, pack=None)); a store packed must state where its documents end. cache holds the
tokenized documents, as images holds its images: packed first-fit or best-fit, cut into the
pieces a row holds, so the cache is seq_len’s alone; concatenated or cut into windows, whole.
A generation published from the preset’s own unbatched first-fit rows (publish) reads its
packing plan in place of the pack, the rows ordered after the preset. batch=None leaves the
rows unbatched, for corpora each laid out alone and then mixed or phased (phases), batched
after.
token_windows
Section titled “token_windows”def token_windows( dataset: Dataset, *, seq_len: int, batch: int | None, stride: int | None = None, records: int | None = None, host: tuple[HostStep, ...] = (),) -> PipelineA token store’s stream of tokens (TokenFiles) cut into windows of seq_len inputs and their
targets, stride apart (seq_len where None), read by index: each window a record of the
store’s, whose rows and valid targets it states, in batches of batch (batch=None, records
and host as images takes them). A shuffle, repeat or take written before it orders and
selects the windows. A store holds one split: validation tokens are a store of their own, its
TokenFiles(url) through the preset as well.
def sft( dataset: Dataset, tokenizer: str | SavedTokenizer, *, seq_len: int, batch: int | None, messages: str = 'messages', long: Literal['chunk', 'truncate', 'drop'] = 'chunk', pack: Literal['first-fit', 'best-fit', 'concat'] = PACK_POLICY, bins: int = PACK_BINS, cache: str | Cached | None = None, records: int | None = None, host: tuple[HostStep, ...] = (),) -> PipelineConversations in messages rendered through the tokenizer’s chat
template, each token’s speaker kept beside it for an assistant-only loss,
and packed into rows of seq_len + 1 tokens by pack, as lm packs,
in batches of batch rows (batch=None, records and host as
images takes them). A conversation longer than a row is cut into pieces that keep
every turn (long="chunk"), cut at its end ("truncate"), or dropped
and counted ("drop"). cache holds the rendered conversations so cut or fitted, as lm holds
its documents, so the cache is that seq_len’s and long’s alone.
prompts
Section titled “prompts”def prompts( dataset: Dataset, tokenizer: str | SavedTokenizer, *, max_prompt_len: int, batch: int | None, pad_id: int = PROMPT_PAD, long: Literal['drop', 'truncate'] = PROMPT_LONG, thinking: bool | None = None, reward_bytes: int | None = None, cache: str | Cached | None = None, records: int | None = None, host: tuple[HostStep, ...] = (),) -> PipelinePrompt rows (verl’s columns: “prompt”, “data_source”, “ground_truth”, …) rendered to
max_prompt_len left-padded ids, keeping a longer prompt’s tail, or dropping it (long="drop"),
with their reward columns padded to reward_bytes, in batches of batch (batch=None,
records and host as images takes them). Without reward_bytes, a memory read measures
it over each store, the widest reward string it holds, so every row fits whole; another read
needs it given, or stated by the store’s metadata. cache holds the rendered prompts, as
images holds its images.
preferences
Section titled “preferences”def preferences( dataset: Dataset, *, seq_len: int, batch: int | None, pad_id: int = 0, cache: str | Cached | None = None, records: int | None = None, host: tuple[HostStep, ...] = (),) -> PipelinePreference pairs of token ids, chosen and rejected, each laid out in seq_len tokens with
its completion’s mask, padded with pad_id (preferences.PreferencePair), in batches of
batch (batch=None, records, host and cache as images takes them). No tokenizer: a
pair of raw text is tokenized by a step before it.
canvas
Section titled “canvas”def canvas( dataset: Dataset, tokenizer: str | SavedTokenizer, *, prompt_length: int, canvas_size: int, batch: int | None, canvases: int = 1, pad_id: int = 0, messages: str = 'messages', cache: str | Cached | None = None, records: int | None = None, host: tuple[HostStep, ...] = (),) -> PipelineConversations in messages rendered through the tokenizer’s chat template, then each
assistant turn one row of a block-diffusion objective (ResponseCanvases): what comes before
it in prompt_length tokens, padded with pad_id, and its response in canvases canvases of
canvas_size, the tokenizer’s EOS filling the rest of its last canvas and pad_id the canvases
after it; a longer prompt or response keeps its first tokens. Never packed, in batches of
batch (batch=None, records, host and cache as images takes them). A tokenizer with
no EOS cannot fill a canvas, and is refused.
def clips( dataset: Dataset, seconds: float, *, frames: int, fps: float, size: int, batch: int | None, sample_rate: int | None = None, margin: float = 0.0, context: float = 0.0, missing: Literal['silence', 'reject'] = 'silence', rejects: bool = True, records: int | None = None, host: tuple[HostStep, ...] = (),) -> PipelineWindows of seconds drawn from the media in mp4, each record’s own
on every read and resume (video.PickClip), as frames frames at fps
fitted into size squares, in batches of batch. With sample_rate,
the window’s sound beside them, mono at that rate, filled with silence
marked invalid where the media has none (audio.DecodeAudio), or with
missing="reject" dropped. Media shorter than the window, or that does not decode, is
counted and dropped, or with rejects=False stops the run. A window is drawn margin
seconds clear of the media’s ends, and its sound is context seconds wider on each side. The
media and its window are read here and leave the record. batch=None, records and host as
images takes them.
def video( dataset: Dataset, seconds: float, *, frames: int, fps: float, size: int, batch: int | None, caption: str = '', extensions: tuple[str, ...] = ('.mp4', '.avi', '.mov', '.webm'), sample_rate: int | None = None, margin: float = 0.0, context: float = 0.0, missing: Literal['silence', 'reject'] = 'silence', rejects: bool = True, records: int | None = None, host: tuple[HostStep, ...] = (),) -> PipelineA listing of media files (Files(url, format="files"), cut to those whose name ends in one
of extensions), or rows with a path or video_path, each read as clips reads its media,
with its caption, caption where it states none. The other settings are clips’.