Skip to content

dew.data.dataset

The compiled Loader, shared reader partitions and checkpoint contracts.

Every stream is opened for a DataPartition, the share of each global batch its reader reads. The trainer gets it from the mesh (DataPartition.of), because processes that a pipeline or a split sequence spans hold the same rows and so read the same share. A loader reads the share it is given and nothing else.

An indexed global order resumes from a record count across process counts. Unknown-length, rejecting and token-budget orders instead save Grain’s per-share state and refuse changed topology before a reader opens.

NameSummary
BudgetedA stream that reports how long stopping it may take.
CheckpointableA data stream that can report where it stopped and resume from there.
CloseableA stream that holds resources a stopped run must release, such as worker processes, file handles or a shared memory block.
DataPartitionThe share of every global batch a reader reads: the indexth of count equal, disjoint shares.
ForwardingForwards a stream wrapper’s stop signal, stop budget, close, identity and where it ends to its source.
GlobalStreamReads a training stream whose saved position is one global record count.
IdentifiedA training stream that names the records it reads and says where it ends (Ends).
LoaderOpens the batches a run trains and validates on.
LoadingRead throughput settings, as Grain’s four knobs.
PhasedStreamReads one global order per phase, switching at step boundaries.
RampGrows the global batch over the run’s first records.
RampedStreamGrows a training stream’s step over the run’s first records.
ReaderOpens a fresh iterator over one share of every global batch, the share the partition names.
RecordsReturns records by index, which is how the loaders read a source.
ResumableA batch stream that can resume where it stopped; this is what a batch ramp wraps.
SourceSliceReads source[start:stop] by index.
StageOne stage of a batch ramp: the global batch a step reads in it, and the record it starts at.
StoppableA stream that can be asked to stop before it is drained.
mappedReturn stream with stage applied to each batch, forwarding stop, close and position.

class source

class Budgeted(Protocol)

A stream that reports how long stopping it may take.

It is separate from Stoppable because a stream can report a budget without accepting a stop request. A wrapper forwards each one separately.

class source

class Checkpointable(Protocol)

A data stream that can report where it stopped and resume from there.

Grain’s iterators satisfy this, and so does GlobalStream, and so does a stream a reader declares to replay (replayed). The state itself is opaque here; dew.position tells which of its two kinds a checkpoint holds.

def get_state() -> Position
def set_state(state: Position) -> None

class source

class Closeable(Protocol)

A stream that holds resources a stopped run must release, such as worker processes, file handles or a shared memory block.

def close() -> None

dataclass source

class DataPartition(
index: int = 0,
count: int = 1,
readers: int = 1,
reader: int = 0,
row_multiple: int = 1,
window: int = 1,
)

The share of every global batch a reader reads: the indexth of count equal, disjoint shares.

A loader cuts its record order as index :: count, so the shares of global batch k together hold the same records at every count, and a record count marks the same place in the stream whatever the count. The trainer gets the share a process reads from the mesh (DataPartition.of). Processes whose devices hold the same rows read the same share, because the axes between them split a sequence or hold a pipeline’s stages, not rows. DataPartition() is a single reader of every row, which is what a single process reads.

readers: int

The number of processes that read this share.

Each reads the same records. A source whose rows can differ between reads of one share, such as a network fetch that drops failed records, must either refuse more than one reader or read on the first reader alone and send its batch to the others (dew.distributed.batches.first_reader_batch).

reader: int

This process’s place among the share’s readers, in process order (0 for the first).

row_multiple: int

Number of distinct row shards this reader feeds within its host.

window: int

The steps an update pools (Trainer’s accumulation): a batch ramp changes stage only where an update begins (Ramp.stages). It cuts the stream into steps and names no data, so it is no part of what a share reads.

def of(mesh: Mesh, window: int = 1) -> DataPartition

Return the share of every global batch this process reads on mesh.

A batch’s rows split over the row axes and no others: those the rules in context give activation_batch (batch_axes). The sequence axis splits positions, the tensor axis splits widths, and the stage axis holds a pipeline’s stages, so processes whose devices hold the same row shards need the same rows. The processes therefore fall into groups by the rows they hold. Each group reads one share, numbered by the first row shard it holds, and every process in the group reads it (readers); reader is this process’s place among them, in process order.

A hand-built device order can produce groups whose rows overlap without being the same rows. No share can then be read whole by each group, so such a mesh is refused.

def bounded(examples: int, records: int | None = None) -> int

This share’s examples of the pool’s first examples, where each share reads records of its own dealt index :: count (cut) from a root of records a pass, each pass over its own: its records in every whole pass the bound spans, and its records among those the last pass reaches. Where no pass’s count is known ahead (a step before the bound drops records), the examples themselves are dealt so, every count-th from its index.

def held(rows: int, offset: int = 0) -> tuple[slice, bool]

Which of rows rows a validation read of this share holds, and whether they are real: every count-th from its index, offset places on as a resume’s count inside a batch leaves it (GlobalStream); where that holds none of rows that hold some, the first in their place, not real (FilledBatch marks them so), so every share reads a batch where any does and none scores another’s row.

def cut(offset: int = 0) -> slice

This share’s rows of an order or a batch: every count-th from its first (first).

def first(offset: int = 0) -> int

This share’s first row of an order or a batch: its index, offset places on as a resume’s count inside a batch leaves it (GlobalStream).

def rows(batch: int) -> int

Return how many rows of a batch-row global batch one share holds.

batch must split evenly into the shares: a remainder would train on fewer records a step than the run reports, and a batch smaller than the share count would leave a share with nothing.

class source

class Forwarding

Forwards a stream wrapper’s stop signal, stop budget, close, identity and where it ends to its source.

Subclasses keep the source at _source and override close for their own cleanup around super().close().

def request_stop() -> None
def close() -> None
def identity() -> str

class source

class GlobalStream(
open_at: Callable[[int], pygrain.DatasetIterator[Batch]],
batch: int,
plan: Plan,
loading: Loading,
*,
resume: Literal['exact', 'non-exact'] = 'exact',
endless: bool = True,
)

Reads a training stream whose saved position is one global record count.

The stream is an endless shuffled order over the whole corpus, and each step is cut from it here. Global batch k is records [k * batch, (k + 1) * batch) of that order, and share p of n reads every nth of them starting at p. The position is then the number of records the run has consumed: one number, the same on every reader, tied to no share. A checkpoint written by two processes resumes on one process or four, on the same records in the same steps.

open_at(offset) starts the share’s read at a record offset. A restore uses it instead of replaying the stream, since an offset is just a slice bound. It is called on the first batch and again after set_state, so a stream restored before it is read does not start its workers twice.

The offset alone would resume that many records into whatever order the run now has, so the plan’s order (order) and the plan are saved with it and a restore whose plan disagrees is refused (Plan.check). dew.position defines the format that both this class and dew.checkpoints read.

Every share steps as many batches of the order, its whole batches where it drops a remainder, and else a share past the order’s last batch reading on with batches of no real row (dew.data.lowering._evened), so where it ends, the shares run out at the same read (ends).

order: str

The order’s description, which a saved position is compared against.

def get_state() -> bytes
def identity() -> str

The plan’s identity (Plan.identity), which a pool agrees before it reads: hosts whose copies of a corpus hold other records, or whose steps make other values, are refused however the sources describe themselves.

def set_state(state: bytes) -> None
def close() -> None

class source

class Identified(Protocol)

A training stream that names the records it reads and says where it ends (Ends).

Each process of a pool opens its own stream, over its own copy of the corpus, and Trainer.fit refuses a pool whose streams name different records before any process reads one. It agrees every read of a stream that may run out on some processes and not others, so a process whose stream ran out stops its peers before they step without it; one that never ends, or ends on every share together, is agreed at its first read (dew.training._fit.FitRun.agrees).

def identity() -> str

dataclass source

class Loader(
train: Reader,
val: Reader | None,
records: int | None,
batch: int,
ramp: Ramp | None = None,
held_out: int = 0,
counts: Mapping[str, CorpusCounts] = dict(),
validation_counts: Mapping[str, CorpusCounts] = dict(),
plan: Plan | None = None,
statistics: Callable[[], Mapping[str, float]] = _data_statistics,
batch_tokens: int | None = None,
device: tuple[DeviceStep, ...] = (),
widths: Mapping[str, object] = dict(),
tail: Tail = 'dropped',
order_records: int | None = None,
examples: int | None = None,
pass_examples: int | None = None,
epochs: int | None = None,
bound_ends: bool = True,
token_ends: tuple[int | None, ...] | None = None,
)

Opens the batches a run trains and validates on.

train(partition) opens an endless shuffled stream. val(partition) opens one pass over the held-out records in a fixed order, which ends by itself; val is None when nothing is held out. Both read the share of each global batch that partition names (DataPartition). batch is the global batch and records the number of training records behind it, so steps_per_epoch is one pass over them. ramped sets ramp when the run grows its batch over its first records, and batch is then the batch the ramp ends at. held_out is how many records of the training split a spec kept back for validation, which fit reports when the run starts; it is 0 when validation is a separate split or there is none.

Each factory call returns a fresh iterator, and the code that opened it owns closing it. Close it after use when it is Closeable; every wrapper around a stream (host steps, ramps, phases, the readers) forwards close to the stream it wraps. Never close the shared dataset or its backing store. A batch an iterator yields stays valid until its consumer asks for the next one. A consumer that keeps batches longer copies them, since a batch made by worker processes lives in shared memory and holds a descriptor for as long as it is kept. A source’s optional request_stop is a separate, thread-safe signal; it does not make it safe to call the final close while another thread is still iterating.

Image and video fields are uint8 in [0, 255], text is the tokenized {"input_ids", "attention_mask"} dict under “text”, and a token window is int32 ids under “text”.

Whether a run resumes its data position depends on the iterator. Grain-backed iterators have get_state and set_state, and host transforms forward the pair. A stream with neither trains, and a resume of its checkpoint is refused, unless its reader declares that it reads the same batches each time it opens (replayed), which positions it by replay. An indexed global-order position resumes on any partition, over one corpus or a weighted mixture. A stream that batches its own records reports its share’s own position, and dew.checkpoints resumes it only on a reader of that share.

device: tuple[DeviceStep, ...]

The training batch’s steps the compiled training step runs, on the accelerator, after the reader delivers it (DeviceStep).

widths: Mapping[str, object]

The run’s static widths its steps bound (run_widths): the facts its objective’s run-scoped steps read (Configured.scope), such as a decision’s slots, which a saved run of the objective records (Objective.bind_widths); empty where they read none.

tail: Tail

What becomes of the records an epoch leaves short of a whole batch: read, carried into the next epoch by an order that reads on or kept and filled by one that keeps them, so an epoch’s steps count their batch; or dropped, so they count none (Tail).

order_records: int | None

The records a training read of a global order that ends hands out over all its passes, the same on every share, by which a batch ramp ends its last step (RampedStream); None where the order never ends or its read counts none ahead.

examples: int | None

The records the training read is bounded to before they are packed (compile’s examples), whose steps a run’s examples= budget takes (example_steps); None where it is not bounded.

pass_examples: int | None

The source examples one pass of a training read with no record count holds (records None: an objective makes rows of them, or a pack lays them out, after the order, or a filter cuts it per share), as its read states them (dew.data.compile._pass_examples): an epoch of it; None where nothing states them, or the read has a record count.

epochs: int | None

The epochs a training read with no record count is bounded to, examples their passes over its source examples (pass_examples), as a run of that many reads it (dew.data.compile.fitted); None for a read bounded otherwise or not at all.

bound_ends: bool

Whether a read bounded in examples ends where its bound does, having read every one: no take or slice after the bound ends it first, which a run on the examples clock refuses (dew.training._fit.clocked), whose last batch consumes the examples no batch counted (drained).

token_ends: tuple[int | None, ...] | None

The committed tokens each phase of a schedule ends at (dew.data.pipeline.schedule), the last where the data ends or None where it runs on, which the trainer reads its commits against and ends the phase at (PhasedStream.ending); None for a read on no token clock.

def epoch_steps(epochs: int = 1, window: int = 1) -> int

Return the number of steps that epochs passes over the records take, in updates of window steps (Ramp.stages), refusing passes too short to fill a batch.

A batch ramp grows the batch over the run’s first records once, so its smaller early steps count once across all the passes, not once a pass. A stream without a record count has no epoch, so a run over it must give its length in steps instead.

def example_steps(examples: int, window: int = 1) -> int | None

The steps that read the examples source examples this loader’s training read is bounded to (examples), with every row the objective makes of each, its last batch short where they end inside one, in whole updates of window steps, where a global order’s plan counts them ahead (order_records); None for one read in its own order, as a pack that runs as rows arrive makes, or one an objective expands after the bound, which no pass reads ahead to count, such as a pool’s read whose shares each read records of their own and so end steps apart: its run keeps its progress in the examples its batches consume (dew.batch.EXAMPLES_CONSUMED), a pool’s every share reading on until each has run out (dew.training._fit.pooled_read). A pool’s read in one global order plans as one process’s does, every share ending at its steps, a share the order’s last batch ends before stepping batches of no real row (dew.data.lowering._evened). Refused where the read is bounded to another count, and where the last steps would make no whole update, so some examples would train nothing.

dataclass source

class Loading(
workers: int | Literal['auto'] = 0,
threads: int = 16,
read_buffer: int = 16,
worker_buffer: int = WORKER_BUFFER,
ram_budget_mb: int | None = None,
max_workers: int | None = None,
cache: bool = True,
)

Read throughput settings, as Grain’s four knobs.

None of the four changes which records a run sees or what they contain, so a host can tune them for its disk and still produce identical batches. The shuffle seed is not one of them: it sets the order records arrive in and keys the per-record rng that augments and captions them.

Each knob counts something different:

  • workers: worker processes;
  • threads: reads one worker keeps in flight;
  • read_buffer: reads one worker holds ahead;
  • worker_buffer: whole batches one worker holds ready for the training process.

threads and read_buffer count the elements a read produces: a whole batch where the order is indexed, which is cut into batches by index before any reader starts, as Grain’s own indexed pipelines are, and a record where the order streams and is batched as it arrives.

The defaults, 16 reads in flight and 16 held ahead, keep enough reads in flight for a remote store’s latency while holding few batches. On an 8-core armada host with two workers, Dew read 128-pixel images at 0.91 to 1.04 times plain Grain under the same settings, within the run-to-run noise of parity, and held about 32 batches, 50 MB for 32 such images, per worker. 64 and 128 buffered 300 MB per worker and matched Grain only because Grain itself slowed down there.

Zero workers is Grain’s own default: threads in the training process read the records. Each worker process imports the program again, so it costs seconds and a process’s memory before the first batch, and pays off only once decoding or augmentation is slower than the threads. On two cores of a shared workstation, the first batch of a 100-record pipeline took 17.8 s and 7.07 GiB resident across 33 processes with 32 workers, against under 0.1 s and 0.23 GiB with none.

cache: bool

Whether a run reads its chains’ leading deterministic steps from the records an earlier run made of the same records (dew.data.cache.automatic_root), which changes none of them.

stop_seconds: float

How long stopping this many workers may take before it counts as a hang.

This is Grain’s own bound. A stop first waits for the batch being read. Grain then stops the worker processes one after another, each finishing its current batch before it exits, and kills a worker that has not exited within Grain’s kill timeout; the read gets the same time. Slow stops are normal: four workers took 5.1 to 7.4 s to stop on 12 vCPUs.

def resolved(
records: pygrain.IterDataset,
*,
reads: int,
host: Host | None = None,
) -> Loading

These settings with automatic workers and read buffer sized for host (this process’s by default) from its CPUs and its memory together, as tf.data’s autotuning budgets both. Grain’s own pick (pick_performance_config) reads memory alone, takes the machine’s CPU count where a container may run on a few of them, and counts one batch for each worker.

A worker for each CPU, as tf.data’s parallelism takes each schedulable one, where there is more than one: on one CPU a worker adds a process and no parallelism, so the reading process reads. On four CPUs of an eight-core host an image read in four workers read as many images a second as in eight, within 2%, and in three 11% fewer (armada, 2026-10-26). At most max_workers of them, and no more than half the memory the process may still take holds (ram_budget_mb where given), each a process of its own (WORKER_PROCESS_BYTES) holding its worker_buffer batches and read_buffer reads, the read buffer itself no more reads than that half holds. records are the batches the read makes without workers, SAMPLED_BATCHES of which are measured; reads is how many reads make one: one where the order is read by batch, its rows where it streams records. A read that makes no batch reads without workers.

def announced_stop() -> Iterator[None]

Stop the workers inside this context, explaining on stderr if the stop takes long.

If the stop runs past 5 seconds (_QUIET_STOP_SECONDS), it prints once what it is waiting for, so a long stop does not look like a hang.

class source

class PhasedStream(
phases: Sequence[tuple[Callable[[tuple[int, ...]], GlobalReader], int | None]],
stop_seconds: float,
*,
tokens: Sequence[int | None] | None = None,
)

Reads one global order per phase, switching at step boundaries.

Each phase opens a GlobalReader, such as a compiled global order, and has the global record count it ends at, or None for the last phase, which runs on. Phase k starts its own order at its own record zero when the run’s count reaches the end of phase k - 1, so the records a step reads depend only on the step and the phase list, at any process count.

The saved position is the run’s record count, the current phase’s order, and each completed phase’s order with its end (position.Global). A restore checks the completed phases and the current phase’s order against this list, and nothing after them. So a run may append phases, or move a boundary it has not reached yet, and still resume where it stopped. A one-order run’s position is phase 0 with no completed phases, so a run that trained on one mixture resumes into a phase list that starts with it. Changing a phase the run has already read, or ending the current phase before the records the run already read in it, is refused.

A run on the committed-token clock (tokens, the committed tokens each phase ends at, the last’s where the data ends or None, dew.data.pipeline.schedule) knows no phase’s end in records ahead: the trainer ends the phase it stands in at the step whose commit reaches the phase’s tokens (ending), and the saved position carries that end, as the end of a phase the run completed. Each phase opens with the record ends of the phases before it (a corpus they read continues in it from where they left it), so one after an end the run has yet to reach opens once the run reaches it. A phase the run’s commits passed within one step ends where it starts, holding no record.

def get_state() -> bytes
def set_state(state: bytes) -> None
def ending(state: bytes) -> bytes

state, a position of this run on the committed-token clock, with the phase it stands in ended where it stands and the next opened there: where the run’s commits reached the phase’s tokens, which the trainer reads (dew.training.Trainer.fit).

def identity() -> str
def close() -> None

dataclass source

class Ramp(start: int, increment: int, samples: int)

Grows the global batch over the run’s first records.

A run starts at start records a step and adds increment each time the records read since the last increment reach samples divided by the number of increments. It stops at the batch the dataset was loaded with, and the difference must be a whole number of increments, as in MaxText’s batch rampup.

MaxText counts the batch per device, while Dew counts the global batch everywhere, so start and increment are records a step: MaxText’s per_device_batch_size_start times the device count is start here.

The stage a run is in depends only on the records it has read, which a checkpoint already holds as the data position, so a resumed run continues the ramp with nothing else saved. A stage lasts ceil(samples / increments / batch) steps, computed as one integer ratio instead of MaxText’s two floating-point divisions, so the boundaries are exact. A run whose updates pool several steps (DataPartition.window) begins each stage where an update does, at the first update boundary at or after the step it would begin at, as MaxText and torchtitan apply a batch schedule per optimizer step: no update pools two stages’ batches, and a boundary moves by less than one update. The records stay the position, so a run resumed under another window cuts its stages again. Where the ramp ends, MaxText’s loader drops what is left of its buffered batch while Dew keeps reading, so from then on MaxText reads later records than Dew in each step, although the batch sizes match.

def stages(final: int, window: int = 1) -> tuple[Stage, ...]

Return every stage of the ramp up to final, the run’s own global batch, each beginning where an update of window steps does.

Each stage’s batch has to split evenly into the mesh’s rows, and so into the shares its readers read; the trainer checks every stage against the mesh before the run starts.

def steps_for(records: int, final: int, window: int = 1, tail: Tail = 'dropped') -> int

Return the number of steps a run of updates of window steps takes to read records records under this ramp, a last step that holds fewer than a batch counted where it is read (tail).

Early steps read fewer records, so a pass over a corpus takes more steps than it would with the final batch from the start.

class source

class RampedStream(
source: Resumable,
stages: Sequence[Stage],
partition: DataPartition,
*,
tail: Tail = 'dropped',
order_records: int | None = None,
)

Grows a training stream’s step over the run’s first records.

It wraps a stream whose position is a global record count (a GlobalStream or a host transform over one) and decides only how many of its records each step reads. Reads run one whole final-size batch ahead, and each step is cut from a rolling buffer, as in MaxText’s loader. A step therefore reads the records the run would have read without the ramp, in the same order, and a stage change neither skips a record nor reads one twice.

Records left in the buffer have not been trained on and are not counted in the position, which is the number of records handed out so far. A restore gives the source that count, so it reopens its read there.

stages: tuple[Stage, ...]

The batch this stream reads at each stage, and where each stage begins.

The trainer reads them from the stream it opened. One compiled step per stage covers every shape a ramped run uses, and the mesh has to divide each of them.

def get_state() -> bytes
def set_state(state: bytes) -> None
def close() -> None

attribute source

Opens a fresh iterator over one share of every global batch, the share the partition names. The iterator is its caller’s to close.

class source

class Records(Protocol)

Returns records by index, which is how the loaders read a source.

A record is one example’s fields, in the shape Batch names, or the packed bytes an ArrayRecord file holds, which the spec’s own transform unpacks into fields. A record may be missing (None). A memory read and a cache leave a missing record out and count it, records held in memory leave it out of an order batched by index, and a stream skips it; any other source read in an order batched by index must hold every record.

Grain’s own RandomAccessDataSource declares the same two methods with the record as a type parameter. That parameter is invariant, so a source declared through it cannot be passed to a loader that names a different record type. This protocol names what the loaders actually read.

class source

class Resumable(Checkpointable, Protocol)

A batch stream that can resume where it stopped; this is what a batch ramp wraps.

class source

class SourceSlice(source: Indexed, start: int, stop: int)

Reads source[start:stop] by index.

It gives the train and validation loaders disjoint index ranges while Grain still handles the shuffle, the epochs and the sharding. The attributes stay plain because Grain pickles the source to its workers.

dataclass source

class Stage(batch: int, records: int)

One stage of a batch ramp: the global batch a step reads in it, and the record it starts at.

class source

class Stoppable(Protocol)

A stream that can be asked to stop before it is drained.

def request_stop() -> None

function source

def mapped(stream: Reader, stage: Callable[[Batch], Batch]) -> Reader

Return stream with stage applied to each batch, forwarding stop, close and position.