Skip to content

dew.distributed.batches

A run’s batches on the devices: the fields a batch carries to the host apart (split_fields, with_host), the shape a batch’s compiled step is keyed by (Signature), its shardings and its placement on a mesh (batch_shardings, shard_batch, first_reader_batch), its host fields gathered across processes by the rows’ origins (row_origins, shared_host_fields), and the prefetch that keeps the next batches in flight within a memory budget (DevicePrefetchIterator). Where the state’s arrays live is dew.distributed.placement’s, whose placed-tree helpers this reads.

NameSummary
DevicePrefetchIteratorReads batches on a worker thread and places them on the mesh ahead of the step.
batch_shardingsReturn a sharding for each leaf of batch, chosen from the leaf’s shape.
placed_abstractThe batch shard_batch would place from this process’s share batch, as shapes with their shardings and nothing transferred: what a run reads a step’s plan and its whole state from before any array of it is on a device (Trainer.fit).

class source

class DevicePrefetchIterator(
iterator: Iterator,
mesh: Mesh,
depth: int = PREFETCH_DEPTH,
source_state: bytes | None = None,
*,
prepare: Callable[[Batch], Batch] | None = None,
reads: Mapping[Signature, tuple[bool, ...]] | None = None,
)

Reads batches on a worker thread and places them on the mesh ahead of the step.

The iterator closes its source when the source runs out or the iterator is closed, so use it as a context manager or call close, even after a loop that ended early. At most depth batches are queued, plus the one being read and placed.

Every call into the source runs on the worker thread: next, reading the checkpoint position, and the final close. The exception is the source’s optional request_stop hook, which runs on the thread that closes the iterator, so it must be thread-safe and must not block.

source_state, when given, is a saved position the worker restores before its first read; if the source is not checkpointable, the first next raises TypeError. After each batch the iterator returns, source_state holds the source’s position after that batch.

prepare, when supplied, transforms the source’s host batch before placement, on the same worker and after its checkpoint position is read. peek returns the first batch so prepared, on the host, before the worker places anything; the worker places it, and reads on, from the first next. reads holds, by the placed batch’s signature, which leaves a compiled step reads (step_reads); a batch of a signature it lacks, its step not yet compiled, is placed whole (shard_batch). Analytical per-row telemetry therefore travels with its own batch and never requires a synchronous device-to-host read in the step loop.

If the worker thread cannot be created, the source is finalized on the caller’s thread before the error is re-raised; no worker has touched it at that point. That path runs the source’s close synchronously, so the close has to return on its own. When a running worker is stuck in source code that cannot be stopped, close raises TimeoutError.

def held() -> Iterator[None]

Place no batch while the caller’s block runs, a placement under way finishing first; the worker reads on meanwhile, and its next placement waits for the block to end.

A run holds its prefetcher while it does device work beside its steps (a compile, an evaluation, a checkpoint): XLA’s GPU pool gives an allocation the lowest block it fits, so a batch placed while that work holds memory lands past it, inside the free block the next step’s temporaries take whole, and stays there until a later step reads it (row 2088: a 4B’s step that fit its 50.35 GiB block before its first validation pass found it split after the pass, by a batch the stream read ahead of the pass). The block must not read this iterator, whose next batch may wait on the placement it holds, but through read_held; closing it stops the worker as ever.

def read_held() -> Batch

The next batch, read by a block that holds this iterator’s placements (held): the hold is let go for the read, whose batch may wait on a placement, and taken again after it, once a placement it let through finishes. A rollout that draws further batches reads them so (dew.training.rollout.rolled_out).

def close(*, timeout: float | None = None) -> None

Stop the worker and wait for it, discarding unread batches but keeping source_state.

The wait lasts timeout seconds, else the source’s own stop_seconds (a grain pipeline joins one worker process after another), else until the worker ends: the source, the external reader, owns its deadline. If the worker is still alive after a deadline, this raises TimeoutError. Cancellation stays requested and a later close may wait again, but the source’s in-flight work and buffers may not have been released. An error from the source’s own close is raised here too.

def peek() -> Batch | None

The first batch as the source and prepare give it, on the host, before the worker places any; None where the source holds none. A run checks what it will train against its shapes before it places its state (dew.distributed.batches.placed_abstract); the first next places this batch and reads on, after which the peeked batch’s buffers are the reader’s again.

function source

def batch_shardings(mesh: Mesh | AbstractMesh, batch: Batch) -> Placement[Batch]

Return a sharding for each leaf of batch, chosen from the leaf’s shape.

Rows split over the row axes, those the rules in context give activation_batch (batch_axes): data, expert and fsdp by default. A leaf of rank 2 or 3 holds a sequence per row: token ids, segment ids, positions, encoded tokens. Its second dimension splits over the sequence axis when the mesh has one that divides it, and otherwise stays whole, the way logical_spec drops a name no dimension can split. An image or a video is not a sequence, so only its rows split.

This only decides placement and never changes values. Until a model constrains its attention to the sequence axis, a sequence split here is gathered again inside the model. Only the shape is read, so a leaf that is already a global array costs no transfer.

function source

def placed_abstract(mesh: Mesh, batch: Batch) -> Batch

The batch shard_batch would place from this process’s share batch, as shapes with their shardings and nothing transferred: what a run reads a step’s plan and its whole state from before any array of it is on a device (Trainer.fit).