Skip to content

dew.data.sources.base

What a pipeline reads from: a store, its identity, and how it can be read.

A Source names a store (records in memory, files at a URL, a Hub dataset, a prepared TFDS build, a torch dataset, a catalogue of URLs) and says which of the three read modes it supports (Read). Memory and indexed reads hand over records by index (dew.data.dataset.Records); a streamed read hands over a ShardManifest, the store’s physical shards with one reader each, which the pipeline slices over its readers and workers and interleaves.

A source states facts about itself (metadata, lengths) and never scans to find them: a step that needs a global fact reads it here or from config, and a streamed source without it is refused before the run opens. describe and fingerprint are its identity: the first is what a saved position names, the second what a cache’s key and a run’s plan record (the URI, the resolved revision, each file’s size and ETag). label is the short name a person reads in a run’s metrics, never its identity.

NameSummary
READS
ConsumedA source whose read of a given kind goes through another source.
ReadHow a run reads every source it names: memory reads each whole into RAM (and may materialize its deterministic stages, design §10.3), indexed reads records by index through an index the store holds, streamed reads shards in order with no index.
ShardOne piece of a streamed source: a file, a byte or row range of one, a Parquet row group, or a disjoint row share.
ShardManifestA streamed source’s shards, in canonical order, and the reader that opens one: a picklable function returning a checkpointable grain IterDataset over the shard’s records in order.
SourceA store a pipeline reads, and the read modes it supports.

attribute source

READS: tuple[Read, ...] = ('memory', 'indexed', 'streamed')

class source

class Consumed(Protocol)

A source whose read of a given kind goes through another source.

Wrappers rebuild over their consumed children, retaining their own policy.

def consumed(read: Read) -> Source

The source a read of this one reads its records from.

attribute source

How a run reads every source it names: memory reads each whole into RAM (and may materialize its deterministic stages, design §10.3), indexed reads records by index through an index the store holds, streamed reads shards in order with no index.

dataclass source

class Shard(
id: str,
uri: str,
start: int = 0,
stop: int | None = None,
rows: int | None = None,
)

One piece of a streamed source: a file, a byte or row range of one, a Parquet row group, or a disjoint row share. id is immutable across runs (path and range, or a Hub revision and path), so a cache keys rows by (id, row).

dataclass source

class ShardManifest(
shards: tuple[Shard, ...],
reader: Callable[[Shard], pygrain.IterDataset],
subdivide: Callable[[int], tuple[Shard, ...]] | None = None,
index_ordered: bool = False,
)

A streamed source’s shards, in canonical order, and the reader that opens one: a picklable function returning a checkpointable grain IterDataset over the shard’s records in order.

When physical shards cannot fill the readers and workers, subdivide receives the needed count and returns exactly that many Shards, each a disjoint part of these records with an exact per-part position read by the same reader. None keeps the physical-shard requirement.

index_ordered states that the shards, one after another, read the records in their index order, as a store read by index streams them (_indexed_manifest).

class source

class Source(Protocol)

A store a pipeline reads, and the read modes it supports.

def describe() -> str

A stable description: store, URI, revision and split, never an address.

def label() -> str

The store’s short name in a run’s metrics: its dataset, builder or file name.

def fingerprint() -> Mapping[str, object]

The store’s identity for a cache key and a run’s plan: the URI, the resolved revision, and each file’s size and ETag or generation.

def supports(read: Read) -> bool

Whether the source can be read this way without provisioning.

def records(read: Read) -> Records

The records by index, for a memory or an indexed read.

def manifest() -> ShardManifest

The shards and their reader, for a streamed read.

def metadata() -> Mapping[str, object]

Facts the store states about itself (class names, a mixture’s shares, slot maxima, per-corpus token counts), never found by a scan.

def lengths() -> np.ndarray | None

Each record’s length when the store states it (a producer’s index), which length grouping ranks by; None when it does not.