dew.data.sources.base
What a pipeline reads from: a store, its identity, and how it can be read.
A Source names a store (records in memory, files at a URL, a Hub dataset,
a prepared TFDS build, a torch dataset, a catalogue of URLs) and says which of
the three read modes it supports (Read). Memory and indexed reads hand over
records by index (dew.data.dataset.Records); a streamed read hands over a
ShardManifest, the store’s physical shards with one reader each, which the
pipeline slices over its readers and workers and interleaves.
A source states facts about itself (metadata, lengths) and never scans to
find them: a step that needs a global fact reads it here or from config, and
a streamed source without it is refused before the run opens. describe and
fingerprint are its identity: the first is what a saved position names, the
second what a cache’s key and a run’s plan record (the URI, the resolved
revision, each file’s size and ETag). label is the short name a person reads
in a run’s metrics, never its identity.
| Name | Summary |
|---|---|
READS | |
Consumed | A source whose read of a given kind goes through another source. |
Read | How a run reads every source it names: memory reads each whole into RAM (and may materialize its deterministic stages, design §10.3), indexed reads records by index through an index the store holds, streamed reads shards in order with no index. |
Shard | One piece of a streamed source: a file, a byte or row range of one, a Parquet row group, or a disjoint row share. |
ShardManifest | A streamed source’s shards, in canonical order, and the reader that opens one: a picklable function returning a checkpointable grain IterDataset over the shard’s records in order. |
Source | A store a pipeline reads, and the read modes it supports. |
READS: tuple[Read, ...] = ('memory', 'indexed', 'streamed')Consumed
Section titled “Consumed”class Consumed(Protocol)A source whose read of a given kind goes through another source.
Wrappers rebuild over their consumed children, retaining their own policy.
Consumed.consumed
Section titled “Consumed.consumed”def consumed(read: Read) -> SourceThe source a read of this one reads its records from.
How a run reads every source it names: memory reads each whole into RAM
(and may materialize its deterministic stages, design §10.3), indexed reads
records by index through an index the store holds, streamed reads shards in
order with no index.
class Shard( id: str, uri: str, start: int = 0, stop: int | None = None, rows: int | None = None,)One piece of a streamed source: a file, a byte or row range of one,
a Parquet row group, or a disjoint row share. id is immutable across runs
(path and range, or a Hub revision and path), so a cache keys rows by (id, row).
ShardManifest
Section titled “ShardManifest”class ShardManifest( shards: tuple[Shard, ...], reader: Callable[[Shard], pygrain.IterDataset], subdivide: Callable[[int], tuple[Shard, ...]] | None = None, index_ordered: bool = False,)A streamed source’s shards, in canonical order, and the reader that
opens one: a picklable function returning a checkpointable grain
IterDataset over the shard’s records in order.
When physical shards cannot fill the readers and workers, subdivide
receives the needed count and returns exactly that many Shards, each
a disjoint part of these records with an exact per-part position read
by the same reader. None keeps the physical-shard requirement.
index_ordered states that the shards, one after another, read the
records in their index order, as a store read by index streams them
(_indexed_manifest).
Source
Section titled “Source”class Source(Protocol)A store a pipeline reads, and the read modes it supports.
Source.describe
Section titled “Source.describe”def describe() -> strA stable description: store, URI, revision and split, never an address.
Source.label
Section titled “Source.label”def label() -> strThe store’s short name in a run’s metrics: its dataset, builder or file name.
Source.fingerprint
Section titled “Source.fingerprint”def fingerprint() -> Mapping[str, object]The store’s identity for a cache key and a run’s plan: the URI, the resolved revision, and each file’s size and ETag or generation.
Source.supports
Section titled “Source.supports”def supports(read: Read) -> boolWhether the source can be read this way without provisioning.
Source.records
Section titled “Source.records”def records(read: Read) -> RecordsThe records by index, for a memory or an indexed read.
Source.manifest
Section titled “Source.manifest”def manifest() -> ShardManifestThe shards and their reader, for a streamed read.
Source.metadata
Section titled “Source.metadata”def metadata() -> Mapping[str, object]Facts the store states about itself (class names, a mixture’s shares, slot maxima, per-corpus token counts), never found by a scan.
Source.lengths
Section titled “Source.lengths”def lengths() -> np.ndarray | NoneEach record’s length when the store states it (a producer’s index), which length grouping ranks by; None when it does not.