Skip to content

dew.data.sources

Storage capabilities and the source boundary every chain reads through.

NameSummary
TFDSA prepared TFDS split, read with TFDS’s native ArrayRecord data source. Documented in dew.data.sources.tfds.
ConcatenateThe records of sources one store after another, as one store: two splits of one Hub dataset read as one (Concatenate((hub.with_split("train"), hub.with_split("validation")))), or stores of any kinds.
FilesFiles at URLs, with producer indexes or explicit provisioning for text. Documented in dew.data.sources.files.
HubA pinned split of a Hub dataset: name is its configuration, as datasets.load_dataset(repo, name=) takes it, and columns the fields each record keeps (all where None). Documented in dew.data.sources.hub.
MemoryRecords, column arrays, or a DataFrame or Arrow table (DataFrame), with immutable positional IDs. Documented in dew.data.sources.memory.
ReadHow a run reads every source it names: memory reads each whole into RAM (and may materialize its deterministic stages, design §10.3), indexed reads records by index through an index the store holds, streamed reads shards in order with no index. Documented in dew.data.sources.base.
ShardOne piece of a streamed source: a file, a byte or row range of one, a Parquet row group, or a disjoint row share. Documented in dew.data.sources.base.
ShardManifestA streamed source’s shards, in canonical order, and the reader that opens one: a picklable function returning a checkpointable grain IterDataset over the shard’s records in order. Documented in dew.data.sources.base.
SourceA store a pipeline reads, and the read modes it supports. Documented in dew.data.sources.base.
TokenFilesA committed token export (dew.data.text.export_tokens, dew data export --tokenizer): its documents of token ids, the metadata its manifest states and where its documents end, each a record its producer wrote, as its length index states it (document_ends).
from_graindataset as a Dew dataset, its content version identity where given (Grain). Documented in dew.data.sources.grain.
from_hfA Hugging Face datasets.Dataset or IterableDataset as a Dew dataset (HF): an IterableDataset with its identity, unshuffled. Documented in dew.data.sources.hf.
from_iterThe records factory() iterates as a Dew dataset, its content version identity where given, resumed through the iterator’s own state and load_state where it has them, or by replay where it declares replay (Iterated). Documented in dew.data.sources.iterator.
from_torchA map-style torch.utils.data.Dataset as a Dew dataset, its samples named by fields where they are tuples. Documented in dew.data.sources.pytorch.
sourceThe store value names: a Source as it is; a tfds://name[:version] or hf://owner/repo[@revision] URL as its TFDS or Hub dataset, split its split (“train” by default); any other URL as its files (Files, format naming their record format where the URL’s suffix does not), or as a committed token export (TokenFiles) where its manifest says it is one; and records in memory (a sequence of rows, a mapping of columns or a data frame) as Memory.

dataclass source

class Concatenate(sources: tuple[Source, ...])

The records of sources one store after another, as one store: two splits of one Hub dataset read as one (Concatenate((hub.with_split("train"), hub.with_split("validation")))), or stores of any kinds. A record’s position is its place in that order, every source’s shards are its shards, and its identity is its sources’ in order, so a run resumes on it as on any store and a changed source refuses the resume as it would alone. It is a store wherever one is read; concat joins stores as a dataset.

def label() -> str
def describe() -> str
def consumed(read: Read) -> Concatenate
def fingerprint() -> Mapping[str, object]
def supports(read: Read) -> bool
def records(read: Read) -> Records
def metadata() -> Mapping[str, object]

What every source states of its records, as one store’s: the counts (COUNTED) added where each source states one, the tokenizer contract where the sources that state it agree (tokenizer_contract, which refuses two), the rules any part’s rows were counted by where every part states its own (count_rules), and any other fact, class names among them, where every source states the one value.

def stated() -> Mapping[str, object]

What every part states of its records without reading them, merged as metadata is.

def lengths() -> np.ndarray | None
def manifest() -> ShardManifest

Each store’s shards, one store after another: their index order where each store’s is, as records joins them.

dataclass source

class TokenFiles(
url: str,
storage_options: Mapping[str, object] | None = None,
field: str | None = None,
)

A committed token export (dew.data.text.export_tokens, dew data export --tokenizer): its documents of token ids, the metadata its manifest states and where its documents end, each a record its producer wrote, as its length index states it (document_ends). An export holds one split; validation tokens are an export of their own.

committed: Mapping[str, object]

The export’s committed manifest, read through this store’s own storage options.

def label() -> str
def describe() -> str
def stated() -> Mapping[str, object]

What the store’s manifest says of its tokens, none of them read (Dataset.facts): their tokenizer, vocabulary and EOS, as token ids under “text”; an export not here states nothing.

def metadata() -> Mapping[str, object]
def fingerprint() -> Mapping[str, object]
def read_alike() -> bool

Whether a Dew before row 3028, which read a token store’s documents by its EOS, read this store’s as its producer’s records: stating no EOS, one record holding every token; stating one, each record ended by that EOS and holding no other (_earlier_ends), a scan of every token of the field read, kept in Dew’s cache under the export’s content (export_content), which no write changes, and that field. Read only to resume a plan that Dew saved (dew.data.plan.earlier_plan). Carried to 1.0 (ledger 1552).

def supports(read: Read) -> bool
def can_provision(read: Read) -> bool
def provision() -> TokenFiles

Nothing to write: the producer’s length index, which every export carries, states where each document ends (document_ends).

def records(read: Read) -> Records

Each document by index (document_ends), in every read.

def document_ends() -> np.ndarray

Where each document ends among the export’s tokens: each record its producer wrote, by its length index (TokenRecords.shards), a shard’s after the shard before it, an EOS its tokens spell within it or not, as the documents it exported were (dew.data.text.export_tokens). An empty record holds no document, so none is read of it.

def manifest() -> ShardManifest
def lengths() -> np.ndarray

function source

def source(
value: object,
*,
format: Format | None = None,
split: str | None = None,
provision: bool = False,
) -> Source

The store value names: a Source as it is; a tfds://name[:version] or hf://owner/repo[@revision] URL as its TFDS or Hub dataset, split its split (“train” by default); any other URL as its files (Files, format naming their record format where the URL’s suffix does not), or as a committed token export (TokenFiles) where its manifest says it is one; and records in memory (a sequence of rows, a mapping of columns or a data frame) as Memory. Files and records read what they hold, whatever split is named: a split is a Hub or TFDS URL’s. With provision, the store builds the indexes its reads need first.