Skip to content

dew.data.sources.files

Indexed files and checkpointable physical-shard readers over filesystem URLs.

NameSummary
Format
FORMATS
declared_formatThe record format value names, as a committed manifest and a native writer state it (dew.data.sinks.Writer): one of FORMATS, never read by a suffix, which “text”, “webdataset” and “files” do not spell.
export_placeWhere a committed export holds a file, its path without the directory its writer named by a nonce of its own (dew.data.sinks.export: shards/<writer>/..., _dew_indexes/<writer>/...): its partitions and its name stay, so two exports of the same rows place their files alike.
export_contentA committed export’s manifest with each file where it is placed (export_place), not where its writer put it, so two exports of the same rows hold the same content, which the manifest’s order, places, sizes and hashes say alike.
FilesFiles at URLs, with producer indexes or explicit provisioning for text.

attribute source

attribute source

FORMATS: tuple[Format, ...] = get_args(Format.__value__)

function source

def declared_format(value: str, store: str) -> Format

The record format value names, as a committed manifest and a native writer state it (dew.data.sinks.Writer): one of FORMATS, never read by a suffix, which “text”, “webdataset” and “files” do not spell.

function source

def export_place(path: str) -> str

Where a committed export holds a file, its path without the directory its writer named by a nonce of its own (dew.data.sinks.export: shards/<writer>/..., _dew_indexes/<writer>/...): its partitions and its name stay, so two exports of the same rows place their files alike.

function source

def export_content(committed: Mapping[str, object]) -> Mapping[str, object]

A committed export’s manifest with each file where it is placed (export_place), not where its writer put it, so two exports of the same rows hold the same content, which the manifest’s order, places, sizes and hashes say alike.

dataclass source

class Files(
url: str | tuple[str, ...],
format: Format | None = None,
index: str | None = None,
columns: tuple[str, ...] | None = None,
where: ds.Expression | None = None,
partitioning: str | None = 'hive',
shard_bytes: int = 64 * 2 ** 20,
storage_options: Mapping[str, object] | None = None,
spark: bool = False,
file_info: Mapping[str, Mapping[str, object]] | None = None,
suffixes: tuple[str, ...] | None = None,
)

Files at URLs, with producer indexes or explicit provisioning for text.

Parquet partitions are discovered with Arrow’s Hive partitioning. A predicate runs within each streamed fragment; indexed predicates need a provisioned accepted-row index and are refused here. columns selects the fields of every format’s rows.

CSV is parsed by Python’s csv module under the process’s field size limit, 128 KiB a field by default; a source with longer fields reads once its caller raises csv.field_size_limit.

suffixes keeps the files whose names end in one of them, any case, with or without the dot, as the store lists them when it is read: a directory’s media, not the lists and notes beside it (dew.data.video.media).

def label() -> str
def describe() -> str
def fingerprint() -> Mapping[str, object]
def supports(read: Read) -> bool
def can_provision(read: Read) -> bool
def provision() -> Files
def records(read: Read) -> Records
def manifest() -> ShardManifest

The files’ shards, each named by its file’s place (_places) and where in it it starts, so a fill keyed by a shard’s rows finds them wherever the files are mounted.

def locality() -> tuple[tuple[int, ...], bool]
def order_fingerprints() -> Mapping[str, str]
def metadata() -> Mapping[str, object]
def stated() -> Mapping[str, object]

What its committed manifest says, or a parquet or Arrow store’s schema names (class names), no record read; files not here state nothing.

def lengths() -> np.ndarray | None