dew.data.sources.files
Indexed files and checkpointable physical-shard readers over filesystem URLs.
| Name | Summary |
|---|---|
Format | |
FORMATS | |
declared_format | The record format value names, as a committed manifest and a native writer state it (dew.data.sinks.Writer): one of FORMATS, never read by a suffix, which “text”, “webdataset” and “files” do not spell. |
export_place | Where a committed export holds a file, its path without the directory its writer named by a nonce of its own (dew.data.sinks.export: shards/<writer>/..., _dew_indexes/<writer>/...): its partitions and its name stay, so two exports of the same rows place their files alike. |
export_content | A committed export’s manifest with each file where it is placed (export_place), not where its writer put it, so two exports of the same rows hold the same content, which the manifest’s order, places, sizes and hashes say alike. |
Files | Files at URLs, with producer indexes or explicit provisioning for text. |
Format
Section titled “Format”FORMATS
Section titled “FORMATS”FORMATS: tuple[Format, ...] = get_args(Format.__value__)declared_format
Section titled “declared_format”def declared_format(value: str, store: str) -> FormatThe record format value names, as a committed manifest and a native writer state it
(dew.data.sinks.Writer): one of FORMATS, never read by a suffix, which “text”,
“webdataset” and “files” do not spell.
export_place
Section titled “export_place”def export_place(path: str) -> strWhere a committed export holds a file, its path without the directory its writer named by a
nonce of its own (dew.data.sinks.export: shards/<writer>/..., _dew_indexes/<writer>/...):
its partitions and its name stay, so two exports of the same rows place their files alike.
export_content
Section titled “export_content”def export_content(committed: Mapping[str, object]) -> Mapping[str, object]A committed export’s manifest with each file where it is placed (export_place), not where
its writer put it, so two exports of the same rows hold the same content, which the manifest’s
order, places, sizes and hashes say alike.
class Files( url: str | tuple[str, ...], format: Format | None = None, index: str | None = None, columns: tuple[str, ...] | None = None, where: ds.Expression | None = None, partitioning: str | None = 'hive', shard_bytes: int = 64 * 2 ** 20, storage_options: Mapping[str, object] | None = None, spark: bool = False, file_info: Mapping[str, Mapping[str, object]] | None = None, suffixes: tuple[str, ...] | None = None,)Files at URLs, with producer indexes or explicit provisioning for text.
Parquet partitions are discovered with Arrow’s Hive partitioning. A
predicate runs within each streamed fragment; indexed predicates need
a provisioned accepted-row index and are refused here. columns
selects the fields of every format’s rows.
CSV is parsed by Python’s csv module under the process’s field size
limit, 128 KiB a field by default; a source with longer fields reads
once its caller raises csv.field_size_limit.
suffixes keeps the files whose names end in one of them, any case, with
or without the dot, as the store lists them when it is read: a directory’s
media, not the lists and notes beside it (dew.data.video.media).
Files.label
Section titled “Files.label”def label() -> strFiles.describe
Section titled “Files.describe”def describe() -> strFiles.fingerprint
Section titled “Files.fingerprint”def fingerprint() -> Mapping[str, object]Files.supports
Section titled “Files.supports”def supports(read: Read) -> boolFiles.can_provision
Section titled “Files.can_provision”def can_provision(read: Read) -> boolFiles.provision
Section titled “Files.provision”def provision() -> FilesFiles.records
Section titled “Files.records”def records(read: Read) -> RecordsFiles.manifest
Section titled “Files.manifest”def manifest() -> ShardManifestThe files’ shards, each named by its file’s place (_places) and where in it it starts, so a
fill keyed by a shard’s rows finds them wherever the files are mounted.
Files.locality
Section titled “Files.locality”def locality() -> tuple[tuple[int, ...], bool]Files.order_fingerprints
Section titled “Files.order_fingerprints”def order_fingerprints() -> Mapping[str, str]Files.metadata
Section titled “Files.metadata”def metadata() -> Mapping[str, object]Files.stated
Section titled “Files.stated”def stated() -> Mapping[str, object]What its committed manifest says, or a parquet or Arrow store’s schema names (class names), no record read; files not here state nothing.
Files.lengths
Section titled “Files.lengths”def lengths() -> np.ndarray | None