dew.data.sources.tfds
Prepared TFDS data sources, with explicit preparation and remote mirrors.
| Name | Summary |
|---|---|
METADATA | What a prepared version directory holds beside its shards. |
PREPARE | |
prepared | Whether name was prepared into Dew’s own data_dir, dew_cache_dir()/tfds. |
read_only_builder | The read-only builder over the prepared data at path, checked against what was asked for. |
TFDS | A prepared TFDS split, read with TFDS’s native ArrayRecord data source. |
METADATA
Section titled “METADATA”METADATA = ('dataset_info.json', 'features.json')What a prepared version directory holds beside its shards.
PREPARE
Section titled “PREPARE”PREPAREprepared
Section titled “prepared”def prepared( name: str, *, config: str | None = None, version: str | None = None,) -> boolWhether name was prepared into Dew’s own data_dir, dew_cache_dir()/tfds.
A missing name is prepared there by download_and_prepare in a
separate process of the same interpreter with no GPU visible, since
TensorFlow imported into a JAX process stays loaded and claims
accelerator memory. Processes on one host take turns under a file lock.
Without TensorFlow, as on Python 3.14, a missing name raises
FileNotFoundError. The directory is not TFDS’s ~/tensorflow_datasets,
where an earlier tfds.load may have prepared the name as TFRecords,
which dew refuses and TFDS would not prepare again.
read_only_builder
Section titled “read_only_builder”def read_only_builder( path: str, *, name: str | None, config: str | None, version: str | None,)The read-only builder over the prepared data at path, checked against
what was asked for.
path is either a version directory, when it holds the metadata, or the
data_dir a preparation run wrote under, where TFDS’s own
builder_from_files finds the builder’s directory from the names given
(the newest prepared version when none is named, the default config when
none is). Without a builder name only the first form resolves, since a
data_dir holds one directory per builder and nothing says which of them
was wanted.
The import is here rather than at module level so import dew.data costs
no TFDS, and so a missing extra names the extra. TFDS reads prepared
ArrayRecords through this builder without importing TensorFlow, which is
why another format is refused rather than converted: a TFRecord split
would pull TensorFlow into the training process.
The metadata is the authority on which dataset this is. Resolving a directory from names and then trusting the names would read whatever happens to sit at that path, so the builder, the config and the version the caller asked for are compared with the ones the prepared data reports.
class TFDS( name: str, split: str = 'train', path: str | None = None, config: str | None = None, version: str | None = None, decoders: DecoderTree | None = None, shard_rows: int = 1024,)A prepared TFDS split, read with TFDS’s native ArrayRecord data source.
TFDS.with_split
Section titled “TFDS.with_split”def with_split(name: str) -> TFDSTFDS.label
Section titled “TFDS.label”def label() -> strTFDS.describe
Section titled “TFDS.describe”def describe() -> strTFDS.fingerprint
Section titled “TFDS.fingerprint”def fingerprint() -> Mapping[str, object]TFDS.supports
Section titled “TFDS.supports”def supports(read: Read) -> boolTFDS.can_provision
Section titled “TFDS.can_provision”def can_provision(read: Read) -> boolTFDS.provision
Section titled “TFDS.provision”def provision() -> TFDSTFDS.records
Section titled “TFDS.records”def records(read: Read) -> RecordsTFDS.manifest
Section titled “TFDS.manifest”def manifest() -> ShardManifestTFDS.stated
Section titled “TFDS.stated”def stated() -> Mapping[str, object]Its builder’s info, no data source opened: the dataset’s name and version, its class names, and the split’s rows the info counts. A split not prepared here, or a remote one not mirrored yet, states nothing: a run’s record and model are built where its data may not be.
TFDS.metadata
Section titled “TFDS.metadata”def metadata() -> Mapping[str, object]TFDS.lengths
Section titled “TFDS.lengths”def lengths()