Skip to content

dew.data.sources.tfds

Prepared TFDS data sources, with explicit preparation and remote mirrors.

NameSummary
METADATAWhat a prepared version directory holds beside its shards.
PREPARE
preparedWhether name was prepared into Dew’s own data_dir, dew_cache_dir()/tfds.
read_only_builderThe read-only builder over the prepared data at path, checked against what was asked for.
TFDSA prepared TFDS split, read with TFDS’s native ArrayRecord data source.

attribute source

METADATA = ('dataset_info.json', 'features.json')

What a prepared version directory holds beside its shards.

attribute source

PREPARE

function source

def prepared(
name: str,
*,
config: str | None = None,
version: str | None = None,
) -> bool

Whether name was prepared into Dew’s own data_dir, dew_cache_dir()/tfds.

A missing name is prepared there by download_and_prepare in a separate process of the same interpreter with no GPU visible, since TensorFlow imported into a JAX process stays loaded and claims accelerator memory. Processes on one host take turns under a file lock. Without TensorFlow, as on Python 3.14, a missing name raises FileNotFoundError. The directory is not TFDS’s ~/tensorflow_datasets, where an earlier tfds.load may have prepared the name as TFRecords, which dew refuses and TFDS would not prepare again.

function source

def read_only_builder(
path: str,
*,
name: str | None,
config: str | None,
version: str | None,
)

The read-only builder over the prepared data at path, checked against what was asked for.

path is either a version directory, when it holds the metadata, or the data_dir a preparation run wrote under, where TFDS’s own builder_from_files finds the builder’s directory from the names given (the newest prepared version when none is named, the default config when none is). Without a builder name only the first form resolves, since a data_dir holds one directory per builder and nothing says which of them was wanted.

The import is here rather than at module level so import dew.data costs no TFDS, and so a missing extra names the extra. TFDS reads prepared ArrayRecords through this builder without importing TensorFlow, which is why another format is refused rather than converted: a TFRecord split would pull TensorFlow into the training process.

The metadata is the authority on which dataset this is. Resolving a directory from names and then trusting the names would read whatever happens to sit at that path, so the builder, the config and the version the caller asked for are compared with the ones the prepared data reports.

dataclass source

class TFDS(
name: str,
split: str = 'train',
path: str | None = None,
config: str | None = None,
version: str | None = None,
decoders: DecoderTree | None = None,
shard_rows: int = 1024,
)

A prepared TFDS split, read with TFDS’s native ArrayRecord data source.

def with_split(name: str) -> TFDS
def label() -> str
def describe() -> str
def fingerprint() -> Mapping[str, object]
def supports(read: Read) -> bool
def can_provision(read: Read) -> bool
def provision() -> TFDS
def records(read: Read) -> Records
def manifest() -> ShardManifest
def stated() -> Mapping[str, object]

Its builder’s info, no data source opened: the dataset’s name and version, its class names, and the split’s rows the info counts. A split not prepared here, or a remote one not mirrored yet, states nothing: a run’s record and model are built where its data may not be.

def metadata() -> Mapping[str, object]
def lengths()