Skip to content

dew.data.sources.pytorch

Map-style torch datasets, imported on use and read through the shared pipeline.

NameSummary
TorchRecordsReads a map-style torch dataset by index, each sample as named fields.
TorchA map-style torch dataset, with explicit content identity for plans and artifacts.
from_torchA map-style torch.utils.data.Dataset as a Dew dataset, its samples named by fields where they are tuples.

class source

class TorchRecords(dataset: TorchDataset, names: Sequence[str] | None)

Reads a map-style torch dataset by index, each sample as named fields.

A dict sample keeps its keys, and names names the values of a tuple sample, or the one value of any other. Tensors come to the host as numpy, and every value is a provider row’s field (fields). The repr names the dataset as a saved position needs (describe).

dataclass source

class Torch(
dataset: TorchDataset,
fields: tuple[str, ...] | None = None,
identity: str | None = None,
known: Mapping[str, object] | None = None,
)

A map-style torch dataset, with explicit content identity for plans and artifacts.

known: Mapping[str, object] | None

Facts of its records the caller states, which its metadata and Dataset.facts carry.

def stated() -> Mapping[str, object]
def label() -> str
def describe() -> str
def fingerprint() -> Mapping[str, object]
def supports(read: Read) -> bool
def records(read: Read) -> Records
def manifest() -> ShardManifest

The map-style records as a stream of consecutive index ranges (_indexed_manifest), as a memory source streams.

def metadata() -> Mapping[str, object]
def lengths() -> np.ndarray | None

function source

def from_torch(
dataset: TorchDataset,
fields: tuple[str, ...] | None = None,
*,
identity: str | None = None,
) -> Torch

A map-style torch.utils.data.Dataset as a Dew dataset, its samples named by fields where they are tuples. Plans and artifacts require a versioned content identity, including non-exact resumes (Torch).