Skip to content

dew.data.cache

Write-through artifacts with per-input completions and conditional publication.

NameSummary
AUTOMATICThe environment variable that names the directory a process’s runs cache their chains’ leading deterministic steps in (automatic_root), or turns that off (off).
automatic_rootWhere a run caches what its chains’ leading deterministic steps make of their records, with no cache= (dew.data.compile._automatic): $DEW_DATA_CACHE, else data in Dew’s cache directory (dew.cache.dew_cache_dir), a local root holding at most Cache.keep_local_bytes, its oldest generations evicted first (LocalRetention).
AbandonedAn upload or a seal the cache’s writer was closed before, which it counts (abandoned).
Transform
CachedValue
CacheStats
CacheCache a deterministic prefix without changing a filling run’s execution plan.

attribute source

AUTOMATIC = 'DEW_DATA_CACHE'

The environment variable that names the directory a process’s runs cache their chains’ leading deterministic steps in (automatic_root), or turns that off (off).

function source

def automatic_root(loading: Loading) -> str | None

Where a run caches what its chains’ leading deterministic steps make of their records, with no cache= (dew.data.compile._automatic): $DEW_DATA_CACHE, else data in Dew’s cache directory (dew.cache.dew_cache_dir), a local root holding at most Cache.keep_local_bytes, its oldest generations evicted first (LocalRetention). None where the run’s loading opts out (Loading(cache=False)) or the process does (DEW_DATA_CACHE=off).

class source

class Abandoned(RuntimeError)

An upload or a seal the cache’s writer was closed before, which it counts (abandoned).

attribute source

attribute source

class source

class CacheStats(TypedDict)

dataclass source

class Cache(
url: str,
spool_bytes: int | str = 256 * 2 ** 20,
mirror_bytes: int | str = 8 * 2 ** 30,
keep_local_bytes: int | str = 8 * 2 ** 30,
on_full: Literal['stop', 'block'] = 'stop',
upload_workers: int = 2,
retry: int = 3,
shard_bytes: int | str = 16 * 2 ** 20,
storage_options: Mapping[str, object] | None = None,
optional: bool = False,
)

Cache a deterministic prefix without changing a filling run’s execution plan.

A fill whose key a generation is published under (CURRENT) reads that generation’s completions in place of running its steps, so a later run reuses an earlier one’s complete fill. Otherwise each process keeps its own writes; it never discovers another filling writer’s rows, and a writer that completes every input of the source it is given publishes that source’s generation. Volatile prefixes run on every read and produce no completions. A shard a remote store took but reads back otherwise is read again up to retry more times (Store.put_immutable); a failed transfer raises after its transport’s own retries, and a write to this host’s disk or memory fails at once.

optional: bool

Whether a run reads on without it where its storage fails (no room, no permission, its index’s database full or read-only): a cache the compiler placed (dew.data.compile._automatic), which a run never needs, then looks nothing up, stops admitting and its records are computed, as they are with no cache (read_on). A cache a chain names stops the run.

def read_on(error: OSError | sqlite3.Error | ValueError | TypeError, doing: str) -> bool

Whether this cache reads on uncached after its storage raised error while doing: it is optional, and logs that it does.

def reference(manifest: DependencyManifest) -> str

The cache a plan reads, by the key of the rows it serves: where the cache lives is no part of the data, so a run resumes against the same rows cached anywhere.

def mapped(
parent: grain.MapDataset,
key: DependencyManifest,
*,
transform: Transform | None = None,
source: Source | None = None,
missing: Sequence[int] = (),
) -> _Mapped

The cached rows of parent, a source’s records by index: a published generation’s, or filled as they are read. A fill that completes every input publishes the generation of source, the source they are. The inputs missing that a read skips (a store’s absent records) are written as a fill starts, so it can complete without reading them.

def streamed(
reader: Callable[[Shard], grain.IterDataset],
key: DependencyManifest,
*,
transform: Transform | None = None,
threads: int | None = None,
) -> Callable[[Shard], grain.IterDataset]
def generation(key: DependencyManifest) -> Generation | None
def publish(
key: DependencyManifest,
source: Source,
*,
plan: Callable[[Generation], Mapping[str, np.ndarray]] | None = None,
) -> Generation

The generation of source under key, once its sealed fills cover every input: the one published already, unless plan adds a packing plan, which a new generation carries.