Skip to content

dew.data.plan

Execution identities and the order a saved record count names.

NameSummary
REBUILD
RECORD
SCHEDULEThe routes a run whose chain changed since it saved its position takes (PlanOrderChanged.routes): built as it was; resumed from a record that holds its chain (RunConfig.train names it where the saved run’s record does); a decision mixture saved before its corpus order was seeded read in that schedule (rows 3870, 3881).
PlanOrderChangedA saved data position’s plan whose order is not the chain’s now: each order key changed, its saved value and its value now (None where one holds none), and the routes its run goes on by (REBUILD, RECORD, SCHEDULE), a mix saved unseeded and seeded now naming SCHEDULE.
DigestA running hash, as hashlib’s are.
array_digestThe digest an array of dtype and shape is fingerprinted by, its header taken (dew._hashing.array_header, which tells apart dtypes of one byte layout): its elements’ bytes follow in C order, whole (fingerprint) or page by page as a published plan writes them (generation._write_plan), to the same hash.
fingerprintHash order metadata, including an array’s shape and dtype.
identifiedWhether a source snapshot states the content of every store in it: a store that holds no version of its own (a Grain dataset, an iterator, a torch dataset) states it by the name it was given, and one given none can’t say a later run reads the same records.
callable_namestep’s qualified name, module.qualname: a function’s or a bound method’s own, else its type’s, as a batch encoder names the reader it runs.
StepIdentityWhat a step computes (semantic), and the bytes of each external module its code comes from (modules): its own, then those of the factory that configured it.
ManifestThe dependency manifest shared by an execution plan and its cache.
same_orderWhether two orders, as a saved position and a stream state them in JSON, are one order: compared without their sources’ descriptions, which name where a host mounted them, as a pool compares its processes’ orders (Plan.identity).
unplacedorder without its sources’ descriptions, which name where a host mounted them, its corpora’s and phases’ included.
earlier_positionA per-share read’s saved position (its data plan and its Grain iterator’s state) as plan, this run’s, reads it: the plan as earlier_plan translates it, and the iterator’s state, where the plan was written before version 8, with the counts version 8 keeps that a Dew of version 7 may not have: of what a read delivered (each corpus’s delivered_rows, which one that counted each row as one example held as accepted), of its cuts by kind (CUT_COUNTS, none), of the examples its filter held pending (consumed, none), and of the examples refused past a held row and those a causal window credited (its carry of _dew_refused, none, beside its _dew_consumed, and credited, none).
COUNTERSThe format of a per-share position’s counters (dew.data.readers): 2 counts a mixture a run bounds in examples by member where its rows are delivered (row 3312), which a position of an earlier format, holding no counters, does not (_by_member).
read_otherwise_at_8Why version 8 reads a version 7 data plan’s streamed order otherwise, None where it reads it alike: where its read is not streamed, one reader reads it, no take reads its order (no Limit among its stages, its mixed corpora’s orders within it), or its takes are no bound in examples and no loading workers read them, a corpus’s own pool among them (pools).
earlier_plansaved, a data plan a saved position records, as plan, this run’s, reads it (Plan.check): the position’s one translation, where a reader’s set_state decodes it.
grown_limitsThe limits the order current sets higher than saved does, each its saved and its current record, where current is saved but for them; None where it differs otherwise.
PlanOrder-defining fields checked before a restored iterator opens readers.

attribute source

REBUILD = 'build its chain as it was'

attribute source

RECORD = 'resume it from its record, which holds the chain it read (`dew train <run directory>/run.json`)'

attribute source

SCHEDULE = 'read its mix unseeded, in the schedule it saved (`mix(..., seed=None)`, a decision mixture\'s `mixture(..., order="schedule")`)'

The routes a run whose chain changed since it saved its position takes (PlanOrderChanged.routes): built as it was; resumed from a record that holds its chain (RunConfig.train names it where the saved run’s record does); a decision mixture saved before its corpus order was seeded read in that schedule (rows 3870, 3881).

class source

class PlanOrderChanged(
changed: Mapping[str, tuple[JSON | None, JSON | None]],
routes: tuple[str, ...],
)

A saved data position’s plan whose order is not the chain’s now: each order key changed, its saved value and its value now (None where one holds none), and the routes its run goes on by (REBUILD, RECORD, SCHEDULE), a mix saved unseeded and seeded now naming SCHEDULE.

def between(saved: Mapping[str, object], now: Mapping[str, object]) -> PlanOrderChanged

The refusal of a position saved under order saved, read under now: each top-level key whose value differs.

def routed(*, recorded: bool) -> PlanOrderChanged

This refusal with the record route first where the saved run’s record holds its chain.

class source

class Digest(Protocol)

A running hash, as hashlib’s are.

def update(chunk: bytes, /) -> None
def hexdigest() -> str

function source

def array_digest(dtype: np.dtype, shape: Sequence[int]) -> Digest

The digest an array of dtype and shape is fingerprinted by, its header taken (dew._hashing.array_header, which tells apart dtypes of one byte layout): its elements’ bytes follow in C order, whole (fingerprint) or page by page as a published plan writes them (generation._write_plan), to the same hash.

function source

def fingerprint(value: JSON | Mapping[str, object] | np.ndarray) -> str

Hash order metadata, including an array’s shape and dtype.

function source

def identified(
value: dict[str, JSON] | list[JSON] | str | int | float | bool | None,
) -> bool

Whether a source snapshot states the content of every store in it: a store that holds no version of its own (a Grain dataset, an iterator, a torch dataset) states it by the name it was given, and one given none can’t say a later run reads the same records.

function source

def callable_name(step: Callable[..., Batch]) -> str

step’s qualified name, module.qualname: a function’s or a bound method’s own, else its type’s, as a batch encoder names the reader it runs. A step’s identity (StepIdentity.of) names a bound method by its type, as plans have always written it; only a function is named by its own name there.

dataclass source

class StepIdentity(
name: str,
config: str,
exact: bool,
modules: tuple[str, ...] = (),
declared: str | None = None,
dependencies: Mapping[str, object] = dict(),
)

What a step computes (semantic), and the bytes of each external module its code comes from (modules): its own, then those of the factory that configured it.

def of(
step: object,
identity: str | None = None,
*,
cache: bool = False,
) -> StepIdentity

step’s identity, refused where a cache keys it (cache) and cannot (keyable).

def keyable(*, function: bool, external: bool) -> bool

Whether a cache can key what the step computes: a stable identity, declared wherever the step is a function or its code is outside Dew, which Dew’s revision does not name.

def semantic() -> dict[str, JSON]

What the step computes, on any host: its identity without its modules’ bytes, which only a manifest’s environment records.

dataclass source

class Manifest(
source: Mapping[str, object],
steps: tuple[StepIdentity, ...] = (),
ran: tuple[StepIdentity, ...] = (),
revision: str = 'unversioned',
lock: str = '',
python: str = platform.python_version(),
platform: str = platform.platform(),
tokenizer: Mapping[str, object] = dict(),
fill: Mapping[str, object] = dict(),
codec: int = 1,
schema: int = 1,
clean: bool = True,
)

The dependency manifest shared by an execution plan and its cache.

ran are the steps a memory read already ran into a source it holds. That source’s snapshot names them by what they compute, so the semantic record leaves them out and the environment takes their modules. They are steps of the plan all the same, so the manifest is exact only when they are as well as its own.

exact: bool

Whether a later run can tell it makes these rows again: every step has a stable identity and every store states its content (identified).

def of(
source: Mapping[str, object],
steps: Sequence[StepIdentity],
*,
ran: Sequence[StepIdentity] = (),
fill: Mapping[str, object] | None = None,
) -> Manifest
def record() -> dict[str, JSON]
def semantic() -> dict[str, JSON]
def environment() -> dict[str, JSON]

function source

def same_order(saved: str, current: str) -> bool

Whether two orders, as a saved position and a stream state them in JSON, are one order: compared without their sources’ descriptions, which name where a host mounted them, as a pool compares its processes’ orders (Plan.identity).

function source

def unplaced(
order: dict[str, JSON] | list[JSON] | str | int | float | bool | None,
) -> JSON

order without its sources’ descriptions, which name where a host mounted them, its corpora’s and phases’ included.

function source

def earlier_position(
saved: Mapping[str, object],
plan: Plan,
) -> tuple[Mapping[str, object], JSON]

A per-share read’s saved position (its data plan and its Grain iterator’s state) as plan, this run’s, reads it: the plan as earlier_plan translates it, and the iterator’s state, where the plan was written before version 8, with the counts version 8 keeps that a Dew of version 7 may not have: of what a read delivered (each corpus’s delivered_rows, which one that counted each row as one example held as accepted), of its cuts by kind (CUT_COUNTS, none), of the examples its filter held pending (consumed, none), and of the examples refused past a held row and those a causal window credited (its carry of _dew_refused, none, beside its _dew_consumed, and credited, none). The position’s one translation; carried to 1.0 (ledger 1552).

attribute source

COUNTERS = 2

The format of a per-share position’s counters (dew.data.readers): 2 counts a mixture a run bounds in examples by member where its rows are delivered (row 3312), which a position of an earlier format, holding no counters, does not (_by_member).

function source

def read_otherwise_at_8(saved: Mapping[str, object]) -> str | None

Why version 8 reads a version 7 data plan’s streamed order otherwise, None where it reads it alike: where its read is not streamed, one reader reads it, no take reads its order (no Limit among its stages, its mixed corpora’s orders within it), or its takes are no bound in examples and no loading workers read them, a corpus’s own pool among them (pools). Version 7 read a take before its loading workers (a .take) on each of them, and a run’s bound in examples (its manifest’s MarkConsumed step) whole on each of a pool’s shares and their workers; version 8 takes the one across them and deals the other to them. So a position 7 wrote of such a read can go on only as 7 read it, which reads the bound once a reader: the over-read 8 ended.

function source

def earlier_plan(saved: Mapping[str, object], plan: Plan) -> Mapping[str, object]

saved, a data plan a saved position records, as plan, this run’s, reads it (Plan.check): the position’s one translation, where a reader’s set_state decodes it. A token store a Dew before row 3028 named without saying how its documents are read is named as today’s names it where that Dew read the same documents of it (_named_as_today); its derived order holds today’s lengths (plan’s) of each source that Dew stated none for (_unstated_lengths), where the plan names the source as today’s does; the saved and current manifests’ sources are numbered as their derived facts number them. Carried to 1.0 (ledger 1552).

function source

def grown_limits(
saved: dict[str, JSON] | list[JSON] | str | int | float | bool | None,
current: dict[str, JSON] | list[JSON] | str | int | float | bool | None,
) -> frozenset[tuple[str, str]] | None

The limits the order current sets higher than saved does, each its saved and its current record, where current is saved but for them; None where it differs otherwise. Only the run’s own chain’s limit grows (never a mixed corpus’s or a phase’s, which the mixture’s own stages read after it), and only where nothing after it but a batch reads the records it holds, as the examples clock’s bound (dew.data.compile._bounded) of a chain that neither packs nor windows: its first records are the saved one’s, read further, where a repeat, a shuffle, a split, a pack or windows after it would read them anew (row 2649).

dataclass source

class Plan(
lowering: Literal['global', 'per-share'],
manifest: Manifest,
order: Mapping[str, object],
derived: Mapping[str, str] = dict(),
topology: Mapping[str, object] = dict(),
rejections: Literal['drop', 'mask'] = 'drop',
cache: tuple[str, ...] = (),
volatile: bool = False,
members: Mapping[str, str] = dict(),
stores: tuple[TokenFiles, ...] = (),
earlier_members: Mapping[str, tuple[JSON, tuple[TokenFiles, ...]]] = dict(),
)

Order-defining fields checked before a restored iterator opens readers.

version names how Dew reads a plan into rows, which its fields do not say: a change that makes other rows or draws from the same plan raises it, and a position saved under another version is refused.

members: Mapping[str, str]

A mixture’s members, each what it reads and runs as one digest under its key, and its own stages under the empty key (dew.data.analysis._member_manifests): what a streamed mix’s resume reads of a member that joined or left (check).

stores: tuple[TokenFiles, ...]

The token stores beneath the plan’s sources, which a plan a Dew before row 3028 saved names otherwise (earlier_plan); no part of the plan’s record.

earlier_members: Mapping[str, tuple[JSON, tuple[TokenFiles, ...]]]

Each of members’ manifests, undigested, with the token stores the member reads, from which a resume makes the digests an earlier Dew made of it (earlier_plan); no part of the plan’s record.

def record() -> dict[str, JSON]
def identity() -> str

What every process of a pool reads, which the pool agrees before any reads: the order, as a reader can follow it, then the rest of the plan as its parts’ digests (parts), so a pool that disagrees is shown which part differs. The order’s sources are named by the manifest’s content, not by their descriptions: hosts that mount one dataset at different paths read the same records.

def parts() -> dict[str, JSON]

A digest of each part of the plan but its order, its environment and the share this process reads, which only the position it saves keeps: the manifest’s parts each apart, and a list’s entries, such as each source’s content and each step’s identity, one by one.

def check(
saved: Mapping[str, object],
*,
resume: Literal['exact', 'non-exact'] = 'exact',
) -> tuple[str, ...]

Refuse a saved plan whose rows this plan does not make again: another version, order, derived order fact, per-share topology or semantic manifest (Manifest.semantic). A changed environment (Manifest.environment) is returned and warned of, and refused for a per-share plan.