dew.data.plan
Execution identities and the order a saved record count names.
| Name | Summary |
|---|---|
REBUILD | |
RECORD | |
SCHEDULE | The routes a run whose chain changed since it saved its position takes (PlanOrderChanged.routes): built as it was; resumed from a record that holds its chain (RunConfig.train names it where the saved run’s record does); a decision mixture saved before its corpus order was seeded read in that schedule (rows 3870, 3881). |
PlanOrderChanged | A saved data position’s plan whose order is not the chain’s now: each order key changed, its saved value and its value now (None where one holds none), and the routes its run goes on by (REBUILD, RECORD, SCHEDULE), a mix saved unseeded and seeded now naming SCHEDULE. |
Digest | A running hash, as hashlib’s are. |
array_digest | The digest an array of dtype and shape is fingerprinted by, its header taken (dew._hashing.array_header, which tells apart dtypes of one byte layout): its elements’ bytes follow in C order, whole (fingerprint) or page by page as a published plan writes them (generation._write_plan), to the same hash. |
fingerprint | Hash order metadata, including an array’s shape and dtype. |
identified | Whether a source snapshot states the content of every store in it: a store that holds no version of its own (a Grain dataset, an iterator, a torch dataset) states it by the name it was given, and one given none can’t say a later run reads the same records. |
callable_name | step’s qualified name, module.qualname: a function’s or a bound method’s own, else its type’s, as a batch encoder names the reader it runs. |
StepIdentity | What a step computes (semantic), and the bytes of each external module its code comes from (modules): its own, then those of the factory that configured it. |
Manifest | The dependency manifest shared by an execution plan and its cache. |
same_order | Whether two orders, as a saved position and a stream state them in JSON, are one order: compared without their sources’ descriptions, which name where a host mounted them, as a pool compares its processes’ orders (Plan.identity). |
unplaced | order without its sources’ descriptions, which name where a host mounted them, its corpora’s and phases’ included. |
earlier_position | A per-share read’s saved position (its data plan and its Grain iterator’s state) as plan, this run’s, reads it: the plan as earlier_plan translates it, and the iterator’s state, where the plan was written before version 8, with the counts version 8 keeps that a Dew of version 7 may not have: of what a read delivered (each corpus’s delivered_rows, which one that counted each row as one example held as accepted), of its cuts by kind (CUT_COUNTS, none), of the examples its filter held pending (consumed, none), and of the examples refused past a held row and those a causal window credited (its carry of _dew_refused, none, beside its _dew_consumed, and credited, none). |
COUNTERS | The format of a per-share position’s counters (dew.data.readers): 2 counts a mixture a run bounds in examples by member where its rows are delivered (row 3312), which a position of an earlier format, holding no counters, does not (_by_member). |
read_otherwise_at_8 | Why version 8 reads a version 7 data plan’s streamed order otherwise, None where it reads it alike: where its read is not streamed, one reader reads it, no take reads its order (no Limit among its stages, its mixed corpora’s orders within it), or its takes are no bound in examples and no loading workers read them, a corpus’s own pool among them (pools). |
earlier_plan | saved, a data plan a saved position records, as plan, this run’s, reads it (Plan.check): the position’s one translation, where a reader’s set_state decodes it. |
grown_limits | The limits the order current sets higher than saved does, each its saved and its current record, where current is saved but for them; None where it differs otherwise. |
Plan | Order-defining fields checked before a restored iterator opens readers. |
REBUILD
Section titled “REBUILD”REBUILD = 'build its chain as it was'RECORD
Section titled “RECORD”RECORD = 'resume it from its record, which holds the chain it read (`dew train <run directory>/run.json`)'SCHEDULE
Section titled “SCHEDULE”SCHEDULE = 'read its mix unseeded, in the schedule it saved (`mix(..., seed=None)`, a decision mixture\'s `mixture(..., order="schedule")`)'The routes a run whose chain changed since it saved its position takes (PlanOrderChanged.routes):
built as it was; resumed from a record that holds its chain (RunConfig.train names it where the saved
run’s record does); a decision mixture saved before its corpus order was seeded read in that schedule
(rows 3870, 3881).
PlanOrderChanged
Section titled “PlanOrderChanged”class PlanOrderChanged( changed: Mapping[str, tuple[JSON | None, JSON | None]], routes: tuple[str, ...],)A saved data position’s plan whose order is not the chain’s now: each order key changed, its saved
value and its value now (None where one holds none), and the routes its run goes on by (REBUILD,
RECORD, SCHEDULE), a mix saved unseeded and seeded now naming SCHEDULE.
PlanOrderChanged.between
Section titled “PlanOrderChanged.between”def between(saved: Mapping[str, object], now: Mapping[str, object]) -> PlanOrderChangedThe refusal of a position saved under order saved, read under now: each top-level key
whose value differs.
PlanOrderChanged.routed
Section titled “PlanOrderChanged.routed”def routed(*, recorded: bool) -> PlanOrderChangedThis refusal with the record route first where the saved run’s record holds its chain.
Digest
Section titled “Digest”class Digest(Protocol)A running hash, as hashlib’s are.
Digest.update
Section titled “Digest.update”def update(chunk: bytes, /) -> NoneDigest.hexdigest
Section titled “Digest.hexdigest”def hexdigest() -> strarray_digest
Section titled “array_digest”def array_digest(dtype: np.dtype, shape: Sequence[int]) -> DigestThe digest an array of dtype and shape is fingerprinted by, its header taken
(dew._hashing.array_header, which tells apart dtypes of one byte layout): its elements’ bytes
follow in C order, whole (fingerprint) or page by page as a published plan writes them
(generation._write_plan), to the same hash.
fingerprint
Section titled “fingerprint”def fingerprint(value: JSON | Mapping[str, object] | np.ndarray) -> strHash order metadata, including an array’s shape and dtype.
identified
Section titled “identified”def identified( value: dict[str, JSON] | list[JSON] | str | int | float | bool | None,) -> boolWhether a source snapshot states the content of every store in it: a store that holds no version of its own (a Grain dataset, an iterator, a torch dataset) states it by the name it was given, and one given none can’t say a later run reads the same records.
callable_name
Section titled “callable_name”def callable_name(step: Callable[..., Batch]) -> strstep’s qualified name, module.qualname: a function’s or a bound method’s own, else its
type’s, as a batch encoder names the reader it runs. A step’s identity (StepIdentity.of) names
a bound method by its type, as plans have always written it; only a function is named by its
own name there.
StepIdentity
Section titled “StepIdentity”class StepIdentity( name: str, config: str, exact: bool, modules: tuple[str, ...] = (), declared: str | None = None, dependencies: Mapping[str, object] = dict(),)What a step computes (semantic), and the bytes of each external
module its code comes from (modules): its own, then those of the
factory that configured it.
StepIdentity.of
Section titled “StepIdentity.of”def of( step: object, identity: str | None = None, *, cache: bool = False,) -> StepIdentitystep’s identity, refused where a cache keys it (cache) and
cannot (keyable).
StepIdentity.keyable
Section titled “StepIdentity.keyable”def keyable(*, function: bool, external: bool) -> boolWhether a cache can key what the step computes: a stable identity, declared wherever the step is a function or its code is outside Dew, which Dew’s revision does not name.
StepIdentity.semantic
Section titled “StepIdentity.semantic”def semantic() -> dict[str, JSON]What the step computes, on any host: its identity without its modules’ bytes, which only a manifest’s environment records.
Manifest
Section titled “Manifest”class Manifest( source: Mapping[str, object], steps: tuple[StepIdentity, ...] = (), ran: tuple[StepIdentity, ...] = (), revision: str = 'unversioned', lock: str = '', python: str = platform.python_version(), platform: str = platform.platform(), tokenizer: Mapping[str, object] = dict(), fill: Mapping[str, object] = dict(), codec: int = 1, schema: int = 1, clean: bool = True,)The dependency manifest shared by an execution plan and its cache.
ran are the steps a memory read already ran into a source it holds.
That source’s snapshot names them by what they compute, so the semantic
record leaves them out and the environment takes their modules. They are
steps of the plan all the same, so the manifest is exact only when they
are as well as its own.
exact: bool-
Whether a later run can tell it makes these rows again: every step has a stable identity and every store states its content (
identified).
Manifest.of
Section titled “Manifest.of”def of( source: Mapping[str, object], steps: Sequence[StepIdentity], *, ran: Sequence[StepIdentity] = (), fill: Mapping[str, object] | None = None,) -> ManifestManifest.record
Section titled “Manifest.record”def record() -> dict[str, JSON]Manifest.semantic
Section titled “Manifest.semantic”def semantic() -> dict[str, JSON]Manifest.environment
Section titled “Manifest.environment”def environment() -> dict[str, JSON]same_order
Section titled “same_order”def same_order(saved: str, current: str) -> boolWhether two orders, as a saved position and a stream state them in JSON, are one order:
compared without their sources’ descriptions, which name where a host mounted them, as a
pool compares its processes’ orders (Plan.identity).
unplaced
Section titled “unplaced”def unplaced( order: dict[str, JSON] | list[JSON] | str | int | float | bool | None,) -> JSONorder without its sources’ descriptions, which name where a host
mounted them, its corpora’s and phases’ included.
earlier_position
Section titled “earlier_position”def earlier_position( saved: Mapping[str, object], plan: Plan,) -> tuple[Mapping[str, object], JSON]A per-share read’s saved position (its data plan and its Grain iterator’s state) as plan, this
run’s, reads it: the plan as earlier_plan translates it, and the iterator’s state, where the plan
was written before version 8, with the counts version 8 keeps that a Dew of version 7 may not have:
of what a read delivered (each corpus’s delivered_rows, which one that counted each row as one
example held as accepted), of its cuts by kind (CUT_COUNTS, none), of the examples its filter
held pending (consumed, none), and of the examples refused past a held row and those a causal
window credited (its carry of _dew_refused, none, beside its _dew_consumed, and credited,
none). The position’s one translation; carried to 1.0 (ledger 1552).
COUNTERS
Section titled “COUNTERS”COUNTERS = 2The format of a per-share position’s counters (dew.data.readers): 2 counts a mixture a run bounds in
examples by member where its rows are delivered (row 3312), which a position of an earlier format, holding
no counters, does not (_by_member).
read_otherwise_at_8
Section titled “read_otherwise_at_8”def read_otherwise_at_8(saved: Mapping[str, object]) -> str | NoneWhy version 8 reads a version 7 data plan’s streamed order otherwise, None where it reads it alike:
where its read is not streamed, one reader reads it, no take reads its order (no Limit among its stages,
its mixed corpora’s orders within it), or its takes are no bound in examples and no loading workers read
them, a corpus’s own pool among them (pools). Version 7 read a take before its loading workers (a
.take) on each of them, and a run’s bound in examples (its manifest’s MarkConsumed step) whole on each
of a pool’s shares and their workers; version 8 takes the one across them and deals the other to them. So
a position 7 wrote of such a read can go on only as 7 read it, which reads the bound once a reader: the
over-read 8 ended.
earlier_plan
Section titled “earlier_plan”def earlier_plan(saved: Mapping[str, object], plan: Plan) -> Mapping[str, object]saved, a data plan a saved position records, as plan, this run’s, reads it (Plan.check): the
position’s one translation, where a reader’s set_state decodes it. A token store a Dew before row 3028
named without saying how its documents are read is named as today’s names it where that Dew read the
same documents of it (_named_as_today); its derived order holds today’s lengths (plan’s) of each
source that Dew stated none for (_unstated_lengths), where the plan names the source as today’s does;
the saved and current manifests’ sources are numbered as their derived facts number them. Carried to
1.0 (ledger 1552).
grown_limits
Section titled “grown_limits”def grown_limits( saved: dict[str, JSON] | list[JSON] | str | int | float | bool | None, current: dict[str, JSON] | list[JSON] | str | int | float | bool | None,) -> frozenset[tuple[str, str]] | NoneThe limits the order current sets higher than saved does, each its saved and its current
record, where current is saved but for them; None where it differs otherwise. Only the run’s own
chain’s limit grows (never a mixed corpus’s or a phase’s, which the mixture’s own stages read after
it), and only where nothing after it but a batch reads the records it holds, as the examples clock’s
bound (dew.data.compile._bounded) of a chain that neither packs nor windows: its first records are
the saved one’s, read further, where a repeat, a shuffle, a split, a pack or windows after it would
read them anew (row 2649).
class Plan( lowering: Literal['global', 'per-share'], manifest: Manifest, order: Mapping[str, object], derived: Mapping[str, str] = dict(), topology: Mapping[str, object] = dict(), rejections: Literal['drop', 'mask'] = 'drop', cache: tuple[str, ...] = (), volatile: bool = False, members: Mapping[str, str] = dict(), stores: tuple[TokenFiles, ...] = (), earlier_members: Mapping[str, tuple[JSON, tuple[TokenFiles, ...]]] = dict(),)Order-defining fields checked before a restored iterator opens readers.
version names how Dew reads a plan into rows, which its fields do not say: a change that
makes other rows or draws from the same plan raises it, and a position saved under another
version is refused.
members: Mapping[str, str]-
A mixture’s members, each what it reads and runs as one digest under its key, and its own stages under the empty key (
dew.data.analysis._member_manifests): what a streamed mix’s resume reads of a member that joined or left (check). stores: tuple[TokenFiles, ...]-
The token stores beneath the plan’s sources, which a plan a Dew before row 3028 saved names otherwise (
earlier_plan); no part of the plan’s record. earlier_members: Mapping[str, tuple[JSON, tuple[TokenFiles, ...]]]-
Each of
members’ manifests, undigested, with the token stores the member reads, from which a resume makes the digests an earlier Dew made of it (earlier_plan); no part of the plan’s record.
Plan.record
Section titled “Plan.record”def record() -> dict[str, JSON]Plan.identity
Section titled “Plan.identity”def identity() -> strWhat every process of a pool reads, which the pool agrees before any
reads: the order, as a reader can follow it, then the rest of the plan
as its parts’ digests (parts), so a pool that disagrees is shown
which part differs. The order’s sources are named by the manifest’s
content, not by their descriptions: hosts that mount one dataset at
different paths read the same records.
Plan.parts
Section titled “Plan.parts”def parts() -> dict[str, JSON]A digest of each part of the plan but its order, its environment and the share this process reads, which only the position it saves keeps: the manifest’s parts each apart, and a list’s entries, such as each source’s content and each step’s identity, one by one.
Plan.check
Section titled “Plan.check”def check( saved: Mapping[str, object], *, resume: Literal['exact', 'non-exact'] = 'exact',) -> tuple[str, ...]Refuse a saved plan whose rows this plan does not make again: another
version, order, derived order fact, per-share topology or semantic
manifest (Manifest.semantic). A changed environment
(Manifest.environment) is returned and warned of, and refused for a
per-share plan.