dew.data.compile
Compile the same pipeline to an elastic order or an exact per-share reader.
The public entry points, compile, iterate, validation, publish and
objective_steps, share one preparation (_prepared): the checks,
generation promotion, the sources’ capabilities and pinned facts, the
tokenizer contract, objective steps and configured steps bound to metadata.
A memory read then runs its eager stages once (dew.data.materialize), the
analysis (dew.data.analysis) settles how the order is cut, and the Grain
lowering (dew.data.lowering) builds the readers. compile saves with a
training read the plan a resume checks: the dependency manifest, the
canonical order and the derived order facts.
| Name | Summary |
|---|---|
compile | Check every leaf before opening readers, then build both branches; widths are the run’s static widths its steps bind to (run_widths), and measured its training branch as their measurement prepared it (_training_measure), which it binds as it stands. |
objective_steps | Put objective record steps on each corpus before its mixture, where the mixture is batched as it is; after a mixture’s own steps (a preset’s tokenize and pack over raw stores), on the mixture. |
iterate | A finite record pass through the same lowering, for publication and export. |
Handed | Data handed to fit as it is, which a run steps on as given (native): a Grain dataset’s elements, or arrays whole. |
native | A reader of the data handed to fit as it is, each element a batch the run steps on, never batched again: a Grain dataset’s elements (batched by Grain, MapDataset.batch), a map dataset’s read endlessly where it trains, or a mapping of arrays read whole, every step the same batch (a full-batch run; a run of minibatches states them, Memory(arrays).batch(size)). |
native_loader | The data handed to fit as it is (native) as the loader the run reads, its global batch its first element’s rows. |
run_widths | A run’s static widths: the facts its objective’s run-scoped steps read (Configured.scope, such as a decision’s slots), measured over its training chain and every held one, each prepared in its own role with the objective’s steps in their places and its source’s own steps run where a memory read runs them (_measure), and merged as a run reads them together (merge_facts: slots as their maxima). |
fitted | dataset and each validation dataset as Trainer.fit reads them, compiled with the objective’s own stages and the run’s data seed, their stores provisioned, or training data handed as it is read as handed (native; dew.training._fit.handed_splits reads validation so). |
held | dataset read once to validate or evaluate on, in its own batches, else in batches of batch, read and loaded as it asks, its random steps drawing from seed; one that states neither batch is refused. |
validation | Compile one finite evaluation branch without constructing a training stream. |
publish | Fill and seal a deterministic prefix, then publish its certified generation, unless one is already published under its key (with the plan of a terminal pack), which it returns. |
compile
Section titled “compile”def compile( train: Pipeline, *, validation: Pipeline | None = None, host: tuple[HostStep, ...] = (), objective: ObjectiveData | None = None, read: Read | None = None, provision: bool = False, loading: Loading | None = None, seed: int = 0, rejections: Literal['drop', 'mask'] = 'drop', resume: Literal['exact', 'non-exact'] = 'exact', examples: int | None = None, widths: Mapping[str, object] | None = None, measured: _Measured | None = None, epochs: int | None = None,) -> LoaderCheck every leaf before opening readers, then build both branches; widths are the run’s
static widths its steps bind to (run_widths), and measured its training branch as their
measurement prepared it (_training_measure), which it binds as it stands.
With no read, the sources choose it (_preferred_read). examples bounds the training read to
its first that many source examples, before they are packed and before the objective makes rows
of them (_bounded), a run’s examples= budget. epochs reads the chain endlessly, or, where its
read has no record count, bounds it to that many passes over the source examples it states a pass
holds once it is prepared (_staged, Loader.pass_examples).
objective_steps
Section titled “objective_steps”def objective_steps( pipeline: Pipeline, objective: ObjectiveData, *, train: bool, read: Read,) -> PipelinePut objective record steps on each corpus before its mixture, where the mixture is batched as it is; after a mixture’s own steps (a preset’s tokenize and pack over raw stores), on the mixture.
iterate
Section titled “iterate”def iterate( pipeline: Pipeline, *, read: Read = 'streamed', provision: bool = False, loading: Loading = _DEFAULT_LOADING,) -> pygrain.IterDatasetA finite record pass through the same lowering, for publication and export.
Handed
Section titled “Handed”Data handed to fit as it is, which a run steps on as given (native): a Grain dataset’s
elements, or arrays whole.
native
Section titled “native”def native(handed: Handed, *, training: bool) -> ReaderA reader of the data handed to fit as it is, each element a batch the run steps on, never
batched again: a Grain dataset’s elements (batched by Grain, MapDataset.batch), a map dataset’s
read endlessly where it trains, or a mapping of arrays read whole, every step the same batch (a
full-batch run; a run of minibatches states them, Memory(arrays).batch(size)).
native_loader
Section titled “native_loader”def native_loader(handed: Handed) -> LoaderThe data handed to fit as it is (native) as the loader the run reads, its global batch
its first element’s rows.
run_widths
Section titled “run_widths”def run_widths( measured: Sequence[_Measured], objective: ObjectiveData,) -> Mapping[str, object]A run’s static widths: the facts its objective’s run-scoped steps read (Configured.scope,
such as a decision’s slots), measured over its training chain and every held one, each prepared
in its own role with the objective’s steps in their places and its source’s own steps run where a
memory read runs them (_measure), and merged as a run reads them together (merge_facts: slots
as their maxima). Training and every validation split bind them alike (fitted), so every split
reads in training’s shapes, one program for them all, where a split once took its own. None are
measured where the objective’s steps read none.
fitted
Section titled “fitted”def fitted( dataset: Dataset | Loader | Handed, validation: Dataset | Mapping[str, Dataset | Entry] | None, *, objective: ObjectiveData, seed: int = 0, examples: int | None = None, epochs: int | None = None,) -> tuple[Loader, Mapping[str, Reader | Entry] | None]dataset and each validation dataset as Trainer.fit reads them,
compiled with the objective’s own stages and the run’s data seed,
their stores provisioned, or training data handed as it is read as
handed (native; dew.training._fit.handed_splits reads validation so).
A training dataset without a repeat repeats before its batch, so a run’s
length is its steps, epochs or tokens, or with examples reads that many
records before they are packed and ends (compile’s examples); a
validation dataset without a batch takes the
training batch, and is read and loaded as it asks. A Loader is already
compiled, and an entry that is no Dataset (a reader, an evaluation suite)
is passed on as it is. A run of epochs over a chain whose read has no
record count, but states the source examples a pass holds
(Loader.pass_examples), reads it bounded to that many passes, as
examples bounds it (Loader.epochs), in the one compile (_staged).
def held( dataset: Dataset, *, objective: ObjectiveData | None = None, batch: int | None = None, seed: int = 0, widths: Mapping[str, object] | None = None, measured: _Measured | None = None,) -> Readerdataset read once to validate or evaluate on, in its own batches, else in batches of batch,
read and loaded as it asks, its random steps drawing from seed; one that states neither batch is
refused. A pass reads every record: a batch it fills only in part keeps the records it holds,
filled out with copies its VALID_ROWS marks, where a training read would drop them. measured
is the chain as its run’s widths measured it (_held_measure), which it reads as it stands.
validation
Section titled “validation”def validation( pipeline: Pipeline, *, batch: int, objective: ObjectiveData | None = None, host: tuple[HostStep, ...] = (), read: Read | None = 'indexed', provision: bool = False, loading: Loading = _DEFAULT_LOADING, seed: int = 0, widths: Mapping[str, object] | None = None, measured: _Measured | None = None,) -> ReaderCompile one finite evaluation branch without constructing a training
stream. With no read, the sources choose it (_preferred_read); widths are the run’s static
widths its steps bind to (run_widths), and measured the branch as they measured it, batched.
publish
Section titled “publish”def publish( pipeline: Pipeline, cache, *, read: Read = 'streamed', provision: bool = False, loading: Loading = _DEFAULT_LOADING,)Fill and seal a deterministic prefix, then publish its certified generation, unless one is already published under its key (with the plan of a terminal pack), which it returns.