Skip to content

dew.data.compile

Compile the same pipeline to an elastic order or an exact per-share reader.

The public entry points, compile, iterate, validation, publish and objective_steps, share one preparation (_prepared): the checks, generation promotion, the sources’ capabilities and pinned facts, the tokenizer contract, objective steps and configured steps bound to metadata. A memory read then runs its eager stages once (dew.data.materialize), the analysis (dew.data.analysis) settles how the order is cut, and the Grain lowering (dew.data.lowering) builds the readers. compile saves with a training read the plan a resume checks: the dependency manifest, the canonical order and the derived order facts.

NameSummary
compileCheck every leaf before opening readers, then build both branches; widths are the run’s static widths its steps bind to (run_widths), and measured its training branch as their measurement prepared it (_training_measure), which it binds as it stands.
objective_stepsPut objective record steps on each corpus before its mixture, where the mixture is batched as it is; after a mixture’s own steps (a preset’s tokenize and pack over raw stores), on the mixture.
iterateA finite record pass through the same lowering, for publication and export.
HandedData handed to fit as it is, which a run steps on as given (native): a Grain dataset’s elements, or arrays whole.
nativeA reader of the data handed to fit as it is, each element a batch the run steps on, never batched again: a Grain dataset’s elements (batched by Grain, MapDataset.batch), a map dataset’s read endlessly where it trains, or a mapping of arrays read whole, every step the same batch (a full-batch run; a run of minibatches states them, Memory(arrays).batch(size)).
native_loaderThe data handed to fit as it is (native) as the loader the run reads, its global batch its first element’s rows.
run_widthsA run’s static widths: the facts its objective’s run-scoped steps read (Configured.scope, such as a decision’s slots), measured over its training chain and every held one, each prepared in its own role with the objective’s steps in their places and its source’s own steps run where a memory read runs them (_measure), and merged as a run reads them together (merge_facts: slots as their maxima).
fitteddataset and each validation dataset as Trainer.fit reads them, compiled with the objective’s own stages and the run’s data seed, their stores provisioned, or training data handed as it is read as handed (native; dew.training._fit.handed_splits reads validation so).
helddataset read once to validate or evaluate on, in its own batches, else in batches of batch, read and loaded as it asks, its random steps drawing from seed; one that states neither batch is refused.
validationCompile one finite evaluation branch without constructing a training stream.
publishFill and seal a deterministic prefix, then publish its certified generation, unless one is already published under its key (with the plan of a terminal pack), which it returns.

function source

def compile(
train: Pipeline,
*,
validation: Pipeline | None = None,
host: tuple[HostStep, ...] = (),
objective: ObjectiveData | None = None,
read: Read | None = None,
provision: bool = False,
loading: Loading | None = None,
seed: int = 0,
rejections: Literal['drop', 'mask'] = 'drop',
resume: Literal['exact', 'non-exact'] = 'exact',
examples: int | None = None,
widths: Mapping[str, object] | None = None,
measured: _Measured | None = None,
epochs: int | None = None,
) -> Loader

Check every leaf before opening readers, then build both branches; widths are the run’s static widths its steps bind to (run_widths), and measured its training branch as their measurement prepared it (_training_measure), which it binds as it stands. With no read, the sources choose it (_preferred_read). examples bounds the training read to its first that many source examples, before they are packed and before the objective makes rows of them (_bounded), a run’s examples= budget. epochs reads the chain endlessly, or, where its read has no record count, bounds it to that many passes over the source examples it states a pass holds once it is prepared (_staged, Loader.pass_examples).

function source

def objective_steps(
pipeline: Pipeline,
objective: ObjectiveData,
*,
train: bool,
read: Read,
) -> Pipeline

Put objective record steps on each corpus before its mixture, where the mixture is batched as it is; after a mixture’s own steps (a preset’s tokenize and pack over raw stores), on the mixture.

function source

def iterate(
pipeline: Pipeline,
*,
read: Read = 'streamed',
provision: bool = False,
loading: Loading = _DEFAULT_LOADING,
) -> pygrain.IterDataset

A finite record pass through the same lowering, for publication and export.

attribute source

Data handed to fit as it is, which a run steps on as given (native): a Grain dataset’s elements, or arrays whole.

function source

def native(handed: Handed, *, training: bool) -> Reader

A reader of the data handed to fit as it is, each element a batch the run steps on, never batched again: a Grain dataset’s elements (batched by Grain, MapDataset.batch), a map dataset’s read endlessly where it trains, or a mapping of arrays read whole, every step the same batch (a full-batch run; a run of minibatches states them, Memory(arrays).batch(size)).

function source

def native_loader(handed: Handed) -> Loader

The data handed to fit as it is (native) as the loader the run reads, its global batch its first element’s rows.

function source

def run_widths(
measured: Sequence[_Measured],
objective: ObjectiveData,
) -> Mapping[str, object]

A run’s static widths: the facts its objective’s run-scoped steps read (Configured.scope, such as a decision’s slots), measured over its training chain and every held one, each prepared in its own role with the objective’s steps in their places and its source’s own steps run where a memory read runs them (_measure), and merged as a run reads them together (merge_facts: slots as their maxima). Training and every validation split bind them alike (fitted), so every split reads in training’s shapes, one program for them all, where a split once took its own. None are measured where the objective’s steps read none.

function source

def fitted(
dataset: Dataset | Loader | Handed,
validation: Dataset | Mapping[str, Dataset | Entry] | None,
*,
objective: ObjectiveData,
seed: int = 0,
examples: int | None = None,
epochs: int | None = None,
) -> tuple[Loader, Mapping[str, Reader | Entry] | None]

dataset and each validation dataset as Trainer.fit reads them, compiled with the objective’s own stages and the run’s data seed, their stores provisioned, or training data handed as it is read as handed (native; dew.training._fit.handed_splits reads validation so). A training dataset without a repeat repeats before its batch, so a run’s length is its steps, epochs or tokens, or with examples reads that many records before they are packed and ends (compile’s examples); a validation dataset without a batch takes the training batch, and is read and loaded as it asks. A Loader is already compiled, and an entry that is no Dataset (a reader, an evaluation suite) is passed on as it is. A run of epochs over a chain whose read has no record count, but states the source examples a pass holds (Loader.pass_examples), reads it bounded to that many passes, as examples bounds it (Loader.epochs), in the one compile (_staged).

function source

def held(
dataset: Dataset,
*,
objective: ObjectiveData | None = None,
batch: int | None = None,
seed: int = 0,
widths: Mapping[str, object] | None = None,
measured: _Measured | None = None,
) -> Reader

dataset read once to validate or evaluate on, in its own batches, else in batches of batch, read and loaded as it asks, its random steps drawing from seed; one that states neither batch is refused. A pass reads every record: a batch it fills only in part keeps the records it holds, filled out with copies its VALID_ROWS marks, where a training read would drop them. measured is the chain as its run’s widths measured it (_held_measure), which it reads as it stands.

function source

def validation(
pipeline: Pipeline,
*,
batch: int,
objective: ObjectiveData | None = None,
host: tuple[HostStep, ...] = (),
read: Read | None = 'indexed',
provision: bool = False,
loading: Loading = _DEFAULT_LOADING,
seed: int = 0,
widths: Mapping[str, object] | None = None,
measured: _Measured | None = None,
) -> Reader

Compile one finite evaluation branch without constructing a training stream. With no read, the sources choose it (_preferred_read); widths are the run’s static widths its steps bind to (run_widths), and measured the branch as they measured it, batched.

function source

def publish(
pipeline: Pipeline,
cache,
*,
read: Read = 'streamed',
provision: bool = False,
loading: Loading = _DEFAULT_LOADING,
)

Fill and seal a deterministic prefix, then publish its certified generation, unless one is already published under its key (with the plan of a terminal pack), which it returns.