dew.data.pipeline
Immutable record pipelines, lowered to Grain by :func:compile.
| Name | Summary |
|---|---|
Rejected | |
CONSUMED | A row’s count of the source examples it is the last row of, set on each source example a run bounded in examples reads (MarkConsumed) and carried through every step to the batch: each example’s count goes to the last row a step makes of it, the next row kept where a step rejects or makes none of it (Owed), and a packed window’s is its rows’ sum. |
REFUSED | The part of a row’s CONSUMED that are source examples a step rejected or made no row of, which the row carries on (Owed): consumed, and none of them trained. |
ACCOUNTS | A row’s two counts of source examples, which every step carries together, as integers no batch stacks. |
MEMBER | The mixture member a row was drawn from, by its key (dew.data.analysis._member_keys), which a member keeps when another joins or leaves, set where a mixture a run bounds in examples draws it (dew.data.lowering._Build.mixed) and carried to every row a step makes of it, its rejection’s marker too (Rejected.member). |
MEMBERS | The source examples a row completes, delivered (CONSUMED less REFUSED), by the mixture member each was drawn from (MEMBER), set where a member’s row first holds counts (consumed_by) and carried wherever its counts go, so a mixture counts each member’s examples where they are delivered (dew.data.mixing.CorpusCounts). |
COUNTING | A row’s fields the run counts it by, which every step carries out of its sight. |
Members | The source examples a row completes by mixture member (MEMBERS). |
member_of | The mixture member row was drawn from (MEMBER), none for a row of none or of several: every read of a row’s member. |
members_of | row’s completed examples by mixture member (MEMBERS), none where it holds no member’s. |
joined | Completed examples by mixture member, summed member by member, none for a member of none. |
DOCUMENT | A layout’s field beside its tokens that says, by an id, the document each token is of (Documents), the id negated at the document’s last token, so a window says which documents it holds and which end in it. |
Documents | The documents a layout lays out of a mixture’s rows, each under the id its tokens carry (DOCUMENT): the member it was drawn from (MEMBER, none for one of several members’ rows) and the examples it completes by member (MEMBERS), until the window that holds its last token takes them. |
accounts | The source examples row consumes and the part of them refused (CONSUMED, REFUSED), none where it holds neither: a rejected row’s are all refused, as the row kept after it is paid them (Owed). |
MarkConsumed | The step that marks each source example a run bounded in examples reads as one consumed by the row it is read in (CONSUMED), set where the bound is (dew.data.compile). |
Owed | The source examples rows a step dropped or made none of were the last of (Rejected.consumed), owed to the next row kept: the one carrier the reading process’s (dew.data.mixing.Accepted), a batched step’s (dew.data.mixing.Carried) and a loading worker’s group (dew.data.lowering) use. |
consumed_by | step’s outputs of given, the fields the run counts it by (COUNTING) kept out of the step’s sight: the mixture member it was drawn from (MEMBER) on every output, a rejected row’s marker too (Rejected.member), and the source examples it consumes (CONSUMED, and of them those it carries for refused ones, REFUSED, those it completes by member, MEMBERS) carried to its last output: a row’s counts beside its own, a rejected row’s on its marker (Rejected.consumed, every one of them refused now, completing no member’s), which the corpus’s next kept row carries on (Accepted). |
as_member | A step’s output of a mixture member’s row as that member’s (MEMBER, Rejected.member), the examples it completes its member’s where it first holds counts (the bound’s marks): a mixture tags each member’s records so (functools.partial(as_member, member=key)). |
members_held | The examples row completes by mixture member (MEMBERS): of its member alone, the consumed it has not refused, where it states no split. |
consumed_forward | elements in their order, each rejected row’s consumed examples (Rejected.consumed) carried on to the next row kept after it, as the corpus’s reading carries them (Accepted), so a group that holds its markers apart from its rows (_Grouped) moves none to an earlier row: only a marker no row follows in the group keeps its count, which _Ungrouped hands on after the group’s rows; the row it moves to carries it as refused (REFUSED). |
Cuts | A record’s cut record (_dew_cut): by what each cut, the rows it shortened and the tokens it removed; characters where the kind names them (<kind>_characters), as a cut of text encoded whole is measured. |
cut | element cut short by a step that removed removed of it (by what it cut, the tokens), its cut record composed with what earlier steps recorded (composed). |
composed | Two cut records of the same rows, one step after another: a kind both cut counts its rows once, the most either shortened, and adds what each removed; a 100-token text cut to 80, then to 60, removed 40 of one row. |
cut_unit | The unit a cut of kind removes: characters for a kind that names them (<kind>_characters, a text encoded whole), none for an event that removes nothing (options_alike, a row whose cut left two options alike), else tokens. |
cut_counts | Several rows’ cuts (each the tokens a cut removed of the row, by what it cut) as one record’s _dew_cut, which dew.data.mixing.CorpusCounts.delivered counts: by what each cut, the rows it shortened and the tokens it removed; distinct rows add, where composed adds one row’s successive cuts. |
Refused | A request a served task refuses, under the reason its training would reject the same record under (Rejected), which a server counts as a run’s data counters count rejections (dew.data.mixing.CorpusCounts). |
Record | |
RecordStep | |
RandomStep | |
HostStep | |
ColumnStep | A function of records read as a dict of columns, each a list, as Hugging Face’s map(batched=True) reads them. |
Rejecting | |
Reasoned | A predicate that names what it drops (images.MinSide’s “too_small”). |
PACK_POLICY | How a pack fills its rows where none is asked for. |
DROP_REMAINDER | Whether a batch a training pass fills only in part is dropped, where none is asked for. |
TOKEN_FIELDS | The per-token fields a pack or a window lays out where none are named. |
PACK_BINS | The rows a pack holds open at once where none is asked for. |
INTERLEAVE | The sources an interleave reads at once where none is asked for. |
PREFETCH | The records a prefetch reads ahead where none is asked for. |
Volatile | |
External | A step that reads the storage a record names by path, as a media decoder does when its field holds a path rather than bytes: what it makes is keyed by the path, not by what the path holds, so no automatic cache’s prefix runs past it (dew.data.compile._automatic), a file replaced at the same path reading anew; a chain’s own cache= keeps what it made, as it is told to. |
Costly | A step that costs more to run on a record than reading what it made back from a cache costs, as measured (tools/benchmark_data.py automatic-cache), and makes it from the record alone: so a chain that runs it first caches what it makes with no cache= (dew.data.compile._automatic). |
SingleReader | |
TokenGeometry | A record step, or a stage that lays rows out (Pack, Windows), whose rows each hold max_length tokens, which a token budget divides into rows (TokenBudget). |
TokenBudget | A batch of tokens tokens: as many rows as the pipeline’s fixed row length (TokenGeometry) divides it into, resolved when it compiles. |
Expansion | |
ExpandStep | |
Declares | A step, or a stage, that says what it makes of the facts of the records it reads (Dataset.facts): the fields it writes and their shapes, the tokenizer it encodes with. |
Stated | A store that states facts of its records without reading them (Dataset.facts): a token store’s tokenizer and vocabulary, say. |
Measure | A fact a configured step needs that a memory read measures over a store’s records where the store states none (Configured.measures): fact names it, and calling it on the records gives its value. |
Configured | Resolve one record step from source facts before any record reader opens. |
Bound | The configured step a record step was resolved from, with the metadata values its factory read. |
Batched | A function of size records at a time, read as a dict of columns, each a list, as Hugging Face’s map(batched=True): with same_rows, its columns of as many rows join the records’ own; otherwise they are the records that replace the group. |
Native | A function of the Grain dataset the stages before it make, for a Grain operation Dew does not name; the rows after it are read per share. |
Map | |
examples_bound | Where a run bounded in examples marks the source examples it reads (MarkConsumed), the stage after the bound’s limit (dew.data.compile._bounded): its index in stages, None for a read the run bounds otherwise. |
RandomMap | |
MapWithIndex | |
Filter | |
FlatMap | |
Shuffle | |
Repeat | |
Slice | |
Interleave | |
BatchStage | |
Pack | Documents packed into windows of seq_len tokens along fields. |
Windows | |
TokenSplit | |
Prefetch | |
Workers | |
Limit | |
DeviceStep | A step over the training batch that runs inside the compiled training step, on the accelerator, rather than in the reader: pure and traceable, a batch of arrays in and out, its randomness only from per-record key fields (images.AugmentationKey), so its draws are the same at any split, microbatch or pool size and on resume. |
MapBatch | |
Cached | What a cache stage fills and reads, a dew.data.cache.Cache, or the published generation it stands in for, a dew.data.generation.Generation: each says where it lives and which rows a plan reads from it. |
Artifact | |
Stage | |
Step | What a stage runs and its identity names (dew.data.plan.StepIdentity): a record, random, indexed, expanding, batch or column step, a predicate, a Grain function or a configured step’s factory. |
Runs | What a stage runs (runs): its callable, the identity written for it, and the configured step it was bound from, the one description its plan identity, its cache keyability and show read. |
runs | What stage runs, None for a stage that runs no step of its own (an order, a batch, a cache). |
Mixture | |
Phases | |
children | The chains pipeline’s mixture or phases read, none where it reads a store. |
chains | Every chain of pipeline’s tree, children first: each corpus under its mixtures and phases, then the chain that mixes or phases them, and pipeline last. |
with_children | pipeline with each chain its mixture or phases read rewritten by rewrite; pipeline itself where it reads a store. |
with_stores | pipeline with each store under its mixtures and phases read through store, every step as it was. |
Dataset | Records a run reads: a store (Files, Hub, TFDS, Memory, Torch), or a store with the steps the verbs added, in the order they are written. |
Piped | A call of Dataset.pipe and what it made: function(input, *args, **settings) made the stages start to end of the chain that keeps it, or, where it made another store of its input (a token store’s windows), the store and the stages to end from input. |
Pipeline | A dataset’s store and the stages its verbs added; what every verb returns. |
concat | The records of datasets, stores read one after another, as one dataset. |
mix | datasets read together, each filling its share of a step, as Grain’s mix: weights are the shares, equal by default, counted in records, or with by="tokens" in the valid target tokens of each member’s field. |
phases | Datasets read one after another: each (dataset, until_step) until the training step it names, the last until the run ends (None). |
schedule | Datasets read one after another on the run’s committed-token clock, the clock fit(tokens=) counts: each (dataset, tokens) for the valid target tokens its steps commit, the last for the rest of the run (None), or for its tokens, where the data ends (the run’s last step). |
ObjectiveData | |
record_stage | The stage a record step is applied by where no verb names it, from what the step says it is: an expansion’s records (FlatMap), a generator’s draws (RandomMap), or one record’s replacement (Map), dropping records where it says it does (rejects). |
Rejected
Section titled “Rejected”class Rejected( reason: str, permanent: bool = True, consumed: int = 0, cut: Mapping[str, int] = dict(), member: str | None = None,)consumed: int-
The source examples a rejected row was the last of (
CONSUMED), which the next row its corpus keeps carries on (dew.data.mixing.Accepted). cut: Mapping[str, int]-
The tokens the cuts of the refused rows removed, by what each cut, which its corpus counts beside the refusal (
dew.data.mixing.CorpusCounts.dropped). member: str | None-
The mixture member the rejected row was drawn from (
MEMBER), which its refusal is counted under.
CONSUMED
Section titled “CONSUMED”CONSUMED = '_dew_consumed'A row’s count of the source examples it is the last row of, set on each source example a run bounded in
examples reads (MarkConsumed) and carried through every step to the batch: each example’s count goes to the
last row a step makes of it, the next row kept where a step rejects or makes none of it (Owed), and a packed
window’s is its rows’ sum. A batch’s sum is the source examples its dispatch consumes
(dew.batch.EXAMPLES_CONSUMED).
REFUSED
Section titled “REFUSED”REFUSED = '_dew_refused'The part of a row’s CONSUMED that are source examples a step rejected or made no row of, which the row
carries on (Owed): consumed, and none of them trained. It travels with CONSUMED through every step, so a
count of the examples delivered rows complete takes their difference (dew.data.mixing.CorpusCounts).
ACCOUNTS
Section titled “ACCOUNTS”ACCOUNTS = (CONSUMED, REFUSED)A row’s two counts of source examples, which every step carries together, as integers no batch stacks.
MEMBER
Section titled “MEMBER”MEMBER = '_dew_member'The mixture member a row was drawn from, by its key (dew.data.analysis._member_keys), which a member
keeps when another joins or leaves, set where a mixture a run bounds in examples draws it
(dew.data.lowering._Build.mixed) and carried to every row a step makes of it, its rejection’s marker too
(Rejected.member).
MEMBERS
Section titled “MEMBERS”MEMBERS = '_dew_members'The source examples a row completes, delivered (CONSUMED less REFUSED), by the mixture member each
was drawn from (MEMBER), set where a member’s row first holds counts (consumed_by)
and carried wherever its counts go, so a mixture counts each member’s examples where they are delivered
(dew.data.mixing.CorpusCounts). Examples a step refused are carried on as refused (Owed) and complete
no member’s: what is refused of a member is counted where it is refused (Rejected.member).
COUNTING
Section titled “COUNTING”COUNTING = (*ACCOUNTS, MEMBER, MEMBERS)A row’s fields the run counts it by, which every step carries out of its sight.
Members
Section titled “Members”The source examples a row completes by mixture member (MEMBERS).
member_of
Section titled “member_of”def member_of(row: Batch) -> str | NoneThe mixture member row was drawn from (MEMBER), none for a row of none or of several: every read
of a row’s member.
members_of
Section titled “members_of”def members_of(row: Batch) -> Membersrow’s completed examples by mixture member (MEMBERS), none where it holds no member’s.
joined
Section titled “joined”def joined(*members: Members = ()) -> dict[str, int]Completed examples by mixture member, summed member by member, none for a member of none.
DOCUMENT
Section titled “DOCUMENT”DOCUMENT = '_dew_document'A layout’s field beside its tokens that says, by an id, the document each token is of (Documents), the
id negated at the document’s last token, so a window says which documents it holds and which end in it.
Documents
Section titled “Documents”class Documents( next: int = 1, held: dict[int, tuple[str | None, dict[str, int]]] = dict(),)The documents a layout lays out of a mixture’s rows, each under the id its tokens carry (DOCUMENT):
the member it was drawn from (MEMBER, none for one of several members’ rows) and the examples it
completes by member (MEMBERS), until the window that holds its last token takes them.
Documents.laid
Section titled “Documents.laid”def laid(member: str | None, members: Members) -> intThe id of a document of member that completes members.
Documents.saved
Section titled “Documents.saved”def saved(documents: Iterable[int]) -> dict[str, JSON]documents’ members and examples as a saved position holds them, by id.
Documents.restore
Section titled “Documents.restore”def restore(saved: JSON) -> NoneHold the documents a saved position held (saved) again.
accounts
Section titled “accounts”def accounts(row: Batch | Rejected) -> tuple[int, int]The source examples row consumes and the part of them refused (CONSUMED, REFUSED), none where
it holds neither: a rejected row’s are all refused, as the row kept after it is paid them (Owed).
MarkConsumed
Section titled “MarkConsumed”class MarkConsumed()The step that marks each source example a run bounded in examples reads as one consumed by the row it
is read in (CONSUMED), set where the bound is (dew.data.compile).
class Owed(examples: int = 0)The source examples rows a step dropped or made none of were the last of (Rejected.consumed),
owed to the next row kept: the one carrier the reading process’s (dew.data.mixing.Accepted), a
batched step’s (dew.data.mixing.Carried) and a loading worker’s group (dew.data.lowering) use.
Owed.owe
Section titled “Owed.owe”def owe(consumed: int) -> NoneOwe consumed more examples to the next row kept.
Owed.paid
Section titled “Owed.paid”def paid(row: Batch) -> Batchrow, kept, taking the examples owed (CONSUMED), every one refused (REFUSED), none owed
after it.
consumed_by
Section titled “consumed_by”def consumed_by( given: Record, step: Callable[[Record], tuple[Record, ...]],) -> tuple[Record, ...]step’s outputs of given, the fields the run counts it by (COUNTING) kept out of the step’s
sight: the mixture member it was drawn from (MEMBER) on every output, a rejected row’s marker too
(Rejected.member), and the source examples it consumes (CONSUMED, and of them those it carries for
refused ones, REFUSED, those it completes by member, MEMBERS) carried to its last output: a row’s
counts beside its own, a rejected row’s on its marker (Rejected.consumed, every one of them refused
now, completing no member’s), which the corpus’s next kept row carries on (Accepted). Every other row
holds none, so the rows of one read hold the same fields. A member’s row that first holds counts (the
bound’s marks, MarkConsumed) completes them as its member’s.
as_member
Section titled “as_member”def as_member(made: Record, member: str | None) -> RecordA step’s output of a mixture member’s row as that member’s (MEMBER, Rejected.member), the
examples it completes its member’s where it first holds counts (the bound’s marks): a mixture tags each
member’s records so (functools.partial(as_member, member=key)). A record of raw bytes holds no field
to say it, so a member’s bytes, whose examples a mixture counting its members’ would credit to none,
are refused, not passed on untagged: a mixture’s members are decoded before the mix, and a member’s
rows stay rows.
members_held
Section titled “members_held”def members_held(row: Batch, member: str | None) -> dict[str, int]The examples row completes by mixture member (MEMBERS): of its member alone, the consumed it
has not refused, where it states no split.
consumed_forward
Section titled “consumed_forward”def consumed_forward(elements: Sequence[Record]) -> list[Record]elements in their order, each rejected row’s consumed examples (Rejected.consumed) carried on
to the next row kept after it, as the corpus’s reading carries them (Accepted), so a group that
holds its markers apart from its rows (_Grouped) moves none to an earlier row: only a marker no row
follows in the group keeps its count, which _Ungrouped hands on after the group’s rows; the row it
moves to carries it as refused (REFUSED).
A record’s cut record (_dew_cut): by what each cut, the rows it shortened and the tokens it removed;
characters where the kind names them (<kind>_characters), as a cut of text encoded whole is measured.
def cut(element: Batch, removed: Mapping[str, int]) -> Batchelement cut short by a step that removed removed of it (by what it cut, the tokens), its cut
record composed with what earlier steps recorded (composed).
composed
Section titled “composed”def composed(earlier: Cuts, later: Cuts) -> dict[str, tuple[int, int]]Two cut records of the same rows, one step after another: a kind both cut counts its rows once, the most either shortened, and adds what each removed; a 100-token text cut to 80, then to 60, removed 40 of one row.
cut_unit
Section titled “cut_unit”def cut_unit(kind: str) -> str | NoneThe unit a cut of kind removes: characters for a kind that names them (<kind>_characters,
a text encoded whole), none for an event that removes nothing (options_alike, a row whose cut
left two options alike), else tokens.
cut_counts
Section titled “cut_counts”def cut_counts(cuts: Iterable[Mapping[str, int]]) -> dict[str, tuple[int, int]]Several rows’ cuts (each the tokens a cut removed of the row, by what it cut) as one record’s
_dew_cut, which dew.data.mixing.CorpusCounts.delivered counts: by what each cut, the rows it
shortened and the tokens it removed; distinct rows add, where composed adds one row’s successive
cuts.
Refused
Section titled “Refused”class Refused(reason: str, message: str)A request a served task refuses, under the reason its training would reject the same record
under (Rejected), which a server counts as a run’s data counters count rejections
(dew.data.mixing.CorpusCounts).
Record
Section titled “Record”RecordStep
Section titled “RecordStep”RandomStep
Section titled “RandomStep”HostStep
Section titled “HostStep”ColumnStep
Section titled “ColumnStep”A function of records read as a dict of columns, each a list, as Hugging
Face’s map(batched=True) reads them.
Rejecting
Section titled “Rejecting”class Rejecting(Protocol)Reasoned
Section titled “Reasoned”class Reasoned(Protocol)A predicate that names what it drops (images.MinSide’s “too_small”).
PACK_POLICY
Section titled “PACK_POLICY”PACK_POLICY: Literal['first-fit', 'best-fit', 'concat'] = 'first-fit'How a pack fills its rows where none is asked for.
DROP_REMAINDER
Section titled “DROP_REMAINDER”DROP_REMAINDER = TrueWhether a batch a training pass fills only in part is dropped, where none is asked for. Grain keeps it
(drop_remainder=False); Dew drops it because the training step is compiled for one batch shape, and a
short batch would compile a second program and take an update over fewer rows. A validation pass never
drops one (dew.data.lowering._Build.batch_order), and a run bounded in examples keeps its last batch,
filled and marked (dew.data.compile._bounded).
TOKEN_FIELDS
Section titled “TOKEN_FIELDS”TOKEN_FIELDS = ('text',)The per-token fields a pack or a window lays out where none are named.
PACK_BINS
Section titled “PACK_BINS”PACK_BINS = 32The rows a pack holds open at once where none is asked for.
INTERLEAVE
Section titled “INTERLEAVE”INTERLEAVE = 16The sources an interleave reads at once where none is asked for.
PREFETCH
Section titled “PREFETCH”PREFETCH = 64The records a prefetch reads ahead where none is asked for.
Volatile
Section titled “Volatile”class Volatile(Protocol)External
Section titled “External”class External(Protocol)A step that reads the storage a record names by path, as a media decoder does when its field
holds a path rather than bytes: what it makes is keyed by the path, not by what the path holds, so
no automatic cache’s prefix runs past it (dew.data.compile._automatic), a file replaced at the same
path reading anew; a chain’s own cache= keeps what it made, as it is told to.
Costly
Section titled “Costly”class Costly(Protocol)A step that costs more to run on a record than reading what it made back from a cache costs,
as measured (tools/benchmark_data.py automatic-cache), and makes it from the record alone: so a
chain that runs it first caches what it makes with no cache= (dew.data.compile._automatic).
An image decode is not one: an image a record names by path is read from storage the record’s
content does not key.
SingleReader
Section titled “SingleReader”class SingleReader(Protocol)TokenGeometry
Section titled “TokenGeometry”class TokenGeometry(Protocol)A record step, or a stage that lays rows out (Pack, Windows), whose rows each hold
max_length tokens, which a token budget divides into rows (TokenBudget).
TokenBudget
Section titled “TokenBudget”class TokenBudget( tokens: int, drop_remainder: bool = DROP_REMAINDER, batch_fn: Callable[[Sequence[Batch]], Batch] | None = None,)A batch of tokens tokens: as many rows as the pipeline’s fixed row
length (TokenGeometry) divides it into, resolved when it compiles.
TokenBudget.declared
Section titled “TokenBudget.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]Rows of the pipeline’s row length, as many a batch as tokens holds.
Expansion
Section titled “Expansion”class Expansion(Protocol)Expansion.flat_map
Section titled “Expansion.flat_map”def flat_map(element: Batch, /) -> Sequence[Batch | Rejected]ExpandStep
Section titled “ExpandStep”Declares
Section titled “Declares”class Declares(Protocol)A step, or a stage, that says what it makes of the facts of the records it reads
(Dataset.facts): the fields it writes and their shapes, the tokenizer it encodes with.
Declares.declared
Section titled “Declares.declared”def declared(facts: Mapping[str, object], /) -> Mapping[str, object]Stated
Section titled “Stated”class Stated(Protocol)A store that states facts of its records without reading them (Dataset.facts): a token
store’s tokenizer and vocabulary, say.
Stated.stated
Section titled “Stated.stated”def stated() -> Mapping[str, object]Measure
Section titled “Measure”class Measure(Protocol)A fact a configured step needs that a memory read measures over a store’s records where the
store states none (Configured.measures): fact names it, and calling it on the records gives
its value.
Configured
Section titled “Configured”class Configured( factory: Callable[[Mapping[str, object]], RecordStep | pygrain.RandomMapTransform | Expansion], needs: tuple[str, ...], scope: Literal['source', 'run'] = 'source', random: bool = False, one_to_one: bool = False, measures: Measure | None = None, expands: bool = False, examples: bool = False,)Resolve one record step from source facts before any record reader opens. one_to_one
states that the step keeps one record for each input whatever the facts, which a cache after it
reads before it binds; binding it to a step that drops or expands records is refused.
measures measures a fact it needs over each store it reads, in a memory read: the whole store,
below any holdout, so the parts a run trains and scores on agree; a store’s own statement of it
gives way to the measure, and a mixture’s step binds the largest of its stores’. random and
expands state what the step it binds to is, before it binds: a random map’s, and an expansion’s
of a record into several (Expansion); binding to another is refused. An objective’s step that
expands is marked as one that makes several rows of each source example (examples, set where the
objective places it): it runs after the chain’s record order, a run’s examples= bound stands
before it, and the expansion it binds to carries the mark (FlatMap.examples).
class Bound(step: Configured, values: Mapping[str, object])The configured step a record step was resolved from, with the metadata values its factory read.
Batched
Section titled “Batched”class Batched( step: ColumnStep, size: int, identity: str | None = None, same_rows: bool = True,)A function of size records at a time, read as a dict of columns, each
a list, as Hugging Face’s map(batched=True): with same_rows, its
columns of as many rows join the records’ own; otherwise they are the
records that replace the group. The last group of a finite pass may be
smaller.
Native
Section titled “Native”class Native( function: Callable[[pygrain.MapDataset | pygrain.IterDataset], pygrain.MapDataset | pygrain.IterDataset], identity: str | None = None,)A function of the Grain dataset the stages before it make, for a Grain operation Dew does not name; the rows after it are read per share.
class Map( step: RecordStep, identity: str | None = None, rejects: bool = False, configured: Bound | None = None,)examples_bound
Section titled “examples_bound”def examples_bound(stages: Sequence[object]) -> int | NoneWhere a run bounded in examples marks the source examples it reads (MarkConsumed), the stage
after the bound’s limit (dew.data.compile._bounded): its index in stages, None for a read the run
bounds otherwise. The one locator of the bound every compile and lowering step asks.
RandomMap
Section titled “RandomMap”class RandomMap( step: RandomStep, identity: str | None = None, rejects: bool = False, configured: Bound | None = None,)MapWithIndex
Section titled “MapWithIndex”class MapWithIndex( step: Callable[[int, Batch], Batch | Rejected], identity: str | None = None, rejects: bool = False,)Filter
Section titled “Filter”class Filter( predicate: Callable[[Batch], bool], reason: str = 'filtered', identity: str | None = None,)FlatMap
Section titled “FlatMap”class FlatMap( step: ExpandStep, identity: str | None = None, configured: Bound | None = None, examples: bool = False,)examples: bool-
Whether this is an objective’s step that makes several rows of each source example (a decision layout’s row a question): a corpus counts its source examples before it (
Accepted), and a run’sexamples=bound stands before it (dew.data.compile._bounded).
Shuffle
Section titled “Shuffle”class Shuffle( seed: int, window: int | None = None, block_shards: int | None = None, streamed: int | None = None,)streamed: int | None-
Where
windowis None, the records a streamed read draws its order from, whose records no index permutes whole (Pipeline.shuffle); None for a shuffle a streamed read refuses.
Repeat
Section titled “Repeat”class Repeat(epochs: int | None = None)class Slice(selection: slice)Interleave
Section titled “Interleave”class Interleave(cycle_length: int = INTERLEAVE)BatchStage
Section titled “BatchStage”class BatchStage( size: int, drop_remainder: bool = DROP_REMAINDER, batch_fn: Callable[[Sequence[Batch]], Batch] | None = None,)BatchStage.declared
Section titled “BatchStage.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]The same records, size of them a batch.
class Pack( seq_len: int, fields: tuple[str, ...] = TOKEN_FIELDS, bins: int = PACK_BINS, policy: Literal['first-fit', 'best-fit', 'concat'] = PACK_POLICY, segments: int | None = None, along: tuple[tuple[str, int], ...] = (), overlap: int = 0,)Documents packed into windows of seq_len tokens along fields.
along packs other fields on axes of their own, each to its own length,
and segments bounds the documents a window holds. A first-fit or
best-fit window closes when any of them is full, as Grain’s packer does
with its length_struct and max_sequences_per_bin.
A concatenating pack cuts documents where its windows end; overlap is
how many tokens before a cut through a document the next window holds
again, so 1 keeps the next-token target at every cut. A fitting pack cuts
no document: one longer than a window is cut before it, into pieces that
overlap as dew.data.text.Chunks says.
packed: tuple[str, ...]-
Every field a window holds:
fieldsand thosealongaxes of their own. max_length: int-
The inputs each window holds, its last token their target, which a token budget divides into rows (
TokenGeometry).
Pack.document
Section titled “Pack.document”def document(element: Batch) -> Batchelement as a document a window packs: its packed fields alone, its token fields of one
length (dew.data.steps.generic.aligned), its along fields beside them.
Pack.declared
Section titled “Pack.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]Rows of seq_len tokens along fields, the last of them a target: seq_len - 1 inputs.
Windows
Section titled “Windows”class Windows( seq_len: int, stride: int | None = None, fields: tuple[str, ...] = TOKEN_FIELDS,)max_length: int-
The inputs each window holds, which a token budget divides into rows (
TokenGeometry).
Windows.declared
Section titled “Windows.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]Windows of seq_len inputs and their targets along fields.
TokenSplit
Section titled “TokenSplit”class TokenSplit(fraction: float, held: bool, fields: tuple[str, ...] = TOKEN_FIELDS)Prefetch
Section titled “Prefetch”class Prefetch(buffer: int = PREFETCH)Workers
Section titled “Workers”class Workers(count: int, buffer: int = WORKER_BUFFER)class Limit(count: int, passes: int | None = None)passes: int | None-
Where set, the limit is that many passes over the rows reaching it, which a memory read that holds them counts as it holds them and sets
countto (dew.data.materialize): a run of epochs’s bound (dew.data.compile._bounded), the stages the read ran stating the count it resolved to.
DeviceStep
Section titled “DeviceStep”class DeviceStepA step over the training batch that runs inside the compiled training
step, on the accelerator, rather than in the reader: pure and traceable,
a batch of arrays in and out, its randomness only from per-record key
fields (images.AugmentationKey), so its draws are the same at any
split, microbatch or pool size and on resume. The reader leaves the fields
it reads in the batch it delivers (Loader.device).
MapBatch
Section titled “MapBatch”class MapBatch(step: HostStep | DeviceStep, identity: str | None = None)Cached
Section titled “Cached”class Cached(Protocol)What a cache stage fills and reads, a dew.data.cache.Cache, or the published generation
it stands in for, a dew.data.generation.Generation: each says where it lives and which rows
a plan reads from it.
Cached.reference
Section titled “Cached.reference”def reference(manifest: DependencyManifest, /) -> strArtifact
Section titled “Artifact”class Artifact(cache: Cached, automatic: bool = False)automatic: bool-
Whether the compiler placed it (
dew.data.compile._automatic) rather than the chain: it holds the records its prefix makes, so a run reads the same records with or without it and its plan names none (dew.data.cache.automatic_root).
What a stage runs and its identity names (dew.data.plan.StepIdentity): a record, random,
indexed, expanding, batch or column step, a predicate, a Grain function or a configured step’s
factory.
class Runs(step: Step, identity: str | None = None, configured: Bound | None = None)What a stage runs (runs): its callable, the identity written for it, and the configured step
it was bound from, the one description its plan identity, its cache keyability and show read.
def runs(stage: Stage) -> Runs | NoneWhat stage runs, None for a stage that runs no step of its own (an order, a batch, a cache).
Mixture
Section titled “Mixture”class Mixture( pipelines: tuple[Pipeline, ...], weights: tuple[float, ...], tokens: Callable[[Batch], int] | None = None, seed: int | None = None,)seed: int | None-
The seed whose permutation of each window’s corpus choices the mix reads in (
Proportions): None reads Grain’s schedule, the same every window in the members’ order.
Phases
Section titled “Phases”class Phases( pipelines: tuple[Pipeline, ...], ends: tuple[int | None, ...], clock: Literal['steps', 'tokens'] = 'steps',)ends: tuple[int | None, ...]-
Where each phase ends, as
clockcounts them: at the training step it names (phases), or at the run’s committed tokens (schedule). The last runs on (None); on the token clock it may end instead, where the data ends.
children
Section titled “children”def children(pipeline: Pipeline) -> tuple[Pipeline, ...]The chains pipeline’s mixture or phases read, none where it reads a store.
chains
Section titled “chains”def chains(pipeline: Pipeline) -> Iterator[Pipeline]Every chain of pipeline’s tree, children first: each corpus under its mixtures and phases,
then the chain that mixes or phases them, and pipeline last.
with_children
Section titled “with_children”def with_children( pipeline: Pipeline, rewrite: Callable[[Pipeline], Pipeline],) -> Pipelinepipeline with each chain its mixture or phases read rewritten by rewrite; pipeline itself
where it reads a store.
with_stores
Section titled “with_stores”def with_stores(pipeline: Pipeline, store: Callable[[Source], Source]) -> Pipelinepipeline with each store under its mixtures and phases read through store, every step as it
was.
Dataset
Section titled “Dataset”class DatasetRecords a run reads: a store (Files, Hub, TFDS, Memory,
Torch), or a store with the steps the verbs added, in the order they
are written. A dataset never changes; every verb returns a new one.
Trainer.fit and iterate lower it to Grain.
The verb shows the cardinality: map is one record to one, random_map
one to one with a generator, filter one to none or one, flat_map
one to many; dataset-level stages, such as text.pack, are functions
of a dataset applied with pipe. The compiler takes each stage where it
stands or refuses it before a reader opens.
batched: bool-
Whether this chain has made its training batch, after which
mapreads batches.
Dataset.holdout
Section titled “Dataset.holdout”def holdout( fraction: float | None = None, *, key: str | None = None, records: int | None = None, split: str | tuple[str, ...] | None = None, tokens: float | None = None, fields: tuple[str, ...] = TOKEN_FIELDS, **named: float = {},) -> tuple[Dataset, ...]This dataset split into the records a run trains on and the ones it scores, with the steps it has replayed on each, so hold out before preparing.
holdout(0.01) returns (train, held), and holdout(val=0.0005, test=0.0005) the remainder and one dataset per name, in order: a
record goes to the part its identity hashes into, the same in every
run, on every machine and after every resume. key names the field
that identifies a document (an id or a url), so its versions and
chunks land together; without one, a record’s identity is its whole
content. records=n holds out the first n records, and
split="validation" the store’s own split, or
split=("validation", "test") each of them, in order, after the
records the store reads by default. tokens=0.01 holds out the first
hundredth of the tokens of the order the steps so far read, cutting at
most one document in two, so the parts are disjoint and together the
corpus; it reads a finite indexed order, before any repeat, and cuts the
per-token fields together, counting the first.
Dataset.facts
Section titled “Dataset.facts”def facts() -> Mapping[str, object]What this dataset’s records are, read without opening a reader: the facts its stores
state (a token store’s tokenizer and vocabulary), then what each step makes of them in
order: fields, each field’s shape a record holds; seq_len and packed for token rows;
tokenizer, vocab_size; batch once batched. A step that changes records and doesn’t
say how leaves their fields unknown.
Dataset.named
Section titled “Dataset.named”def named(name: str) -> PipelineThis dataset under name, which its counts log under in a mix.
Dataset.pipe
Section titled “Dataset.pipe”def pipe( stages: Callable[Concatenate[Dataset, Options], Dataset], *args: Options.args = (), **kwargs: Options.kwargs = {},) -> Datasetstages(self, *args, **kwargs): a dataset-level stage, such as text.pack. The chain
it returns keeps the call beside the stages it made (Piped), which a run’s record writes
in their place (dew.data.records).
Dataset.map
Section titled “Dataset.map”def map( *steps: RecordStep | Configured | HostStep = (), batched: bool = False, batch_size: int = 1000, identity: str | None = None,) -> PipelineOne record to one: each step returns its record’s replacement. A
step that drops records (returns Rejected) goes through filter_map.
With batched, a step reads batch_size records at a time as a dict
of columns, each a list, and returns columns of as many rows, which
replace or join the record’s own (Hugging Face’s map(batched=True)).
After batch, a step reads each training batch on the host.
Dataset.random_map
Section titled “Dataset.random_map”def random_map(*steps: RandomStep = (), identity: str | None = None) -> PipelineOne record to one, each step drawing from the record’s generator
(fn(record, rng) or a grain.RandomMapTransform). A draw is keyed by
the record’s place in the run’s order, its epoch included: the same in
every read mode and reader for one order, and on resume; a different
shuffle draws differently, and a read cut per share (one that drops
records, or a streamed one) keys it by the record’s place in its share.
Dataset.filter_map
Section titled “Dataset.filter_map”def filter_map(*steps: RecordStep = (), identity: str | None = None) -> PipelineOne record to none or one: each step returns its record’s
replacement or Rejected(reason), which drops it, counted by reason
in the reader’s checkpointed counts, as images.fetch and
images.DecodeImage do.
Dataset.random_filter_map
Section titled “Dataset.random_filter_map”def random_filter_map(*steps: RandomStep = (), identity: str | None = None) -> PipelineOne record to none or one, each step drawing from the record’s
generator (fn(record, rng) or a grain.RandomMapTransform) and
returning its record’s replacement or Rejected(reason), which drops
it, counted by reason, as video.PickClip drops a clip shorter than
its window. Its draws are keyed as random_map’s are, by the record’s
place in the run’s order.
Dataset.map_with_index
Section titled “Dataset.map_with_index”def map_with_index( step: Callable[[int, Batch], Batch | Rejected], *, identity: str | None = None,) -> PipelineMap each row with its position in its source, as the read numbers it.
That is the index of an indexed source, the rows of the earlier shards
plus the row’s place in its own for a streamed source, and a cache’s
input index. A filter keeps it, so a rejected row leaves a gap, a
flat_map’s outputs share their input’s, and shuffles, repeats and
slices carry it with the row, so it is the same in every read, under
a cache, on resume and at any share or worker count. A row built from
several (after pack, windows, group or batch) or a mixture of corpora
has no single position, so the step is refused there. A step that
drops rows says so with rejects = True and returns Rejected(reason).
Dataset.filter
Section titled “Dataset.filter”def filter( predicate: Callable[[Batch], bool], *, reason: str | None = None, identity: str | None = None,) -> PipelineKeep the records predicate accepts; the rest are counted under
reason, by default the predicate’s own (images.MinSide’s “too_small”).
Dataset.flat_map
Section titled “Dataset.flat_map”def flat_map( step: ExpandStep | ColumnStep, *, batched: bool = False, batch_size: int = 1000, identity: str | None = None,) -> PipelineOne record to any number: the step returns a record’s records. With
batched, it reads batch_size records at a time as a dict of columns
and returns columns of any length, the records that replace them.
Dataset.shuffle
Section titled “Dataset.shuffle”def shuffle( seed: int = 0, *, window: int | None = None, block_shards: int | None = None, streamed: int | None = None,) -> PipelineThe records in an order drawn from seed: whole, within a window of records, or by blocks of
block_shards physical shards; or whole where the read is indexed and within streamed records
where it streams, as the read the run takes decides. The seed is 0 where none is given, where Grain
derives an unseeded shuffle from its pipeline’s seed (ds.seed), in Dew the run’s
(compile(seed=)): this stage’s
order is the chain’s own, the one every run of the chain reads and its saved plan names, so a run’s
seed changes neither it nor what a resume reads by it. What the run’s seed does draw are its random
steps and the order a streamed read deals its physical shards in (dew.data.shards.streamed).
Dataset.repeat
Section titled “Dataset.repeat”def repeat(epochs: int | None = None) -> PipelineDataset.interleave
Section titled “Dataset.interleave”def interleave(cycle_length: int = INTERLEAVE) -> PipelineDataset.batch
Section titled “Dataset.batch”def batch( size: int | None = None, *, tokens: int | None = None, drop_remainder: bool = DROP_REMAINDER, batch_fn: Callable[[Sequence[Batch]], Batch] | None = None,) -> PipelineDataset.pack
Section titled “Dataset.pack”def pack( seq_len: int, *, fields: tuple[str, ...] = TOKEN_FIELDS, bins: int = PACK_BINS, policy: Literal['first-fit', 'best-fit', 'concat'] = PACK_POLICY, segments: int | None = None, along: tuple[tuple[str, int], ...] = (), overlap: int = 0,) -> PipelineRecords laid end to end in rows of seq_len tokens: each row holds
the per-token fields of the records it packs, each along field on
an axis of its own, and at most segments records; a field that is
one value a record has no place in a row of several and is left out.
A concatenating pack’s row starts again overlap tokens before a cut
through a record (Pack).
Dataset.windows
Section titled “Dataset.windows”def windows( seq_len: int, *, stride: int | None = None, fields: tuple[str, ...] = TOKEN_FIELDS,) -> PipelineDataset.prefetch
Section titled “Dataset.prefetch”def prefetch(buffer: int = PREFETCH) -> PipelineDataset.workers
Section titled “Dataset.workers”def workers(count: int, *, buffer: int = WORKER_BUFFER) -> PipelineDataset.take
Section titled “Dataset.take”def take(count: int) -> PipelineDataset.cache
Section titled “Dataset.cache”def cache(cache: str | Cached) -> PipelineFill cache with the deterministic record steps before it, or read a
published Generation as the source in their place, with its published
packing in place of a pack that follows.
Dataset.snapshot
Section titled “Dataset.snapshot”def snapshot( url: str, *, format: Literal['array_record', 'parquet'] | Writer = 'array_record', rows_per_shard: int = 100000, deduplicate: NearDuplicates | None = None,) -> Mapping[str, object]Run this dataset once over every record and seal what it makes at
url: immutable shards and their committed manifest, which
Files(url) reads back, written as ArrayRecord, as Parquet or by a
native writer (dew.data.sinks.Writer). A fetch’s outcomes are sealed with it, so a
run that reads the snapshot resumes exactly. With deduplicate
(dew.data.text.NearDuplicates, the curation extra), each group of
near-duplicates in that one pass is sealed as one record, and the
manifest’s metadata records the rule, DataTrove’s release and the
records removed. Returns the manifest.
Dataset.rename_columns
Section titled “Dataset.rename_columns”def rename_columns(names: Mapping[str, str]) -> PipelineEach record with its fields renamed, as rename_columns({"txt": "caption"}).
Dataset.select_columns
Section titled “Dataset.select_columns”def select_columns(*names: str = ()) -> PipelineEach record with the fields names alone.
Dataset.remove_columns
Section titled “Dataset.remove_columns”def remove_columns(*names: str = ()) -> PipelineEach record without the fields names.
Dataset.grain
Section titled “Dataset.grain”def grain( function: Callable[[pygrain.MapDataset | pygrain.IterDataset], pygrain.MapDataset | pygrain.IterDataset], *, identity: str | None = None,) -> Pipelinefunction of the Grain dataset this chain makes, for a Grain
operation Dew does not name (zip, Grain’s own interleave,
FlatMapMapDataset). The chain after it is read per share and
checkpoints through Grain’s own state; identity names it in a run’s
plan, its code and closure by default.
Dataset.stats
Section titled “Dataset.stats”def stats(records: int = 1000) -> dict[str, float]The counts a run logs from a fresh, lazy pass over at most records accepted records:
the rows delivered and the drops by reason (dew.data.mixing.CorpusCounts), the rows the
pass read a second once its reader was open (data/records_per_second: the loading
settings default to none, while workers or prefetching the chain writes run as written),
and for token rows, under “text”, the tokens they hold (data/tokens) and the share of
their places that pad a row (data/padding).
Dataset.read
Section titled “Dataset.read”def read(mode: Read) -> PipelineRead the stores this way, rather than as they prefer: “memory” holds every record and runs the deterministic steps once, “indexed” reads by index, “streamed” reads shards in order.
Dataset.threads
Section titled “Dataset.threads”def threads(count: int) -> PipelineRead with count threads a worker, rather than the measured default.
Dataset.elastic
Section titled “Dataset.elastic”def elastic() -> PipelineRead it for training so a run resumes on any number of readers that splits its batch, as
a global order does, where its own order is read per share (a stream, or steps that drop,
expand, pack or draw). Every reader reads the order one reader reads, through its last step
that is not a map of each record alone, and keeps every count-th row of each batch, its
share; the maps written after that step run on its own rows alone, on its read threads. So
its rows are a one-reader run’s, and its position, the same on every reader, resumes on any
count, as HF’s split_dataset_by_node reads a stream on every node; each reader reads the
steps before those maps over every record. Its loading workers read its inputs in their
order, by index or each reading the whole stream and keeping its groups
(dew.data.elastic_order), so their count is no part of its position either.
Dataset.loading
Section titled “Dataset.loading”def loading(loading: Loading) -> PipelineRead with these workers, threads and buffers (Loading), rather than the measured
defaults, recorded with the chain; threads sets its read threads alone.
Dataset.show
Section titled “Dataset.show”def show(count: int = 8) -> NonePrint the chain, each verb and step a preset or function expanded to in order, then
its first count records.
class Piped( function: Callable[..., Dataset], args: tuple[object, ...], settings: tuple[tuple[str, object], ...], start: int, end: int, input: Pipeline | None = None,)A call of Dataset.pipe and what it made: function(input, *args, **settings) made the
stages start to end of the chain that keeps it, or, where it made another store of its
input (a token store’s windows), the store and the stages to end from input.
Pipeline
Section titled “Pipeline”class Pipeline( source: Source | Mixture | Phases, stages: tuple[Stage, ...] = (), name: str | None = None, mode: Read | None = None, _loading: Loading | None = None, _elastic: bool = False, piped: tuple[Piped, ...] = (),)A dataset’s store and the stages its verbs added; what every verb returns.
piped: tuple[Piped, ...]-
The calls of
pipethat made its stages (Piped), outermost last; no plan reads them.
concat
Section titled “concat”def concat(datasets: Sequence[Dataset]) -> PipelineThe records of datasets, stores read one after another, as one dataset.
def mix( datasets: Sequence[Dataset], weights: Sequence[float] | None = None, *, by: Literal['records', 'tokens'] = 'records', field: str = 'text', seed: int | None = None,) -> Pipelinedatasets read together, each filling its share of a step, as Grain’s
mix: weights are the shares, equal by default, counted in records,
or with by="tokens" in the valid target tokens of each member’s field.
Each member logs under its name, its store’s by default (Dataset.named).
Counted in records, the mix reads its members in Grain’s schedule, which every window of
the mix (its proportions’ sum of positions) repeats in the members’ order; a seed
permutes each window’s choices by a permutation of its own, every window still holding
each member at its share (dew.data.mixing.Proportions).
phases
Section titled “phases”def phases(*entries: tuple[Dataset, int | None] = ()) -> PipelineDatasets read one after another: each (dataset, until_step) until the training step it
names, the last until the run ends (None). Each phase is a whole chain, laid out and batched
as it reads, each of the same batch.
schedule
Section titled “schedule”def schedule(stages: Sequence[tuple[Dataset, float | None]]) -> PipelineDatasets read one after another on the run’s committed-token clock, the clock fit(tokens=)
counts: each (dataset, tokens) for the valid target tokens its steps commit, the last for the
rest of the run (None), or for its tokens, where the data ends (the run’s last step). A stage
ends at the step whose commit reaches its tokens, on every process alike, so its last step may
commit past them; one that a single step’s commit passes reads nothing. A corpus that an earlier
stage read continues from where it left it, and what the run had read ahead of the end is not
trained on. Each stage is a whole chain, laid out and batched as it reads.
ObjectiveData
Section titled “ObjectiveData”class ObjectiveData( train: tuple[RecordStep | pygrain.RandomMapTransform | Expansion | Configured, ...] = (), evaluate: tuple[RecordStep | Expansion | Configured, ...] = (), pack: Pack | None = None, packed: tuple[RecordStep, ...] = (), train_batch: tuple[HostStep, ...] = (), evaluate_batch: tuple[HostStep, ...] = (), geometry: RecordStep | None = None,)geometry: RecordStep | None-
The rows the objective reads where a chain lays out none of its own, after its other record steps; rows a chain packs or cuts into windows (
Pack,Windows) keep the length it gives them, as a schedule’s stages each give theirs (dew.data.compile.objective_steps).
record_stage
Section titled “record_stage”def record_stage( step: RecordStep | pygrain.RandomMapTransform | Expansion, *, identity: str | None = None, configured: Bound | None = None,) -> Map | RandomMap | FlatMapThe stage a record step is applied by where no verb names it, from what the step says it is:
an expansion’s records (FlatMap), a generator’s draws (RandomMap), or one record’s
replacement (Map), dropping records where it says it does (rejects). An objective’s steps
and a configured step’s binding are read through it; a verb (map, random_map, filter_map,
random_filter_map, flat_map) builds the stage its name states, which a step of another
kind is refused.