dew.data.steps.generic
Field selection, formatting and record filtering, steps any chain takes.
| Name | Summary |
|---|---|
aligned | element’s fields, which hold one entry a token of the record and so one length, as arrays, beside its beside fields, on axes of their own and read as they are. |
Fields | A record’s fields names (every field where None), renamed by rename, without remove. |
Coordinated | A packed training or validation batch with each window’s documents as the model reads them: the <ids>_segment_ids and <ids>_positions a packer writes for its fields become <ids>_coordinates, the model’s dew.batch.Coordinates built on the host, holding at most max_segments documents a row, or as many as a window holds tokens where None (static, so a compiled step is specialized to it). |
Formatting |
aligned
Section titled “aligned”def aligned( element: Batch, fields: tuple[str, ...], beside: tuple[str, ...] = (),) -> Batchelement’s fields, which hold one entry a token of the record and so one length, as arrays,
beside its beside fields, on axes of their own and read as they are. A record whose fields
differ in length is refused: packed or cut into windows, each field would be cut at other places,
one field’s tokens standing beside another’s from elsewhere, which no reader tells.
Fields
Section titled “Fields”class Fields( names: tuple[str, ...] | None = None, rename: tuple[tuple[str, str], ...] = (), remove: tuple[str, ...] = (), bookkeeping: bool = False,)A record’s fields names (every field where None), renamed by rename, without remove.
bookkeeping keeps Dew’s own record of the record (_dew_*) whatever the projection keeps,
its counted targets among it (dew.data.steps.text.rebound), as a pipeline’s column steps do;
a projection Dew reads for a packer keeps exactly the fields it names.
Fields.declared
Section titled “Fields.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]The shapes of the fields kept, under their new names.
Fields.map
Section titled “Fields.map”def map(element: Batch) -> BatchCoordinated
Section titled “Coordinated”class Coordinated()A packed training or validation batch with each window’s documents as
the model reads them: the <ids>_segment_ids and <ids>_positions a
packer writes for its fields become <ids>_coordinates, the model’s
dew.batch.Coordinates built on the host, holding at most
max_segments documents a row, or as many as a window holds tokens where
None (static, so a compiled step is specialized to it). A window holding
more than its read declared is refused, which a sequence kernel would run
into its neighbours’ sequences. ids is the packed ids field, the first
of fields.
Coordinated.map
Section titled “Coordinated.map”def map(element: Batch) -> BatchFormatting
Section titled “Formatting”class Formatting()Formatting.map
Section titled “Formatting.map”def map(element: Batch) -> Batch