Skip to content

dew.data.steps.text

Over-length policies, fixed-length checks and target counts of token records.

NameSummary
LengthPolicy
ValidTargets
CountTargetsEach record’s targets the loss scores, counted once and carried on the record with the rule that counted them (recounted), so a token mixture weighs it without counting it again.
reboundWhere a projection of a counted element (Fields) leaves the fields its count was counted from: renamed where it renames one, empty where it drops one, which no step can count from again; nothing for a record no count rides on.
targets_changedWhether a step that made output of given changed what its targets are counted from: a field the count reads that given held and output holds as a different array, or a counted field that output holds and given did not, as a map adding a row’s blocks does.
recountedoutput with the targets counted on given (CountTargets): carried onto a record the step made anew without them, with the rule and the names they were counted by; kept while the step left every field they were counted from as it was, as a map of a record’s other fields or a projection does; and counted again by the rule that counted them where it changed one in place, as a crop does, from the fields as they are named now.
CountRuleThe rule a record’s targets were counted by (CountTargets) and the name its roles go by (rebound): a windows or pack stage counts the windows it makes of such records by it, as the loss scores them, since their documents’ counts describe none of them.
count_rulesThe rules other than every target’s that rows’ targets were counted by (CountRule), as a store states them (count_rules): each a role and a pad id.
stated_rulesThe rules a store’s count_rules states, each a role and a pad id (CountRule), either None, as a generation writes them.
counted_windowA window a pack or windows stage made, its targets of its token field counted as the loss scores them and carried on it, by the rule its documents were counted by (CountRule), or every target’s where nothing counted them: a later step, the read that accepts it (a memory read’s, dew.data.lowering._Build.targeted) and a mixture by tokens read them as a counted record’s.
LayoutRuleThe one rule the documents a pack or windows stage lays out were counted by (CountRule), or None where nothing counted them, as each document passes.
FixedTokensRows of exactly max_length + extra tokens of field, the max_length a token budget divides into rows (TokenGeometry).

dataclass source

class LengthPolicy(
field: str = 'text',
long: Literal['truncate', 'drop'] = 'truncate',
side: Literal['left', 'right'] = 'right',
aligned: tuple[str, ...] = (),
)
rejects: bool

Whether a long record is dropped (filter_map) rather than cut (map).

def map(element: Batch) -> Batch | Rejected

dataclass source

class ValidTargets(
field: str = 'text',
role: int | None = None,
pad_id: int | None = None,
measured: bool = True,
)

dataclass source

class CountTargets(role: int | None = None, pad_id: int | None = None)

Each record’s targets the loss scores, counted once and carried on the record with the rule that counted them (recounted), so a token mixture weighs it without counting it again.

def declared(facts: Mapping[str, object]) -> Mapping[str, object]

The records as they were, their targets counted.

def map(element: Batch) -> Batch | Rejected

function source

def rebound(element: Batch, kept: Batch, renamed: Mapping[str, str]) -> Batch

Where a projection of a counted element (Fields) leaves the fields its count was counted from: renamed where it renames one, empty where it drops one, which no step can count from again; nothing for a record no count rides on.

function source

def targets_changed(given: Batch, output: Batch) -> bool

Whether a step that made output of given changed what its targets are counted from: a field the count reads that given held and output holds as a different array, or a counted field that output holds and given did not, as a map adding a row’s blocks does. Each is read where the record names it before the step and where output names it after (rebound), so a projection that moves a field, or names another field as it was, changes nothing; a field the step removed pins the count, which nothing it still holds could give again.

function source

def recounted(given: Batch, output: Batch) -> Batch

output with the targets counted on given (CountTargets): carried onto a record the step made anew without them, with the rule and the names they were counted by; kept while the step left every field they were counted from as it was, as a map of a record’s other fields or a projection does; and counted again by the rule that counted them where it changed one in place, as a crop does, from the fields as they are named now. Every read carries a count so, a memory read’s among them, which keeps no count of its own beside its records.

dataclass source

class CountRule()

The rule a record’s targets were counted by (CountTargets) and the name its roles go by (rebound): a windows or pack stage counts the windows it makes of such records by it, as the loss scores them, since their documents’ counts describe none of them.

def of(element: Batch) -> CountRule | None

The rule element’s targets were counted by, or None where it is every target’s, as an uncounted row’s are (ValidTargets): no role and no padding.

def counted(window: Batch, field: str) -> Batch

window, made of records counted by this rule from its token field, its own targets counted by the rule and carrying it, so a later step or layout counts them by it again.

function source

def count_rules(rows: Iterable[Batch | bytes | None]) -> list[list[int | None]]

The rules other than every target’s that rows’ targets were counted by (CountRule), as a store states them (count_rules): each a role and a pad id.

function source

def stated_rules(value: object) -> set[tuple[int | None, int | None]]

The rules a store’s count_rules states, each a role and a pad id (CountRule), either None, as a generation writes them.

function source

def counted_window(window: Batch, rule: CountRule | None, field: str) -> Batch

A window a pack or windows stage made, its targets of its token field counted as the loss scores them and carried on it, by the rule its documents were counted by (CountRule), or every target’s where nothing counted them: a later step, the read that accepts it (a memory read’s, dew.data.lowering._Build.targeted) and a mixture by tokens read them as a counted record’s.

class source

class LayoutRule()

The one rule the documents a pack or windows stage lays out were counted by (CountRule), or None where nothing counted them, as each document passes. Documents counted by two rules, or some by none, make windows no one rule counts, and are refused.

def of(documents: Iterable[Batch]) -> CountRule | None

The one rule documents were counted by, all of them passing.

def state() -> Mapping[str, object]
def restore(state: Mapping[str, object]) -> None

dataclass source

class FixedTokens(extra: int = 0, field: str = 'text')

Rows of exactly max_length + extra tokens of field, the max_length a token budget divides into rows (TokenGeometry).

def map(element: Batch) -> Batch