dew.data.steps.windows
Overlapping causal windows, with the boundary target retained.
Grain’s concat-and-split packer has disjoint windows. Causal LM windows share their boundary token so every target is predicted exactly once.
| Name | Summary |
|---|---|
TokenSpan | |
IndexedWindows | Complete causal windows of aligned fields backed by contiguous token spans. |
WindowSource | Causal windows of length tokens, stride apart, over a token store’s tokens. |
CausalWindows | Windows over a stream of documents; a mixture’s (documents) say, by id, which documents each holds and which end in it (DOCUMENT). |
TokenSpan
Section titled “TokenSpan”class TokenSpan(Protocol)IndexedWindows
Section titled “IndexedWindows”class IndexedWindows(dtype: str | None = None)Complete causal windows of aligned fields backed by contiguous token spans.
IndexedWindows.from_records
Section titled “IndexedWindows.from_records”def from_records(rows: Sequence[Batch], windows: Windows) -> IndexedWindowsWindowSource
Section titled “WindowSource”class WindowSource(rows: Rows = 'derived')Causal windows of length tokens, stride apart, over a token store’s tokens.
max_length: int-
The inputs each window holds, which a token budget divides into rows (
dew.data.pipeline.TokenGeometry).
WindowSource.describe
Section titled “WindowSource.describe”def describe() -> strWindowSource.fingerprint
Section titled “WindowSource.fingerprint”def fingerprint() -> Mapping[str, object]WindowSource.can_provision
Section titled “WindowSource.can_provision”def can_provision(read: Read) -> boolWindowSource.provision
Section titled “WindowSource.provision”def provision() -> WindowSourceProvision the token store, whose length index these windows need to read it by span.
WindowSource.records
Section titled “WindowSource.records”def records(read: Read) -> IndexedWindowsWindowSource.metadata
Section titled “WindowSource.metadata”def metadata() -> Mapping[str, object]WindowSource.lengths
Section titled “WindowSource.lengths”def lengths() -> np.ndarray | NoneWindowSource.manifest
Section titled “WindowSource.manifest”def manifest() -> ShardManifestCausalWindows
Section titled “CausalWindows”class CausalWindows( parent: grain.IterDataset[Batch], windows: Windows, documents: Documents | None = None,)Windows over a stream of documents; a mixture’s (documents) say, by id, which documents each holds
and which end in it (DOCUMENT).