dew.data.holdout
Held-out records: a dataset split into the part a run trains on and the parts it scores.
| Name | Summary |
|---|---|
RecordSlice | The records [start, stop) of source, by index (holdout(records=)): the first start held out of the part a run trains on, or those a run scores. |
HashPart | The rows of source whose identity (its key field, or its content) hashes into [low, high) of the unit interval (held), or the rows that do not: a row is in exactly one part, the same in every run, read, machine and resume (holdout(fraction)). |
holdout | Dataset.holdout. |
selecting | source, then each source its holdout selections select from, down to their store, each with the selections above it, outermost first. |
SelectedGeneration | The rows generation holds for the inputs of source, the source it was published from, that the holdout selections keep (outermost first, source below the last), in their order: the generation standing in for its steps under selections written before it, each picking the inputs it picks uncached (dew.data.analysis._in_place). |
RecordSlice
Section titled “RecordSlice”class RecordSlice(source: Source, start: int, stop: int | None = None)The records [start, stop) of source, by index (holdout(records=)): the first start
held out of the part a run trains on, or those a run scores. A run’s record names it.
RecordSlice.describe
Section titled “RecordSlice.describe”def describe() -> strRecordSlice.fingerprint
Section titled “RecordSlice.fingerprint”def fingerprint() -> Mapping[str, object]RecordSlice.records
Section titled “RecordSlice.records”def records(read: Read)RecordSlice.admits
Section titled “RecordSlice.admits”def admits(position: int, row: Batch | bytes | None) -> boolWhether the record at position among its source’s is in this slice.
RecordSlice.admitted
Section titled “RecordSlice.admitted”def admitted(records: int) -> NoneRefuse a slice past the records it was offered (bounds).
RecordSlice.bounds
Section titled “RecordSlice.bounds”def bounds(records: int) -> tuple[int, int]The first and the one past the last of records it keeps, refusing a slice past them.
RecordSlice.metadata
Section titled “RecordSlice.metadata”def metadata() -> Mapping[str, object]RecordSlice.lengths
Section titled “RecordSlice.lengths”def lengths() -> np.ndarray | NoneRecordSlice.manifest
Section titled “RecordSlice.manifest”def manifest() -> ShardManifestHashPart
Section titled “HashPart”class HashPart( source: Source, low: float, high: float, held: bool, key: str | None = None,)The rows of source whose identity (its key field, or its content)
hashes into [low, high) of the unit interval (held), or the rows
that do not: a row is in exactly one part, the same in every run, read,
machine and resume (holdout(fraction)). A run’s record names it.
HashPart.describe
Section titled “HashPart.describe”def describe() -> strHashPart.fingerprint
Section titled “HashPart.fingerprint”def fingerprint() -> Mapping[str, object]HashPart.records
Section titled “HashPart.records”def records(read: Read)HashPart.keeps
Section titled “HashPart.keeps”def keeps(row: Batch | bytes | None) -> boolWhether this part holds row: one that is there, its identity in the part.
HashPart.admits
Section titled “HashPart.admits”def admits(position: int, row: Batch | bytes | None) -> boolWhether this part holds row, wherever it is (keeps).
HashPart.admitted
Section titled “HashPart.admitted”def admitted(records: int) -> NoneAny number of rows holds a hash part.
HashPart.lengths
Section titled “HashPart.lengths”def lengths() -> np.ndarray | NoneHashPart.manifest
Section titled “HashPart.manifest”def manifest() -> ShardManifestThe source’s shards, each read through the hash. Which of a shard’s rows the hash keeps
is known only once they are read, so no shard states its rows; the rows it keeps stream in
the order its source’s do, which is its index order where theirs is (records).
holdout
Section titled “holdout”def holdout( dataset: Dataset, fraction: float | None = None, *, key: str | None = None, records: int | None = None, split: str | tuple[str, ...] | None = None, **named: float = {},) -> tuple[Pipeline, ...]selecting
Section titled “selecting”def selecting( source: Source,) -> Iterator[tuple[tuple[RecordSlice | HashPart, ...], Source]]source, then each source its holdout selections select from, down to their store, each with
the selections above it, outermost first.
SelectedGeneration
Section titled “SelectedGeneration”class SelectedGeneration( generation: Generation, selections: tuple[RecordSlice | HashPart, ...], source: Source, placing: Read,)The rows generation holds for the inputs of source, the source it was published from, that
the holdout selections keep (outermost first, source below the last), in their order: the
generation standing in for its steps under selections written before it, each picking the
inputs it picks uncached (dew.data.analysis._in_place). The selections name the part, as the
run wrote them; source is read as the fill read it (_consumed), numbering its inputs by
their place in a memory or indexed read (placing), or by its shards’ rows in a stream, as the
fill did (_source_ids). Ids that name other inputs than the fill’s are refused, never read as none.
whole: Source-
The generation’s rows for every input: the whole store this part’s measured facts are measured over (
dew.data.analysis._stated), as uncached they are over the store below its selections.
SelectedGeneration.label
Section titled “SelectedGeneration.label”def label() -> strSelectedGeneration.describe
Section titled “SelectedGeneration.describe”def describe() -> strSelectedGeneration.fingerprint
Section titled “SelectedGeneration.fingerprint”def fingerprint() -> Mapping[str, object]SelectedGeneration.supports
Section titled “SelectedGeneration.supports”def supports(read: Read) -> boolSelectedGeneration.can_provision
Section titled “SelectedGeneration.can_provision”def can_provision(read: Read) -> boolSelectedGeneration.provision
Section titled “SelectedGeneration.provision”def provision() -> SelectedGenerationSelectedGeneration.records
Section titled “SelectedGeneration.records”def records(read: Read) -> RecordsSelectedGeneration.manifest
Section titled “SelectedGeneration.manifest”def manifest() -> ShardManifestSelectedGeneration.metadata
Section titled “SelectedGeneration.metadata”def metadata() -> Mapping[str, object]The selections’ facts, as uncached (what they hold out included), with this part’s rows and the generation’s count rules.
SelectedGeneration.stated
Section titled “SelectedGeneration.stated”def stated() -> Mapping[str, object]SelectedGeneration.order_fingerprints
Section titled “SelectedGeneration.order_fingerprints”def order_fingerprints() -> Mapping[str, str]Which of the generation’s rows this part reads, in their order.
SelectedGeneration.lengths
Section titled “SelectedGeneration.lengths”def lengths() -> np.ndarray | NoneNone: a part packs as it does uncached, where the steps the generation stands for come between it and the pack.