dew.data.text
Tokenizers for language-model data, utf-8 bytes or any HF tokenizer, and
the text record steps: tokenizing, chunking, dropping, normalizing and
filtering text, and curating it through DataTrove’s filters (Language,
Quality, Classifier, Decontaminated) and its MinHash deduplication
(NearDuplicates, which Dataset.snapshot applies; the curation extra).
import dew.data stays cheap. Nothing here imports transformers at module
scope, and ByteTokenizer needs nothing but numpy. HFTokenizer loads its
tokenizer on first use, so a host without the hub cache still imports
dew.data.text (and everything that re-exports it) fine.
| Name | Summary |
|---|---|
RENAMED_TOKENIZERS | Tokenizer classes a release ships as remote code that are a transformers class under another name: load_tokenizer loads such a release with that class, and runs none of the release’s code. |
PinnedTokenizer | A Hugging Face tokenizer resolved once to immutable files (pin_tokenizer): the snapshot its identity hashes and the directory it reads its vocabulary and templates from, which are one revision by construction. |
pin_tokenizer | Resolve name once (pin_files), with every file its loader can read hashed. |
pin_files | Resolve name once: a local directory, or a hub repo (optionally repo@revision) downloaded at the commit its revision names, with every file of patterns hashed: the snapshot an identity holds (its repo, the commit and each file’s hash) and the directory to read those files from, one commit by construction. |
load_tokenizer | The HF tokenizer of a hub repo or local directory, loaded once per process, or once per contents, the digest of a directory’s files where a caller pinned them (PinnedTokenizer). |
TokenArray | Holds ids as an array rather than as a list. |
Tokenizer | What a text vocabulary offers, as ByteTokenizer and HFTokenizer do: text to ids, with or without the vocabulary’s special tokens, ids back to text, and the id a sequence starts with, or None. |
Offsets | A vocabulary that says which characters each token of a text covers (offsets): a layout that encodes its prompt whole locates its spans by them (dew.decision.renderer). |
ByteTokenizer | Encodes text as one id per utf-8 byte, over a vocabulary of 256. |
HFTokenizer | A Hugging Face tokenizer, resolved from its hub name or local directory on first use. |
SavedTokenizer | A tokenizer object Dew names by the files it saves (tokenizer_name), such as a transformers tokenizer: a preset or a step given one reads it by that name. |
tokenizer_name | A tokenizer as the name Dew’s steps carry: a name as it is, and a transformers tokenizer object saved once to a directory named for its files’ hash, which every worker reads and every run hashes alike. |
tokenize | A record step that encodes the text in field to token ids under “text”, each document ended by the tokenizer’s EOS with eos, or where the tokenizer has one with None. |
Tokenize | The text in field as token ids under “text”, each document ended by the tokenizer’s EOS with eos, or where the tokenizer has one with None. |
Chunks | A document’s token fields cut into pieces of at most length, each starting length - overlap after the last (flat_map), before pack, which refuses a document longer than its window. |
Dropped | The text in field emptied with probability p, as classifier-free guidance trains its unconditioned branch. |
Normalized | The text in field in Unicode normal form form (unicodedata.normalize), so text written with composed, decomposed or compatibility characters (a ligature, a full-width digit) reads and tokenizes alike; other fields pass through. |
MinChars | Whether the text in field holds at least count characters. |
Curated | A record step over one of DataTrove’s document filters: the record where the filter keeps the text in field, else Rejected with the reason the filter names, or the step’s own. |
Language | The text in field kept where DataTrove’s language identification scores it above min_score in one of languages (DataTrove’s codes; any language when none is named), over fastText’s lid.176 (backend="ft176") or GlotLID ("glotlid"), which DataTrove downloads on first use into its asset cache. |
QUALITY_RULES | The heuristic quality rules Quality runs: Gopher’s (Rae et al. |
Quality | The text in field kept where DataTrove’s rule passes it: Gopher’s, C4’s or FineWeb’s quality filter at DataTrove’s own thresholds, any of which settings replaces as (parameter, value) pairs of the filter’s constructor. |
Classifier | The text in field kept by a fastText classifier, DataTrove’s fastText filter over the model at model (a path or a URL, which DataTrove downloads on first use): where one of keep’s (label, score) pairs scores at least its score, or where none of remove’s does. |
Decontaminated | The text in field kept unless it shares an n-gram with one of the evaluation texts against, by DataTrove’s n-gram decontamination: both normalized, split into words and hashed as its filter reads documents, and a record with a shared n-gram Rejected("contaminated"). |
NearDuplicates | MinHash’s near-duplicate rule, as DataTrove applies it (Broder 1997; FineWeb’s defaults): a record’s text in field is shingled into n-word n-grams, buckets bands of hashes minhashes each sign it, and two records whose signatures agree in a whole band are near-duplicates, which for Jaccard similarity s happens with probability 1 - (1 - s^hashes)^buckets: about 0.72 at half odds by default. |
near_duplicates_removed | The records each pass of records opens, one sealed snapshot’s in one order, with each group of near-duplicates (near) kept as one record by DataTrove’s four MinHash stages: signatures over a pass, band matches bucket by bucket, their clusters, and the filter that drops every record a cluster removes over a second pass. |
export_tokens | dataset’s documents in field tokenized by tokenizer and written to url as a committed token store (ArrayRecord shards, each record a document’s ids under “text”, its manifest naming the tokenizer, vocabulary and EOS), which dew.data.sources.TokenFiles(url) reads back and presets.lm(ds, None, ...) packs. |
train_tokenizer | A tokenizer of vocab_size tokens trained on the text in field of dataset’s records, the first records where given, by the algorithm, normalization and special tokens of base (a Hub repo or directory: a BPE tokenizer’s learns merges, a unigram’s pieces), saved at path, which it returns for tokenize(path) and the lm and sft presets to read. |
tokenizer_for | Builds the tokenizer name asks for. |
RENAMED_TOKENIZERS
Section titled “RENAMED_TOKENIZERS”RENAMED_TOKENIZERS: Mapping[str, str] = {'DreamTokenizer': 'Qwen2Tokenizer'}Tokenizer classes a release ships as remote code that are a transformers
class under another name: load_tokenizer loads such a release with that
class, and runs none of the release’s code.
PinnedTokenizer
Section titled “PinnedTokenizer”class PinnedTokenizer(snapshot: Mapping[str, object], directory: str)A Hugging Face tokenizer resolved once to immutable files
(pin_tokenizer): the snapshot its identity hashes and the directory it
reads its vocabulary and templates from, which are one revision by
construction.
PinnedTokenizer.transformers
Section titled “PinnedTokenizer.transformers”def transformers() -> PreTrainedTokenizerBaseThe transformers tokenizer of these files, whose chat template
renders conversations, as transformers’ class cls where a reader
names one (a CLIP tower’s slow CLIPTokenizer). A directory is read
where it is, so the load reaches no hub, and it is the one load every
reader of these files shares (load_tokenizer): keyed by their hashes,
so a directory whose files changed since a load loads again.
pin_tokenizer
Section titled “pin_tokenizer”def pin_tokenizer(name: str, *, local_files_only: bool = False) -> PinnedTokenizerResolve name once (pin_files), with every file its loader can read hashed.
pin_files
Section titled “pin_files”def pin_files( name: str, patterns: Sequence[str], *, local_files_only: bool = False,) -> tuple[Mapping[str, object], str]Resolve name once: a local directory, or a hub repo (optionally repo@revision) downloaded at
the commit its revision names, with every file of patterns hashed: the snapshot an identity holds
(its repo, the commit and each file’s hash) and the directory to read those files from, one commit by
construction. The Hub cache keeps a commit’s files in a directory named for the commit, which is the
revision recorded; offline (HF_HUB_OFFLINE), or with local_files_only, the name resolves to the
commit the local cache holds.
load_tokenizer
Section titled “load_tokenizer”def load_tokenizer( name: str, *, revision: str | None = None, local_files_only: bool = False, contents: str | None = None,) -> PreTrainedTokenizerBaseThe HF tokenizer of a hub repo or local directory, loaded once per process, or once per
contents, the digest of a directory’s files where a caller pinned them (PinnedTokenizer).
Every Dew path that needs an HF tokenizer loads it here, so one name at
one revision is one object however many readers share it. A release whose
tokenizer is a transformers class renamed in its own remote code loads as
that class (RENAMED_TOKENIZERS); a reader that needs one class of
transformers’ own names it (cls, a CLIP tower’s slow CLIPTokenizer,
ledger 3013), a load of its own.
TokenArray
Section titled “TokenArray”class TokenArray(Protocol)Holds ids as an array rather than as a list.
numpy’s and jax’s arrays both hand their ids over through tolist, and a
decode takes either beside a plain sequence, because a sampler returns a
row of a device array while a caller writes a list. A row has a length;
the scalars ArrayLike also covers are one id and have none.
TokenArray.tolist
Section titled “TokenArray.tolist”def tolist() -> list[int]Tokenizer
Section titled “Tokenizer”class Tokenizer(Protocol)What a text vocabulary offers, as ByteTokenizer and HFTokenizer do:
text to ids, with or without the vocabulary’s special tokens, ids back
to text, and the id a sequence starts with, or None.
Tokenizer.encode
Section titled “Tokenizer.encode”def encode(text: str, *, add_special_tokens: bool = True) -> list[int]Tokenizer.decode
Section titled “Tokenizer.decode”def decode(ids: ArrayLike | Sequence[int]) -> strOffsets
Section titled “Offsets”class Offsets(Protocol)A vocabulary that says which characters each token of a text covers (offsets): a layout that
encodes its prompt whole locates its spans by them (dew.decision.renderer).
Offsets.offsets
Section titled “Offsets.offsets”def offsets(text: str) -> tuple[list[int], list[int], list[int]]text’s ids without special tokens, and each token’s first character and the character
after its last.
ByteTokenizer
Section titled “ByteTokenizer”class ByteTokenizer()Encodes text as one id per utf-8 byte, over a vocabulary of 256.
It needs no training and no download, so it is the default for small corpora and for tests. Its eos id is 255, a byte that never occurs in utf-8 text, and it has no bos id. Decoding inverts encoding for any unicode text. Ids that are not valid utf-8, which a model can generate, decode with replacement characters.
name-
The name a run records for this vocabulary, which
tokenizer_forresolves back to it.
ByteTokenizer.dependencies
Section titled “ByteTokenizer.dependencies”def dependencies() -> Mapping[str, object]ByteTokenizer.encode
Section titled “ByteTokenizer.encode”def encode(text: str, *, add_special_tokens: bool = True) -> list[int]Return the text’s utf-8 bytes as ids.
add_special_tokens has no effect, because this vocabulary has no
special tokens to insert.
ByteTokenizer.decode
Section titled “ByteTokenizer.decode”def decode(ids: ArrayLike | Sequence[int]) -> strByteTokenizer.offsets
Section titled “ByteTokenizer.offsets”def offsets(text: str) -> tuple[list[int], list[int], list[int]]text’s bytes, each covering its character.
ByteTokenizer.document
Section titled “ByteTokenizer.document”def document(text: str, *, eos: bool) -> list[int]text as one document’s ids, its utf-8 bytes, ended by the EOS with eos where it holds
any.
HFTokenizer
Section titled “HFTokenizer”class HFTokenizer(name: str, *, local_files_only: bool = False)A Hugging Face tokenizer, resolved from its hub name or local directory on first use.
It resolves its name once (pinned), and its identity (dependencies)
and every encoding read that one resolution, here and in any process it
is pickled to, so a hub name that moves, or another revision of it a
process already loaded, never encodes rows under the identity of this
one. Because it resolves lazily, import dew.data.text (and
import dew.data) does not import transformers or look anything up on
the hub until a caller uses the tokenizer.
pinned: PinnedTokenizer-
The name resolved once to immutable files (
pin_tokenizer). eos_id: int | None-
The eos token’s id, or None for a tokenizer that declares none.
Raises
TypeErrorwhen the tokenizer reports something other than one id. bos_id: int | None-
The id the tokenizer starts a sequence with, or None where it has none.
Raises
TypeErrorwhen the tokenizer reports something other than one id.
HFTokenizer.dependencies
Section titled “HFTokenizer.dependencies”def dependencies() -> Mapping[str, object]HFTokenizer.encode
Section titled “HFTokenizer.encode”def encode(text: str, *, add_special_tokens: bool = True) -> list[int]HFTokenizer.offsets
Section titled “HFTokenizer.offsets”def offsets(text: str) -> tuple[list[int], list[int], list[int]]text’s ids and each token’s character span, by the fast tokenizer’s offsets; a slow
tokenizer keeps none and is refused (TypeError).
HFTokenizer.decode
Section titled “HFTokenizer.decode”def decode(ids: ArrayLike | Sequence[int]) -> strHFTokenizer.document
Section titled “HFTokenizer.document”def document(text: str, *, eos: bool) -> list[int]text as one document’s ids, by the tokenizer’s own special tokens (its BOS where it adds
one), ended by its EOS exactly once with eos where it holds any, and by none without,
whether or not the tokenizer’s post-processor appends its own, as T5’s and an EOS-inserting
Llama’s do. Raises ValueError for eos on a tokenizer without one.
HFTokenizer.save_pretrained
Section titled “HFTokenizer.save_pretrained”def save_pretrained(directory) -> NoneWrite this tokenizer’s own files into an export directory.
The tokenizer writes tokenizer.json, tokenizer_config.json and its vocabulary itself, so the export copies no files by hand and the result loads in anything that reads the HF layout.
SavedTokenizer
Section titled “SavedTokenizer”class SavedTokenizer(Protocol)A tokenizer object Dew names by the files it saves (tokenizer_name), such as a transformers
tokenizer: a preset or a step given one reads it by that name.
SavedTokenizer.save_pretrained
Section titled “SavedTokenizer.save_pretrained”def save_pretrained(save_directory: str, /) -> tuple[str, ...]tokenizer_name
Section titled “tokenizer_name”def tokenizer_name(tokenizer: str | SavedTokenizer) -> strA tokenizer as the name Dew’s steps carry: a name as it is, and a transformers tokenizer object saved once to a directory named for its files’ hash, which every worker reads and every run hashes alike.
tokenize
Section titled “tokenize”def tokenize( tokenizer: str | SavedTokenizer = 'byte', *, field: str = 'text', eos: bool | None = None,) -> TokenizeA record step that encodes the text in field to token ids under
“text”, each document ended by the tokenizer’s EOS with eos, or where
the tokenizer has one with None.
tokenizer is “byte” (Dew’s utf-8 vocabulary), a Hub repo or directory,
or a transformers tokenizer object.
Tokenize
Section titled “Tokenize”class Tokenize(tokenizer: str = 'byte', field: str = 'text', eos: bool | None = False)The text in field as token ids under “text”, each document ended by
the tokenizer’s EOS with eos, or where the tokenizer has one with None.
vocabulary: ByteTokenizer | HFTokenizer-
The tokenizer
tokenizernames, which resolves its name once (HFTokenizer.pinned): the step’s identity and its encoding both read that one resolution, in this process and in any worker it is sent to. costly: bool-
Whether encoding costs more than reading the ids back from a cache (
dew.data.pipeline.Costly): a learned tokenizer’s does, Dew’s bytes’ do not, astools/benchmark_data.py cachingmeasures (docs/design/data.md, section 8).
Tokenize.declared
Section titled “Tokenize.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]Token ids of each document’s length under “text”, by tokenizer, vocab_size of them.
Tokenize.dependencies
Section titled “Tokenize.dependencies”def dependencies() -> Mapping[str, object]Tokenize.map
Section titled “Tokenize.map”def map(element: Batch) -> BatchChunks
Section titled “Chunks”class Chunks(length: int, *, fields: tuple[str, ...] = TOKEN_FIELDS, overlap: int = 0)A document’s token fields cut into pieces of at most length, each
starting length - overlap after the last (flat_map), before pack,
which refuses a document longer than its window. With overlap=1 every
token’s next-token target stays in one piece: pack to length with
seq_len=length.
Chunks.declared
Section titled “Chunks.declared”def declared(facts: Mapping[str, object]) -> Mapping[str, object]Pieces of the documents, each of fields up to length long.
Chunks.flat_map
Section titled “Chunks.flat_map”def flat_map(element: Batch) -> Sequence[Batch]Dropped
Section titled “Dropped”class Dropped(field: str = 'caption', p: float = 0.1)The text in field emptied with probability p, as classifier-free
guidance trains its unconditioned branch.
Dropped.random_map
Section titled “Dropped.random_map”def random_map(element: Batch, rng: np.random.Generator) -> BatchNormalized
Section titled “Normalized”class Normalized( form: Literal['NFC', 'NFKC', 'NFD', 'NFKD'] = 'NFKC', *, field: str = 'text',)The text in field in Unicode normal form form (unicodedata.normalize), so text written
with composed, decomposed or compatibility characters (a ligature, a full-width digit) reads and
tokenizes alike; other fields pass through.
Normalized.map
Section titled “Normalized.map”def map(element: Batch) -> BatchMinChars
Section titled “MinChars”class MinChars(count: int, field: str = 'text')Whether the text in field holds at least count characters.
Curated
Section titled “Curated”class Curated(abc.ABC)A record step over one of DataTrove’s document filters: the record where the filter keeps the
text in field, else Rejected with the reason the filter names, or the step’s own. The filter
is built once a process (curator), never pickled with the step, and the step’s identity
records DataTrove’s release, which pins the filter’s rules and the models it names.
Curated.curator
Section titled “Curated.curator”def curator() -> BaseFilterThe DataTrove filter the step runs.
Curated.reads
Section titled “Curated.reads”def reads() -> strWhat the filter reads beyond the step’s own fields, by its content (a model’s SHA-256),
which the process’s filter is keyed by beside the step (_curator); none by default.
Curated.dependencies
Section titled “Curated.dependencies”def dependencies() -> Mapping[str, object]Language
Section titled “Language”class Language( languages: tuple[str, ...] = (), min_score: float = 0.65, backend: Literal['ft176', 'glotlid'] = 'ft176', field: str = 'text',)The text in field kept where DataTrove’s language identification scores it above
min_score in one of languages (DataTrove’s codes; any language when none is named), over
fastText’s lid.176 (backend="ft176") or GlotLID ("glotlid"), which DataTrove downloads on
first use into its asset cache.
Language.curator
Section titled “Language.curator”def curator() -> BaseFilterQUALITY_RULES
Section titled “QUALITY_RULES”QUALITY_RULES = ('gopher', 'c4', 'fineweb')The heuristic quality rules Quality runs: Gopher’s (Rae et al. 2021), C4’s (Raffel et al.
2020) and FineWeb’s (Penedo et al. 2024), as DataTrove implements them.
Quality
Section titled “Quality”class Quality( rule: Literal['gopher', 'c4', 'fineweb'], settings: tuple[tuple[str, float | int | bool | str | tuple[str, ...] | None], ...] = (), field: str = 'text',)The text in field kept where DataTrove’s rule passes it: Gopher’s, C4’s or FineWeb’s
quality filter at DataTrove’s own thresholds, any of which settings replaces as
(parameter, value) pairs of the filter’s constructor. A rejected record is counted under the
rule’s reason (gopher_short_doc, too_few_sentences, line_punct_ratio, …).
settings: tuple[tuple[str, float | int | bool | str | tuple[str, ...] | None], ...]-
The filter’s constructor arguments, a sequence (Gopher’s
stop_words) as a tuple.
Quality.curator
Section titled “Quality.curator”def curator() -> BaseFilterClassifier
Section titled “Classifier”class Classifier( model: str, keep: tuple[tuple[str, float], ...] = (), remove: tuple[tuple[str, float], ...] = (), field: str = 'text',)The text in field kept by a fastText classifier, DataTrove’s fastText filter over the model at
model (a path or a URL, which DataTrove downloads on first use): where one of keep’s
(label, score) pairs scores at least its score, or where none of remove’s does. The model is
read once a step, to a copy named by its SHA-256 (pinned), which the filter loads and the
step’s identity records, so a model replaced at its path later reaches neither.
pinned: Path-
The model’s bytes as this step first read them, under the system’s temporary directory by their SHA-256: a local file’s own, a URL’s as DataTrove downloads it once into its asset cache, which holds a local path’s first bytes too.
Classifier.reads
Section titled “Classifier.reads”def reads() -> strClassifier.curator
Section titled “Classifier.curator”def curator() -> BaseFilterClassifier.dependencies
Section titled “Classifier.dependencies”def dependencies() -> Mapping[str, object]Decontaminated
Section titled “Decontaminated”class Decontaminated(against: tuple[str, ...], n: int = 12, field: str = 'text')The text in field kept unless it shares an n-gram with one of the evaluation texts
against, by DataTrove’s n-gram decontamination: both normalized, split into words and hashed
as its filter reads documents, and a record with a shared n-gram Rejected("contaminated").
The evaluation texts are indexed once, with no evaluation harness, under the system’s
temporary directory by what they hold.
Decontaminated.curator
Section titled “Decontaminated.curator”def curator() -> BaseFilterNearDuplicates
Section titled “NearDuplicates”class NearDuplicates( n: int = 5, buckets: int = 14, hashes: int = 8, field: str = 'text',)MinHash’s near-duplicate rule, as DataTrove applies it (Broder 1997; FineWeb’s defaults): a
record’s text in field is shingled into n-word n-grams, buckets bands of hashes minhashes
each sign it, and two records whose signatures agree in a whole band are near-duplicates, which
for Jaccard similarity s happens with probability 1 - (1 - s^hashes)^buckets: about 0.72 at
half odds by default.
near_duplicates_removed
Section titled “near_duplicates_removed”def near_duplicates_removed( records: Callable[[], Iterator[Batch]], near: NearDuplicates,) -> tuple[Iterator[Batch], int]The records each pass of records opens, one sealed snapshot’s in one order, with each group
of near-duplicates (near) kept as one record by DataTrove’s four MinHash stages: signatures
over a pass, band matches bucket by bucket, their clusters, and the filter that drops every
record a cluster removes over a second pass. Returns the records kept, lazily, and how many the
clusters remove.
export_tokens
Section titled “export_tokens”def export_tokens( dataset: Dataset, url: str, tokenizer: str, *, field: str = 'text', eos: bool = True, rows_per_shard: int = 100000, read: Read = 'streamed', provision: bool = False,) -> Mapping[str, object]dataset’s documents in field tokenized by tokenizer and written to url as a committed
token store (ArrayRecord shards, each record a document’s ids under “text”, its manifest naming
the tokenizer, vocabulary and EOS), which dew.data.sources.TokenFiles(url) reads back and
presets.lm(ds, None, ...) packs. Each document is ended by the tokenizer’s EOS with eos.
Returns the store’s manifest.
train_tokenizer
Section titled “train_tokenizer”def train_tokenizer( dataset: Dataset, base: str, path: str, *, vocab_size: int, field: str = 'text', records: int | None = None,) -> strA tokenizer of vocab_size tokens trained on the text in field of dataset’s records, the
first records where given, by the algorithm, normalization and special tokens of base (a Hub
repo or directory: a BPE tokenizer’s learns merges, a unigram’s pieces), saved at path, which it
returns for tokenize(path) and the lm and sft presets to read. A vocab_size smaller than
the base’s alphabet and special tokens, which its algorithm always keeps, is refused, and so is a
base without a fast tokenizer, which has no algorithm to train.
tokenizer_for
Section titled “tokenizer_for”def tokenizer_for( name: str, *, local_files_only: bool = False,) -> ByteTokenizer | HFTokenizerBuilds the tokenizer name asks for.
byte is dew’s own utf-8 vocabulary; any other name is the HF tokenizer
of that repo or local directory. Resolving names here alone keeps a
training run and an export of what it trained reading one name one way.
An export passes local_files_only, since writing a checkpoint out is no
reason to reach the hub.