dew.sampling.decoding
Logit transforms and stopping criteria for the decode loop.
Three concepts extend decoding. A LogitsTransform is a pure callable from
the step state and [rows, vocab] logits to new logits. A Stopping is a
pure callable from the step state and the tokens just drawn to a per-row
finished flag; criteria combine with OR after every committed token. A
Strategy in dew.sampling.strategies runs the device loop.
Transforms and criteria read StepState, which holds the token history and
nothing about the model: no parameters and no cache. Every built-in here is a
pytree, so a configuration holding arrays crosses jax.jit as data instead of
entering a compilation cache key, and so do its numbers where they fix no
shape (as_pytree): another temperature, penalty or length runs the same
program. A plain function works as well, and jax.tree_util.Partial can hold
array configuration for one.
The numerical reference is Transformers 5.16.1
generation/logits_process.py and generation/stopping_criteria.py, with two
differences that follow from Dew’s decode loop. Each row reads its own
unpadded history rather than the batch’s padded width, and frequency and
presence penalties follow vLLM’s formula
(model_executor/layers/utils.py), which Transformers does not implement.
| Name | Summary |
|---|---|
BeginSuppressTokens | Removes tokens at one generated position, as SuppressTokensAtBeginLogitsProcessor. |
EndOfSequence | Finishes a row that drew one of the EOS ids, as EosTokenCriteria. |
EpsilonCutoff | Removes tokens below an absolute probability, as EpsilonLogitsWarper. |
EtaCutoff | Entropy-scaled cutoff, as EtaLogitsWarper, which takes the square root of its float32 epsilon on the device. |
ExponentialDecayLengthPenalty | Raises the EOS score after start drawn tokens, as ExponentialDecayLengthPenalty. |
Finish | The rows a step’s criteria finished, and those of them a criterion ended, as a stop does, rather than one cut short at a length (Stopping’s truncates): the terminated rows. |
Finishing | A request’s criteria as pytrees (components), those that end a row apart from those that cut it short at a length, which is told from each criterion as given (Truncates) before it is one: a built-in’s numbers ride as data (as_pytree). |
ForcedBOS | Forces one token as the first of the whole sequence, as ForcedBOSTokenLogitsProcessor. |
ForcedEOS | Forces EOS one step before the end, as ForcedEOSTokenLogitsProcessor. |
FrequencyPenalty | Subtracts penalty times each token’s count among the drawn tokens. |
Greedy | The argmax as a distribution: zero on the best token, -inf elsewhere. |
LogitsChain | The ordered transforms, keeping a terminal Greedy for the sampler. |
LogitsTransform | A pure rewrite of [rows, vocab] scores, applied before the draw. |
MaxLength | Finishes a row once prompt and generated tokens reach length, as MaxLengthCriteria, which truncates it. |
MaxNewTokens | Finishes a row once it has drawn count tokens, which truncates it. |
MinLength | Suppresses EOS until the whole sequence reaches length, as MinLengthLogitsProcessor. |
MinNewTokens | Suppresses EOS until count tokens are drawn, as MinNewTokensLengthLogitsProcessor. |
MinP | Relative filtering at p times the top probability, as MinPLogitsWarper. |
NoRepeatNGram | Bans tokens that would repeat an n-gram of the row’s own history. |
PresencePenalty | Subtracts penalty from every token already drawn. |
PromptNoRepeatNGram | Bans tokens that would repeat an n-gram of the prompt. |
PromptRepetitionPenalty | Raises the scores of prompt tokens, as EncoderRepetitionPenaltyLogitsProcessor. |
RemoveInvalidValues | Maps NaN to zero and infinities to the float range, as InfNanRemoveLogitsProcessor. |
Renormalize | Replaces scores by their log softmax, as LogitNormalization. |
RepetitionPenalty | Divides the positive scores of seen tokens by the penalty and multiplies negative ones. |
SequenceBias | Adds a bias to the token that would complete each biased sequence. |
StepState | What a transform or a criterion sees at one decode step. |
StopStrings | Finishes a row whose text ends with one of the compiled stop strings. |
Stopping | A pure per-row finish test over the tokens a step just drew. |
SuppressTokens | Removes a fixed set of tokens, as SuppressTokensLogitsProcessor. |
Temperature | Divides the logits by value, as TemperatureLogitsWarper. |
TopH | Entropy-budget filtering, as TopHLogitsWarper. |
TopK | Keeps the k highest scores, as TopKLogitsWarper. |
TopP | Nucleus filtering, as TopPLogitsWarper. |
Truncates | Marks a stopping criterion that cuts a row short at a length rather than ending it. |
Typical | Locally typical filtering, as TypicalLogitsWarper. |
BeginSuppressTokens
Section titled “BeginSuppressTokens”class BeginSuppressTokens( forced_bos: bool = struct.field(pytree_node=False, default=False),)Removes tokens at one generated position, as SuppressTokensAtBeginLogitsProcessor.
The reference suppresses the first drawn token, or the one after it where
forced_bos (ForcedBOS) forces the first: transformers moves the begin
index past the forced token only for a one-token prompt
(generation/utils.py:1278-1282, 5.16.1), the rows where ForcedBOS fires.
Each row decides by its own prompt, so rows of mixed prompt lengths
suppress each at its own position.
BeginSuppressTokens.numerical
Section titled “BeginSuppressTokens.numerical”def numerical() -> jax.tree_util.PartialBeginSuppressTokens.apply
Section titled “BeginSuppressTokens.apply”def apply( tokens: jax.Array, forced_bos: bool, state: StepState, logits: jax.Array,) -> jax.ArrayEndOfSequence
Section titled “EndOfSequence”class EndOfSequence()Finishes a row that drew one of the EOS ids, as EosTokenCriteria.
EpsilonCutoff
Section titled “EpsilonCutoff”class EpsilonCutoff(epsilon: float = struct.field(pytree_node=False, default=0.0))Removes tokens below an absolute probability, as EpsilonLogitsWarper.
EpsilonCutoff.numerical
Section titled “EpsilonCutoff.numerical”def numerical() -> jax.tree_util.PartialEpsilonCutoff.apply
Section titled “EpsilonCutoff.apply”def apply(epsilon: float, state: StepState, logits: jax.Array) -> jax.ArrayEtaCutoff
Section titled “EtaCutoff”class EtaCutoff(epsilon: float = struct.field(pytree_node=False, default=0.0))Entropy-scaled cutoff, as EtaLogitsWarper, which takes the square
root of its float32 epsilon on the device.
EtaCutoff.numerical
Section titled “EtaCutoff.numerical”def numerical() -> jax.tree_util.PartialEtaCutoff.apply
Section titled “EtaCutoff.apply”def apply(epsilon: float, state: StepState, logits: jax.Array) -> jax.ArrayExponentialDecayLengthPenalty
Section titled “ExponentialDecayLengthPenalty”class ExponentialDecayLengthPenalty( start: int = struct.field(pytree_node=False, default=0), factor: float = struct.field(pytree_node=False, default=1.0), eos: jax.Array = struct.field(default_factory=lambda: jnp.zeros((0,), jnp.int32)),)Raises the EOS score after start drawn tokens, as ExponentialDecayLengthPenalty.
The reference measures from start_index + prompt_width, which is the
generated count used here, and adds |score| * (factor ** index - 1) so a
negative score also rises. A removed EOS (-inf, a grammar’s mask) stays
removed.
ExponentialDecayLengthPenalty.numerical
Section titled “ExponentialDecayLengthPenalty.numerical”def numerical() -> jax.tree_util.PartialExponentialDecayLengthPenalty.apply
Section titled “ExponentialDecayLengthPenalty.apply”def apply( start: int, factor: float, eos: jax.Array, state: StepState, logits: jax.Array,) -> jax.ArrayFinish
Section titled “Finish”class Finish(NamedTuple)The rows a step’s criteria finished, and those of them a criterion ended, as a stop does,
rather than one cut short at a length (Stopping’s truncates): the terminated rows.
Finishing
Section titled “Finishing”class Finishing(NamedTuple)A request’s criteria as pytrees (components), those that end a row apart from those that
cut it short at a length, which is told from each criterion as given (Truncates) before it is
one: a built-in’s numbers ride as data (as_pytree).
Finishing.of
Section titled “Finishing.of”def of(criteria: Sequence[Stopping]) -> FinishingForcedBOS
Section titled “ForcedBOS”class ForcedBOS(token: int = struct.field(pytree_node=False, default=0))Forces one token as the first of the whole sequence, as ForcedBOSTokenLogitsProcessor.
ForcedBOS.numerical
Section titled “ForcedBOS.numerical”def numerical() -> jax.tree_util.PartialForcedBOS.apply
Section titled “ForcedBOS.apply”def apply(token: int, state: StepState, logits: jax.Array) -> jax.ArrayForcedEOS
Section titled “ForcedEOS”class ForcedEOS( eos: jax.Array = struct.field(default_factory=lambda: jnp.zeros((0,), jnp.int32)), max_length: int | None = struct.field(pytree_node=False, default=None),)Forces EOS one step before the end, as ForcedEOSTokenLogitsProcessor.
eos may name several ids, all of which the forced step allows, as the
reference allows every id its tensor holds. max_length counts prompt and
generated tokens together. Left as None, the end is the request’s own
token budget (StepState.budgets), so a caller that changes the budget
per call forces EOS at the new end rather than at a length the task was
built with, and a loop that scans past the budget forces it inside it.
ForcedEOS.numerical
Section titled “ForcedEOS.numerical”def numerical() -> jax.tree_util.PartialForcedEOS.apply
Section titled “ForcedEOS.apply”def apply( eos: jax.Array, max_length: int | None, state: StepState, logits: jax.Array,) -> jax.ArrayFrequencyPenalty
Section titled “FrequencyPenalty”class FrequencyPenalty(penalty: float = struct.field(pytree_node=False, default=0.0))Subtracts penalty times each token’s count among the drawn tokens.
This is vLLM’s formula, logits -= frequency_penalties * output_bin_counts
(model_executor/layers/utils.py), which is also OpenAI’s
frequency_penalty. It counts generated tokens only, not the prompt.
FrequencyPenalty.numerical
Section titled “FrequencyPenalty.numerical”def numerical() -> jax.tree_util.PartialFrequencyPenalty.apply
Section titled “FrequencyPenalty.apply”def apply(penalty: float, state: StepState, logits: jax.Array) -> jax.ArrayGreedy
Section titled “Greedy”class Greedy()The argmax as a distribution: zero on the best token, -inf elsewhere.
Sampling(temperature=0) compiles to this, so the draw stays the
deterministic argmax with a behaviour log probability of exactly zero; a
terminal Greedy lets the sampler take the argmax without building the
point mass. Transforms before it still shape the argmax, as in greedy
search. A row without a distribution (all removed, or NaN or +inf)
passes through for the draw to refuse, rather than becoming a confident
token.
LogitsChain
Section titled “LogitsChain”class LogitsChain(transforms: tuple[LogitsTransform, ...])The ordered transforms, keeping a terminal Greedy for the sampler.
An arbitrary transform after Greedy may restore a nondegenerate
distribution, so only the final built-in Greedy proves an argmax draw.
Like a function closure, the callable is hashed by identity, not by its
captured processor arrays, when passed directly to jax.jit.
LogitsTransform
Section titled “LogitsTransform”class LogitsTransform(Protocol)A pure rewrite of [rows, vocab] scores, applied before the draw.
MaxLength
Section titled “MaxLength”class MaxLength(length: int = struct.field(pytree_node=False, default=0))Finishes a row once prompt and generated tokens reach length, as MaxLengthCriteria, which
truncates it.
MaxLength.numerical
Section titled “MaxLength.numerical”def numerical() -> jax.tree_util.PartialMaxLength.apply
Section titled “MaxLength.apply”def apply(length: int, state: StepState, tokens: jax.Array) -> jax.ArrayMaxNewTokens
Section titled “MaxNewTokens”class MaxNewTokens(count: int = struct.field(pytree_node=False, default=0))Finishes a row once it has drawn count tokens, which truncates it.
MaxNewTokens.numerical
Section titled “MaxNewTokens.numerical”def numerical() -> jax.tree_util.PartialMaxNewTokens.apply
Section titled “MaxNewTokens.apply”def apply(count: int, state: StepState, tokens: jax.Array) -> jax.ArrayMinLength
Section titled “MinLength”class MinLength( length: int = struct.field(pytree_node=False, default=0), eos: jax.Array = struct.field(default_factory=lambda: jnp.zeros((0,), jnp.int32)),)Suppresses EOS until the whole sequence reaches length, as MinLengthLogitsProcessor.
MinLength.numerical
Section titled “MinLength.numerical”def numerical() -> jax.tree_util.PartialMinLength.apply
Section titled “MinLength.apply”def apply(length: int, eos: jax.Array, state: StepState, logits: jax.Array) -> jax.ArrayMinNewTokens
Section titled “MinNewTokens”class MinNewTokens( count: int = struct.field(pytree_node=False, default=0), eos: jax.Array = struct.field(default_factory=lambda: jnp.zeros((0,), jnp.int32)),)Suppresses EOS until count tokens are drawn, as MinNewTokensLengthLogitsProcessor.
MinNewTokens.numerical
Section titled “MinNewTokens.numerical”def numerical() -> jax.tree_util.PartialMinNewTokens.apply
Section titled “MinNewTokens.apply”def apply(count: int, eos: jax.Array, state: StepState, logits: jax.Array) -> jax.Arrayclass MinP(p: float = struct.field(pytree_node=False, default=0.0))Relative filtering at p times the top probability, as MinPLogitsWarper.
MinP.numerical
Section titled “MinP.numerical”def numerical() -> jax.tree_util.PartialMinP.apply
Section titled “MinP.apply”def apply(p: float, state: StepState, logits: jax.Array) -> jax.ArrayNoRepeatNGram
Section titled “NoRepeatNGram”class NoRepeatNGram(size: int = struct.field(pytree_node=False, default=0))Bans tokens that would repeat an n-gram of the row’s own history.
This is the tensorised form of NoRepeatNGramLogitsProcessor: the current
suffix is matched against every window, and a matching window bans the
token that followed it. The window starting at the suffix itself needs one token more
than the row has, so a suffix never bans its own successor.
PresencePenalty
Section titled “PresencePenalty”class PresencePenalty(penalty: float = struct.field(pytree_node=False, default=0.0))Subtracts penalty from every token already drawn.
This is vLLM’s logits -= presence_penalties * output_mask, which is also
OpenAI’s presence_penalty. It reads generated tokens only, not the prompt.
PresencePenalty.numerical
Section titled “PresencePenalty.numerical”def numerical() -> jax.tree_util.PartialPresencePenalty.apply
Section titled “PresencePenalty.apply”def apply(penalty: float, state: StepState, logits: jax.Array) -> jax.ArrayPromptNoRepeatNGram
Section titled “PromptNoRepeatNGram”class PromptNoRepeatNGram(size: int = struct.field(pytree_node=False, default=0))Bans tokens that would repeat an n-gram of the prompt.
EncoderNoRepeatNGramLogitsProcessor builds its table from the encoder
input and matches it against the decoder’s suffix. The prompt is the
encoder input of a decoder-only model.
PromptRepetitionPenalty
Section titled “PromptRepetitionPenalty”class PromptRepetitionPenalty( penalty: float = struct.field(pytree_node=False, default=1.0),)Raises the scores of prompt tokens, as EncoderRepetitionPenaltyLogitsProcessor.
The reference inverts its argument, so a penalty above one rewards repeating the prompt. A decoder-only prompt is the encoder input here.
PromptRepetitionPenalty.numerical
Section titled “PromptRepetitionPenalty.numerical”def numerical() -> jax.tree_util.PartialPromptRepetitionPenalty.apply
Section titled “PromptRepetitionPenalty.apply”def apply(inverted: float, state: StepState, logits: jax.Array) -> jax.ArrayRemoveInvalidValues
Section titled “RemoveInvalidValues”class RemoveInvalidValues()Maps NaN to zero and infinities to the float range, as InfNanRemoveLogitsProcessor.
Nothing else in the chain repairs a broken distribution; an undefined draw raises instead. Add this transform if you want one sanitized.
Renormalize
Section titled “Renormalize”class Renormalize()Replaces scores by their log softmax, as LogitNormalization.
RepetitionPenalty
Section titled “RepetitionPenalty”class RepetitionPenalty(penalty: float = struct.field(pytree_node=False, default=1.0))Divides the positive scores of seen tokens by the penalty and multiplies negative ones.
The history is the row’s valid prompt and drawn tokens, as
RepetitionPenaltyLogitsProcessor reads the whole input_ids.
RepetitionPenalty.numerical
Section titled “RepetitionPenalty.numerical”def numerical() -> jax.tree_util.PartialRepetitionPenalty.apply
Section titled “RepetitionPenalty.apply”def apply(penalty: float, state: StepState, logits: jax.Array) -> jax.ArraySequenceBias
Section titled “SequenceBias”class SequenceBias()Adds a bias to the token that would complete each biased sequence.
This is SequenceBiasLogitsProcessor as a table: sequences is [count, width]
right-aligned token ids, lengths their real lengths and bias the value
added to the last id when the row’s suffix matches the preceding ones. A
sequence longer than the row’s history is skipped, as the reference skips
one longer than the context.
StepState
Section titled “StepState”class StepState( prompt_width: int = struct.field(pytree_node=False, default=0), budget: jax.Array | None = None,)What a transform or a criterion sees at one decode step.
tokens is the fixed-capacity buffer of the prompt followed by the draw
slots, [rows, prompt_width + budget], and valid marks the slots that
hold a real token. A row’s history is therefore its own, whatever padding
the prompt batch needed. step counts the tokens the row has committed,
active marks the rows still generating, and keys holds one PRNG key
per row. budget is the tokens each row’s request may draw, [rows],
which can be fewer than the draw slots: a task scans a bucket of trips
past it so budgets share one compiled loop, and a server’s slots hold
its whole capacity. None leaves it the draw slots.
width: int-
Slots in the buffer, prompt plus budget.
StepState.budgets
Section titled “StepState.budgets”def budgets() -> jax.ArrayReturn the tokens each row’s request may draw (budget), [rows].
StepState.total
Section titled “StepState.total”def total() -> jax.ArrayReturn the number of real tokens each row holds, prompt and generated together.
StepState.history
Section titled “StepState.history”def history() -> tuple[jax.Array, jax.Array]Return each row’s real tokens left-aligned, and how many there are.
Prompts pad wherever their batch needed it, so a transform that reads order (n-grams, biased sequences, stop strings) needs the row’s own tokens without holes. Padding slots hold -1, which no token id equals.
StepState.prompt_history
Section titled “StepState.prompt_history”def prompt_history() -> tuple[jax.Array, jax.Array]Return the prompt region’s real tokens left-aligned, and how many there are.
StepState.commit
Section titled “StepState.commit”def commit(tokens: jax.Array, drawn: jax.Array) -> StepStateReturn the state after the drawn rows append tokens at their next slot.
StopStrings
Section titled “StopStrings”class StopStrings( positions: int = struct.field(pytree_node=False, default=1), ends: int = struct.field(pytree_node=False, default=1), span: int = struct.field(pytree_node=False, default=1),)Finishes a row whose text ends with one of the compiled stop strings.
The tables come from stop_strings, which reads the tokenizer once. The
device check is StopStringCriteria’s: it goes through the row’s tokens
backwards, requires the last token to overlap the end of a stop string,
and keeps matching earlier tokens against the positions they can occupy. A match
counts only when the string touches the final token, so a string produced
earlier does not stop the row later.
Stopping
Section titled “Stopping”class Stopping(Protocol)A pure per-row finish test over the tokens a step just drew.
A criterion that cuts a row short at a length rather than ending it, as
MaxNewTokens and MaxLength do, is a Truncates: a row it finishes
is not terminated (Finish).
SuppressTokens
Section titled “SuppressTokens”class SuppressTokens()Removes a fixed set of tokens, as SuppressTokensLogitsProcessor.
Temperature
Section titled “Temperature”class Temperature(value: float = struct.field(pytree_node=False, default=1.0))Divides the logits by value, as TemperatureLogitsWarper.
Temperature.numerical
Section titled “Temperature.numerical”def numerical() -> jax.tree_util.PartialTemperature.apply
Section titled “Temperature.apply”def apply(value: float, state: StepState, logits: jax.Array) -> jax.Arrayclass TopH(fraction: float = struct.field(pytree_node=False, default=1.0))Entropy-budget filtering, as TopHLogitsWarper.
Tokens enter in probability order while the cumulative entropy of the
truncated head stays within fraction of its total entropy, and the best
token always enters, out of the reference’s fixed head of 100.
The two entropies are computed the way the reference computes them, and
they are not the same expression. The budget is
torch.distributions.Categorical.entropy, which clamps the log
probabilities to the dtype’s minimum so a removed token contributes
nothing. The running sum is the reference’s own -p * log(p), whose
removed tokens are NaN, and a NaN ends the selection because every
comparison against it is false. Substituting one for the other keeps a
token the reference drops.
TopH.numerical
Section titled “TopH.numerical”def numerical() -> jax.tree_util.PartialTopH.apply
Section titled “TopH.apply”def apply(fraction: float, state: StepState, logits: jax.Array) -> jax.Arrayclass TopK(k: int = struct.field(pytree_node=False, default=1))Keeps the k highest scores, as TopKLogitsWarper.
class TopP(p: float = struct.field(pytree_node=False, default=1.0))Nucleus filtering, as TopPLogitsWarper.
The ascending tail holding cumulative mass at most 1 - p is removed and
the best token always survives.
TopP.numerical
Section titled “TopP.numerical”def numerical() -> jax.tree_util.PartialTopP.apply
Section titled “TopP.apply”def apply(excluded_mass: float, state: StepState, logits: jax.Array) -> jax.ArrayTruncates
Section titled “Truncates”class TruncatesMarks a stopping criterion that cuts a row short at a length rather than ending it.
Typical
Section titled “Typical”class Typical(mass: float = struct.field(pytree_node=False, default=1.0))Locally typical filtering, as TypicalLogitsWarper.
Typical.numerical
Section titled “Typical.numerical”def numerical() -> jax.tree_util.PartialTypical.apply
Section titled “Typical.apply”def apply(mass: float, state: StepState, logits: jax.Array) -> jax.Array