dew.diffusion.discrete
Masked (absorbing-state) discrete diffusion, with the interface of the Gaussian one.
The forward process replaces each token with a mask id independently, with
a probability that grows with t in [0, 1]. MaskingSchedule.alpha(t) is
the fraction of tokens still visible at t. Training minimizes the
continuous-time negative ELBO of MDLM (Sahoo et al. 2024, “Simple and
Effective Masked Diffusion Language Models”), which is the cross entropy of
the model’s prediction at the masked positions, weighted by
-alpha’(t) / (1 - alpha(t)).
Sampling reverses the process one interval at a time. A masked token is
revealed with probability (alpha(s) - alpha(t)) / (1 - alpha(t)), and a
revealed token is drawn from the model’s categorical distribution. This is
MDLM’s _ddpm_update.
DiscreteProcess provides what dew.sampling.sample needs from a process:
a time grid, an initial state and a denoiser. The denoiser’s two outputs
sit where a Gaussian denoiser returns x_0 and epsilon. They are the model’s
argmax fill of the masked positions and the log-probabilities the solver
draws from.
| Name | Summary |
|---|---|
MDLM | Builds MDLM’s masked diffusion process on the log-linear schedule. |
Confident | LLaDA2’s reveal, as its release’s generate takes it (inclusionAI/LLaDA2.0-mini@dad945c, modeling_llada2_moe.py): each step draws every masked position’s token from the model’s distribution at temperature, its top_k best kept, and reveals the positions whose drawn token holds more than threshold of that distribution where they number at least the step’s quota, else the quota’s most probable, ties taken in slot order. |
DiscreteDenoiser | Maps (x_t, t) to (argmax fill, log-probabilities) for model under params. |
DiscreteProcess | Masks tokens of a vocabulary whose mask token is mask_id. |
LogLinear | MDLM’s log-linear schedule, alpha(t) = 1 - (1 - eps) t. |
MaskingSchedule | Sets how fast tokens are masked as t grows. |
Unmask | Integrates a DiscreteProcess with MDLM’s reverse step from t to s < t. |
aligned | scores [B, S, ...] per position as the model’s predictions of each position’s token: its own for a model whose logits line up with its inputs (LLaDA’s, MDLM’s), and for one that declares shifted_logits (Dream-v0’s) the position before’s, as its reference shifts them. |
class MDLM(mask_id: int, eps: float = 0.001, endpoint: float | None = None)Builds MDLM’s masked diffusion process on the log-linear schedule.
MDLM is Sahoo et al. 2024.
endpoint: float | None-
The time the sampling grid runs to (
DiscreteProcess.endpoint), None for MDLM’s.
Confident
Section titled “Confident”class Confident( threshold: float = 0.95, temperature: float = 1.0, top_k: int | None = None,)LLaDA2’s reveal, as its release’s generate takes it (inclusionAI/LLaDA2.0-mini@dad945c,
modeling_llada2_moe.py): each step draws every masked position’s token from the model’s
distribution at temperature, its top_k best kept, and reveals the positions whose drawn token
holds more than threshold of that distribution where they number at least the step’s quota, else
the quota’s most probable, ties taken in slot order. The quota spreads the walked block’s slots, its
prompt’s among them (DiscreteDenoiser.block; the slots masked at the walk’s first step where no
block is walked), over the walk’s steps as evenly as they divide, the first steps one more, so a
block is whole by its last step. The mask token is no draw (DiscreteDenoiser).
Two choices are Dew’s where the release’s code does otherwise: temperature 0 takes each
position’s best token, as Dew’s samplers do, where the release’s generate documents greedy
decoding but still draws, unscaled; and a tie at the quota’s edge goes to the earlier slot, which
torch.topk leaves unspecified. At top_k 1, where each position’s best token is unique and the
threshold leaves no tie at the quota’s edge, the two agree, which tests/test_blockwise_walk.py
holds over its recording. A step with nothing masked reveals nothing and draws nothing.
Confident.init
Section titled “Confident.init”def init(x, times, process, *, key) -> tuple[jax.Array, jax.Array, jax.Array]The step reached, the walk’s step count, and each row’s slots the quota spreads (set at the
first step, -1 until then). A process that masks no tokens is refused, as Unmask refuses it.
Confident.step
Section titled “Confident.step”def step(x, t, t_next, denoised, log_probs, state, key, process, denoise)Confident.step_rows
Section titled “Confident.step_rows”def step_rows(x, t, t_next, denoised, log_probs, state, keys, process, denoise)step, row i drawing its tokens from keys[i] alone (RowSolver).
DiscreteDenoiser
Section titled “DiscreteDenoiser”class DiscreteDenoiser( process: DiscreteProcess, model: nn.Module, params: Variables, inputs: ModelInputs | None = None, mutable_mask: jax.Array | None = None, block: jax.Array | None = None, visible: jax.Array | None = None,)Maps (x_t, t) to (argmax fill, log-probabilities) for model under params.
The denoiser ignores t, because the masked model is conditioned on the
corruption it sees and not on the time. Unmask.step reads the time
from the process. t stays in the signature because sample calls
every denoiser as (x_t, t). A model with full-sequence logits
(Logits) is read through them, and any other through its call.
The model’s own logits at an unmasked position do not matter, since the
position keeps its token (MDLM’s carry-over parameterization). The mask
token gets zero probability, whatever score the model gives it, because
it marks corruption and a revealed position must never draw it. When
mutable_mask is given, only masked positions inside it can change.
A blockwise walk (DiscreteProcess.generate’s block) names the block it
walks, block [B, S], its prompt slots among them, which a quota of
reveals spreads over (Confident), and the keys each position reads,
visible [B, S, S]: its document’s real slots through its own block,
which the model takes as the call’s attention_pairwise_mask. Once the
walked block is revealed, a step runs no model.
DiscreteDenoiser.masked
Section titled “DiscreteDenoiser.masked”def masked(tokens)DiscreteProcess
Section titled “DiscreteProcess”class DiscreteProcess( schedule: MaskingSchedule, mask_id: int, endpoint: float | None = None,)Masks tokens of a vocabulary whose mask token is mask_id.
endpoint: float | None-
The time a release’s sampling grid runs to (Dream’s releases’
eps):timeswalkslinspace(T, endpoint, steps + 1)with its last point taken to 0, where anUnmaskstep reveals every position still masked, as their last step does (p_transfer = 1); None is MDLM’s grid toNOISE_REMOVAL_TIMEand its closing fill. T-
The fully masked end of the time domain, named
Tas in a GaussianProcess.
DiscreteProcess.sample_t
Section titled “DiscreteProcess.sample_t”def sample_t(key, n: int) -> jax.ArrayReturn n training times stratified over [SAMPLING_EPS, 1), MDLM’s antithetic draw.
Row i draws its own point in the i-th of n strata, so the weights 1 / t of one batch cover the whole trajectory. The floor at SAMPLING_EPS keeps every weight at most 1 / SAMPLING_EPS.
DiscreteProcess.stratified
Section titled “DiscreteProcess.stratified”def stratified(uniforms, strata, n: int) -> jax.Arraysample_t from a uniform for each row and its stratum of n.
DiscreteProcess.corrupt
Section titled “DiscreteProcess.corrupt”def corrupt(key, tokens, t) -> tuple[jax.Array, jax.Array]Return (masked tokens, is_masked) at t, with one t per row.
DiscreteProcess.masked
Section titled “DiscreteProcess.masked”def masked(draws, tokens, t) -> tuple[jax.Array, jax.Array]corrupt from a uniform draw for each of the tokens: those under 1 - alpha(t) masked.
DiscreteProcess.weight
Section titled “DiscreteProcess.weight”def weight(t) -> jax.ArrayReturn the NELBO weight -alpha’(t) / (1 - alpha(t)) on the masked cross entropy.
The weight is exactly zero at t = 0. Nothing is masked there, so no token contributes, and the quotient itself is undefined.
DiscreteProcess.times
Section titled “DiscreteProcess.times”def times(steps: int) -> jax.ArrayMDLM’s sampling grid for steps reveal steps: steps + 1 times from
T to NOISE_REMOVAL_TIME, at which sample’s closing argmax fills what
is still masked, as MDLM’s noise removal does; with an endpoint, the
release’s grid to it, its last point 0.
DiscreteProcess.noise
Section titled “DiscreteProcess.noise”def noise(key, shape) -> jax.ArrayReturn x_T, with every position masked.
key is not used, because the fully masked state is a single point
and needs no random draw.
DiscreteProcess.denoiser
Section titled “DiscreteProcess.denoiser”def denoiser( model: nn.Module, params: Variables, conditions: Mapping[str, Conditioning] | None = None, unconditional: Mapping[str, Conditioning] | None = None, *, inputs: ModelInputs | None = None, mutable_mask: jax.Array | None = None,) -> DiscreteDenoiserDiscreteProcess.generate
Section titled “DiscreteProcess.generate”def generate( model: nn.Module, variables: Variables, inputs: ModelInputs | jax.typing.ArrayLike, max_new_tokens: int, *, key: int | jax.Array, n: int = 1, steps: int = MDLM_STEPS, solver: Unmask | Confident | None = None, eos_token_ids: tuple[int, ...] = (), pad_token_id: int = 0, block: int | None = None,) -> CanvasGenerationGenerate max_new_tokens tokens after each prompt with native MDLM.
The whole response span starts masked, and solver (Unmask by
default) unmasks it over a grid of steps times. Each prompt row
gets n continuations in the returned CanvasGeneration. Prompt
tokens never change, including literal mask ids. An EOS token trims
the finished response, and the tokens after it become
pad_token_id, but it does not end the bidirectional refinement
early.
The response continues the prompt’s token fields
(ModelInputs.extended): its slots are text, at the positions after
the prompt’s, and the conditioning is the prompt’s own.
With block, the response is walked a block at a time, as LLaDA2’s
releases generate (their generate(block_length=)): the row’s tokens cut
into blocks of block places, counted over its document’s real tokens
from its first (dew.nn.inputs.numbered_blocks), and each block holding
response slots unmasked over its own steps with every other slot
held, in order, each position reading its own block’s keys and the
earlier blocks’ alone (the call’s attention_pairwise_mask), so a block
sees the clean blocks before it and none after. The last block is
walked whole, past max_new_tokens where it ends inside it, and the
response kept to max_new_tokens; the canvas holds that block’s
block - 1 slots past the response at most, so the model’s length
must too. A row stops once its response holds an end token: a block
after the one that revealed it is not walked. A block walks at most
max_new_tokens steps, as the release caps its steps.
Raises ValueError when the model does not read the whole row
(refuse_causal), when the prompt plus max_new_tokens exceeds the
max_seq_len it declares, or when the inputs include a token field
a response has no rule to continue.
LogLinear
Section titled “LogLinear”class LogLinear(eps: float = 0.001)MDLM’s log-linear schedule, alpha(t) = 1 - (1 - eps) t.
The masking rate -log alpha is then linear in log space and the NELBO weight is 1 / t.
LogLinear.alpha
Section titled “LogLinear.alpha”def alpha(t)LogLinear.alpha_prime
Section titled “LogLinear.alpha_prime”def alpha_prime(t)MaskingSchedule
Section titled “MaskingSchedule”class MaskingSchedule(ABC)Sets how fast tokens are masked as t grows.
alpha(t) in (0, 1] is the fraction of tokens left unmasked at t, with
alpha(0) = 1.
MaskingSchedule.alpha
Section titled “MaskingSchedule.alpha”def alpha(t) -> jax.ArrayMaskingSchedule.alpha_prime
Section titled “MaskingSchedule.alpha_prime”def alpha_prime(t) -> jax.ArrayReturn d alpha / dt, which is negative.
Unmask
Section titled “Unmask”class Unmask(temperature: float = 1.0)Integrates a DiscreteProcess with MDLM’s reverse step from t to s < t.
Each masked position is revealed with probability
(alpha(s) - alpha(t)) / (1 - alpha(t)), and a revealed position takes a
token drawn from the model’s categorical distribution at temperature,
its best token at 0 (Dream’s releases’ alg='origin' at their default
temperature 0, sample_tokens). The rest stay masked, as does a position
whose scores hold no distribution (a NaN), so a walk never commits an
arbitrary token: generation refuses a response left masked. Stepping any
other kind of process raises ValueError.
Unmask.init
Section titled “Unmask.init”def init(x, times, process, *, key) -> tupleUnmask.step
Section titled “Unmask.step”def step(x, t, t_next, denoised, log_probs, state, key, process, denoise)Unmask.step_rows
Section titled “Unmask.step_rows”def step_rows(x, t, t_next, denoised, log_probs, state, keys, process, denoise)step, row i drawing its reveals and tokens from keys[i] alone (RowSolver), as the
row sampled by itself draws them.
aligned
Section titled “aligned”def aligned( model: nn.Module, scores: jax.Array, fields: Mapping[str, jax.Array | np.ndarray | Coordinates] | None = None,) -> jax.Arrayscores [B, S, ...] per position as the model’s predictions of each position’s token: its
own for a model whose logits line up with its inputs (LLaDA’s, MDLM’s), and for one that declares
shifted_logits (Dream-v0’s) the position before’s, as its reference shifts them. A position
that opens its text, the row’s first, the first real one after padding and each packed
document’s first in the token fields’ layout (token_layout), reads its own, as the
reference repeats a row’s first: no document reads another’s.