dew.objectives.world_model
Dreamer 4’s world model at a small scale (arXiv 2509.24527, docs/research/dreamer4.md): a causal tokenizer of video frames, and an interactive dynamics transformer over its latents trained by shortcut forcing (Eqs. 4, 6-8) or, as the baseline the paper improves on, diffusion forcing.
The tokenizer turns each frame’s patches and latents learned tokens into latents x bottleneck
values through a tanh, masking each patch with probability drawn from U(0, 0.9) per frame (MAE); it
reads frames one at a time, so it is causal in time. The dynamics model reads each timestep’s
action token, shortcut token (its signal level’s and step size’s indices) and the latents regrouped
into spatial tokens, attending block-causally (a timestep’s tokens to each other and to the past),
and predicts each frame’s clean latents (x-prediction).
Choices the paper leaves open (docs/research/dreamer4.md, last section) are this module’s: the
widths and depths, K_max = 64, one attention over space and time in every layer, learned position
embeddings, the ramp weight multiplying each frame’s loss, no LPIPS term in the tokenizer’s loss,
and context frames at signal level 1 - tau_ctx when generating.
| Name | Summary |
|---|---|
CausalTokenizer | A frame [..., H, W, C] in [-1, 1] to latents [..., latents, bottleneck] and back, one frame at a time (Section 3.1): patches of patch pixels and latents learned tokens through depth blocks, the latents to bottleneck channels through a tanh; the decoder’s readout tokens, one a patch, attend to themselves and the latents, the latents only to latents. |
Dynamics | The interactive dynamics transformer (Section 3.2) over frames of latents x bottleneck tokenizer latents regrouped into spatial tokens, with one action token a step (a learned embedding plus a projection of action_size actions, none for unlabeled video) and one shortcut token, the embeddings of its signal level’s and step size’s grid indices at levels levels (K_max = 2 ** (levels - 1)), predicting each frame’s clean latents. |
TokenizerObjective | The tokenizer’s MAE reconstruction (Eq. |
WorldModelObjective | Trains Dynamics over a frozen tokenizer’s latents of a batch’s video (and actions, where the model reads them) by shortcut forcing, or with shortcut False by diffusion forcing. |
diffusion_forcing_loss | The baseline: each frame at its own signal level, its x-prediction regressed at the smallest step size alone, the ramp weighting it as shortcut forcing’s. |
ramp | Eq. |
sampled_steps | Eq. |
shortcut_forcing_loss | Eqs. |
unit_frames | A batch’s video as the objectives read it, floats in [-1, 1]: uint8 pixels, as the data workers write clips (presets.clips), are scaled to it (dew.inputs.unit_range), and floats are taken as the caller made them, already in the range. |
CausalTokenizer
Section titled “CausalTokenizer”class CausalTokenizer( patch: int = 4, width: int = 64, depth: int = 2, heads: int = 4, latents: int = 16, bottleneck: int = 8,)A frame [..., H, W, C] in [-1, 1] to latents [..., latents, bottleneck] and back, one frame
at a time (Section 3.1): patches of patch pixels and latents learned tokens through depth
blocks, the latents to bottleneck channels through a tanh; the decoder’s readout tokens, one a
patch, attend to themselves and the latents, the latents only to latents.
CausalTokenizer.setup
Section titled “CausalTokenizer.setup”def setup() -> NoneCausalTokenizer.encode
Section titled “CausalTokenizer.encode”def encode( frames: jax.Array, mask_rate: jax.Array | None = None, key: jax.Array | None = None,) -> jax.ArrayEach frame’s latents; with mask_rate [...], each patch replaced by the mask token with
that frame’s probability, drawn from key.
CausalTokenizer.decode
Section titled “CausalTokenizer.decode”def decode(latents: jax.Array) -> jax.ArrayDynamics
Section titled “Dynamics”class Dynamics( latents: int = 16, bottleneck: int = 8, spatial: int = 8, action_size: int = 0, width: int = 128, depth: int = 4, heads: int = 4, levels: int = 7, frames: int = 32,)The interactive dynamics transformer (Section 3.2) over frames of latents x bottleneck
tokenizer latents regrouped into spatial tokens, with one action token a step (a learned
embedding plus a projection of action_size actions, none for unlabeled video) and one shortcut
token, the embeddings of its signal level’s and step size’s grid indices at levels levels
(K_max = 2 ** (levels - 1)), predicting each frame’s clean latents.
TokenizerObjective
Section titled “TokenizerObjective”class TokenizerObjective(model: CausalTokenizer, *, max_mask: float = 0.9)The tokenizer’s MAE reconstruction (Eq. 5 without LPIPS) of a batch’s video [B, T, H, W, C],
each frame’s mask rate drawn from U(0, 0.9).
TokenizerObjective.fresh_variables
Section titled “TokenizerObjective.fresh_variables”def fresh_variables(key: jax.Array, held: Variables | None) -> VariablesTokenizerObjective.loss
Section titled “TokenizerObjective.loss”def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux]WorldModelObjective
Section titled “WorldModelObjective”class WorldModelObjective( model: Dynamics, tokenizer: CausalTokenizer, tokenizer_variables: Variables, *, shortcut: bool = True,)Trains Dynamics over a frozen tokenizer’s latents of a batch’s video (and actions, where
the model reads them) by shortcut forcing, or with shortcut False by diffusion forcing.
WorldModelObjective.fresh_variables
Section titled “WorldModelObjective.fresh_variables”def fresh_variables(key: jax.Array, held: Variables | None) -> VariablesWorldModelObjective.latents
Section titled “WorldModelObjective.latents”def latents(video: jax.Array) -> jax.ArrayWorldModelObjective.loss
Section titled “WorldModelObjective.loss”def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux]WorldModelObjective.generate
Section titled “WorldModelObjective.generate”def generate( variables: Variables, context: jax.Array, frames: int, key: jax.Array, *, steps: int = 4, tau_ctx: float = 0.1, actions: jax.Array | None = None,) -> jax.Arrayframes latent frames after the context latents [B, C, ...], one at a time, each from
noise in steps steps of d = 1 / steps (a shortcut model’s step size; the diffusion-forcing
baseline is told the smallest), the context at signal level 1 - tau_ctx.
diffusion_forcing_loss
Section titled “diffusion_forcing_loss”def diffusion_forcing_loss( f: Predictor, clean: jax.Array, noise: jax.Array, tau: jax.Array, d_min: float,) -> jax.ArrayThe baseline: each frame at its own signal level, its x-prediction regressed at the smallest step size alone, the ramp weighting it as shortcut forcing’s.
def ramp(tau: jax.Array) -> jax.ArrayEq. 8’s weight, 0.9 tau + 0.1.
sampled_steps
Section titled “sampled_steps”def sampled_steps( key: jax.Array, shape: tuple[int, ...], k_max: int,) -> tuple[jax.Array, jax.Array]Eq. 4: each frame’s step size d, the reciprocal of a power of two up to k_max drawn uniformly,
and its signal level tau drawn uniformly from {0, d, …, 1 - d}.
shortcut_forcing_loss
Section titled “shortcut_forcing_loss”def shortcut_forcing_loss( f: Predictor, clean: jax.Array, noise: jax.Array, tau: jax.Array, d: jax.Array, d_min: float,) -> jax.ArrayEqs. 6-8: each frame’s loss [B, T], its x-prediction’s squared error at the smallest step and
else its velocity’s against the average of its own two half steps, held still and scaled back to
x-space by (1 - tau)^2, every frame weighted by the ramp.
unit_frames
Section titled “unit_frames”def unit_frames(video: jax.typing.ArrayLike) -> jax.ArrayA batch’s video as the objectives read it, floats in [-1, 1]: uint8 pixels, as the data workers write
clips (presets.clips), are scaled to it (dew.inputs.unit_range), and floats are taken as the caller
made them, already in the range.