Skip to content

dew.config.world_model

Dreamer 4’s world model as a run over video (docs/design/post-training.md §14.4): its tokenizer, then its dynamics over the frozen tokenizer’s latents.

Both stages read a clip’s video from the run’s data, so both are runs over data, not over an environment (dew.objectives.rl.config). The tokenizer’s stage (TokenizerObjective) trains CausalTokenizer. The dynamics’ stage (WorldModelObjective) trains Dynamics over a tokenizer that no field of the record can hold as a value: its weights are the saved tokenizer run’s, read at the step the record pins (TokenizerConfig), as an environment run reads its teacher’s.

What the data and the tokenizer state, the run derives and refuses a record that states: the tokenizer’s frame from the data’s clips, and the dynamics’ latents and bottleneck from the tokenizer and its action width from the data’s actions, since a width the record restates and gets wrong fails inside Flax, far from the record.

NameSummary
TokenizerConfigThe frozen tokenizer a dynamics run trains over: the saved run in the directory run, a run of this kind that trained a TokenizerObjective, read at its checkpoint step (a step, latest, best or best:<metric>; None for its latest), which the run pins as it starts, so its record names the step it read (WorldModelRunConfig.pinned).
WorldModelRunConfigA run over clips that trains Dreamer 4’s world model: the tokenizer where objective is a TokenizerObjective (the default), the dynamics where it is a WorldModelObjective, which names the saved tokenizer it trains over (tokenizer).

dataclass source

class TokenizerConfig(run: str, step: int | str | None = None)

The frozen tokenizer a dynamics run trains over: the saved run in the directory run, a run of this kind that trained a TokenizerObjective, read at its checkpoint step (a step, latest, best or best:<metric>; None for its latest), which the run pins as it starts, so its record names the step it read (WorldModelRunConfig.pinned). Its model is that run’s, built for the frames of this run’s data.

dataclass source

class WorldModelRunConfig(
model: ModelConfig = (lambda: ModelConfig('dew.objectives.world_model:CausalTokenizer'))(),
data: Annotated[Dataset | None, tyro.conf.Suppress] = None,
validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress] = None,
data_seed: int = 0,
source: SourceOptions = SourceOptions(),
optim: OptimConfig = OptimConfig(),
trainer: TrainerConfig = TrainerConfig(),
objective: ObjectiveConfig = (lambda: ObjectiveConfig('dew.objectives.world_model:TokenizerObjective'))(),
lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None = None,
val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None = None,
val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])] = dict(),
val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])] = (),
tokenizer: Annotated[TokenizerConfig, tyro.conf.subcommand('tokenizer')] | None = None,
)

A run over clips that trains Dreamer 4’s world model: the tokenizer where objective is a TokenizerObjective (the default), the dynamics where it is a WorldModelObjective, which names the saved tokenizer it trains over (tokenizer).

The data is a chain whose records hold a video [T, H, W, C] (dew.data.presets.clips, or stored clips whose export states their shape), and beside it an actions [T, A] where the dynamics are labeled. The objectives read frames in [-1, 1]; uint8 pixels, as the data workers write them, are scaled to it (dew.objectives.world_model.unit_frames).

model: ModelConfig

The network the stage trains, a CausalTokenizer or a Dynamics. The run derives each width the class declares from what the data and the tokenizer state (frame; latents, bottleneck and action_size); a record that states one is refused. A Dynamics.frames holds the data’s windows.

objective: ObjectiveConfig

The stage’s objective: TokenizerObjective, or WorldModelObjective and its shortcut, whose tokenizer, and the weights of it, are tokenizer’s.

tokenizer: Annotated[TokenizerConfig, tyro.conf.subcommand('tokenizer')] | None

The saved tokenizer run of a dynamics stage (TokenizerConfig), turned on by its flags: --tokenizer.run runs/tokenizer --tokenizer.step 4000. A tokenizer stage names none.

def build_run() -> Prepared

The stage’s network and objective for the data’s clips, on the run’s chain.

def clips() -> tuple[tuple[int, int, int, int], int]

The data’s window [T, H, W, C] and the width of its actions (0 where it holds none), as the chain states them (Dataset.facts). Refuses data that is no clips of one size, or whose chain states no shapes.

def trains(kind: type[Objective]) -> None

Refuse a model of a class other than the one kind’s constructor declares for its model.

def derived(widths: Mapping[str, Configured]) -> dict[str, Configured]

The widths the model’s class declares, which the data or the tokenizer state; refused where the record states one of them.

def tokenized(frame: tuple[int, int, int]) -> tuple[nn.Module, Variables]

The saved tokenizer tokenizer names, built for the data’s frame from its run’s model, and its weights at the step the record names. Refuses a dynamics stage with none, a saved run of another kind or objective, and weights that do not fit a tokenizer of this frame.

def pinned() -> Self

This run with its sources pinned (RunConfig.pinned), and the step of its saved tokenizer named as it resolves now (Checkpoints.pinned: the latest, or the best), so its record names the step it read and a resume reads it again.