dew.config.world_model
Dreamer 4’s world model as a run over video (docs/design/post-training.md §14.4): its tokenizer, then its dynamics over the frozen tokenizer’s latents.
Both stages read a clip’s video from the run’s data, so both are runs over data, not over an environment
(dew.objectives.rl.config). The tokenizer’s stage (TokenizerObjective) trains CausalTokenizer. The
dynamics’ stage (WorldModelObjective) trains Dynamics over a tokenizer that no field of the record can
hold as a value: its weights are the saved tokenizer run’s, read at the step the record pins
(TokenizerConfig), as an environment run reads its teacher’s.
What the data and the tokenizer state, the run derives and refuses a record that states: the tokenizer’s
frame from the data’s clips, and the dynamics’ latents and bottleneck from the tokenizer and its action
width from the data’s actions, since a width the record restates and gets wrong fails inside Flax, far from
the record.
| Name | Summary |
|---|---|
TokenizerConfig | The frozen tokenizer a dynamics run trains over: the saved run in the directory run, a run of this kind that trained a TokenizerObjective, read at its checkpoint step (a step, latest, best or best:<metric>; None for its latest), which the run pins as it starts, so its record names the step it read (WorldModelRunConfig.pinned). |
WorldModelRunConfig | A run over clips that trains Dreamer 4’s world model: the tokenizer where objective is a TokenizerObjective (the default), the dynamics where it is a WorldModelObjective, which names the saved tokenizer it trains over (tokenizer). |
TokenizerConfig
Section titled “TokenizerConfig”class TokenizerConfig(run: str, step: int | str | None = None)The frozen tokenizer a dynamics run trains over: the saved run in the directory run, a run of this
kind that trained a TokenizerObjective, read at its checkpoint step (a step, latest, best or
best:<metric>; None for its latest), which the run pins as it starts, so its record names the step it
read (WorldModelRunConfig.pinned). Its model is that run’s, built for the frames of this run’s data.
WorldModelRunConfig
Section titled “WorldModelRunConfig”class WorldModelRunConfig( model: ModelConfig = (lambda: ModelConfig('dew.objectives.world_model:CausalTokenizer'))(), data: Annotated[Dataset | None, tyro.conf.Suppress] = None, validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress] = None, data_seed: int = 0, source: SourceOptions = SourceOptions(), optim: OptimConfig = OptimConfig(), trainer: TrainerConfig = TrainerConfig(), objective: ObjectiveConfig = (lambda: ObjectiveConfig('dew.objectives.world_model:TokenizerObjective'))(), lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None = None, val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None = None, val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])] = dict(), val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])] = (), tokenizer: Annotated[TokenizerConfig, tyro.conf.subcommand('tokenizer')] | None = None,)A run over clips that trains Dreamer 4’s world model: the tokenizer where objective is a
TokenizerObjective (the default), the dynamics where it is a WorldModelObjective, which names the
saved tokenizer it trains over (tokenizer).
The data is a chain whose records hold a video [T, H, W, C] (dew.data.presets.clips, or stored
clips whose export states their shape), and beside it an actions [T, A] where the dynamics are
labeled. The objectives read frames in [-1, 1]; uint8 pixels, as the data workers write them, are
scaled to it (dew.objectives.world_model.unit_frames).
model: ModelConfig-
The network the stage trains, a
CausalTokenizeror aDynamics. The run derives each width the class declares from what the data and the tokenizer state (frame;latents,bottleneckandaction_size); a record that states one is refused. ADynamics.framesholds the data’s windows. objective: ObjectiveConfig-
The stage’s objective:
TokenizerObjective, orWorldModelObjectiveand itsshortcut, whose tokenizer, and the weights of it, aretokenizer’s. tokenizer: Annotated[TokenizerConfig, tyro.conf.subcommand('tokenizer')] | None-
The saved tokenizer run of a dynamics stage (
TokenizerConfig), turned on by its flags:--tokenizer.run runs/tokenizer --tokenizer.step 4000. A tokenizer stage names none.
WorldModelRunConfig.build_run
Section titled “WorldModelRunConfig.build_run”def build_run() -> PreparedThe stage’s network and objective for the data’s clips, on the run’s chain.
WorldModelRunConfig.clips
Section titled “WorldModelRunConfig.clips”def clips() -> tuple[tuple[int, int, int, int], int]The data’s window [T, H, W, C] and the width of its actions (0 where it holds none), as the
chain states them (Dataset.facts). Refuses data that is no clips of one size, or whose chain
states no shapes.
WorldModelRunConfig.trains
Section titled “WorldModelRunConfig.trains”def trains(kind: type[Objective]) -> NoneRefuse a model of a class other than the one kind’s constructor declares for its model.
WorldModelRunConfig.derived
Section titled “WorldModelRunConfig.derived”def derived(widths: Mapping[str, Configured]) -> dict[str, Configured]The widths the model’s class declares, which the data or the tokenizer state; refused where the
record states one of them.
WorldModelRunConfig.tokenized
Section titled “WorldModelRunConfig.tokenized”def tokenized(frame: tuple[int, int, int]) -> tuple[nn.Module, Variables]The saved tokenizer tokenizer names, built for the data’s frame from its run’s model, and its
weights at the step the record names. Refuses a dynamics stage with none, a saved run of another
kind or objective, and weights that do not fit a tokenizer of this frame.
WorldModelRunConfig.pinned
Section titled “WorldModelRunConfig.pinned”def pinned() -> SelfThis run with its sources pinned (RunConfig.pinned), and the step of its saved tokenizer named
as it resolves now (Checkpoints.pinned: the latest, or the best), so its record names the step it
read and a resume reads it again.