Skip to content

dew.objectives.rl.dreamer

DreamerV3 (Hafner et al., arXiv 2301.04104 v2 and Nature 2025) for vector observations, as its code computes it (danijar/dreamerv3 at e01491fad6434b2245a3b8ca201dd7faedcc458c, cited by file:line).

The world model is an RSSM: a block-diagonal GRU’s deterministic state deter (rssm.py:135-158), a posterior over stoch categoricals of classes classes from it and the encoded observation, a prior from it alone (1% unimix, straight-through draws, rssm.py:161-176), and heads off the two concatenated: the symlog-MSE decoder, the two-hot reward, the Bernoulli continue. It is trained on replayed windows by the decoder’s, reward’s and continue’s losses and the dynamics and representation KLs with one free nat (rssm.py:120-133, agent.py:156-182).

The world model is an Env whose state is the latent and whose step is the prior (LatentEnv), so the actor and critic learn in imagination through the same env_rollout acting in a simulator runs: from every replayed posterior state, 15 imagined steps (agent.py:189-201). The critic is a two-hot head trained toward the imagination’s lambda-returns and toward its own slow EMA (agent.py:382-447), and toward the replay’s lambda-returns at 0.3 (agent.py:449-480); the actor by REINFORCE on the returns less the values, over the 5th-to-95th percentile range of the returns floored at 1 (utils.py:16-72), with an entropy bonus of 3e-4.

Two differences from the code: a window starts from a zero latent at its first step, which the code’s replay context instead resumes from a latent stored at acting time; and the computation runs in float32 where the code runs bfloat16.

NameSummary
AUXILIARY_SCALEAn auxiliary head’s loss weight, FlyDreamer’s aux_state and aux_goal (configs.yaml loss_scales).
AgentMemoryAn agent’s memory between its steps in an environment: the block GRU’s deterministic state or the TSSM’s cache, held; the last posterior draw, stoch; and the action it took, previous.
ConvDecoderDreamerV3’s image decoder (rssm.py:253-355, the simple decoder with bspace 8): from the deterministic state through blocks block-diagonal maps and from the categoricals through two dense layers to the smallest map, its side the image’s over 2 to the stages, summed, normed and a SiLU; then a stage for each but the last of mults, deepest first: a 2x nearest upsampling, a convolution, an RMS norm and a SiLU; a last upsampling and convolution to the image’s channels; a sigmoid, the pixels in [0, 1].
ConvEncoderDreamerV3’s image encoder (rssm.py:179-248, the simple encoder): pixels [..., H, W, C] in [0, 255] scaled to [-0.5, 0.5], then a stage for each of mults: a kernel-square same-padded convolution to depth times the multiplier channels, a 2x2 max pool, an RMS norm and a SiLU; the last stage’s map flattened.
CriticThe two-hot value head, 3 layers then zero-initialized logits (configs.yaml value).
DreamerRolloutThe trainer’s rollout of a DreamerV3Objective (dew.training.rollout.Rollout): its agent acts in env, a batched Env driven as Driven drives it or a Simulator driven alike (DrivenSimulator), writing each step to a device replay of capacity steps an environment; each update trains on batch windows of length replayed steps while the agent takes environment steps so that each replays train_ratio steps (the code’s run.train_ratio), after prefill steps of each environment taken uniformly at random.
DreamerV3ObjectiveDreamerV3 over a WorldModel (the module docstring).
LatentEnvThe world model as an Env: its state the latent and a key, its step the GRU and a draw from the prior, its observation the latent’s feature, its reward the reward head’s two-hot mean and its terminated never set: the continue head’s probability weighs an imagined step instead (agent.py:396-405).
WorldModelDreamerV3’s networks over observations of observation_size and continuous actions of action_size in [-1, 1].
demonstration_replayDemonstrations as a replay to pretrain from, one stream of every episode in order, in the convention the agent records its own steps (Driven): an episode’s row t holds the observation t, the reward that came with it (none at its first), and the action taken at it, after it was seen (FlyDreamer’s recorder stored each action one row later, with the observation it produced, and so shifted every action-conditioned prediction).
episodes_returnThe mean undiscounted return of the first episode of each of env’s environments, an Env or a Simulator, over steps steps of the agent’s draws from fresh episodes: the whole episode where steps is its length (dew.rl.evaluation.episodes, the environment driven for the agent).
lambda_returnThe code’s lambda-returns [B, T - 1] of sequences [B, T] (agent.py:482-490): from the last bootstrap backward, each step’s reward plus the discounted blend of its bootstrap and the next return, cut where the episode ended (last) or terminated (term).
normal_entropy
normal_log_prob

attribute source

AUXILIARY_SCALE = 0.5

An auxiliary head’s loss weight, FlyDreamer’s aux_state and aux_goal (configs.yaml loss_scales).

dataclass source

class AgentMemory()

An agent’s memory between its steps in an environment: the block GRU’s deterministic state or the TSSM’s cache, held; the last posterior draw, stoch; and the action it took, previous.

Flax module source

class ConvDecoder(
depth: int = 64,
units: int = 1024,
mults: tuple[int, ...] = (2, 3, 4, 4),
kernel: int = 5,
blocks: int = 8,
dtype: Dtype = jnp.float32,
)

DreamerV3’s image decoder (rssm.py:253-355, the simple decoder with bspace 8): from the deterministic state through blocks block-diagonal maps and from the categoricals through two dense layers to the smallest map, its side the image’s over 2 to the stages, summed, normed and a SiLU; then a stage for each but the last of mults, deepest first: a 2x nearest upsampling, a convolution, an RMS norm and a SiLU; a last upsampling and convolution to the image’s channels; a sigmoid, the pixels in [0, 1].

Flax module source

class ConvEncoder(
depth: int = 64,
mults: tuple[int, ...] = (2, 3, 4, 4),
kernel: int = 5,
dtype: Dtype = jnp.float32,
)

DreamerV3’s image encoder (rssm.py:179-248, the simple encoder): pixels [..., H, W, C] in [0, 255] scaled to [-0.5, 0.5], then a stage for each of mults: a kernel-square same-padded convolution to depth times the multiplier channels, a 2x2 max pool, an RMS norm and a SiLU; the last stage’s map flattened. The defaults are configs.yaml’s enc.simple at its 200M size.

Flax module source

class Critic(dtype: Dtype = jnp.float32)

The two-hot value head, 3 layers then zero-initialized logits (configs.yaml value).

dataclass source

class DreamerRollout(
objective: DreamerV3Objective,
env: Env | Simulator,
capacity: int = 100000,
batch: int = 16,
length: int = 64,
train_ratio: float = 1024.0,
prefill: int = 64,
replay: ReplayBuffer | None = None,
checkpointed_replay: bool = False,
teacher: Teacher | None = None,
demonstrator: Teacher | None = None,
)

The trainer’s rollout of a DreamerV3Objective (dew.training.rollout.Rollout): its agent acts in env, a batched Env driven as Driven drives it or a Simulator driven alike (DrivenSimulator), writing each step to a device replay of capacity steps an environment; each update trains on batch windows of length replayed steps while the agent takes environment steps so that each replays train_ratio steps (the code’s run.train_ratio), after prefill steps of each environment taken uniformly at random. A replay given (demonstrations recorded in the agent’s convention from as many environments, [capacity, envs, ...]) is the replay it starts from and writes on, in place of the prefill. metrics reports the mean return of the episodes that ended since the last update. Given a privileged teacher (dew.rl.aggregation.Teacher), a policy of the environment’s observation, every step the rollout takes, the prefill’s too, is written with the teacher’s action at it, its label (dew.rl.aggregation.labelled), which an objective that imitates it learns: DAgger, the replay aggregating the student’s steps beside the teacher’s own flights it was given, labelled alike. Given a demonstrator (a Teacher too, a policy of the environment’s observation, such as a teacher flying its policy’s draws, DART), it flies the prefill in place of the uniform actions, so the replay starts from its demonstrations, labelled where there is a teacher; the agent’s memory passes through its steps untouched, as through the uniform ones. The prefill is flown length steps at a time, so a long prefill of images holds one window’s flight beside the replay, not its whole.

The run’s checkpoints hold the environment steps it has taken (saved), so a resumed run takes the steps it still owes; its environments begin again and its replay is made again, from replay or the prefill. A rollout owes each update from the step it goes on from at its own pace, so a run resumed with another batch, length or train ratio (a later phase’s longer windows) takes the steps of its updates from there at that pace alone. Resumed from a step that holds none of its state (one an earlier Dew saved), it owes the steps of the updates after that step alone. A cooldown branch runs on copies of its environments and replay (branch), a Simulator keeping its own (dew.rl.simulator.branched). With checkpointed_replay, its checkpoints also hold its replay, its environments mid episode and their running returns, and a resumed run samples what the uninterrupted run did, at the replay’s size a checkpoint (a JAX Env’s alone: a Simulator’s environments live outside JAX).

environment_steps: int

The steps every environment has taken so far, summed.

held_replay: ReplayBuffer | None

The replay the rollout writes and samples, None before its first call.

def saved() -> Saved | None

What the run’s checkpoints hold of the rollout (Rollout’s saved): the environment steps it has taken and those its prefills took, and with checkpointed_replay its replay, its environments mid episode and their running returns; None before its first call.

def saved_shapes() -> Mapping[str, object]

saved’s leaves’ shapes and dtypes, before any call.

def restored(tree: Saved) -> None

The rollout carrying on from a saved tree: owing the steps after those it had taken, and with checkpointed_replay from its replay and its environments as they stood.

def branch(name: str) -> Iterator[None]

A cooldown branch’s calls (Rollout’s branch), on copies of the rollout’s environments and replay, after which the rollout stands as the run left it: its live state is donated to the steps that advance it, so the run’s is copied aside first, at the replay’s size again for the branch; a begun Simulator keeps its own environments (DrivenSimulator.branch).

def sample(key: jax.Array) -> Batch

batch windows of length steps drawn from the replay as an update draws them, fetched to the host.

class source

class DreamerV3Objective(
model: WorldModel,
*,
horizon: float = 333.0,
imagination: int = 15,
lam: float = 0.95,
entropy: float = 0.0003,
free_nats: float = 1.0,
slow_decay: float = 0.98,
return_rate: float = 0.01,
variables: Variables | None = None,
scales: Mapping[str, float] | None = None,
imitation: float = 0.0,
imitates: str = 'action',
slow_targets: bool = False,
smoothness: float = 0.0,
)

DreamerV3 over a WorldModel (the module docstring). A batch holds replayed windows [B, T] of DreamerV3’s steps (Driven): observation, the action taken at it, the reward that came with it, and whether it was its episode’s first, last or terminal. The scales weigh the losses (configs.yaml loss_scales); horizon sets the continue target’s discount, 1 - 1/333; imagination is the imagined steps from each replayed state, lam the lambda-returns’ lambda, entropy the actor’s bonus, slow_decay the slow critic’s EMA and return_rate the rate of the returns’ percentile EMA. The slow critic is the EMA slot over the critic alone; with slow_targets the imagined returns bootstrap from it and the advantage is taken over it (configs.yaml slowtar, agent.py’s tarval, as SkyDreamer sets it). A decoder scale of 0 rebuilds no observation: an informed world model’s auxiliary heads decode privileged entries alone (SkyDreamer, Sec II-B). smoothness weighs the squared change of the policy’s mean between imagined steps, added to the policy’s loss (SkyDreamer, arXiv 2510.14783 Eq 11, its 0.002: on the mean alone, so the entropy bonus is untouched).

resumable

The entropy bonus weighs how the policy explores, which SkyDreamer lowers for its last phase; the horizon, lambda and the rest say what the critic and world model estimate.

def held_variables() -> Variables | None
def fresh_variables(key: jax.Array, held: Variables | None) -> Variables
def observe(
model: WorldModel,
deter: jax.Array,
stoch: jax.Array,
observation: Observation,
previous: jax.Array,
first: jax.Array,
) -> tuple[jax.Array, jax.Array, jax.Array]

One posterior step: the latent and action masked where an episode starts, the GRU, then the posterior’s draw; the new latent, beside the posterior’s log-probabilities.

def observed(
variables: Variables,
observation: Observation,
previous: jax.Array,
first: jax.Array,
key: jax.Array,
) -> tuple[jax.Array, jax.Array, jax.Array]

The posterior latents over windows [B, T, ...] from a zero latent: each step’s deterministic state, drawn categoricals and posterior log-probabilities, each step drawing from its own of T keys split from key, as the code’s scan seeds each step’s sample (rssm.py:63-90).

def act(
variables: Params,
observed: Observation,
key: jax.Array,
memory: Memory,
) -> tuple[jax.Array, Tree, AgentMemory]

The agent’s step in an environment (dew.rl.env.Policy over Driven): the posterior given the step’s observation, then an action drawn from the policy, clipped to [-1, 1].

def greedy(
variables: Params,
observed: Observation,
key: jax.Array,
memory: Memory,
) -> tuple[jax.Array, Tree, AgentMemory]

act’s step with the policy’s mean in place of its draw, as a control objective’s greedy scores a policy: the latent is still the posterior’s draw, as the world model reads it.

def memory(count: int) -> AgentMemory

The agent’s memory at the start for each of count environments: a zero latent, or a TSSM’s empty cache of its context steps, and a zero previous latent and action.

def model_flops(model, batch: Batch) -> float

One update’s forward work on batch, its loss’s traced program’s (traced_update_flops).

def training_flops(model, batch: Batch) -> float

One update’s work on batch, its loss’s gradient’s traced program’s (traced_update_flops).

def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux]

dataclass source

class LatentEnv(model: WorldModel = struct.field(pytree_node=False))

The world model as an Env: its state the latent and a key, its step the GRU and a draw from the prior, its observation the latent’s feature, its reward the reward head’s two-hot mean and its terminated never set: the continue head’s probability weighs an imagined step instead (agent.py:396-405). It starts where it is given (at), never by reset.

def at(
deter: jax.Array,
stoch: jax.Array,
key: jax.Array,
cache: Cache | None = None,
) -> tuple[_Latent, jax.Array]

The latent imagination starts from, with the cache a TSSM’s next step attends to.

def reset(key: jax.Array) -> tuple[_Latent, TimeStep]
def step(state: _Latent, action: jax.Array) -> tuple[_Latent, TimeStep]

Flax module source

class WorldModel(
image: tuple[int, int, int] | None = None,
conv_depth: int = 64,
reads: str | None = None,
proprio: tuple[tuple[str, int], ...] = (),
auxiliary: tuple[tuple[str, int, int], ...] = (),
dynamics: TSSM | None = None,
deter: int = 512,
hidden: int = 64,
stoch: int = 32,
classes: int = 4,
blocks: int = 8,
units: int = 64,
bins: int = 255,
unimix: float = 0.01,
min_std: float = 0.1,
max_std: float = 1.0,
dtype: Dtype = jnp.float32,
)

DreamerV3’s networks over observations of observation_size and continuous actions of action_size in [-1, 1]. The defaults are the code’s 1M model (configs.yaml size1m, which its DM Control proprio preset trains): a 512-wide deterministic state in 8 blocks, 32 categoricals of 4 classes, 64 hidden and 64 units a layer.

image: tuple[int, int, int] | None

An image observation’s shape (H, W, C), uint8 pixels, its sides divisible by 16: encoded and decoded by convolutions (ConvEncoder, ConvDecoder, conv_depth channels); None for a vector of observation_size.

reads: str | None

Where observations are dicts, the entry the encoder reads and the decoder rebuilds (FlyDreamer’s image); the others are privileged, read by the auxiliary heads alone.

proprio: tuple[tuple[str, int], ...]

Vector entries the encoder reads beside the image, each (entry, width), symlogged, concatenated and through an MLP beside the convolutions (DreamerV3’s encoder over mlp_keys, nets.py; SkyDreamer’s body rates, motor speeds and flight plan, arXiv 2510.14783 Fig 3): sensed, so the policy reads them; decoding them is the auxiliary heads’.

auxiliary: tuple[tuple[str, int, int], ...]

Privileged entries the latent learns to predict, each (entry, width, units): two layers of units then a linear map to the entry’s width, its squared error summed (FlyDreamer’s aux_state_head and aux_goal_head, dreamerv3/agent.py:72-78 and :207-217, configs.yaml {layers: 2, output: mse}).

dynamics: TSSM | None

TransDreamer’s transformer dynamics (dew.objectives.rl.tssm.TSSM) in place of the block GRU, its own posterior and prior; None for the GRU’s (RSSM).

dtype: Dtype

The networks’ computation dtype, bfloat16 for speed (DreamerV3’s COMPUTE_DTYPE): parameters stay at least float32, and norms, distributions, heads’ outputs and losses at least float32. A TSSM’s dynamics compute in its own dtype.

def setup() -> None
def example(batch: int) -> jax.Array | dict[str, jax.Array]

batch zero observations as the encoder reads them (seen): a vector, an image, or an image beside its proprioceptive entries.

def initial(batch: tuple[int, ...]) -> tuple[jax.Array, jax.Array]
def seen(observation: Observation) -> jax.Array | dict[str, jax.Array]

What the encoder reads of an observation: its reads view, or the observation whole; beside an image, its proprio entries concatenated (example’s structure).

def predicted(name: str, feature: jax.Array) -> jax.Array

The auxiliary head name’s prediction of its privileged entry.

def embed(observation: Observation) -> jax.Array

The encoder’s tokens of an observation (seen): a vector’s symlogged first (rssm.py:218), an image’s pixels through the convolutions, its proprioceptive entries symlogged through an MLP and joined to them.

def core(deter: jax.Array, stoch: jax.Array, action: jax.Array) -> jax.Array

The block GRU’s next deterministic state (rssm.py:135-158).

def posterior(deter: jax.Array | None, tokens: jax.Array) -> jax.Array

The posterior’s log-probabilities [..., stoch, classes] (rssm.py:75-92); the TSSM’s reads the tokens alone, the block GRU’s the deterministic state too.

def prior(deter: jax.Array) -> jax.Array
def categorical(log_probs: jax.Array) -> jax.Array
def feature(deter: jax.Array, stoch: jax.Array) -> jax.Array
def decode(deter: jax.Array, stoch: jax.Array) -> jax.Array

The observation decoded from a latent: a vector’s symlogged values, an image’s pixels in [0, 1].

def reconstruction(
deter: jax.Array,
stoch: jax.Array,
observation: Observation,
) -> jax.Array

The decoder’s squared error, summed over the observation: against a vector’s symlog, or an image’s pixels over 255 (rssm.py:345-355, outs.MSE summed over the image’s axes); the proprioceptive entries are the auxiliary heads’ to decode.

def reward(feature: jax.Array) -> jax.Array
def continue_(feature: jax.Array) -> jax.Array
def policy(feature: jax.Array) -> tuple[jax.Array, jax.Array]

The bounded normal’s mean, tanh of its head, and its deviation between the floor and the ceiling (heads.py:146-155).

function source

def demonstration_replay(episodes: Sequence[Mapping[str, np.ndarray]]) -> ReplayBuffer

Demonstrations as a replay to pretrain from, one stream of every episode in order, in the convention the agent records its own steps (Driven): an episode’s row t holds the observation t, the reward that came with it (none at its first), and the action taken at it, after it was seen (FlyDreamer’s recorder stored each action one row later, with the observation it produced, and so shifted every action-conditioned prediction). An episode is observation, action and reward along its steps (an observation an array or a dict of them), and terminal where it ended in a terminal state (only its last step can).

function source

def episodes_return(
objective: DreamerV3Objective,
env: Env | Simulator,
variables: Variables,
key: jax.Array,
steps: int,
) -> float

The mean undiscounted return of the first episode of each of env’s environments, an Env or a Simulator, over steps steps of the agent’s draws from fresh episodes: the whole episode where steps is its length (dew.rl.evaluation.episodes, the environment driven for the agent).

function source

def lambda_return(
last: jax.Array,
term: jax.Array,
reward: jax.Array,
value: jax.Array,
boot: jax.Array,
discount: float,
lam: float,
) -> jax.Array

The code’s lambda-returns [B, T - 1] of sequences [B, T] (agent.py:482-490): from the last bootstrap backward, each step’s reward plus the discounted blend of its bootstrap and the next return, cut where the episode ended (last) or terminated (term).

function source

def normal_entropy(std: jax.Array) -> jax.Array

function source

def normal_log_prob(action: jax.Array, mean: jax.Array, std: jax.Array) -> jax.Array