dew.objectives.rl.tssm
TransDreamer’s transformer state-space model (Chen et al., arXiv 2202.09481): a Dreamer world model’s dynamics as a causal transformer over the latents and actions before each step.
At step t the posterior reads the observation’s tokens alone, q(z_t | x_t), so a window’s latents are
drawn at once; the transformer reads each step’s previous latent and action, (z_{t-1}, a_{t-1}), and
answers the deterministic state h_t, from which the prior predicts z_t, p(z_t | h_t). h_t therefore
holds nothing of x_t, and the prior is a prediction. Attention stays within an episode: a step that
starts one reads a learned start embedding and attends to nothing before it. Training runs the window in
parallel; acting and imagination run the same attention a step at a time over a cache of keys and
values (Cache), so the policy and the imagined rollouts read the function the model was trained as.
The sizes default to FlyDreamer’s (deter 512, 32 categoricals of 32 classes, 4 layers of 8 heads, rotary positions, RMS-normed queries and keys, a gated feed-forward four widths wide before its 2/3).
| Name | Summary |
|---|---|
TSSM | TransDreamer’s dynamics (module docstring) over tokens [..., E] and actions of action_size. |
Cache | The keys and values a step attends to, [B, layers, C, heads, head] each, with whether each slot is valid [B, C], the slot the next step writes (index, wrapping round the C slots) and that step’s position within its episode [B]; for imagination, beside them the window its starts share. |
Window | A window’s keys and values [B, layers, T, heads, head] that imagination from each of its B * T steps attends to, held once for all of them, and which of the window’s steps each start reads [B * T, T] (those of its episode up to itself). |
class TSSM( stoch: int = 32, classes: int = 32, width: int = 512, layers: int = 4, heads: int = 8, ffup: int = 4, unimix: float = 0.01, context: int = 64, dtype: Dtype = jnp.float32,)TransDreamer’s dynamics (module docstring) over tokens [..., E] and actions of action_size.
context: int-
The steps an acting cache holds (
empty_cache), the windows’ length it trained on. dtype: Dtype-
The computation’s dtype, bfloat16 for speed; parameters stay at least float32 (
at_least_fp32), so a bfloat16 computation updates float32 weights (DreamerV3’s COMPUTE_DTYPE beside its float32 parameters), and the latents’ distributions are at least float32.
TSSM.setup
Section titled “TSSM.setup”def setup() -> NoneTSSM.posterior
Section titled “TSSM.posterior”def posterior(tokens: jax.Array) -> jax.Arrayq(z_t | x_t): the latent’s log-probabilities from an observation’s tokens alone.
TSSM.prior
Section titled “TSSM.prior”def prior(deter: jax.Array) -> jax.Arrayp(z_t | h_t): the latent’s log-probabilities predicted from the state.
TSSM.inputs
Section titled “TSSM.inputs”def inputs(stoch: jax.Array, previous: jax.Array, first: jax.Array) -> jax.ArrayEach step’s input [..., width]: its previous latent stoch [..., S, K] (each step’s own,
shifted by one along the step axis by the caller) and action, embedded, or the start embedding
where the step first starts an episode.
TSSM.states
Section titled “TSSM.states”def states(x: jax.Array, first: jax.Array) -> jax.ArrayThe states h [B, T, width] of a window’s inputs x [B, T, width], each step attending to
the steps of its episode up to itself; the window’s first step starts its own segment.
TSSM.window_cache
Section titled “TSSM.window_cache”def window_cache(x: jax.Array, first: jax.Array, horizon: int) -> CacheFor each step k of a window [B, T], the cache imagination from it attends to: the window’s keys
and values (Window, held once), of which k reads its episode’s steps up to k, and horizon slots
of its own for the steps it imagines, the next at the position after k’s.
TSSM.empty_cache
Section titled “TSSM.empty_cache”def empty_cache(rows: int, size: int) -> CacheA cache of size slots holding nothing, for acting from fresh episodes (context slots).
Within an episode’s first size steps acting reads what training’s window does; past them the
cache holds the last size steps’ keys, each computed over the context before it.
TSSM.step
Section titled “TSSM.step”def step(x: jax.Array, first: jax.Array, cache: Cache) -> tuple[jax.Array, Cache]One step’s state [B, width] from its input x [B, width], attending to the cache and to
itself; a step that starts an episode first attends to nothing cached. The step’s keys and
values are written at the cache’s next slot, the oldest where it is full.
class Cache(window: Window | None = None)The keys and values a step attends to, [B, layers, C, heads, head] each, with whether each slot is
valid [B, C], the slot the next step writes (index, wrapping round the C slots) and that step’s
position within its episode [B]; for imagination, beside them the window its starts share.
Window
Section titled “Window”class Window()A window’s keys and values [B, layers, T, heads, head] that imagination from each of its B * T
steps attends to, held once for all of them, and which of the window’s steps each start reads [B * T, T]
(those of its episode up to itself).