dew.rl.replay
A replay of environment steps held on the device, written and sampled by jittable functions.
The replay keeps the last capacity steps of each of envs environments, a ring per environment
over a pytree of step fields [capacity, envs, ...], its leading axes shardable over the
environment axis as the batch is. sample draws uniformly among the windows of consecutive steps
the replay holds whole (DreamerV3’s uniform replay, arXiv 2301.04104, of length-step chunks).
| Name | Summary |
|---|---|
ReplayBuffer | The replay’s state: the held steps’ leaves [capacity, envs, ...], the slot the next step writes (cursor) and the steps written so far, at most capacity (filled). |
Steps | A pytree of step fields, each leaf leading with its step and environment axes. |
ReplayBuffer
Section titled “ReplayBuffer”class ReplayBuffer()The replay’s state: the held steps’ leaves [capacity, envs, ...], the slot the next step writes
(cursor) and the steps written so far, at most capacity (filled).
ReplayBuffer.empty
Section titled “ReplayBuffer.empty”def empty(example: Steps, capacity: int) -> ReplayBufferAn empty replay of capacity steps of what example, one step [envs, ...], holds.
ReplayBuffer.add
Section titled “ReplayBuffer.add”def add(steps: Steps) -> ReplayBufferThe replay after steps, leaves [T, envs, ...] in order, the oldest overwritten.
ReplayBuffer.sample
Section titled “ReplayBuffer.sample”def sample(key: jax.Array, batch: int, length: int) -> Stepsbatch windows of length consecutive steps [batch, length, ...], each from an
environment and a start drawn uniformly among those the replay holds whole, its steps in the
order they were written. The replay must hold length steps.
A pytree of step fields, each leaf leading with its step and environment axes.