dew.rl.distributions
DreamerV3’s output transforms and distributions on plain arrays (arXiv 2301.04104 and its code, danijar/dreamerv3 at e01491fad6434b2245a3b8ca201dd7faedcc458c, embodied/jax/nets.py and outs.py).
symlog compresses a target’s magnitude and symexp inverts it. A two-hot distribution over the
bins symexp(linspace(-20, 20, 255)) predicts a scalar of any scale as its bins’ expectation, summed
from both ends toward the middle so uniform probabilities predict zero exactly, and is trained by
the cross-entropy of a target’s two-hot encoding, its weight split between its two neighbouring bins.
A categorical latent mixes its softmax with 1% uniform (unimix) and is drawn one-hot, its gradient
passed straight through the probabilities.
| Name | Summary |
|---|---|
onehot_sample | A one-hot draw from the categorical log_probs, its gradient the probabilities’ (straight through). |
symexp | sign(x) (exp(|x|) - 1), symlog’s inverse. |
symlog | sign(x) log(1 + |x|). |
twohot_bins | The two-hot bins: symexp of count points evenly spaced over [-limit, limit], the negative half mirrored so the bins are exactly symmetric (embodied/jax/heads.py:132-144). |
twohot_loss | The cross-entropy of softmax(logits) against target’s two-hot encoding over bins: its weight on the bins either side of it, each the other’s share of the distance between them, all of it on the first or last bin past the range (outs.py:311-323). |
twohot_mean | The expectation of bins under softmax(logits), summed outward-in from both ends to the middle bin, so symmetric bins under uniform probabilities give exactly zero (outs.py:285-309). |
unimix_log_probs | A categorical’s log-probabilities, its softmax mixed with unimix of the uniform: without the mixture the log-softmax itself, finite wherever the logits are, where the log of a probability that underflowed would be -inf. |
onehot_sample
Section titled “onehot_sample”def onehot_sample(key: jax.Array, log_probs: jax.Array) -> jax.ArrayA one-hot draw from the categorical log_probs, its gradient the probabilities’ (straight
through).
symexp
Section titled “symexp”def symexp(x: jax.Array) -> jax.Arraysign(x) (exp(|x|) - 1), symlog’s inverse.
symlog
Section titled “symlog”def symlog(x: jax.Array) -> jax.Arraysign(x) log(1 + |x|).
twohot_bins
Section titled “twohot_bins”def twohot_bins(count: int = 255, limit: float = 20.0) -> jax.ArrayThe two-hot bins: symexp of count points evenly spaced over [-limit, limit], the negative half
mirrored so the bins are exactly symmetric (embodied/jax/heads.py:132-144).
twohot_loss
Section titled “twohot_loss”def twohot_loss(logits: jax.Array, target: jax.Array, bins: jax.Array) -> jax.ArrayThe cross-entropy of softmax(logits) against target’s two-hot encoding over bins: its
weight on the bins either side of it, each the other’s share of the distance between them, all
of it on the first or last bin past the range (outs.py:311-323).
twohot_mean
Section titled “twohot_mean”def twohot_mean(logits: jax.Array, bins: jax.Array) -> jax.ArrayThe expectation of bins under softmax(logits), summed outward-in from both ends to the
middle bin, so symmetric bins under uniform probabilities give exactly zero (outs.py:285-309).
unimix_log_probs
Section titled “unimix_log_probs”def unimix_log_probs(logits: jax.Array, unimix: float = 0.01) -> jax.ArrayA categorical’s log-probabilities, its softmax mixed with unimix of the uniform: without the
mixture the log-softmax itself, finite wherever the logits are, where the log of a probability that
underflowed would be -inf.