Skip to content

dew.objectives.rl.config

The environment run as one typed record: the environment an agent acts in, the agent’s networks and objective, the rollout that collects its batches there, and a teacher that labels or demonstrates its steps (docs/design/post-training.md §14).

The environment is its class and arguments (EnvConfig) and the rollout its class and settings (RolloutConfig), each held as its import path, the arguments the run states and the class’s other defaults as they were recorded, as a run holds its model and its objective. What the environment states, the run derives: the agent’s action width, the width a policy reads of each observation and the image a world model sees. The rollout replaces every batch whole, so the run reads no data, and the trainer is handed a loader of empty rows (idle).

NameSummary
EnvConfigAn environment’s class and the arguments the run gives it, and how the run wraps it.
EnvRunConfigA run that trains an agent in an environment: the shared run fields, the environment (env), the agent’s networks (model: a WorldModel, its dynamics a nested TSSM record where it has one, or an ActorCritic, each by its import path) and objective (objective: a DreamerV3Objective or a ControlPPOObjective, by its import path, its imitation among its arguments), the rollout’s settings (rollout), an optional teacher (teacher), the saved run the agent starts from (pretrained) and the episodes each validation flies (episodes).
EpisodesConfigA validation split scored by flying the agent (dew.training.Episodes over dew.rl.evaluation.episodes): its greedy policy, or with drawn the policy’s draws as it acts in training (act), flown steps steps from fresh episodes of env, the run’s own where None, each environment’s first episode scored, logged as <split>/return and <split>/steps, and for each observation view in reached its mean at each first episode’s last step (a race’s gates passed), <split>/<view>.
RolloutConfigThe rollout that collects the run’s batches (dew.training.rollout.Rollout): name an alias of dew.registry.rollouts (dreamer, control) or an import path, held as the path, and fields the settings the run gives it, as records: a DreamerRollout’s batch, length, train_ratio, prefill, capacity and checkpointed_replay, a ControlRollout’s unroll_length, unrolls, epochs and minibatches.
TeacherConfigA privileged teacher: the saved environment run in the directory run, a ControlPPOObjective’s, read at its checkpoint step (a step, latest, best or best:<metric>; None for its latest), which the run pins as it starts, so its record names the step it read (EnvRunConfig.pinned).
idleThe loader of a run whose rollout makes every batch whole (§14): a batch of no records, one value for the step, which any mesh places whole, the same each time it opens (replayed), so a resume reads on; with no record count, so no cadence is an epoch of it.

dataclass source

class EnvConfig(
name: str,
fields: JsonDict = dict(),
defaults: JsonDict = dict(),
count: int | None = None,
episode_length: int | None = None,
action_repeat: int = 1,
)

An environment’s class and the arguments the run gives it, and how the run wraps it.

name is an alias of dew.registry.envs or an import path, held as the path: a JAX Env class, a host Simulator class, or a function or classmethod that returns one (Playground.load). fields are its arguments as given, a nested value as its record (a race’s Racing, a GateRace’s renderer as {"class": ..., "fields": ...}), set one at a time by --set env.<argument>=...; defaults holds every other argument the class defaults, as a model’s are held. An Env of one environment is wrapped as dew.rl.playground.playground wraps one: each episode cut after episode_length steps of action_repeat repeats where that is set (TimeLimit), then count of them stepped together (Batched). A Simulator is a batch of its own, and takes neither.

def build() -> Env | Simulator

The environment the record names, wrapped as it says.

dataclass source

class EnvRunConfig(
model: ModelConfig = (lambda: ModelConfig('dew.objectives.rl.dreamer:WorldModel'))(),
data: Annotated[Dataset | None, tyro.conf.Suppress] = None,
validation: Annotated[Dataset | Mapping[str, Dataset] | None, tyro.conf.Suppress] = None,
data_seed: int = 0,
source: SourceOptions = SourceOptions(),
optim: OptimConfig = OptimConfig(),
trainer: TrainerConfig = (lambda: TrainerConfig(eval_every=None, checkpoint_every=None))(),
objective: ObjectiveConfig = (lambda: ObjectiveConfig('dew.objectives.rl.dreamer:DreamerV3Objective'))(),
lora: Annotated[LoRA, tyro.conf.subcommand(lora)] | None = None,
val_metrics: Annotated[tuple[MetricConfig, ...], record_argument(tuple[MetricConfig, ...])] | None = None,
val_every: Annotated[Mapping[str, int | str], record_argument(Mapping[str, int | str])] = dict(),
val_summaries: Annotated[tuple[Summary, ...], record_argument(tuple[Summary, ...])] = (),
env: Annotated[EnvConfig, tyro.conf.subcommand('env')] | None = None,
rollout: RolloutConfig = RolloutConfig(),
teacher: Annotated[TeacherConfig, tyro.conf.subcommand('teacher')] | None = None,
pretrained: str | None = None,
episodes: Annotated[Mapping[str, EpisodesConfig], record_argument(Mapping[str, EpisodesConfig])] = dict(),
)

A run that trains an agent in an environment: the shared run fields, the environment (env), the agent’s networks (model: a WorldModel, its dynamics a nested TSSM record where it has one, or an ActorCritic, each by its import path) and objective (objective: a DreamerV3Objective or a ControlPPOObjective, by its import path, its imitation among its arguments), the rollout’s settings (rollout), an optional teacher (teacher), the saved run the agent starts from (pretrained) and the episodes each validation flies (episodes).

The environment is what the run reads, as data is a run’s, and the rollout how it collects and replays it, so a run resumed under its name may change them (RESUMABLE): a curriculum resumes the run at a new difficulty, a later phase of training at longer windows (SkyDreamer’s), the rollout owing its updates from there at its own pace. The run has no epochs, so its trainer validates and checkpoints at the cadences it names, none by default, and at its end.

model: ModelConfig

The agent’s networks. The run derives from its environment each width the class declares: its action_size, and a world model’s observation_size and image from the view it reads; a record that states one is refused. A run that starts from a saved run trains that run’s model, and names none.

objective: ObjectiveConfig

The agent’s objective and its arguments (--objective.imitation 1 --objective.imitates label); a ControlPPOObjective’s observation_size is derived, the width of the views it reads.

env: Annotated[EnvConfig, tyro.conf.subcommand('env')] | None

The environment the agent acts in (EnvConfig): --env.name flight --env.count 4096 --env.fields '{"racing": {"hz": 60}}', an argument at a time --set env.difficulty=0.5.

rollout: RolloutConfig

The rollout and its settings (RolloutConfig): --rollout.name control --rollout.fields '{"unroll_length": 64}', a setting at a time --set rollout.train_ratio=128.

teacher: Annotated[TeacherConfig, tyro.conf.subcommand('teacher')] | None

A privileged teacher’s saved run (TeacherConfig), turned on by its flags: --teacher.run runs/teacher --teacher.demonstrates.

pretrained: str | None

A saved environment run’s directory the agent starts from, its model and its weights at the step source.step selects (the latest unless it names one), which the run pins as it starts; its live weights, or its average with source.ema. A stage of a chain starts from the stage before it.

episodes: Annotated[Mapping[str, EpisodesConfig], record_argument(Mapping[str, EpisodesConfig])]

The episodes each validation flies, a split of each (EpisodesConfig) by its name, one JSON value on the command line: --episodes '{"episodes": {"steps": 1000}}'; at trainer.eval_every, or a split’s own cadence in val_every.

def build_run() -> Prepared

The agent the record names, acting in its environment through its rollout, on the loader the rollout replaces (idle), validated by its episodes where it names them.

def trained_model() -> ModelConfig

The model the run trains: its own, or where it starts from a saved run that run’s.

def agent(env: Env | Simulator, variables: Variables | None = None) -> Agent

The run’s objective over its model (trained_model) acting in env, from variables, a fresh draw where None: each width env states derived for both, and refused where the record states it. Variables that do not fit the agent are refused.

def started() -> Variables | None

The weights the agent starts from: those of the run pretrained names, at the step source.step selects, live unless source.ema; None where it names none, its source then refused (built_fresh). A model the run names beside it, or a source option other than the step and the average, is refused.

def taught(objective: Agent, env: Env | Simulator) -> dict[str, Teacher]

The rollout’s teacher and demonstrator (TeacherConfig), each the teacher run’s policy acting in env: its greedy action labelling each step where objective imitates labels, and its draws flying the prefill where it demonstrates. An objective imitating labels with no teacher, a teacher that neither labels nor demonstrates, and a teacher run of another objective than PPO’s are refused.

def pinned() -> Self

This run with its sources pinned (RunConfig.pinned), and its teacher’s step and the step of the run it starts from named as each resolves now (Checkpoints.pinned: the latest, or the best), so its record names the steps it read and a resume reads them again.

dataclass source

class EpisodesConfig(
steps: int,
env: EnvConfig | None = None,
reached: tuple[str, ...] = (),
drawn: bool = False,
)

A validation split scored by flying the agent (dew.training.Episodes over dew.rl.evaluation.episodes): its greedy policy, or with drawn the policy’s draws as it acts in training (act), flown steps steps from fresh episodes of env, the run’s own where None, each environment’s first episode scored, logged as <split>/return and <split>/steps, and for each observation view in reached its mean at each first episode’s last step (a race’s gates passed), <split>/<view>. A JAX Env is flown in one program; a Simulator from the host, and only one the record names for the split: the run’s own simulator holds the rollout’s environments mid episode, which a flight would move.

def split(objective: Agent, env: Env | Simulator) -> Episodes

The split of objective’s flights, in env where the record names no environment of its own.

dataclass source

class RolloutConfig(
name: str = 'dreamer',
fields: JsonDict = dict(),
defaults: JsonDict = dict(),
)

The rollout that collects the run’s batches (dew.training.rollout.Rollout): name an alias of dew.registry.rollouts (dreamer, control) or an import path, held as the path, and fields the settings the run gives it, as records: a DreamerRollout’s batch, length, train_ratio, prefill, capacity and checkpointed_replay, a ControlRollout’s unroll_length, unrolls, epochs and minibatches. defaults holds the class’s others as recorded. The run builds the rest, the objective, the environment and its teacher (_BUILT); a train_ratio of inf trains on the prefill alone.

def build(objective: Agent, env: Env | Simulator, **taught: Teacher = {}) -> Rollout

The rollout of objective in env, with the teacher and demonstrator taught names. Refuses a setting the run builds, an objective of another class than the rollout’s, and a teacher it takes none of.

dataclass source

class TeacherConfig(run: str, step: int | str | None = None, demonstrates: bool = False)

A privileged teacher: the saved environment run in the directory run, a ControlPPOObjective’s, read at its checkpoint step (a step, latest, best or best:<metric>; None for its latest), which the run pins as it starts, so its record names the step it read (EnvRunConfig.pinned). Its policy acts in the run’s own environment and reads there what it read in its own. It labels every step the rollout takes with its greedy action where the run’s objective imitates labels (imitates="label", DAgger), and where it demonstrates, it flies the rollout’s prefill with its policy’s draws (DART), the replay’s first steps. A teacher that does neither is refused.

function source

def idle() -> Loader

The loader of a run whose rollout makes every batch whole (§14): a batch of no records, one value for the step, which any mesh places whole, the same each time it opens (replayed), so a resume reads on; with no record count, so no cadence is an epoch of it. A run composed in Python passes it to Trainer.fit as the run kind does.