dew.objectives.rl.scheduler
Collect rollouts and admit complete groups into the trainer’s batches, synchronously or ahead of it.
RolloutScheduler is the trainer’s Rollout for token sessions. Each task
becomes one group of groups rollouts, which the scheduler admits, fits
into its rows, packs (sessions.pack) and reports, whichever way they were
acquired:
Synchronous(collector)draws every task’s sessions at the call from the variables the step trains (sessions.Collector):PolicyCollectorfor single-turn prompts,EpisodeCollectorfor tool episodes in lock step. Its draws are one program every process of the trainer’s mesh enters, so every process collects and the first reader of each data share packs. Their raw likelihoods are the proximal policy’s, soold_log_probsare those and no rescoring runs. A collected sample is never drawn again: a group with an INFRA_ERROR or CANCELLED session is abandoned.RolloutScheduler.sampledandRolloutScheduler.episodesbuild this case.Asynchronous(source, weights, objective, ...)keeps rollouts in flight from anySessionSource(an in-process environment, a single-turn prompt set, a harness behind a recording gateway) within a staleness bound, as below.
Asynchronously, wrap the task dataset with scheduler.tasks(dataset). The
wrapped stream registers each task batch when the trainer’s prefetch reads
it. When the trainer then passes batch i to the scheduler, the scheduler
submits batches i + 1 ... i + ahead under the weight version the engines
serve at that moment. Batch i’s own rollouts were submitted ahead calls
earlier and have kept running since, through any weight pushes in between.
The scheduler reads no data beyond the trainer’s own prefetch, so the data
position in a checkpoint is still the trainer’s. A resumed run reads again
and resubmits whatever was in flight, and reopening the stream cancels the
rollouts the old stream left running.
The scheduler relabels each rollout with its own group id, sample index and attempt number, so a resubmitted sample rejoins its group. Each finished rollout is admitted or replaced according to its status:
- COMPLETED and AGENT_ERROR are admitted, because the verifier scored them.
- TRUNCATED is admitted, and
truncationsets how it trains:mask(the default, for agentic context, turn and wall-clock limits),scoreon its verifier reward (single-turn RLVR), orzero(a length penalty); seedew.objectives.rl.sessions. Underscore, a truncated rollout without a reward is resubmitted as a failed attempt (causeunscored), like an INFRA_ERROR. Its verifier never ran, so masking it would drop a sample for a reason unrelated to the policy, and there is no reward to train it on. - INFRA_ERROR and CANCELLED are never scored. Asynchronously, the sample is
submitted again under the weights being served. After
max_attemptsfailures of one sample, the scheduler abandons the whole group instead of training it incomplete. - A rollout whose oldest call is more than
max_lagupdates behind is discarded and resubmitted. A rollout still running when its submission falls past that bound is cancelled without waiting for it to finish, because its first call was made under the submission’s version.
The scheduler keeps no clock of its own: a deadline belongs to the
operation that can stop when it passes. The remote rollout servers bound
each request, a sandboxed environment (SubprocessEnvironment) its
wall_seconds, and HarnessSource each trial (trial_timeout). A rollout
that runs out of time there comes back as an INFRA_ERROR, which the
scheduler retries. A reward’s program past its sandbox’s wall clock is a
TIMEOUT verdict that the reward scores, not a failure.
A source that raises an exception instead of returning a status is broken. The scheduler cancels the batch’s remaining rollouts and lets the exception propagate.
Two options cut the long tail of slow rollouts, and both keep the batch
shape fixed. With oversample, each group runs that many extra samples and
is admitted once its first groups rollouts finish; the scheduler cancels
the rest. This is APRIL’s active partial rollouts (arXiv:2509.18521)
without the carry-over. With admit below the task batch size, the
scheduler admits the first admit groups to complete and fit, and cancels
the others, as slime’s over-sampling batch does. Both select by completion
time, which favors short rollouts; that bias is the price of not waiting on
the tail. With varied, DAPO’s dynamic sampling (arXiv:2503.14476 equation
11 and Algorithm 1), a complete group whose members’ returns all agree is
left out, since it gives each of them a zero advantage, and admission goes
on through the batch’s other groups, drawn under the same weights, until
admit groups are admitted or the batch has none left: a batch of more
tasks than admit is DAPO’s generation batch over its training batch. With
generation_batches above one, a call still short of its groups draws the
data stream’s next batch, as many as that in all, and admits from its groups
too, until the step’s batch is full (DAPO’s Algorithm 1, verl’s
max_num_gen_batches): every process draws while any share’s first reader
is short, so the streams read alike, and the step’s checkpoint saves the
stream’s position after the last batch drawn.
Rollouts that are running when new weights are pushed keep running. Their later calls are stamped with the new version, and a rollout’s staleness is that of its oldest call, as in Kimi K2’s partial rollouts (arXiv:2507.20534 section 3.3.4).
Asynchronously, the scheduler pushes weights through weights when the
served version falls sync_every updates behind the trainer, or is ahead
of it, as after a resume from an earlier checkpoint. A first submission is
therefore at most ahead + sync_every - 1 updates stale when the trainer
consumes it, and the constructor refuses a max_lag below that. If a
complete group’s chains do not fit in rows beside the groups admitted
before it, the scheduler cuts that group, so an overfull batch never
reaches the step. The proximal policy is the trainer’s current weights. The
scheduler rescores the packed rows under them with
GRPOObjective.packed_log_probs (decoupled PPO, AReaL arXiv:2505.24298).
Sources report only behavior likelihoods, and the objective’s
behavior_importance weights each token by the ratio of its proximal to
its behavior likelihood.
In a multi-process trainer, every process runs its own scheduler over the
task rows its data stream reads, and packs its own rows. The step’s batch
is all the processes’ rows together, sharded as the trainer shards any
batch, and no rollout moves between processes. The processes act together
at a few points in each call, each an agreement point over the trainer’s
mesh where every process raises if one failed, in this order: the call’s
preparation, the acquisition and its admission, the global estimator’s
moments, which every share’s pool so its advantages are whitened over the
step’s whole batch, the packing, the engine records’ layout, the first
reader’s rows and the proximal rescoring; with a reward-side reference
penalty (reference), the moments wait for it, and the rows’ scoring
under the reference, the moments, the repacked advantages and the first
reader’s rows follow the rescoring. A
weight push is one call every process makes, so the publisher has to agree
on it across processes. When a tensor or sequence axis spans processes,
several processes read the same share of the data. Then only the share’s
first reader packs, and the others train on the rows it packed
(first_reader_batch), its recorded likelihoods included, which
independent draws would not give them. An asynchronous source samples on
the first reader alone; a collector’s draw is one program, which every
reader enters. Without a mesh nothing is agreed, and each process collects
on its own.
| Name | Summary |
|---|---|
Asynchronous | Keeps rollouts from source in flight ahead of the trainer (the module docstring’s asynchronous case), pushing the trainer’s weights through weights. |
Publisher | Serves the trainer’s weights to the rollout engines. |
RolloutScheduler | Admits complete groups of rollouts from acquisition and packs them into the trainer’s batches. |
SchedulerRecord | What one trainer call consumed and what it cost. |
Synchronous | Draws each call’s groups at the call from the trainer’s own variables through collector (the module docstring’s synchronous case); Snapshot is what the collector draws from. |
task_ids | Return one task per integer task_id row of batch (local_task_ids), named by the id in decimal. |
Asynchronous
Section titled “Asynchronous”class Asynchronous( source: SessionSource, weights: Publisher, objective: GRPOObjective, *, width: int, rows: int, ahead: int = 1, max_lag: int = 1, sync_every: int = 1, max_attempts: int = 3,)Keeps rollouts from source in flight ahead of the trainer (the module docstring’s
asynchronous case), pushing the trainer’s weights through weights.
width and rows fix the packed batch shape; rows is this process’s
share of a multi-process trainer’s batch. objective rescores the
packed rows under the trainer’s weights, the proximal policy, and a
max_lag above 0 needs its behavior_importance (a TIS cap or an
IcePop band); the constructor raises ValueError without it.
Asynchronous.rows
Section titled “Asynchronous.rows”def rows(tasks: int, samples: int) -> intAsynchronous.tasks
Section titled “Asynchronous.tasks”def tasks(dataset: Loader, tasks_of: Callable[[Batch], Sequence[Task]]) -> Loaderdataset with a training stream that registers each task batch ahead of the step.
Opening the stream again, as a resume does, cancels every rollout the previous stream left in flight.
Asynchronous.close
Section titled “Asynchronous.close”def close() -> NoneCancel every rollout in flight. The caller closes the source; this method does not.
Asynchronous.begin
Section titled “Asynchronous.begin”def begin(snapshot: Snapshot, partition: DataPartition) -> NoneFold the call’s key for the engines and push the weights where the served version is
sync_every behind the trainer’s, or ahead of it.
The engines draw with JAX’s default PRNG on devices of their own, so
every word of the trainer’s key, of whatever implementation, folds
into an uncommitted default key, and this data share’s index after
them, so two shares’ readers draw apart. A served version ahead of
updates is out of date as well: a fit restored from an earlier
checkpoint must not sample from weights newer than its own.
Asynchronous.entries
Section titled “Asynchronous.entries”def entries(batch: Batch, tasks_of: Callable[[Batch], Sequence[Task]]) -> list[_Entry]Asynchronous.acquires
Section titled “Asynchronous.acquires”def acquires(partition: DataPartition) -> boolThe share’s first reader: independent draws would hand its readers different rows.
Asynchronous.open
Section titled “Asynchronous.open”def open(entry: _Entry, samples: int, snapshot: Snapshot) -> NoneSubmit the entry’s groups, unless an earlier call opened them; the snapshot’s weights
reach the engines through begin.
Asynchronous.redrawn
Section titled “Asynchronous.redrawn”def redrawn(group: _Group, sample: _Sample, cause: str) -> Callable[[], None] | NoneA resubmission under the weights being served; None once one sample has failed
max_attempts times. A stale sample is not a failure.
Asynchronous.cancel
Section titled “Asynchronous.cancel”def cancel(samples: Sequence[_Sample]) -> NoneAsynchronous.stale
Section titled “Asynchronous.stale”def stale(sample: _Sample, updates: int) -> boolAsynchronous.rescore
Section titled “Asynchronous.rescore”def rescore( variables: Variables, packed: Mapping[str, np.ndarray], progress: Progress,) -> np.ndarrayThe packed ids’ proximal log-probabilities under variables, the trainer’s weights, at the
run’s progress the step’s loss reads (GRPOObjective.loss), so the two policies agree where
their weights do.
Over a mesh the rows are placed as the step places the batch, every process’s rows together, so the rescoring runs once over the pool’s batch and each process reads its own rows back.
Publisher
Section titled “Publisher”class Publisher(Protocol)Serves the trainer’s weights to the rollout engines.
load(variables, version) makes the engines serve variables under
version, and the version property is the version they serve now. A
RolloutServer is a publisher, and so is an engine fleet’s publish
sequence.
Publisher.load
Section titled “Publisher.load”def load(variables: Variables, version: int) -> NoneRolloutScheduler
Section titled “RolloutScheduler”class RolloutScheduler( acquisition: Synchronous | Asynchronous, *, tasks: Callable[[Batch], Sequence[Task]] = task_ids, groups: int = 4, oversample: int = 0, admit: int | None = None, estimator: str = 'group', truncation: str = 'mask', credit: str = 'session', support_capacity: int | None = None, log: Callable[[SchedulerRecord], None] | None = None, varied: bool = False, overlong: tuple[int, int] | None = None, reference: GRPOObjective | None = None, generation_batches: int = 1,)Admits complete groups of rollouts from acquisition and packs them into the trainer’s batches.
The module docstring describes groups, oversample, admit and the
two acquisitions. The other arguments:
tasksturns one batch into its tasks (task_idsfor integertask_idrows,prompt_tasksfor prompt rows).estimator,truncationandcreditarepack’s advantage family, truncation policy and credit (per session, or per turn from the step rewards still ahead), andsupport_capacityis its per-row support length, which a filtered-sampling source requires.log, when given, receives aSchedulerRecordof this process’s rollouts after each call. A process that is not the first reader of its share packs nothing and logs nothing.variedis DAPO’s dynamic sampling (the module docstring), andoverlongits soft overlong punishment,(L_max, L_cache)(sessions.overlong_penalty).generation_batchesis the most data batches a call admits groups from, its step’s and those it draws further (further, the trainer’s read of its stream).referenceis the objective whose reward-side reference penalty (GRPOObjective.reward_kl) each call’s reward pays before the estimator compares returns: the packed rows are scored under its reference, the state’s frozen starting policy, and repacked withsessions.reference_penalties.
metrics holds the latest call’s numbers, which the trainer logs as
rollout/<name>.
draws_further: bool-
Whether a call may draw data batches past its step’s (
generation_batches), which the trainer hands it a read of (dew.training.rollout.Rollout).
RolloutScheduler.sampled
Section titled “RolloutScheduler.sampled”def sampled( objective: TokenScorer, reward: Reward, *, decode: Callable[[Sequence[int]], str] = lambda ids: ' '.join(str(token) for token in ids), groups: int = 4, max_new_tokens: int = 32, estimator: str = 'group', truncation: str = 'score', sampling: Sampling | None = None, log: Callable[[SchedulerRecord], None] | None = None, varied: bool = False, overlong: tuple[int, int] | None = None, admit: int | None = None, generation_batches: int = 1,) -> RolloutSchedulerDraw groups completions of each prompt row from objective’s policy at every call and
train on them (PolicyCollector over prompt_tasks; GRPO’s synchronous loop). A completion
that runs out of max_new_tokens trains on its reward, as truncation='score' says. A reward-side
reference penalty (GRPOObjective.reward_kl) is objective’s own (reference).
The calls’ raw likelihoods are the proximal policy’s (Synchronous.rescore), the policy at
temperature 1 over the whole vocabulary: a GRPO objective whose ratio reads its policy at another
sampling_temperature is refused, since its unchanged weights would not give a ratio of 1.
RolloutScheduler.episodes
Section titled “RolloutScheduler.episodes”def episodes( policy: EpisodeInference, environment: EnvironmentFactory, verifier: Verifier, *, max_prompt_tokens: int, max_new_tokens: int, max_turns: int, sampling: Sampling, groups: int = 2, credit: str = 'session', record: EpisodeRecorder | None = None, journal: EpisodeJournal | None = None, log: Callable[[SchedulerRecord], None] | None = None,) -> RolloutSchedulerCollect groups tool episodes of each task_id row at every call, in lock step under the
step’s variables, and train on them (EpisodeCollector; truncated episodes are masked).
RolloutScheduler.tasks
Section titled “RolloutScheduler.tasks”def tasks(dataset: Loader) -> Loaderdataset as the acquisition reads it: asynchronously, a stream that registers each task
batch ahead of the step; synchronously, dataset itself.
RolloutScheduler.close
Section titled “RolloutScheduler.close”def close() -> NoneCancel every rollout in flight. The caller closes the source; this method does not.
RolloutScheduler.branch
Section titled “RolloutScheduler.branch”def branch(name: str) -> Iterator[None]The calls of the cooldown branch name (dew.training.rollout): drawn under the branch’s
states, on its own branch of a collector’s records (Snapshot.branch: an EpisodeJournal’s
cohorts, which a resumed branch reads back as its own), its records reach no log, and the
run’s latest metrics stand after it. The run closes its own stream for the branch and opens
it again at its position after (Trainer.fit), and an asynchronous acquisition’s stream
cancels the rollouts in flight as it opens (tasks): the run’s as the branch’s opens, which
the branch’s weights would otherwise finish, and the branch’s, cancelled here, as it ends. The
engines serve the branch’s weights meanwhile, and the run’s again from its next call
(Asynchronous.begin: a served version ahead of the run’s updates is out of date).
SchedulerRecord
Section titled “SchedulerRecord”class SchedulerRecord( updates: int, version: int, lag: int, groups: int, resubmitted: Mapping[str, int], cancelled: int, abandoned: int, cut: int, waited: float, metrics: Mapping[str, float],)What one trainer call consumed and what it cost.
updatesis the trainer’s update count at the call.versionis the version of the oldest admitted call, andlagis how many updates it is behindupdates.groupscounts the admitted groups.resubmittedcounts resubmissions by cause (infra_error,cancelled,stale,unscored).cancelledcounts in-flight rollouts cancelled because they were surplus, stale or in an abandoned group.abandonedcounts groups given up aftermax_attemptsfailures.cutcounts complete groups left out because their chains did not fit the batch’srowsbeside the groups admitted before them.waitedis the number of seconds the trainer waited.metricsissession_metricsover the admitted rollouts and their packed batch: merge ratio, status shares and masked shares, mean reward and reward components, the submission-to-finish latency tail and token lag.
The trainer-engine mismatch is not in this record; the loss reports it.
Synchronous
Section titled “Synchronous”class Synchronous(collector: Collector)Draws each call’s groups at the call from the trainer’s own variables through collector
(the module docstring’s synchronous case); Snapshot is what the collector draws from.
rescore-
None: the collected calls’ raw likelihoods are the proximal policy’s, drawn from the variables the step trains, so
old_log_probsare those and no rescoring forward runs.
Synchronous.rows
Section titled “Synchronous.rows”def rows(tasks: int, samples: int) -> intSynchronous.tasks
Section titled “Synchronous.tasks”def tasks(dataset: Loader, tasks_of: Callable[[Batch], Sequence[Task]]) -> Loaderdataset itself: a call’s tasks are its own batch’s, and nothing is opened ahead.
Synchronous.close
Section titled “Synchronous.close”def close() -> NoneNothing is in flight between calls.
Synchronous.begin
Section titled “Synchronous.begin”def begin(snapshot: Snapshot, partition: DataPartition) -> NoneNothing is published: the collector draws from the snapshot itself.
Synchronous.entries
Section titled “Synchronous.entries”def entries(batch: Batch, tasks_of: Callable[[Batch], Sequence[Task]]) -> list[_Entry]Synchronous.acquires
Section titled “Synchronous.acquires”def acquires(partition: DataPartition) -> boolEvery process: the draw is one program every process of the mesh enters.
Synchronous.open
Section titled “Synchronous.open”def open(entry: _Entry, samples: int, snapshot: Snapshot) -> NoneCollect the entry’s groups. A share’s later reader enters the draw, one program for the
share, and its collector returns nothing past it (sessions.Collector).
Synchronous.redrawn
Section titled “Synchronous.redrawn”def redrawn(group: _Group, sample: _Sample, cause: str) -> NoneNone: the draw was one program every process entered, which no process reruns alone.
Synchronous.cancel
Section titled “Synchronous.cancel”def cancel(samples: Sequence[_Sample]) -> NoneNothing runs: every collected session is finished.
Synchronous.stale
Section titled “Synchronous.stale”def stale(sample: _Sample, updates: int) -> booltask_ids
Section titled “task_ids”def task_ids(batch: Batch) -> list[Task]Return one task per integer task_id row of batch (local_task_ids), named by the id in
decimal.