Skip to content

dew.objectives.rl.scheduler

Collect rollouts and admit complete groups into the trainer’s batches, synchronously or ahead of it.

RolloutScheduler is the trainer’s Rollout for token sessions. Each task becomes one group of groups rollouts, which the scheduler admits, fits into its rows, packs (sessions.pack) and reports, whichever way they were acquired:

  • Synchronous(collector) draws every task’s sessions at the call from the variables the step trains (sessions.Collector): PolicyCollector for single-turn prompts, EpisodeCollector for tool episodes in lock step. Its draws are one program every process of the trainer’s mesh enters, so every process collects and the first reader of each data share packs. Their raw likelihoods are the proximal policy’s, so old_log_probs are those and no rescoring runs. A collected sample is never drawn again: a group with an INFRA_ERROR or CANCELLED session is abandoned. RolloutScheduler.sampled and RolloutScheduler.episodes build this case.
  • Asynchronous(source, weights, objective, ...) keeps rollouts in flight from any SessionSource (an in-process environment, a single-turn prompt set, a harness behind a recording gateway) within a staleness bound, as below.

Asynchronously, wrap the task dataset with scheduler.tasks(dataset). The wrapped stream registers each task batch when the trainer’s prefetch reads it. When the trainer then passes batch i to the scheduler, the scheduler submits batches i + 1 ... i + ahead under the weight version the engines serve at that moment. Batch i’s own rollouts were submitted ahead calls earlier and have kept running since, through any weight pushes in between. The scheduler reads no data beyond the trainer’s own prefetch, so the data position in a checkpoint is still the trainer’s. A resumed run reads again and resubmits whatever was in flight, and reopening the stream cancels the rollouts the old stream left running.

The scheduler relabels each rollout with its own group id, sample index and attempt number, so a resubmitted sample rejoins its group. Each finished rollout is admitted or replaced according to its status:

  • COMPLETED and AGENT_ERROR are admitted, because the verifier scored them.
  • TRUNCATED is admitted, and truncation sets how it trains: mask (the default, for agentic context, turn and wall-clock limits), score on its verifier reward (single-turn RLVR), or zero (a length penalty); see dew.objectives.rl.sessions. Under score, a truncated rollout without a reward is resubmitted as a failed attempt (cause unscored), like an INFRA_ERROR. Its verifier never ran, so masking it would drop a sample for a reason unrelated to the policy, and there is no reward to train it on.
  • INFRA_ERROR and CANCELLED are never scored. Asynchronously, the sample is submitted again under the weights being served. After max_attempts failures of one sample, the scheduler abandons the whole group instead of training it incomplete.
  • A rollout whose oldest call is more than max_lag updates behind is discarded and resubmitted. A rollout still running when its submission falls past that bound is cancelled without waiting for it to finish, because its first call was made under the submission’s version.

The scheduler keeps no clock of its own: a deadline belongs to the operation that can stop when it passes. The remote rollout servers bound each request, a sandboxed environment (SubprocessEnvironment) its wall_seconds, and HarnessSource each trial (trial_timeout). A rollout that runs out of time there comes back as an INFRA_ERROR, which the scheduler retries. A reward’s program past its sandbox’s wall clock is a TIMEOUT verdict that the reward scores, not a failure.

A source that raises an exception instead of returning a status is broken. The scheduler cancels the batch’s remaining rollouts and lets the exception propagate.

Two options cut the long tail of slow rollouts, and both keep the batch shape fixed. With oversample, each group runs that many extra samples and is admitted once its first groups rollouts finish; the scheduler cancels the rest. This is APRIL’s active partial rollouts (arXiv:2509.18521) without the carry-over. With admit below the task batch size, the scheduler admits the first admit groups to complete and fit, and cancels the others, as slime’s over-sampling batch does. Both select by completion time, which favors short rollouts; that bias is the price of not waiting on the tail. With varied, DAPO’s dynamic sampling (arXiv:2503.14476 equation 11 and Algorithm 1), a complete group whose members’ returns all agree is left out, since it gives each of them a zero advantage, and admission goes on through the batch’s other groups, drawn under the same weights, until admit groups are admitted or the batch has none left: a batch of more tasks than admit is DAPO’s generation batch over its training batch. With generation_batches above one, a call still short of its groups draws the data stream’s next batch, as many as that in all, and admits from its groups too, until the step’s batch is full (DAPO’s Algorithm 1, verl’s max_num_gen_batches): every process draws while any share’s first reader is short, so the streams read alike, and the step’s checkpoint saves the stream’s position after the last batch drawn.

Rollouts that are running when new weights are pushed keep running. Their later calls are stamped with the new version, and a rollout’s staleness is that of its oldest call, as in Kimi K2’s partial rollouts (arXiv:2507.20534 section 3.3.4).

Asynchronously, the scheduler pushes weights through weights when the served version falls sync_every updates behind the trainer, or is ahead of it, as after a resume from an earlier checkpoint. A first submission is therefore at most ahead + sync_every - 1 updates stale when the trainer consumes it, and the constructor refuses a max_lag below that. If a complete group’s chains do not fit in rows beside the groups admitted before it, the scheduler cuts that group, so an overfull batch never reaches the step. The proximal policy is the trainer’s current weights. The scheduler rescores the packed rows under them with GRPOObjective.packed_log_probs (decoupled PPO, AReaL arXiv:2505.24298). Sources report only behavior likelihoods, and the objective’s behavior_importance weights each token by the ratio of its proximal to its behavior likelihood.

In a multi-process trainer, every process runs its own scheduler over the task rows its data stream reads, and packs its own rows. The step’s batch is all the processes’ rows together, sharded as the trainer shards any batch, and no rollout moves between processes. The processes act together at a few points in each call, each an agreement point over the trainer’s mesh where every process raises if one failed, in this order: the call’s preparation, the acquisition and its admission, the global estimator’s moments, which every share’s pool so its advantages are whitened over the step’s whole batch, the packing, the engine records’ layout, the first reader’s rows and the proximal rescoring; with a reward-side reference penalty (reference), the moments wait for it, and the rows’ scoring under the reference, the moments, the repacked advantages and the first reader’s rows follow the rescoring. A weight push is one call every process makes, so the publisher has to agree on it across processes. When a tensor or sequence axis spans processes, several processes read the same share of the data. Then only the share’s first reader packs, and the others train on the rows it packed (first_reader_batch), its recorded likelihoods included, which independent draws would not give them. An asynchronous source samples on the first reader alone; a collector’s draw is one program, which every reader enters. Without a mesh nothing is agreed, and each process collects on its own.

NameSummary
AsynchronousKeeps rollouts from source in flight ahead of the trainer (the module docstring’s asynchronous case), pushing the trainer’s weights through weights.
PublisherServes the trainer’s weights to the rollout engines.
RolloutSchedulerAdmits complete groups of rollouts from acquisition and packs them into the trainer’s batches.
SchedulerRecordWhat one trainer call consumed and what it cost.
SynchronousDraws each call’s groups at the call from the trainer’s own variables through collector (the module docstring’s synchronous case); Snapshot is what the collector draws from.
task_idsReturn one task per integer task_id row of batch (local_task_ids), named by the id in decimal.

class source

class Asynchronous(
source: SessionSource,
weights: Publisher,
objective: GRPOObjective,
*,
width: int,
rows: int,
ahead: int = 1,
max_lag: int = 1,
sync_every: int = 1,
max_attempts: int = 3,
)

Keeps rollouts from source in flight ahead of the trainer (the module docstring’s asynchronous case), pushing the trainer’s weights through weights.

width and rows fix the packed batch shape; rows is this process’s share of a multi-process trainer’s batch. objective rescores the packed rows under the trainer’s weights, the proximal policy, and a max_lag above 0 needs its behavior_importance (a TIS cap or an IcePop band); the constructor raises ValueError without it.

def rows(tasks: int, samples: int) -> int
def tasks(dataset: Loader, tasks_of: Callable[[Batch], Sequence[Task]]) -> Loader

dataset with a training stream that registers each task batch ahead of the step.

Opening the stream again, as a resume does, cancels every rollout the previous stream left in flight.

def close() -> None

Cancel every rollout in flight. The caller closes the source; this method does not.

def begin(snapshot: Snapshot, partition: DataPartition) -> None

Fold the call’s key for the engines and push the weights where the served version is sync_every behind the trainer’s, or ahead of it.

The engines draw with JAX’s default PRNG on devices of their own, so every word of the trainer’s key, of whatever implementation, folds into an uncommitted default key, and this data share’s index after them, so two shares’ readers draw apart. A served version ahead of updates is out of date as well: a fit restored from an earlier checkpoint must not sample from weights newer than its own.

def entries(batch: Batch, tasks_of: Callable[[Batch], Sequence[Task]]) -> list[_Entry]
def acquires(partition: DataPartition) -> bool

The share’s first reader: independent draws would hand its readers different rows.

def open(entry: _Entry, samples: int, snapshot: Snapshot) -> None

Submit the entry’s groups, unless an earlier call opened them; the snapshot’s weights reach the engines through begin.

def redrawn(group: _Group, sample: _Sample, cause: str) -> Callable[[], None] | None

A resubmission under the weights being served; None once one sample has failed max_attempts times. A stale sample is not a failure.

def cancel(samples: Sequence[_Sample]) -> None
def stale(sample: _Sample, updates: int) -> bool
def rescore(
variables: Variables,
packed: Mapping[str, np.ndarray],
progress: Progress,
) -> np.ndarray

The packed ids’ proximal log-probabilities under variables, the trainer’s weights, at the run’s progress the step’s loss reads (GRPOObjective.loss), so the two policies agree where their weights do.

Over a mesh the rows are placed as the step places the batch, every process’s rows together, so the rescoring runs once over the pool’s batch and each process reads its own rows back.

class source

class Publisher(Protocol)

Serves the trainer’s weights to the rollout engines.

load(variables, version) makes the engines serve variables under version, and the version property is the version they serve now. A RolloutServer is a publisher, and so is an engine fleet’s publish sequence.

def load(variables: Variables, version: int) -> None

class source

class RolloutScheduler(
acquisition: Synchronous | Asynchronous,
*,
tasks: Callable[[Batch], Sequence[Task]] = task_ids,
groups: int = 4,
oversample: int = 0,
admit: int | None = None,
estimator: str = 'group',
truncation: str = 'mask',
credit: str = 'session',
support_capacity: int | None = None,
log: Callable[[SchedulerRecord], None] | None = None,
varied: bool = False,
overlong: tuple[int, int] | None = None,
reference: GRPOObjective | None = None,
generation_batches: int = 1,
)

Admits complete groups of rollouts from acquisition and packs them into the trainer’s batches.

The module docstring describes groups, oversample, admit and the two acquisitions. The other arguments:

  • tasks turns one batch into its tasks (task_ids for integer task_id rows, prompt_tasks for prompt rows).
  • estimator, truncation and credit are pack’s advantage family, truncation policy and credit (per session, or per turn from the step rewards still ahead), and support_capacity is its per-row support length, which a filtered-sampling source requires.
  • log, when given, receives a SchedulerRecord of this process’s rollouts after each call. A process that is not the first reader of its share packs nothing and logs nothing.
  • varied is DAPO’s dynamic sampling (the module docstring), and overlong its soft overlong punishment, (L_max, L_cache) (sessions.overlong_penalty). generation_batches is the most data batches a call admits groups from, its step’s and those it draws further (further, the trainer’s read of its stream).
  • reference is the objective whose reward-side reference penalty (GRPOObjective.reward_kl) each call’s reward pays before the estimator compares returns: the packed rows are scored under its reference, the state’s frozen starting policy, and repacked with sessions.reference_penalties.

metrics holds the latest call’s numbers, which the trainer logs as rollout/<name>.

draws_further: bool

Whether a call may draw data batches past its step’s (generation_batches), which the trainer hands it a read of (dew.training.rollout.Rollout).

def sampled(
objective: TokenScorer,
reward: Reward,
*,
decode: Callable[[Sequence[int]], str] = lambda ids: ' '.join(str(token) for token in ids),
groups: int = 4,
max_new_tokens: int = 32,
estimator: str = 'group',
truncation: str = 'score',
sampling: Sampling | None = None,
log: Callable[[SchedulerRecord], None] | None = None,
varied: bool = False,
overlong: tuple[int, int] | None = None,
admit: int | None = None,
generation_batches: int = 1,
) -> RolloutScheduler

Draw groups completions of each prompt row from objective’s policy at every call and train on them (PolicyCollector over prompt_tasks; GRPO’s synchronous loop). A completion that runs out of max_new_tokens trains on its reward, as truncation='score' says. A reward-side reference penalty (GRPOObjective.reward_kl) is objective’s own (reference).

The calls’ raw likelihoods are the proximal policy’s (Synchronous.rescore), the policy at temperature 1 over the whole vocabulary: a GRPO objective whose ratio reads its policy at another sampling_temperature is refused, since its unchanged weights would not give a ratio of 1.

def episodes(
policy: EpisodeInference,
environment: EnvironmentFactory,
verifier: Verifier,
*,
max_prompt_tokens: int,
max_new_tokens: int,
max_turns: int,
sampling: Sampling,
groups: int = 2,
credit: str = 'session',
record: EpisodeRecorder | None = None,
journal: EpisodeJournal | None = None,
log: Callable[[SchedulerRecord], None] | None = None,
) -> RolloutScheduler

Collect groups tool episodes of each task_id row at every call, in lock step under the step’s variables, and train on them (EpisodeCollector; truncated episodes are masked).

def tasks(dataset: Loader) -> Loader

dataset as the acquisition reads it: asynchronously, a stream that registers each task batch ahead of the step; synchronously, dataset itself.

def close() -> None

Cancel every rollout in flight. The caller closes the source; this method does not.

def branch(name: str) -> Iterator[None]

The calls of the cooldown branch name (dew.training.rollout): drawn under the branch’s states, on its own branch of a collector’s records (Snapshot.branch: an EpisodeJournal’s cohorts, which a resumed branch reads back as its own), its records reach no log, and the run’s latest metrics stand after it. The run closes its own stream for the branch and opens it again at its position after (Trainer.fit), and an asynchronous acquisition’s stream cancels the rollouts in flight as it opens (tasks): the run’s as the branch’s opens, which the branch’s weights would otherwise finish, and the branch’s, cancelled here, as it ends. The engines serve the branch’s weights meanwhile, and the run’s again from its next call (Asynchronous.begin: a served version ahead of the run’s updates is out of date).

dataclass source

class SchedulerRecord(
updates: int,
version: int,
lag: int,
groups: int,
resubmitted: Mapping[str, int],
cancelled: int,
abandoned: int,
cut: int,
waited: float,
metrics: Mapping[str, float],
)

What one trainer call consumed and what it cost.

  • updates is the trainer’s update count at the call.
  • version is the version of the oldest admitted call, and lag is how many updates it is behind updates.
  • groups counts the admitted groups.
  • resubmitted counts resubmissions by cause (infra_error, cancelled, stale, unscored).
  • cancelled counts in-flight rollouts cancelled because they were surplus, stale or in an abandoned group.
  • abandoned counts groups given up after max_attempts failures.
  • cut counts complete groups left out because their chains did not fit the batch’s rows beside the groups admitted before them.
  • waited is the number of seconds the trainer waited.
  • metrics is session_metrics over the admitted rollouts and their packed batch: merge ratio, status shares and masked shares, mean reward and reward components, the submission-to-finish latency tail and token lag.

The trainer-engine mismatch is not in this record; the loss reports it.

class source

class Synchronous(collector: Collector)

Draws each call’s groups at the call from the trainer’s own variables through collector (the module docstring’s synchronous case); Snapshot is what the collector draws from.

rescore

None: the collected calls’ raw likelihoods are the proximal policy’s, drawn from the variables the step trains, so old_log_probs are those and no rescoring forward runs.

def rows(tasks: int, samples: int) -> int
def tasks(dataset: Loader, tasks_of: Callable[[Batch], Sequence[Task]]) -> Loader

dataset itself: a call’s tasks are its own batch’s, and nothing is opened ahead.

def close() -> None

Nothing is in flight between calls.

def begin(snapshot: Snapshot, partition: DataPartition) -> None

Nothing is published: the collector draws from the snapshot itself.

def entries(batch: Batch, tasks_of: Callable[[Batch], Sequence[Task]]) -> list[_Entry]
def acquires(partition: DataPartition) -> bool

Every process: the draw is one program every process of the mesh enters.

def open(entry: _Entry, samples: int, snapshot: Snapshot) -> None

Collect the entry’s groups. A share’s later reader enters the draw, one program for the share, and its collector returns nothing past it (sessions.Collector).

def redrawn(group: _Group, sample: _Sample, cause: str) -> None

None: the draw was one program every process entered, which no process reruns alone.

def cancel(samples: Sequence[_Sample]) -> None

Nothing runs: every collected session is finished.

def stale(sample: _Sample, updates: int) -> bool

function source

def task_ids(batch: Batch) -> list[Task]

Return one task per integer task_id row of batch (local_task_ids), named by the id in decimal.