dew.interop.harness
Agent harnesses as a session source: one harness command per sample, its model calls recorded.
HarnessSource runs each sample as one trial of a Harness: a command
whose agent reaches the policy only through its own gateway session
(dew.interop.gateway), at the OpenAI base URL the trial is given. Dew runs
the command as a subprocess in a process group of its own, reads the
session’s calls back from the gateway, and asks the harness how the trial
ended. The harness decides what its command is and what its artifacts say;
the source owns the gateway session, the trial’s lifecycle and its
cancellation, the same for every harness.
Harbor(dew.interop.harbor) runs a Harbor trial: Harbor starts the task’s sandbox, runs its agent harness (mini-swe-agent by default) and then its verifier.Commandruns any agent CLI that reads its model’s base URL fromOPENAI_BASE_URL, in a working directory of its own, and scores the directory it leaves with a verifier.
How a trial ends:
COMPLETEDwhen the agent exited cleanly and the verifier scored it.AGENT_ERROR, with the verifier’s score, when the agent failed and the verifier still scored it.TRUNCATEDwhen the agent ran out of time, context or output budget, hit its own step limit, or its last call stopped at its length limit. It has the verifier’s reward when the verifier ran, and the scheduler’struncationpolicy decides whether it trains.INFRA_ERRORfor anything else: a sandbox, gateway, engine or verifier failure, a trace without ids or likelihoods, an aborted call, or a session with no call. The scheduler retries these and never trains on them.
Cancelling a trial stops its harness; it does not stop a model call the engine is still decoding for it (rllm-model-gateway 3b40c37 holds a non-streaming call’s upstream connection after its client goes, rllm-org/rllm#938), so a cancelled trial’s last call may run on to its token budget.
| Name | Summary |
|---|---|
Command | Any agent CLI that reaches its model through OPENAI_BASE_URL, scored on what it leaves. |
Harness | One kind of agent harness, run as one command per sample against a gateway session. |
HarnessSource | Runs each sample as one trial of harness behind a recording gateway. |
Judged | How a finished trial ended: its status, the verifier’s reward and reward components, and a detail naming what happened. |
Trial | One sample’s run of a harness: its command, the environment it adds, the directory it runs in (the source’s own when None) and the directory its artifacts are under. |
Command
Section titled “Command”class Command( argv: Callable[[Task, Path], Sequence[str]], verify: Callable[[Task, Path, int], float | None], workspace: Path, environment: Mapping[str, str] = dict(),)Any agent CLI that reaches its model through OPENAI_BASE_URL, scored on what it leaves.
argv(task, directory) is the command that runs task in directory,
a fresh directory under workspace that is the trial’s working
directory. It runs with this process’s environment plus environment
and the session’s OPENAI_BASE_URL, so a command that must not see the
trainer’s credentials or files runs its agent in a sandbox of its own
(a container, a remote sandbox). verify(task, directory, returncode)
scores the finished directory, or returns None when it cannot (the
verifier itself failed), which makes the trial infra.
A trial whose command exited 0 and scored is COMPLETED, one that
exited otherwise and scored is AGENT_ERROR. The engine’s own events
come first: an engine failure makes it infra, and a refused prompt or a
last call stopped at its length limit truncates it.
Command.check
Section titled “Command.check”def check(task: Task) -> NoneCommand.trial
Section titled “Command.trial”def trial(task: Task, url: str, name: str) -> TrialCommand.judge
Section titled “Command.judge”def judge( task: Task, trial: Trial, log: str, returncode: int, recorded: Recorded,) -> JudgedHarness
Section titled “Harness”class Harness(Protocol)One kind of agent harness, run as one command per sample against a gateway session.
Harness.check
Section titled “Harness.check”def check(task: Task) -> NoneRaise ValueError when task is not one this harness runs, before any trial starts.
Harness.trial
Section titled “Harness.trial”def trial(task: Task, url: str, name: str) -> TrialThe command that runs task once with its model at the OpenAI base URL url, writing its
artifacts under a directory named name.
Harness.judge
Section titled “Harness.judge”def judge( task: Task, trial: Trial, log: str, returncode: int, recorded: Recorded,) -> JudgedHow the trial ended: from what it wrote, its output log, its exit code and the calls
the gateway recorded.
HarnessSource
Section titled “HarnessSource”class HarnessSource( harness: Harness, gateway: Gateway, *, workers: int = 8, grace: float = 60.0, trial_timeout: float | None = None, ready_timeout: float = 900.0, ready_poll: float = 2.0,)Runs each sample as one trial of harness behind a recording gateway.
workersis how many trials run at once.trial_timeout, when set, bounds each trial in seconds from its start. A trial past this deadline is interrupted ascancelinterrupts it, and resolvesINFRA_ERROR, which the scheduler retries.graceis how long a trial has to wind down after each signal of a cancel or a deadline (seecancel).
Gateway sessions are named {task}:{group}:{sample}:{token}. group is
unique to one submit across runs, and token is 128 secret bits, so a
sandbox can address only the session it was given. As a
SessionSource, every future it returns resolves to a Session. A
failure of the harness, its sandbox, the gateway or the engine gives an
INFRA_ERROR session, not an exception, and a cancelled trial gives a
CANCELLED one. Each session’s attempt is 0, because a retry is a
fresh submit and the scheduler, which assigns group identity, relabels
it.
The first submit waits up to ready_timeout seconds for the gateway
to report a healthy worker (Gateway.ready). So a source started beside
engines that are still loading does not launch trials that the gateway
would answer with a 500 and no trace.
Use it as a context manager, or call close, to cancel every trial when
you are done. A trainer that exits without either, for example on an
uncaught KeyboardInterrupt, still stops its trials. Trials run on daemon
threads, and an atexit hook cancels the queued trials and interrupts the
running ones, then kills their process groups after grace seconds. The
threads start, and the hook registers, at the first submit:
constructing the source starts nothing.
HarnessSource.submit
Section titled “HarnessSource.submit”def submit( task: Task, samples: int, *, version: int, key: jax.Array,) -> list[Future[Session]]HarnessSource.cancel
Section titled “HarnessSource.cancel”def cancel(futures: Sequence[Future[Session]]) -> NoneInterrupt the given trials, each of which resolves CANCELLED; a queued one never starts.
A trial whose output is still open grace seconds after the
interrupt gets SIGTERM, and SIGKILL after another grace seconds,
each sent to its whole process group. Output still open a grace
after the SIGKILL is held by a process that left the group, which
keeps running; the trial stops reading it and resolves.
HarnessSource.close
Section titled “HarnessSource.close”def close() -> NoneCancel every unresolved trial, queued or running, and wait for the running ones to tear down.
Judged
Section titled “Judged”class Judged( status: Status, reward: float | None = None, components: Mapping[str, float] = dict(), detail: str = '',)How a finished trial ended: its status, the verifier’s reward and reward components, and a detail naming what happened.
class Trial( argv: tuple[str, ...], artifacts: Path, environment: Mapping[str, str] = dict(), cwd: Path | None = None,)One sample’s run of a harness: its command, the environment it adds, the directory it runs in (the source’s own when None) and the directory its artifacts are under.