dew.interop.harbor
Harbor trials as a harness: each sample one harbor trials start, its model calls recorded.
Harbor is the Harness (dew.interop.harness) that runs a sample as one
Harbor trial. Harbor starts the task’s sandbox, installs and runs the agent
harness (mini-swe-agent by default), then runs the task’s verifier. Its
agent reaches the model through the trial’s gateway session
(dew.interop.gateway), given to it as the agent environment’s
OPENAI_BASE_URL. Harbor is a separate service in its own environment: Dew
runs harbor trials start as a subprocess and reads the trial’s
result.json; it imports nothing of Harbor.
outcome decides how a trial ended from Harbor’s result, the harness’s own
exit status and the calls, as dew.interop.harness lays out: Harbor’s
exception names say which limit an agent hit, mini-swe-agent’s exit status
says when it stopped at its own limit or its model client failed, and the
engine’s own events come first.
| Name | Summary |
|---|---|
HARBOR_KEY | The Task.data key that holds the Harbor task directory a trial runs. |
Harbor | Runs each sample as one Harbor trial (a Harness for HarnessSource). |
outcome | How one finished trial ended: its status, reward, reward components and failure detail. |
HARBOR_KEY
Section titled “HARBOR_KEY”HARBOR_KEY = 'harbor'The Task.data key that holds the Harbor task directory a trial runs.
Harbor
Section titled “Harbor”class Harbor( harbor: str | os.PathLike[str], model: str, trials: os.PathLike[str], agent: str = 'mini-swe-agent', environment: Mapping[str, str] = dict(), arguments: Sequence[str] = (),)Runs each sample as one Harbor trial (a Harness for HarnessSource).
harboris the Harbor executable, installed in its own environment.agentandmodelare its--agentand--model.trialsis the directory trials are written under.environmentis passed to the harness as agent environment (--ae), beside the per-trialOPENAI_BASE_URLthat names the session.argumentsare furtherharbor trials startoptions, such as the environment provider, timeouts and agent kwargs. Harbor bounds the environment build, the agent’s setup and run, and the verifier itself (--agent-timeout,--verifier-timeout,--timeout-multiplier), but no option of its own bounds the whole trial:HarnessSource’strial_timeoutdoes.
A task names its Harbor task directory under data[HARBOR_KEY]. The
agent must speak the OpenAI chat API through OPENAI_BASE_URL, as
Harbor’s mini-swe-agent does.
Harbor.check
Section titled “Harbor.check”def check(task: Task) -> NoneHarbor.trial
Section titled “Harbor.trial”def trial(task: Task, url: str, name: str) -> TrialHarbor.judge
Section titled “Harbor.judge”def judge( task: Task, trial: Trial, log: str, returncode: int, recorded: Recorded,) -> Judgedoutcome
Section titled “outcome”def outcome( trial: JSON, recorded: Recorded, *, harness_exit: str | None = None,) -> JudgedHow one finished trial ended: its status, reward, reward components and failure detail.
trial is Harbor’s TrialResult as JSON, recorded the session’s calls
and engine errors, and harness_exit the harness’s own exit status when
it reports one. An engine that refused an overflowing prompt truncated
the rollout; any other engine error makes it infra, even when the harness
retried and went on, and so does an aborted call or a failed model
client, whatever limit the trial also hit. A result whose fields have the
wrong JSON types is infra: nothing in it can be trusted to score.