Skip to content

dew.interop.harbor

Harbor trials as a harness: each sample one harbor trials start, its model calls recorded.

Harbor is the Harness (dew.interop.harness) that runs a sample as one Harbor trial. Harbor starts the task’s sandbox, installs and runs the agent harness (mini-swe-agent by default), then runs the task’s verifier. Its agent reaches the model through the trial’s gateway session (dew.interop.gateway), given to it as the agent environment’s OPENAI_BASE_URL. Harbor is a separate service in its own environment: Dew runs harbor trials start as a subprocess and reads the trial’s result.json; it imports nothing of Harbor.

outcome decides how a trial ended from Harbor’s result, the harness’s own exit status and the calls, as dew.interop.harness lays out: Harbor’s exception names say which limit an agent hit, mini-swe-agent’s exit status says when it stopped at its own limit or its model client failed, and the engine’s own events come first.

NameSummary
HARBOR_KEYThe Task.data key that holds the Harbor task directory a trial runs.
HarborRuns each sample as one Harbor trial (a Harness for HarnessSource).
outcomeHow one finished trial ended: its status, reward, reward components and failure detail.

attribute source

HARBOR_KEY = 'harbor'

The Task.data key that holds the Harbor task directory a trial runs.

dataclass source

class Harbor(
harbor: str | os.PathLike[str],
model: str,
trials: os.PathLike[str],
agent: str = 'mini-swe-agent',
environment: Mapping[str, str] = dict(),
arguments: Sequence[str] = (),
)

Runs each sample as one Harbor trial (a Harness for HarnessSource).

  • harbor is the Harbor executable, installed in its own environment.
  • agent and model are its --agent and --model.
  • trials is the directory trials are written under.
  • environment is passed to the harness as agent environment (--ae), beside the per-trial OPENAI_BASE_URL that names the session.
  • arguments are further harbor trials start options, such as the environment provider, timeouts and agent kwargs. Harbor bounds the environment build, the agent’s setup and run, and the verifier itself (--agent-timeout, --verifier-timeout, --timeout-multiplier), but no option of its own bounds the whole trial: HarnessSource’s trial_timeout does.

A task names its Harbor task directory under data[HARBOR_KEY]. The agent must speak the OpenAI chat API through OPENAI_BASE_URL, as Harbor’s mini-swe-agent does.

def check(task: Task) -> None
def trial(task: Task, url: str, name: str) -> Trial
def judge(
task: Task,
trial: Trial,
log: str,
returncode: int,
recorded: Recorded,
) -> Judged

function source

def outcome(
trial: JSON,
recorded: Recorded,
*,
harness_exit: str | None = None,
) -> Judged

How one finished trial ended: its status, reward, reward components and failure detail.

trial is Harbor’s TrialResult as JSON, recorded the session’s calls and engine errors, and harness_exit the harness’s own exit status when it reports one. An engine that refused an overflowing prompt truncated the rollout; any other engine error makes it infra, even when the harness retried and went on, and so does an aborted call or a failed model client, whatever limit the trial also hit. A result whose fields have the wrong JSON types is infra: nothing in it can be trusted to score.