dew.interop.gateway
The seam between an agent harness and the policy: rllm-model-gateway, recording every model call.
A harness reaches the model only through a recording gateway,
rllm-model-gateway, at a base URL that names the session in its path,
/sessions/{session}/v1. The gateway forwards each call to a vLLM or SGLang
replica, asks it for token ids and log-probabilities, and records them under
that session. The session is part of the URL because a CLI harness may drop
a header. The gateway is a separate service in its own environment; Dew
reads a session’s traces over HTTP and imports none of it. It must persist
traces before it answers (sync_traces: true), so a session’s traces are
complete once its harness has exited.
calls turns gateway traces into Call records, in submission order. Each
holds the engine’s prompt ids, sampled ids and behavior log-probabilities
as recorded, and the version the gateway stamped when the request arrived
(the publication sets that stamp; see Publication). The gateway extracts
vLLM’s ids and SGLang’s prompt ids. SGLang’s sampled ids
(choices[0].response_token_ids) are read from the raw response the
gateway keeps, and each call’s Transcript from the chat messages the
harness sent and the message the engine answered, which the gateway records
beside the ids. engine_failure and refused_prompt read what the engine
itself said of a session, which every harness’s verdict weighs the same way
(dew.interop.harness).
| Name | Summary |
|---|---|
SESSION | What a session id may hold: it goes in a URL’s path. |
Gateway | Connects a session source to an rllm-model-gateway. |
Recorded | One session as the gateway recorded it. |
calls | The gateway’s traces of one session as Calls in submission order, and its engine errors. |
engine_failure | Why the engine failed the session, or None: an error reply other than a prompt past its context length, or an aborted call. |
number | value, a JSON number named name, as a float; ValueError for anything else, a bool included. |
refused_prompt | Why the engine cut the session short, or None: a prompt it refused as past its context length. |
SESSION
Section titled “SESSION”SESSION = re.compile('[A-Za-z0-9._:-]+')What a session id may hold: it goes in a URL’s path.
Gateway
Section titled “Gateway”class Gateway( url: str, *, sandbox_url: str | None = None, client: httpx.Client | None = None,)Connects a session source to an rllm-model-gateway.
It builds each session’s base URL, waits for the gateway to be ready,
stamps the weight version and reads a session’s traces. url is the
gateway’s root as Dew reaches it. sandbox_url is the address the
sandboxes reach it at, and a harness’s base URL is built from it.
rllm-model-gateway (3b40c37) has no authentication. One port serves the
model proxy and also GET /sessions, every session’s traces, POST /traces/query, session deletes, POST /admin/workers and POST /admin/weight_version. A sandbox runs the policy’s own commands, so a
policy that reaches that port can read its group members’ transcripts,
reset the version stamp the staleness bound reads, or register a worker
that returns fabricated ids and likelihoods. To prevent that:
- Put
urlon an interface only Dew reaches, by binding the gateway to loopback or a private trainer network. - Make
sandbox_urla reverse proxy in front of the gateway, on its own address, that forwards onlyPOST /sessions/<session>/v1/chat/completions(nginxlocation ~ ^/sessions/[^/]+/v1/chat/completions$) and answers 403 to everything else. - Let the task’s agent reach that address and nothing else. Under
Harbor, that is
[agent] network_mode = "allowlist"andallowed_hostsnaming the proxy, or--allow-agent-host; this needs a provider that supports allowlists (see Harbor’stasks/network-policypage; Docker needs nftablesfibsupport). Set[verifier] network_mode = "no-network"as well. Harbor’s allowlist filters by host and not by path, which is why the proxy is needed.
sandbox_url defaults to url. Leave it unset only for trusted
harnesses and tests.
Gateway.session
Section titled “Gateway.session”def session(session: str) -> strReturn the OpenAI base URL that makes the gateway record a harness’s calls under session.
Raises ValueError unless session is URL-path safe: letters, digits
and ._:-.
Gateway.traces
Section titled “Gateway.traces”def traces(session: str) -> Sequence[object]Gateway.ready
Section titled “Gateway.ready”def ready(timeout: float, *, poll: float = 2.0) -> NoneWait until the gateway routes to a healthy worker that answers, or raise after timeout seconds.
Ready means /health/workers and GET /v1/models, proxied through the gateway to an engine,
answer 200. Only their status is read; the bodies are closed unread. The remaining deadline
bounds each network operation, and a status received after it is never accepted. The check
runs every poll seconds, and expiry raises RuntimeError.
rllm-model-gateway marks a worker dead after three failed health checks, which happens to an engine that is still loading when the gateway starts. Until a later check revives the worker, every proxied call gets a plain-text 500 that leaves no trace. A session submitted in that window would fail because of the gateway, not the policy.
Gateway.stamp
Section titled “Gateway.stamp”def stamp(version: int) -> NoneMake the gateway label the calls it records from now on with version (a Publication stamp).
Gateway.forget
Section titled “Gateway.forget”def forget(session: str) -> NoneRecorded
Section titled “Recorded”class Recorded(calls: tuple[Call, ...], errors: tuple[str, ...])One session as the gateway recorded it.
calls are its model calls in submission order. errors are the
messages of the error replies the engine sent instead of a completion.
def calls(traces: Sequence[object], *, unstamped: int) -> RecordedThe gateway’s traces of one session as Calls in submission order, and its engine errors.
A trace records its arrival indirectly: timestamp is when the answer
was stored and latency_ms how long the engine took, so the calls are
ordered by their difference. A trace without a version stamp, from a
gateway no publication has stamped, takes unstamped, the version the
session was submitted under: no later push can have served it anything
older. The gateway also records an engine’s error reply (a prompt past
the context length, an engine fault): such a trace carries error in its
raw response and is an event of the session, returned as its message, not
a call. A successful reply without ids, with one likelihood too few or
too many, or with a field of the wrong JSON type raises ValueError;
training on it would mean guessing or re-tokenizing text.
engine_failure
Section titled “engine_failure”def engine_failure(recorded: Recorded) -> str | NoneWhy the engine failed the session, or None: an error reply other than a prompt past its context length, or an aborted call. Either makes the session infra, whatever the harness did after it, since the policy did not cause it.
number
Section titled “number”def number(name: str, value: object) -> floatvalue, a JSON number named name, as a float; ValueError for anything else, a bool included.
refused_prompt
Section titled “refused_prompt”def refused_prompt(recorded: Recorded) -> str | NoneWhy the engine cut the session short, or None: a prompt it refused as past its context length. The policy’s own context ran out, so the session is truncated.