Skip to content

dew.eval.harness

Run a saved run as an lm-evaluation-harness model.

DewLM wraps a TextGeneration in lm-eval-harness’s TemplateLM interface, so any task suite can run against a run directory.

lm_eval is an optional extra (pip install dewml[eval-harness]). Only this module imports it, and dew.eval does not import this module. Importing this module registers the adapter under the name dew. lm_eval 0.4 has no plugin discovery, so the import has to happen in the process that runs the command. python -m dew.eval makes that import and then runs lm_eval’s own command line:

python -m dew.eval --model dew --model_args run=runs/shakespeare \
--tasks hellaswag --limit 4

lm-eval’s own code decides which tokens are scored (TemplateLM.loglikelihood, get_rolling_token_windows). This module adds _loglikelihood_tokens, which builds the row HFLM builds from each (context, continuation) pair and scores it with the model’s forward under jax.jit. Slot i of the logits predicts token i + 1, so a continuation of n tokens is read at the n slots that end one before the row’s last token.

NameSummary
DewLMRuns a TextGeneration as an lm-eval-harness model, through the TemplateLM interface.

class source

class DewLM(task: TextGeneration, *, batch_size: int = 1)

Runs a TextGeneration as an lm-eval-harness model, through the TemplateLM interface.

task is the run’s own generation task, with its model, its weights and its processor. A task without a processor raises ValueError, because the harness passes text. batch_size is how many rows one scoring call runs at once. Answers come back in request order, because the harness matches them to its documents by order.

Likelihoods are exact: each is the sum of the model’s log-softmax at the continuation’s own targets. Generation is greedy unless a request’s gen_kwargs ask for sampling with a temperature, read the way HFLM reads them. Greedy is the harness’s own default, and a suite’s reported numbers assume it.

eot_token_id: int

The id a row with no context is conditioned on: the sampling policy’s first EOS. A task whose policy names none has no such id, and is refused (ValueError) rather than given one.

prefix_token_id: int

The id a first token is conditioned on: the tokenizer’s BOS, or eot_token_id without one.

This matches HFLM.prefix_token_id. A vocabulary that starts every sequence with BOS scores its first token after BOS, and conditioning it on EOS instead would change every rolling and empty-context score.

max_length: int

The number of ids one scoring row may hold, as the model declares it (2048 if it does not).

Generation sizes its cache from the same declared field, so a harness row and a generated row have the same bound.

def from_run(
run: str,
*,
batch_size: int = 1,
ema: bool | None = None,
step: int | str | None = None,
mesh: MeshSpec | Mesh | None = None,
layout: Layout | None = None,
dtype: str | None = None,
param_dtype: str | None = None,
trust: Sequence[str] = (),
) -> DewLM

Load the run directory run as a harness model, the way dew.pipeline builds it.

With ema=None, it uses the run’s averaged weights when the run kept them, and its live weights otherwise. mesh, layout, dtype, param_dtype and trust are as TextGeneration.from_run takes them: a run sharded over a mesh, its weights stored narrower. --model_args names the mesh by its axes’ sizes, run=<directory>,fsdp=4,param_dtype=bfloat16 (MeshSpec’s fields).

def create_from_arg_string(
arg_string: str,
additional_config: dict | None = None,
) -> DewLM

Build the model from --model_args run=<directory>,batch_size=4.

def create_from_arg_obj(arg_dict: dict, additional_config: dict | None = None) -> DewLM

Build the model from arguments that are already parsed, as the CLI does.

def tok_encode(
string: str,
add_special_tokens: bool | None = None,
left_truncate_len: int | None = None,
**kwargs: int | None = {},
) -> list[int]

Encode string with the run’s own tokenizer into one row of ids, as HFLM.tok_encode does.

add_special_tokens=True is the tokenizer’s own encoding, its special tokens around the text (a BOS before it, T5’s EOS after it); False is the text’s ids alone. None, the default every scored context and continuation reads, opens the text as the tokenizer opens a sequence and closes it with nothing (_opening): a context and its continuation encode as one row, so an EOS a tokenizer closes a sequence with would end the context before its continuation, shifting where the continuation starts and scoring an EOS it never wrote. The opening is left off where the text already starts with its spelling, as HFLM leaves its BOS off where the text starts with the BOS’s decoded text (Processor.spelling), so no row starts with two. left_truncate_len keeps that many of the last ids. Any other option raises TypeError, since HFLM reads none.

def tok_decode(tokens: Sequence[int]) -> str
def loglikelihood_rolling(
requests: list[Instance],
disable_tqdm: bool = False,
) -> list[float]

Return each string’s log-probability, with every token scored once.

lm-eval’s helpers cut the string into windows. The first window is conditioned on the prefix token and each later one on the max_length ids before it, and no two windows score the same token.

def generate_until(requests: list[Instance], disable_tqdm: bool = False) -> list[str]

Continue each context until one of its stop strings or its token budget, and return the texts.

Each request is read the way HFLM.generate_until reads it, through lm-eval’s own helpers. normalize_gen_kwargs accepts the budget under any of its names and makes do_sample=False greedy whatever the temperature. The EOS token’s text is added to the stop strings, and a context longer than the window minus the budget keeps its last ids. The stop strings cut the decoded text, so a stop sequence that spans two tokens ends the answer where the harness expects. A budget that fills the whole window raises ValueError, as it does in HFLM.

A beam search (num_beams above 1) raises ValueError. HFLM always stops on its EOS’s text as well as its stop strings, and its criterion reads the leading candidate alone and ends every beam together, which neither Dew’s per-beam search nor a cut of its answer reproduces. A request’s max_length is the total length HFLM hands generate, context included, in place of the budget. Its temperature is 0 where it names none, which decodes greedily unless the request asks to sample; sampling at 0 raises ValueError, as generate does. The other controls are the ones HFLM hands Transformers’ generate, and they decode as a source’s generation config does (dew.interop.generation_config.source_decoding), over the task’s own EOS and pad ids where the request names none: sampling filters, penalties and logits processors. One Dew has no counterpart for raises ValueError, as generate refuses a keyword it does not use.