dew.eval.harness
Run a saved run as an lm-evaluation-harness model.
DewLM wraps a TextGeneration in lm-eval-harness’s TemplateLM
interface, so any task suite can run against a run directory.
lm_eval is an optional extra (pip install dewml[eval-harness]). Only this
module imports it, and dew.eval does not import this module. Importing this
module registers the adapter under the name dew. lm_eval 0.4 has no plugin
discovery, so the import has to happen in the process that runs the command.
python -m dew.eval makes that import and then runs lm_eval’s own command
line:
python -m dew.eval --model dew --model_args run=runs/shakespeare \ --tasks hellaswag --limit 4lm-eval’s own code decides which tokens are scored (TemplateLM.loglikelihood,
get_rolling_token_windows). This module adds _loglikelihood_tokens, which
builds the row HFLM builds from each (context, continuation) pair and
scores it with the model’s forward under jax.jit. Slot i of the logits
predicts token i + 1, so a continuation of n tokens is read at the n
slots that end one before the row’s last token.
| Name | Summary |
|---|---|
DewLM | Runs a TextGeneration as an lm-eval-harness model, through the TemplateLM interface. |
class DewLM(task: TextGeneration, *, batch_size: int = 1)Runs a TextGeneration as an lm-eval-harness model, through the TemplateLM interface.
task is the run’s own generation task, with its model, its weights and
its processor. A task without a processor raises ValueError, because the
harness passes text. batch_size is how many rows one scoring call runs
at once. Answers come back in request order, because the harness matches
them to its documents by order.
Likelihoods are exact: each is the sum of the model’s log-softmax at the
continuation’s own targets. Generation is greedy unless a request’s
gen_kwargs ask for sampling with a temperature, read the way HFLM
reads them. Greedy is the harness’s own default, and a suite’s reported
numbers assume it.
eot_token_id: int-
The id a row with no context is conditioned on: the sampling policy’s first EOS. A task whose policy names none has no such id, and is refused (ValueError) rather than given one.
prefix_token_id: int-
The id a first token is conditioned on: the tokenizer’s BOS, or
eot_token_idwithout one.This matches
HFLM.prefix_token_id. A vocabulary that starts every sequence with BOS scores its first token after BOS, and conditioning it on EOS instead would change every rolling and empty-context score. max_length: int-
The number of ids one scoring row may hold, as the model declares it (2048 if it does not).
Generation sizes its cache from the same declared field, so a harness row and a generated row have the same bound.
DewLM.from_run
Section titled “DewLM.from_run”def from_run( run: str, *, batch_size: int = 1, ema: bool | None = None, step: int | str | None = None, mesh: MeshSpec | Mesh | None = None, layout: Layout | None = None, dtype: str | None = None, param_dtype: str | None = None, trust: Sequence[str] = (),) -> DewLMLoad the run directory run as a harness model, the way dew.pipeline builds it.
With ema=None, it uses the run’s averaged weights when the run kept
them, and its live weights otherwise. mesh, layout, dtype, param_dtype and
trust are as TextGeneration.from_run takes them: a run sharded over a mesh, its
weights stored narrower. --model_args names the mesh by its axes’ sizes,
run=<directory>,fsdp=4,param_dtype=bfloat16 (MeshSpec’s fields).
DewLM.create_from_arg_string
Section titled “DewLM.create_from_arg_string”def create_from_arg_string( arg_string: str, additional_config: dict | None = None,) -> DewLMBuild the model from --model_args run=<directory>,batch_size=4.
DewLM.create_from_arg_obj
Section titled “DewLM.create_from_arg_obj”def create_from_arg_obj(arg_dict: dict, additional_config: dict | None = None) -> DewLMBuild the model from arguments that are already parsed, as the CLI does.
DewLM.tok_encode
Section titled “DewLM.tok_encode”def tok_encode( string: str, add_special_tokens: bool | None = None, left_truncate_len: int | None = None, **kwargs: int | None = {},) -> list[int]Encode string with the run’s own tokenizer into one row of ids, as HFLM.tok_encode does.
add_special_tokens=True is the tokenizer’s own encoding, its special tokens around the text
(a BOS before it, T5’s EOS after it); False is the text’s ids alone. None, the default every
scored context and continuation reads, opens the text as the tokenizer opens a sequence and
closes it with nothing (_opening): a context and its continuation encode as one row, so an
EOS a tokenizer closes a sequence with would end the context before its continuation,
shifting where the continuation starts and scoring an EOS it never wrote. The opening is
left off where the text already starts with its spelling, as HFLM leaves its BOS off where
the text starts with the BOS’s decoded text (Processor.spelling), so no row starts with two.
left_truncate_len keeps that many of the last ids. Any other option raises TypeError, since
HFLM reads none.
DewLM.tok_decode
Section titled “DewLM.tok_decode”def tok_decode(tokens: Sequence[int]) -> strDewLM.loglikelihood_rolling
Section titled “DewLM.loglikelihood_rolling”def loglikelihood_rolling( requests: list[Instance], disable_tqdm: bool = False,) -> list[float]Return each string’s log-probability, with every token scored once.
lm-eval’s helpers cut the string into windows. The first window is
conditioned on the prefix token and each later one on the
max_length ids before it, and no two windows score the same token.
DewLM.generate_until
Section titled “DewLM.generate_until”def generate_until(requests: list[Instance], disable_tqdm: bool = False) -> list[str]Continue each context until one of its stop strings or its token budget, and return the texts.
Each request is read the way HFLM.generate_until reads it, through
lm-eval’s own helpers. normalize_gen_kwargs accepts the budget under
any of its names and makes do_sample=False greedy whatever the
temperature. The EOS token’s text is added to the stop strings, and a
context longer than the window minus the budget keeps its last ids.
The stop strings cut the decoded text, so a stop sequence that spans
two tokens ends the answer where the harness expects. A budget that
fills the whole window raises ValueError, as it does in HFLM.
A beam search (num_beams above 1) raises ValueError. HFLM always
stops on its EOS’s text as well as its stop strings, and its criterion
reads the leading candidate alone and ends every beam together, which
neither Dew’s per-beam search nor a cut of its answer reproduces.
A request’s max_length is the total length HFLM hands generate,
context included, in place of the budget. Its temperature is 0 where
it names none, which decodes greedily unless the request asks to
sample; sampling at 0 raises ValueError, as generate does. The other
controls are the ones HFLM hands Transformers’ generate, and they
decode as a source’s generation config does
(dew.interop.generation_config.source_decoding), over the task’s own
EOS and pad ids where the request names none: sampling filters,
penalties and logits processors. One Dew has no counterpart for raises
ValueError, as generate refuses a keyword it does not use.