Skip to content

dew.decision

Decision models: one forward pass answers typed questions about a state.

A request is a state (text, or any JSON) and named questions, each a Noul, a Choice or a Score (TypeSafe Jev’s three, docs.typesafe.ai). The model lays the questions out with the state (Laya’s MarkerLayout, the StateFirstLayout for a causal backbone, or the JointLayout, every question of a request in one row), reads the rows with a backbone, scores every option with a head (Laya’s DecisionHead, or the JointSchemaHead, which decides a row’s questions together, or a readout of options written under Labels: LabelReadout, the backbone’s own logits at them, and a slot release’s SlotReadout), and answers with a distribution over exactly the options each question names. Decide is the task that answers requests, and DecisionObjective trains one on proper scoring rules. Pretrained.load reads released Laya and joint-head sources, and Decide.of turns their declared settings into a decision task.

NameSummary
AURCThe area under the risk-coverage curve.
ECEExpected calibration error: the gap between the top probability and the accuracy.
KINDSThe question types in the order a laid-out row numbers them, Laya’s (laya/common.py QTYPES: choice 0, score 1, noul 2).
NONE_OF_THE_ABOVEDocumented in dew.decision.rows.
TASK_FILEThe file Decide.save writes into a run: its calibration, budget and name. Documented in dew.decision.task.
AbstentionThe lowest calibrated top probability an answer needs, per bucket, or else default.
AccuracyThe share of answered questions whose most likely option is the right one.
Answer
AnswerCueThe cue a readout answers after: template filled with the options’ labels joined by join, many past at options. Documented in dew.decision.layout.
AnsweredThe questions of a Decisions with a known answer, one row each: their probabilities [M, K], real options [M, K], right options [M], gold distributions [M, K] and whether [M] they ask a score.
BinningHistogram binning of the calibrated top probability, per bucket.
BrierThe quadratic score, sum_k (p_k - t_k)^2, between 0 and 2.
BudgetHow much one forward pass may hold: at most tokens tokens in at most rows windows. Documented in dew.decision.task.
CalibrationHow a decision model’s raw logits become reported answers.
ChoiceA choice among named options, 1 to 255 of them.
ChoiceAnswerOne probability per option, and the most likely option.
CombinedA weighted sum of rules, itself proper when every weight is positive.
ConfidenceA statistic of an answer’s distribution, between 0 and 1.
DecideAnswers typed questions about a state, with one forward pass per batch. Documented in dew.decision.task.
DecisionHeadScores each option of a question at its marker in the backbone’s states.
DecisionInputsA batch of laid-out rows: packed windows of whole rows, or rows padded to the longest. Documented in dew.decision.layout.
DecisionModelAny backbone with final states, read by a decision head (HeadedModel).
DecisionObjectiveTrains a decision model on questions with known answers. Documented in dew.decision.objective.
EncodedOne row: its tokens and the questions laid out in it. Documented in dew.decision.layout.
EncodingHow a labelled example becomes the row a step packs and trains on. Documented in dew.decision.rows.
EntropyConfidenceLaya’s confidence: one minus the distribution’s entropy over log n (laya/common.py, confidence_from_probs), for every type but a Noul, which Laya gives its top probability.
ExampleOne state, its questions, and the right answer to some of them. Documented in dew.decision.data.
FramingHow a joint row frames its text as a conversation: a system message, the user’s turn that holds the state and the schema, and the assistant’s turn opened with prefill. Documented in dew.decision.layout.
GroupQuestions of one example that ask the same thing (negated, paraphrased, its options re-ordered), named name: members maps each one’s every option key onto a canonical key the members share, so a consistency term (DecisionObjective.consistency) can hold their answers to one distribution over the canonical keys. Documented in dew.decision.data.
GroupedExamplesExamples whose rows train together, so a consistency term (DecisionObjective.consistency) holds questions asked across them to one answer: the same question negated, paraphrased or its options re-ordered in rows of their own, under any layout. Documented in dew.decision.data.
HeadWhat every decision head offers: its call, whether it reads the output table, and the record a run keeps of it.
InstructionsA question’s instructions under a header of its kind (headers, {} the rendered instructions), rendered by render (rendered); defaults are a kind’s instructions where its question’s render empty, as instructions of whitespace alone do under a release’s entry. Documented in dew.decision.layout.
JevConfidenceTypeSafe Jev’s confidence (docs.typesafe.ai/confidence).
JointLayoutEvery question of a request in one row, each option a span of its own. Documented in dew.decision.layout.
JointSchemaHeadThe joint schema head: decides every question of a row together.
LabelReadoutDew’s readout of labels: each option slot’s logit is the backbone’s own logit for its label at the question’s answer cue (dew.decision.layout.Labels), through the output table’s input QDQ, scale, bias and softcap, computed in fp32 (DecisionModel.readout_logits, OutputTable.columns).
LabelsOptions rendered as lettered lines, read at the row’s answer cue by a readout of the labels’ logits (dew.decision.head.LabelReadout, SlotReadout): slot s as {name}. Documented in [dew.decision.layout`](/api/dew.decision.layout/).
LaidOne question as a row lays it out. Documented in dew.decision.layout.
LayaCheckpointOne released Laya checkpoint: its model, variables, layout and tokenizer, and its agent’s temperatures.
LayoutHow a state and its questions become rows of tokens that fit max_len (fits). Documented in dew.decision.layout.
LogLossThe cross entropy, -sum_k t_k log p_k: the logarithmic score.
MacroF1The mean over classes of each class’s F1, the classes those the split’s answered questions are or are answered as (sklearn’s f1_score(average="macro")): a class is a question’s option (Decisions.classes), the same in every batch and split that asks it, so each class’s true and false positives and false negatives add over the whole pass before its F1 is taken.
MarkerLayoutLaya’s layout, for a bidirectional backbone: the question, a marker before each option, then the state. Documented in dew.decision.layout.
MediaWhere a joint row places a request’s images, each as the backbone’s processor writes one (Processor.image_text) and expands it; a request without images places nothing here. Documented in dew.decision.layout.
NoulA yes-or-no question.
NoulAnswerThe answer to a Noul: P(false) and P(true), and a confidence computed from them.
OptionLinesA question’s options, one line each, joined by join: line filled with the option’s label (QuestionLayout.labels), its name and its description rendered by render, bare where that renders empty, and {option} its whole text as the built-in rows write it (QuestionLayout.options: key: description, level i: description, false: ...); each option’s span is its whole line. Documented in dew.decision.layout.
QuestionOne question about a state, with its options in the order the model scores them.
QuestionLayoutA layout of one row per question, scored at one marker token per option. Documented in dew.decision.layout.
QuestionSegmentA part of a question row’s text (QuestionLayout.segments). Documented in dew.decision.layout.
RankedProbabilityThe ranked probability score of ordered levels.
SchemaEvery question of the request as a field of the schema, never cut: its id, type and instructions, then each allowed option, each rendered as compact JSON with sorted keys, {"description": ..., "option_id": ...}. Documented in dew.decision.layout.
ScoreA rating on ordered levels, 2 to 10 of them, lowest first.
ScoreAnswerOne probability per level, lowest first, and the expected level.
ScoredOne held-out answer, with its question’s type and the model’s raw logits.
ScoredQuestionsHow many questions with a known answer the split scored, the count every other metric of it is taken over.
ScoringRuleA proper scoring rule, lower is better.
SegmentA part of a joint row’s user turn (JointLayout.segments). Documented in dew.decision.layout.
SlotReadoutA published slot release’s head, its law as the release’s runtime computes it: slot i’s score is <slot_i, h> + <q(h), k(m_i)> / sqrt(head_dim), h the question’s answer state (DecisionInputs.readout), m_i the mean of the final states over option i’s whole rendered line (its span), q and k each a [head_dim, features] projection, with no slot bias and no softcap on the sum.
SpecialsThe special tokens a layout places. Documented in dew.decision.layout.
SphericalThe spherical score as a charge, 1 - (t .
StateThe state, as compact JSON with sorted keys, its start kept: cut to its own budget (max_tokens, or a joint layout’s max_state_tokens where it gives none) and to its share of what the rest of the row leaves of its max_len (dew.data.tokens.share_tokens). Documented in dew.decision.layout.
StateFirstLayoutThe layout for a causal backbone: the state, the question, then each option followed by its marker. Documented in dew.decision.layout.
TemperaturesThe temperature each question’s logits are divided by.
TextText a joint row writes as given, at its place in the user’s turn. Documented in dew.decision.layout.
TopProbabilityThe answer’s own probability, the largest one.
UsageWhat a request cost: the tokens every question row read, how much of the state the rows kept, the smallest share any question kept, and each cut by what it cut, the rows it shortened and the tokens it removed, as training counts a row’s cuts (cut_counts). Documented in dew.decision.task.
WeightsThe checkpoint a task’s weights came from: its step, and whether they are the run’s average. Documented in dew.decision.task.
bucketReturn the calibration bucket of question: its type and its option count.
corpusCorpus name’s split of the mixture directory root_url as decision loading reads it, from the shards and index its manifest names (or the corpus’s one JSONL file), each row an Example under example; None when the corpus has no such split. Documented in dew.decision.data.
decisionsdataset’s rows as decisions, each an Example under example, for a chain to batch: a flat table’s text and label columns read as one question over options (its ClassLabel names, or read whole in memory, where none are given), its context column, where named, each example’s context, or rows that hold their examples already. Documented in dew.decision.data.
kind_ofReturn the number a laid-out row gives question’s type (KINDS).
mixtureThe decision corpora of the mixture directory root_url (its mixture.json): every corpus of a positive weight, the manifest’s weights over weights, mixed by them for training, and their held rows together, or None where none has any. Documented in dew.decision.data.
renderReturn a value as the model reads it: text as it is, and anything structured as JSON. Documented in dew.decision.layout.

dataclass source

class AURC()

The area under the risk-coverage curve.

It is the error rate among the rows at or above each confidence, weighted by how many rows that confidence adds, so a group of equal confidences counts as a whole in any row order (Laya’s aurc, laya/evals.py). Lower is better; it rewards a confidence that ranks right answers above wrong ones.

def merge(
accumulated: tuple[np.ndarray, np.ndarray],
contribution: tuple[np.ndarray, np.ndarray],
) -> tuple[np.ndarray, np.ndarray]
def finalize(accumulated: tuple[np.ndarray, np.ndarray]) -> float

dataclass source

class ECE(bins: int = 15)

Expected calibration error: the gap between the top probability and the accuracy.

It is taken over bins equal bins on (0, 1], closed on the right, and weighted by how many rows each bin holds (Laya’s ece_score, 15 bins).

def binned(tops: np.ndarray, correct: np.ndarray) -> np.ndarray

Per bin, the row count, the summed top probability tops and the number correct, for answers however they were drawn: finalize reads these, from a run’s validation pass or a served benchmark’s (reliability_bins).

def merge(accumulated: np.ndarray, contribution: np.ndarray) -> np.ndarray
def finalize(accumulated: np.ndarray) -> float

attribute source

KINDS: tuple[type[Question], ...] = (Choice, Score, Noul)

The question types in the order a laid-out row numbers them, Laya’s (laya/common.py QTYPES: choice 0, score 1, noul 2). A head whose type embedding orders them otherwise maps these numbers to its own rows.

dataclass source

class Abstention(thresholds: Mapping[str, float] = dict(), default: float | None = None)

The lowest calibrated top probability an answer needs, per bucket, or else default.

An answer below it abstains. None gates nothing.

def threshold(kind: str, count: int) -> float | None
def fit(
scored: Sequence[Scored],
temperatures: Temperatures,
binning: Binning | None = None,
*,
target_error: float = 0.1,
bucket_minimum: int = 100,
conservative: bool = True,
) -> Abstention

Return the smallest threshold that meets target_error, per bucket.

Only buckets with at least bucket_minimum answers get one. A threshold meets the target when the accepted answers err at most target_error, counting a group of tied answers as a whole; with conservative, the error is estimated as (errors + 1) / (accepted + 1). A bucket that no threshold satisfies gets 1.0 (Laya’s fit_abstention_thresholds).

dataclass source

class Accuracy()

The share of answered questions whose most likely option is the right one.

attribute source

dataclass source

class Answered(
probabilities: np.ndarray,
options: np.ndarray,
labels: np.ndarray,
targets: np.ndarray,
ordinal: np.ndarray,
)

The questions of a Decisions with a known answer, one row each: their probabilities [M, K], real options [M, K], right options [M], gold distributions [M, K] and whether [M] they ask a score.

def of(artifact: Artifact) -> Answered

dataclass source

class Binning(maps: Mapping[str, tuple[float, ...]] = dict())

Histogram binning of the calibrated top probability, per bucket.

bins equal bins on [0, 1] each map to the held-out accuracy of the answers that fell in them (Laya’s fit_binning_map). A bucket without a map keeps the probability.

def apply(kind: str, count: int, top: float) -> float
def fit(
scored: Sequence[Scored],
temperatures: Temperatures,
*,
bins: int = 15,
bucket_minimum: int = 200,
) -> Binning

Return a map for every bucket with at least bucket_minimum answers.

An empty bin keeps its midpoint.

dataclass source

class Brier()

The quadratic score, sum_k (p_k - t_k)^2, between 0 and 2.

def charge(logits, target, options, ordinal)

dataclass source

class Calibration(
temperatures: Temperatures = Temperatures(),
binning: Binning | None = None,
abstention: Abstention | None = None,
)

How a decision model’s raw logits become reported answers.

def probabilities(question: Question, logits: np.ndarray) -> np.ndarray
def abstains(question: Question, probabilities: np.ndarray) -> bool

Return whether the answer’s calibrated top probability is below its threshold.

dataclass source

class Choice(instructions: JSON, criteria: Mapping[str, JSON] | Sequence[str])

A choice among named options, 1 to 255 of them.

criteria maps each option’s key to what it means, or to None where the key says enough; a list of keys describes none of them.

def answer(probabilities: np.ndarray, confidence: Confidence) -> ChoiceAnswer
def wire() -> dict[str, JSON]

dataclass source

class ChoiceAnswer(
options: tuple[str, ...],
probabilities: np.ndarray,
confidence: float,
abstained: bool = False,
)

One probability per option, and the most likely option.

choice: str

The most likely option; the first of several equally likely ones.

dataclass source

class Combined(terms: tuple[tuple[float, ScoringRule], ...])

A weighted sum of rules, itself proper when every weight is positive.

def charge(logits, target, options, ordinal)

class source

class Confidence(ABC)

A statistic of an answer’s distribution, between 0 and 1.

class source

class DecisionHead(Head)

Scores each option of a question at its marker in the backbone’s states.

This is Laya’s head (laya/common.py DecisionModel at NandhaKishorM/laya a4a8921). The question type’s embedding is added to every state, then layers pre-norm transformer blocks run over the row, as nn.TransformerEncoderLayer(norm_first=True) builds them: LayerNorm with bias at 1e-5, biased attention with features // 64 heads and no position encoding, a biased ReLU feed-forward four times as wide, and padding keys masked. Then a LayerNorm, Dense, erf GELU, Dense scorer reads the state at each marker. The blocks are Dew’s DecoderBlock. Laya’s action head is left out, because its own model card reports that it carries no signal.

It reads one question per row (QuestionLayout), the first token of each option slot being its marker, and a window’s rows apart from each other.

def decision_flops(lengths, questions, options) -> float

Full-row encoder layers and the scorer at real option markers; it pools no spans.

def setup()

dataclass source

class DecisionModel(
backbone: nn.Module,
head: Head,
context_encoder: ContextEncoder = 'table',
)

Any backbone with final states, read by a decision head (HeadedModel).

The rows a window packs reach the backbone as its documents (Coordinates). In the parallel layout an option attends to no other option’s tokens, and the backbone’s attention windows measure distance in positions (attention_pairwise_mask, attention_key_positions).

context_encoder: ContextEncoder

How a row’s context becomes the states its cross-attention reads (encode_context): table, its tokens through the backbone’s own embedding table, or backbone, the backbone’s final states over it in a pass of its own, as a schema read in context always is.

def reads_progress() -> bool

Whether its backbone’s computation changes over a run (dew.nn.progress.ReadsProgress).

def initialize_head(key: jax.Array, backbone: Variables, tokens: jax.Array) -> Variables

Draw the head over a held backbone, at tokens’ shape, without running the backbone.

Readouts draw from its output table or vocabulary rows as their own contract declares; a tied readout has no local parameters. The token values are only initialization geometry.

def states(
variables: Variables,
inputs: DecisionInputs,
*,
media: ModelInputs | None = None,
train: bool = False,
rngs: dict[str, jax.Array] | None = None,
progress: Progress = END,
) -> jax.Array

Return the backbone’s [B, L, features] final states, each packed row read apart from the others in its window.

media are the backbone’s own inputs for images a request’s one row holds (Images.inputs), with which it reads them. Rows that carry a context (DecisionInputs.context) hand the backbone’s cross-attention each row’s context, embedded by the backbone’s own table (stream_embeddings), or, where the context holds the schema (DecisionInputs.spans_in_context), the backbone’s own final states over it (encode_context). The backbone reads the run’s progress where it reads it (dew.nn.progress), the run’s end by default.

def logits(
variables: Variables,
inputs: DecisionInputs,
*,
media: ModelInputs | None = None,
train: bool = False,
rngs: dict[str, jax.Array] | None = None,
progress: Progress = END,
) -> jax.Array

Return the [B, Q, K] fp32 option logits, with -inf past each question’s options, the backbone at the run’s progress (states).

def observed_logits(
variables: Variables,
inputs: DecisionInputs,
*,
media: ModelInputs | None = None,
train: bool = False,
rngs: dict[str, jax.Array] | None = None,
progress: Progress = END,
opened: Sequence[str] = (),
) -> tuple[jax.Array, Variables]

The option logits (logits), and what the row’s pass sowed into the opened collections (dew.objectives.base.Step.sown).

def readout_logits(
variables: Variables,
states: jax.Array,
inputs: DecisionInputs,
labels: tuple[tuple[int, ...], ...],
) -> jax.Array

The backbone’s own logits [B, Q, K, V] at each label’s ids of each question’s answer state (DecisionInputs.readout), its first K labels as many as its option slots (HeadedModel.label_logits).

class source

class EntropyConfidence(Confidence)

Laya’s confidence: one minus the distribution’s entropy over log n (laya/common.py, confidence_from_probs), for every type but a Noul, which Laya gives its top probability.

Flax module source

class Head(
reads_table: bool = False,
reads_labels: bool = False,
draws_from_rows: bool = False,
pools_spans: bool = False,
one_question: bool = False,
width_field: str = 'features',
)

What every decision head offers: its call, whether it reads the output table, and the record a run keeps of it.

A run records a head’s fields, its numerics among them, but not its width, which is the backbone’s.

reads_labels: bool

Whether the head reads the answer state of each question at its row’s answer cue (DecisionInputs.readout) against labels (Labels), the backbone’s output table its table (OutputTable, None where no matrix alone is the backbone’s head).

draws_from_rows: bool

Whether the head draws its parameters from the backbone’s vocabulary rows (VocabularyRows), its table when it is drawn, and reads no table after.

pools_spans: bool

Whether the head pools states over the questions’ spans (DecisionInputs.spans, option_spans), from the pass that holds them: the context’s states (spanned) where the spans are the row’s context’s (DecisionInputs.spans_in_context), its pooling then counted over the context’s tokens (decision_flops(pooled=)).

one_question: bool

Whether the head reads one question a row, refusing a joint row’s several questions at the host boundary (validate_rows).

width_field: str

The field holding the width of the states the head reads.

def tied_labels() -> tuple[tuple[int, ...], ...] | None

The label ids, [K][V], at which the head reads the backbone’s own logits (tied), or None where it reads none.

def labelled(labels: Labels, tokenizer: Tokenizer) -> tuple[Head, Labels]

The head reading labels as tokenizer spells them, and the labels as named, where it reads labels and was drawn before its tokenizer was known, with no ids (a run’s letter readout, LabelReadout); itself and labels otherwise.

def validate_rows(rows: Sequence[Encoded]) -> None

Refuse encoded host rows that ask more questions than this head reads.

def record() -> Record

Return the head as a run records it, the registry’s class record (dew.registry.component): its class by import path and its fields but the width, which the backbone it reads gives, so any head reloads, one of a package the run trusts too.

def from_record(record: JSON | Mapping[str, object], width: int) -> Head

Rebuild the head a run recorded, reading states width wide, in the numerics it recorded. An earlier run’s record is read through dew.inference.tasks.earlier_shapes.

class source

class JevConfidence(Confidence)

TypeSafe Jev’s confidence (docs.typesafe.ai/confidence).

For a Choice of n options, it is how far the top probability sits above an even split, (p_max - 1/n) / (1 - 1/n). For a Score, it weighs each level’s distance from the most likely level m, 1 - sum_i p_i |i - m| / MAD, where MAD is the same average distance for an even spread measured from the middle level, floored at 0. Jev reports none for a Noul; this gives the Choice formula over its two options, |2 p - 1|, as Jev’s page suggests. A choice with one option is certain.

class source

class JointSchemaHead(Head)

The joint schema head: decides every question of a row together.

Over the backbone’s normed states it pools each question’s instructions and each option’s tokens, and each option’s output-table rows (the lexical vector). The options attend to the row in routing_layers routing layers; each question becomes a field from its own pooled tokens, an attention-weighted summary of its options, the row’s last token and its type; the fields attend to each other and to the row in layers decoder layers. An option’s logit is a lexical prior (the cosine of its lexical vector with its question and the last token) plus a gated joint score (the cosine of field and option, and a residual MLP over the pair). The three scales are learned in log space and held to 100 at most.

It reads every question of a row (JointLayout), and the backbone’s output table. In a window of packed rows each question and option reads its own row alone, and the row’s last token is its own. Each option is computed on its own but for its weight in its question’s summary, so where the inputs compact each window’s real options (DecisionInputs.option_slots) it computes those alone, and lays their routed states (for the summaries) and their logits back on the grid.

def decision_flops(lengths, questions, options, *, pooled=None) -> float

Memory/option projections, evidence O-by-L and field Q-by-L attention, and the span pooling over each row, or over pooled tokens a row where the spans are a context’s.

Each attention has query/output maps on its queries and key/value maps on memory. The field layers also have Q-by-Q self-attention. The residual scorer reads four width-sized features per option. Span pooling is a masked matrix product and is counted. Lookup embeddings and pointwise normalization/gates are excluded.

def setup()

class source

class LabelReadout(Head)

Dew’s readout of labels: each option slot’s logit is the backbone’s own logit for its label at the question’s answer cue (dew.decision.layout.Labels), through the output table’s input QDQ, scale, bias and softcap, computed in fp32 (DecisionModel.readout_logits, OutputTable.columns). A label read as several token variants (label_ids[k], one id a variant) is their grouped logit, the logsumexp of theirs. Untrained, the head is the backbone’s own readout.

untied gives each slot’s every variant a trainable row and bias of its own, drawn from its variant’s output row (scale folded in) and bias, read through the same input QDQ and softcap and grouped alike, so at its draw it is the tied readout at every state; it needs the backbone’s output table to draw from (table). Its rows and their product compute in at least float32, its table’s or dtype where wider.

A row’s labels are the head’s, its first as many as the question has options (validate_rows); option slots past the labels, a batch’s padding, read -inf. A head drawn with no ids reads count labels, named where its tokenizer is known: the layout’s own names, or where they name none, the first count its tokenizer spells as one token each (labelled, Labels.spelled); with no count, every label it letters (Labels.capacity).

def tied_labels() -> tuple[tuple[int, ...], ...] | None
def labelled(labels: Labels, tokenizer: Tokenizer) -> tuple[LabelReadout, Labels]
def validate_rows(rows: Sequence[Encoded]) -> None

Refuse rows whose labels are not the head’s (_labels_of).

def decision_flops(lengths, questions, options) -> float

The labels’ product over each answer state: its slots’ variants, a row each.

dataclass source

class LayaCheckpoint(
model: DecisionModel,
variables: Variables,
layout: MarkerLayout,
tokenizer: HFTokenizer,
specials: Specials,
temperatures: tuple[float, float, float],
bucket_temperatures: dict[str, float],
)

One released Laya checkpoint: its model, variables, layout and tokenizer, and its agent’s temperatures.

temperatures is rl_agent_config.json’s per-type list (choice, score, noul), and bucket_temperatures its per-(type, option count) map, as the file states them.

def save(directory: str | Path, *, max_shard_size: int | str = MAX_SHARD_SIZE) -> None

Write this checkpoint in Laya’s layout, which Pretrained.load and llama.cpp’s converter read.

That is model.safetensors (the encoder’s ModernBertModel tensors under encoder. and the head’s under Laya’s names), encoder/config.json, the tokenizer under tokenizer/, and rl_agent_config.json with the head’s depth, the layout’s budgets and the temperatures. The backbone must be a ModernBERT encoder and the head Laya’s. A backbone under LoRA is written with its factors merged into their kernels (Pretrained.bound). Laya’s config states a layout’s window, question budgets and option layout alone, so a layout that chooses more (an order, labels, segments, blocks) is refused before anything is written; a run saved by Decide.save keeps it.

dataclass source

class LogLoss()

The cross entropy, -sum_k t_k log p_k: the logarithmic score.

def charge(logits, target, options, ordinal)

dataclass source

class MacroF1()

The mean over classes of each class’s F1, the classes those the split’s answered questions are or are answered as (sklearn’s f1_score(average="macro")): a class is a question’s option (Decisions.classes), the same in every batch and split that asks it, so each class’s true and false positives and false negatives add over the whole pass before its F1 is taken.

def merge(
accumulated: dict[int, np.ndarray],
contribution: dict[int, np.ndarray],
) -> dict[int, np.ndarray]
def finalize(accumulated: dict[int, np.ndarray]) -> float

dataclass source

class Noul(instructions: JSON, criteria: Mapping[str, JSON] = dict())

A yes-or-no question. criteria may say what true and false mean.

Its options are false then true, so an answer’s second probability is the probability of yes.

def answer(probabilities: np.ndarray, confidence: Confidence) -> NoulAnswer
def wire() -> dict[str, JSON]

dataclass source

class NoulAnswer(probabilities: np.ndarray, confidence: float, abstained: bool = False)

The answer to a Noul: P(false) and P(true), and a confidence computed from them.

abstained: bool

Whether a calibration’s abstention threshold gated the answer out.

noul: float

The probability that the answer is yes.

class source

class Question(ABC)

One question about a state, with its options in the order the model scores them.

kind: str

Jev’s name for the question type: noul, choice or score.

options: tuple[str, ...]

Return the answer keys, one per option, in the order the model scores them.

descriptions: tuple[JSON, ...]

Return what each option means, as the request gave it, with None where it gave nothing.

def answer(probabilities: np.ndarray, confidence: Confidence) -> Answer

Return this question’s answer for probabilities, one per option.

def wire() -> dict[str, JSON]

Return the question in a Jev request’s format.

def from_wire(request: Mapping[str, object]) -> Question

Build the question from a Jev request’s type, instructions and criteria.

def index(label: object) -> int

Return the position of the option that label names, given as its key or as an int index.

dataclass source

class RankedProbability()

The ranked probability score of ordered levels.

It sums the squared gaps between the forecast’s and the target’s cumulative distributions over the K - 1 cuts between K levels, sum_k (P_k - T_k)^2 / (K - 1). It reads only a score question’s levels and charges any other row nothing, since its options have no order. As a metric it averages over the scored rows.

def charge(logits, target, options, ordinal)

dataclass source

class Score(instructions: JSON, criteria: Sequence[JSON])

A rating on ordered levels, 2 to 10 of them, lowest first.

criteria describes each level; the answer’s score is the expected level, so it can land between two.

def answer(probabilities: np.ndarray, confidence: Confidence) -> ScoreAnswer
def wire() -> dict[str, JSON]

dataclass source

class ScoreAnswer(
legend: tuple[JSON, ...],
probabilities: np.ndarray,
confidence: float,
abstained: bool = False,
)

One probability per level, lowest first, and the expected level.

score: float

The expected level, which can land between two.

dataclass source

class Scored(kind: str, logits: np.ndarray, target: np.ndarray)

One held-out answer, with its question’s type and the model’s raw logits.

It holds the question’s type (Question.kind), the model’s raw logits over its options, and the gold distribution over them (Example.distribution): a target’s, or all of the mass on the answer.

label: int

The right option: the gold’s most likely one, as Example.labels reads a target.

dataclass source

class ScoredQuestions()

How many questions with a known answer the split scored, the count every other metric of it is taken over.

def merge(accumulated: int, contribution: int) -> int
def finalize(accumulated: int) -> float

class source

class ScoringRule(ABC)

A proper scoring rule, lower is better.

def charge(
logits: jax.Array,
target: jax.Array,
options: jax.Array,
ordinal: jax.Array,
) -> jax.Array

Return the [B] charge of each row.

The inputs are the [B, K] logits (any value past a row’s options), the [B, K] target distributions, the [B, K] real options, and the [B] rows whose options are ordered levels.

class source

class SlotReadout(Head)

A published slot release’s head, its law as the release’s runtime computes it: slot i’s score is <slot_i, h> + <q(h), k(m_i)> / sqrt(head_dim), h the question’s answer state (DecisionInputs.readout), m_i the mean of the final states over option i’s whole rendered line (its span), q and k each a [head_dim, features] projection, with no slot bias and no softcap on the sum. Its slot rows are drawn from the backbone’s output rows of the slots’ label ids, q from a normal of std features ** -0.5 and k at zero, so at its draw it scores the labels’ rows alone; a release’s reader loads all three (readout_head.pt).

slot_rows is the rows the head holds, the labels’ count where unset: a release may hold more rows than its tokenizer spells labels for, the rows past the labels kept, drawn at zero, and read by no option. Drawn fresh, its slot rows are the backbone’s vocabulary rows at its labels (VocabularyRows, its table then), or a normal of std features ** -0.5 over a backbone that gives none. It computes in at least float32, its states’ or dtype where wider; option slots past the labels, a batch’s padding, read -inf.

def validate_rows(rows: Sequence[Encoded]) -> None

Refuse rows whose labels are not the head’s (_labels_of).

def decision_flops(lengths, questions, options, *, pooled=None) -> float

Each real option’s slot product and key, its affinity with its question’s query, and its line’s pooling over the row, or over pooled tokens a row where the lines are a context’s; each question’s query.

dataclass source

class Spherical()

The spherical score as a charge, 1 - (t . p) / |p|, between 0 and 1.

def charge(logits, target, options, ordinal)

dataclass source

class Temperatures(
types: Mapping[str, float] = dict(),
buckets: Mapping[str, float] = dict(),
lowest: float = 0.5,
highest: float = 5.0,
)

The temperature each question’s logits are divided by.

The temperatures are held within [lowest, highest], as Laya’s agent holds a checkpoint’s: it refuses to sharpen past 0.5 (its clamp_temperature).

def of(kind: str, count: int) -> float

Return the temperature for a question of type kind (Question.kind) with count options.

That is its bucket’s temperature, or else its type’s, or else 1.

def applied() -> Temperatures

Return these temperatures as of applies them, each held within [lowest, highest].

A reader that does not hold them itself, as llama.cpp’s server does not, then answers as this does.

def fit(
scored: Sequence[Scored],
*,
type_minimum: int = TYPE_MINIMUM,
bucket_minimum: int = BUCKET_MINIMUM,
lowest: float | None = None,
highest: float | None = None,
) -> Temperatures

Return the temperatures that minimize held-out log loss.

There is one per question type with at least type_minimum answers, and one per bucket with at least bucket_minimum (TYPE_MINIMUM, BUCKET_MINIMUM), each held within [lowest, highest], the class’s own bounds where not given.

class source

class TopProbability(Confidence)

The answer’s own probability, the largest one.

This is what a temperature fit calibrates and what expected calibration error measures.

function source

def bucket(question: Question) -> str

Return the calibration bucket of question: its type and its option count.

The counts are grouped as 2, 3-5, 6-10 or 11+ (Laya’s temp_bucket).

function source

def kind_of(question: Question) -> int

Return the number a laid-out row gives question’s type (KINDS).