dew.objectives
| Name | Summary |
|---|---|
Aux | Everything a loss returns besides its statistics. Documented in dew.objectives.base. |
DistillationObjective | Mixes the student’s own loss with a frozen teacher’s soft targets. |
EMASpec | Which leaves the EMA copy tracks, and how fast it follows them. Documented in dew.objectives.base. |
Objective | Defines what is learned: the parameters, the loss and what evaluation produces. Documented in dew.objectives.base. |
Prediction | What a token objective scored a batch with, for comparison with a teacher. Documented in dew.objectives.base. |
Ratio | A loss numerator and its denominator, kept apart until the loss is reduced. Documented in dew.objectives.base. |
Shown | How the training display shows one reported metric. Documented in dew.objectives.base. |
Step | What the trainer tells an objective about the current step. Documented in dew.objectives.base. |
Weighted | A loss of several terms, each a Ratio normalized by its own mass, summed with weights: sum_i weight_i * term_i.mean(), which has mass where any term of nonzero weight has. |
DistillationObjective
Section titled “DistillationObjective”class DistillationObjective( student: Objective[Loss, Effects], teacher: Objective[Loss, Effects], *, alpha: Weight = 0.5, temperature: Weight = 1.0, features: Sequence[tuple[int, int]] = (), beta: Weight = 0.0, feature_loss: FeatureLoss = 'cosine', divergence: Literal['full', 'selected'] = 'full', tokenizers: tuple[str, str] | None = None,)Mixes the student’s own loss with a frozen teacher’s soft targets.
DistillationObjective.data
Section titled “DistillationObjective.data”def data(tokenizer: Tokenizer | None = None) -> ObjectiveDataThe student’s record and batch steps: the teacher scores the batch the student reads.
DistillationObjective.real_lengths
Section titled “DistillationObjective.real_lengths”def real_lengths(batch: Batch) -> np.ndarrayThe student’s real lengths of the batch it reads.
DistillationObjective.scored_tokens
Section titled “DistillationObjective.scored_tokens”def scored_tokens(batch: Batch) -> np.ndarrayThe targets the student scores of the batch it reads.
DistillationObjective.row_tokens
Section titled “DistillationObjective.row_tokens”def row_tokens(batch: Batch) -> int | NoneThe tokens each row of the batch the student reads holds.
DistillationObjective.inference_record
Section titled “DistillationObjective.inference_record”def inference_record() -> JSONThe student’s record: a saved distillation loads as the student it trained, on the
student’s own tree, the teacher and the projections left unread (UNREAD).
DistillationObjective.program_key
Section titled “DistillationObjective.program_key”def program_key() -> tuple[ProgramModule, ...]The student’s modules, then the teacher’s, which the step does not train. The step reads
both models’ whole logits (predict), so no head of either is fused.
DistillationObjective.towers
Section titled “DistillationObjective.towers”def towers() -> tuple[Held, ...]The student’s towers, then the teacher’s.
DistillationObjective.substitute
Section titled “DistillationObjective.substitute”def substitute(modules: Sequence[nn.Module]) -> NoneDistillationObjective.tile_head
Section titled “DistillationObjective.tile_head”def tile_head(tile: tuple[int, int] | None = None) -> str | NoneTile the student’s head, the vocabulary head this objective trains.
DistillationObjective.restore_head
Section titled “DistillationObjective.restore_head”def restore_head(tile: tuple[int, int] | None, *, fused: bool) -> NoneDistillationObjective.held_variables
Section titled “DistillationObjective.held_variables”def held_variables() -> Variables | NoneReturn the student’s held tree, if any, with the teacher’s under teacher.
DistillationObjective.init
Section titled “DistillationObjective.init”def init(key: jax.Array, variables: Variables | None = None) -> VariablesDistillationObjective.stage_sites
Section titled “DistillationObjective.stage_sites”def stage_sites(store: Variables, stages: int) -> tuple[DecoderBank, ...]The student’s runs a pipeline over stages stacks (Objective.stage_sites), which its tree,
the run’s without the teacher and the projections, holds where the run’s does: the teacher reads
no update, whose layers stay whole.
DistillationObjective.student_variables
Section titled “DistillationObjective.student_variables”def student_variables(params: Variables) -> VariablesReturn the student’s own tree, without the teacher or the feature projections.
The student’s methods read this tree, and a distilled checkpoint passes it on to a plain student run.
DistillationObjective.model_variables
Section titled “DistillationObjective.model_variables”def model_variables(variables: Variables) -> VariablesReturn the student’s model tree, without the teacher or the feature projections.
DistillationObjective.loss
Section titled “DistillationObjective.loss”def loss(variables: Variables, batch: Batch, step: Step) -> tuple[Ratio, Aux[Effects]]DistillationObjective.apply_effects
Section titled “DistillationObjective.apply_effects”def apply_effects(variables: Variables, effects: Effects) -> VariablesDistillationObjective.projected
Section titled “DistillationObjective.projected”def projected(params: Variables, *, updated: bool) -> VariablesThe student’s projection of its own parameters; the feature projections as they are.
DistillationObjective.evaluate
Section titled “DistillationObjective.evaluate”def evaluate(params: Variables, batch: Batch, step: Step) -> Artifacts | NoneDistillationObjective.preview
Section titled “DistillationObjective.preview”def preview( params: Variables, batch: Batch, step: Step, *, scored: Artifacts | None = None,) -> Artifacts | NoneDistillationObjective.build_task
Section titled “DistillationObjective.build_task”def build_task(variables: Variables, *, processor: Processor | None | Omitted = OMITTED)Return the student’s task, without the teacher.
Weighted
Section titled “Weighted”class Weighted(weights: tuple[float, ...] = struct.field(pytree_node=False))A loss of several terms, each a Ratio normalized by its own mass, summed with weights:
sum_i weight_i * term_i.mean(), which has mass where any term of nonzero weight has.
Each term sums across microbatches and devices as a Ratio does, so every term is its mean over the
whole update before the weights combine them (Objective.reduce_loss).
Weighted.mean
Section titled “Weighted.mean”def mean() -> tuple[jax.Array, jax.Array]Return the weighted sum of the terms’ means, and whether any term of nonzero weight has mass.