Skip to content

dew.nn.wrapping

Models around a backbone: a module around one causal decoder, answering the decoder’s training, cache and serving hooks (dew.nn.protocols) as the decoder answers them, its decoder’s parameters under the field that holds it (a multimodal model’s language model, DiffusionGemma’s text decoder); and a backbone with a head over its final states, two modules over one variables tree (HeadedModel).

NameSummary
DecoderWrapperThe hooks a module around one decoder answers as the decoder does: its geometry, its banks and cache under the wrapper’s field, a clone over the decoder each hook returns, and the decoder’s projection of its own parameters.
HeadedModelAny backbone with final states (HiddenStates) and a head over them, applied as two modules over one variables tree: the two under backbone and head in every collection (dew.objectives.base.joined).

Flax module source

class DecoderWrapper(wraps: str)

The hooks a module around one decoder answers as the decoder does: its geometry, its banks and cache under the wrapper’s field, a clone over the decoder each hook returns, and the decoder’s projection of its own parameters. A subclass names the field (wraps) and keeps its own modules beside it as they are.

wraps: str

The field that holds the decoder, the name its parameters sit under in the wrapper’s tree.

def reads_progress() -> bool

Whether its decoder’s computation changes over a run (dew.nn.progress.ReadsProgress), which a run’s objective then passes or refuses.

def replicated_flops(stages: int, lengths) -> float
def bank_sites(store: Variables) -> tuple[DecoderBank, ...]
def stage_sites(store: Variables, stages: int) -> tuple[DecoderBank, ...]
def with_cache_capacity(capacity: int) -> Self
def with_kv_cache(layout: KVCache) -> Self
def projected(params, *, updated: bool)

The decoder’s projection of its own parameters (ProjectsParameters); the wrapper’s other parameters as they are.

def generation_refusal() -> str | None
def block_chunks() -> tuple[int | None, ...]

The decoder’s trunk layers that read blocks, by their attention chunk (BlockReading); the wrapper’s own modules read none.

def with_remat(policy: RematPolicy | None) -> Self

The decoder’s layer policy; the wrapper’s other modules as they were.

def recompute_record() -> JSON
def recompute_more() -> Self | None
def recompute_half() -> Self | None
def restore_recompute(record: JSON) -> Self

dataclass source

class HeadedModel(backbone: nn.Module, head: nn.Module)

Any backbone with final states (HiddenStates) and a head over them, applied as two modules over one variables tree: the two under backbone and head in every collection (dew.objectives.base.joined). Each is applied as its own module over its part, so a backbone that Pretrained.adapt gave LoRA factors runs with its own adapter, with its base weights under frozen. A task’s model reads its rows through it (dew.decision.DecisionModel); a head reads the backbone’s final states, its vocabulary rows or its output table, and its own logits at a label’s tokens (label_logits).

context_length: int | None

The most context tokens a row carries: the backbone’s cross-attention’s (CrossAttending), or None for a backbone that reads no context.

def stage_sites(store: Variables, stages: int) -> tuple[DecoderBank, ...]

The backbone’s runs a pipeline over stages stacks, under backbone (StagedStacks); a head has none.

def table(variables: Variables) -> jax.Array | np.ndarray

Return the backbone’s head as [V, D] rows (VocabularyRows), which a head may read.

def output_table(variables: Variables) -> OutputTable | None

The backbone’s head as one matrix (AffineHead), or None where no matrix alone is it.

def label_logits(
variables: Variables,
answers: jax.Array,
ids: tuple[tuple[int, ...], ...],
) -> jax.Array

The backbone’s own logits [B, N, K, V] at each label’s ids of each answer state of answers [B, N, features] (dew.nn.protocols.label_logits).

def head_size(backbone: nn.Module) -> tuple[int, Dtype | None]

Return the width of backbone’s final states, and the dtype a head over them computes in.

The backbone is traced, not run, so this costs no compute. A backbone that computes in reduced precision has its head compute in it too; over full-precision states a head computes at its parameters’ precision.