Skip to content

dew.nn.dex

Differential extension (DEX, arXiv 2505.16333): a learned differential on chosen heads’ outputs.

On each chosen head h of a softmax attention layer, the head’s output O_h (its attention-weighted values, before the output projection) becomes

O'_h = O_h - lambda(t) * O_h W_D,h (paper Eq. 3, 5)

with W_D,h a learned [Dv, Dv] projection of the head’s own, and the other heads unchanged. lambda is annealed over the first anneal_steps optimizer updates t from zero to a learned scalar (Eq. 4):

lambda(t) = (1 - a) * (t / T) * lambda_init + a * lambda_learn, a = min(t / T, 1)

so a pretrained model computes its own function at the start (lambda(0) = 0) and the extension enters as it trains. lambda_init defaults to Differential Transformer’s depth schedule, 0.8 - 0.6 exp(-0.3 (l - 1)) for the layer’s 1-based depth l (arXiv 2410.05258, which DEX adopts). Outside training (no step, dew.nn.progress) a layer reads lambda_learn, the anneal’s end.

The heads are chosen per layer (heads), as the paper picks the half of each layer’s heads of least importance or most attention entropy; a run picks them from a model’s own statistics before it attaches the extension. A layer holds its chosen heads and its lambda_init as variables of its own (DEX_CONSTANTS, DexStart), so layers that choose as many heads share one spec (LayerDex) and a scanned run or a pipeline stage holds them as it holds any layer’s weights. The paper’s recipe trains the attention’s key, value and output projections with the extension and freezes the rest (dew.optim.ParamGroup).

NameSummary
DEX_COLLECTIONThe collection an attention layer holds DEX’s start in (DEX_CONSTANTS), which never trains.
DEX_CONSTANTSThe variables an attention layer holds for DEX under DEX_COLLECTION, by name: the heads it acts on and its lambda_init (DexStart).
DEX_PARAMETERSThe parameters an attention layer adds for DEX, by name: its chosen heads’ projections and its learned lambda (dew.nn.mixers.attention.CausalSelfAttention).
DexStartThe values one layer’s extension holds from its start (DEX_CONSTANTS): the heads it acts on, ascending, and its lambda_init.
DifferentialExtensionWhich heads of which layers carry the differential extension (this module’s doc), as (layer, heads) pairs, each layer once; lambda_init for every chosen layer, or None for Differential Transformer’s depth schedule (diff_lambda_init); and the optimizer updates lambda anneals over.
FoldedKernelA layer’s o_proj kernel with its differential folded in (folded), computed as it is read (__array__): the kernel read home (read_home), each chosen head’s rows K_h made (I - strength W_h) K_h in float64 and rounded once to the kernel’s dtype.
HeadOutputsA mixer whose heads’ outputs DEX extends, before its output projection: softmax attention (dew.nn.mixers.attention.CausalSelfAttention).
LayerDexOne layer’s differential extension as every layer that chooses as many heads shares it: how many of its heads it acts on and the updates its lambda anneals over (DifferentialExtension).
annealedlambda(t) after updates optimizer updates (this module’s Eq.
diff_lambda_initDifferential Transformer’s lambda_init for a layer of 1-based depth: 0.8 - 0.6 exp(-0.3 (l - 1)).
differentialThe heads’ outputs attention [B, S, H, Dv] with each of heads less strength times its own projection of itself (projection [len(heads), Dv, Dv], in heads’ order), the other heads as they are: one product over the chosen heads, scattered back.
foldedvariables with each DEX layer’s differential at its learned lambda folded into its output projection, the extension’s parameters and constants dropped: a layer’s o_proj kernel [H Dv, D] has each chosen head h’s rows K_h made (I - lambda_learn W_h) K_h, in float64 and rounded once to the kernel’s dtype, so the source’s own layer computes what the extended one does outside a run (annealed), O'_h K_h = O_h (I - lambda W_h) K_h.

attribute source

DEX_COLLECTION = 'constants'

The collection an attention layer holds DEX’s start in (DEX_CONSTANTS), which never trains.

attribute source

DEX_CONSTANTS = ('dex_heads', 'dex_lambda_init')

The variables an attention layer holds for DEX under DEX_COLLECTION, by name: the heads it acts on and its lambda_init (DexStart).

attribute source

DEX_PARAMETERS = ('dex_projection', 'dex_lambda')

The parameters an attention layer adds for DEX, by name: its chosen heads’ projections and its learned lambda (dew.nn.mixers.attention.CausalSelfAttention).

dataclass source

class DexStart(heads: tuple[int, ...], lambda_init: float)

The values one layer’s extension holds from its start (DEX_CONSTANTS): the heads it acts on, ascending, and its lambda_init.

dataclass source

class DifferentialExtension(
layers: tuple[tuple[int, tuple[int, ...]], ...],
lambda_init: float | None = None,
anneal_steps: int = 1000,
trains: Literal['attention', 'all'] = 'attention',
)

Which heads of which layers carry the differential extension (this module’s doc), as (layer, heads) pairs, each layer once; lambda_init for every chosen layer, or None for Differential Transformer’s depth schedule (diff_lambda_init); and the optimizer updates lambda anneals over.

trains: Literal['attention', 'all']

What of the model trains beside the extension: the paper’s recipe, every attention layer’s key, value and output projections, the rest frozen (trainable); or all of it.

def trainable(
path: tuple[str, ...],
decoder: tuple[str, ...],
attention: Callable[[str], bool],
) -> bool

Whether the parameter at path (("params", ...)) trains under the extension’s recipe (trains): every parameter with “all”; else the extension’s own, and the key, value and output projections of the decoder’s softmax attention layers, whose modules’ names attention takes (layers_3, or a scanned run’s layers_3_7), under the decoder’s parameters (decoder, ("params",) for a bundle’s own decoder, deeper in a wrapper’s).

def layer(index: int) -> LayerDex | None

Layer index’s extension as its spec holds it, or None where it carries none.

def start(index: int) -> DexStart | None

The values layer index’s extension starts from, or None where it carries none.

dataclass source

class FoldedKernel(
kernel: jax.Array | np.ndarray,
projection: np.ndarray,
strength: np.ndarray,
heads: np.ndarray,
)

A layer’s o_proj kernel with its differential folded in (folded), computed as it is read (__array__): the kernel read home (read_home), each chosen head’s rows K_h made (I - strength W_h) K_h in float64 and rounded once to the kernel’s dtype. An export reads one at a time, so a host holds one folded kernel, not every layer’s.

class source

class HeadOutputs(Protocol)

A mixer whose heads’ outputs DEX extends, before its output projection: softmax attention (dew.nn.mixers.attention.CausalSelfAttention).

def head_outputs() -> tuple[int, int]

Its heads’ count and the width of each one’s output.

def dex_fold_refusal() -> str | None

Why DEX on its heads does not fold into its output projection (folded): what its heads’ outputs pass through between the two; None where nothing does.

dataclass source

class LayerDex(count: int, anneal_steps: int)

One layer’s differential extension as every layer that chooses as many heads shares it: how many of its heads it acts on and the updates its lambda anneals over (DifferentialExtension). Which heads, and lambda’s start, are the layer’s own variables (DexStart).

function source

def annealed(
updates: jax.Array | None,
layer: LayerDex,
start: jax.Array,
learned: jax.Array,
) -> jax.Array

lambda(t) after updates optimizer updates (this module’s Eq. 4), from the layer’s lambda_init start; learned past the anneal, and outside training (updates None).

function source

def diff_lambda_init(depth: int) -> float

Differential Transformer’s lambda_init for a layer of 1-based depth: 0.8 - 0.6 exp(-0.3 (l - 1)).

function source

def differential(
attention: jax.Array,
heads: jax.Array | tuple[int, ...],
projection: jax.Array,
strength: jax.Array,
precision=None,
) -> jax.Array

The heads’ outputs attention [B, S, H, Dv] with each of heads less strength times its own projection of itself (projection [len(heads), Dv, Dv], in heads’ order), the other heads as they are: one product over the chosen heads, scattered back.

function source

def folded(
variables: Mapping[str, Mapping],
compute: DTypeLike | None,
) -> dict[str, Mapping]

variables with each DEX layer’s differential at its learned lambda folded into its output projection, the extension’s parameters and constants dropped: a layer’s o_proj kernel [H Dv, D] has each chosen head h’s rows K_h made (I - lambda_learn W_h) K_h, in float64 and rounded once to the kernel’s dtype, so the source’s own layer computes what the extended one does outside a run (annealed), O'_h K_h = O_h (I - lambda W_h) K_h. Lambda and W_h are the ones the layer multiplies by, rounded to its compute dtype as differential rounds them (None for its parameters’ own). A layer whose heads’ outputs pass through more before o_proj does not fold (HeadOutputs.dex_fold_refusal). Each layer’s lambda, W_h and heads are read home as a pool’s export reads them (read_home), the layers in order, the same on every process, and each kernel folds as it is read (FoldedKernel).