Skip to content

dew.nn.gemma3n

Gemma 3n’s residual stream: AltUp, the LAuReL block and activation sparsity.

Gemma3nTextModel (modeling_gemma3n.py) keeps altup_num_inputs copies of the residual. The first copy is the embeddings, and each other copy is its own projection of them, rescaled to the embeddings’ RMS. Every layer predicts all copies from the active one (Gemma3nTextAltUp.predict), runs the block on that prediction, corrects every copy by the block’s innovation (correct), and adds the per-layer input to the copies past the first. After the last layer, the copies are projected back and rescaled to the first copy’s magnitude, and their mean goes into the final norm. Gemma3nTextLaurelBlock is a rank-laurel_rank map of the block’s normed input, normed and added back; its sum with the attention residual is divided by sqrt(2). Gemma3nTextMLP._gaussian_topk keeps the gate activations that are more than norm.ppf(sparsity) standard deviations above the mean, which any gated MLP runs (dew.nn.mlp.gaussian_topk).

NameSummary
MAGNITUDE_EPSILON
AltUpHow many copies of the residual stream a model keeps, and which one its blocks run on.
AltUpLayerRuns one layer’s AltUp: predict before the block and correct after it.
LaurelBlockComputes Gemma 3n’s learned augmented residual (LAuReL).
clamped_predictionsparams with every AltUp layer’s stored prediction coefficients clamped to +-clip, as the reference clamps prediction_coefs.weight in place before a training forward (modeling_gemma3n.py:1027-1028), so the optimizer updates the clamped weight.
rescale_toReturn x scaled per token to target’s RMS magnitude.

attribute source

MAGNITUDE_EPSILON = 1e-05

dataclass source

class AltUp(
num_inputs: int = 4,
active_idx: int = 0,
coef_clip: float | None = 120.0,
correct_scale: bool = True,
)

How many copies of the residual stream a model keeps, and which one its blocks run on.

The fields are the reference’s altup_* config entries without the prefix (configuration_gemma3n.py).

Flax module source

class AltUpLayer(
norm_eps: float = 1e-06,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

Runs one layer’s AltUp: predict before the block and correct after it.

The stream is [num_inputs, B, S, D]. Both steps read a token’s modalities: the tanh of the modality_router output for the token’s normed, 1 / D-scaled active copy. predict maps the modalities to a num_inputs by num_inputs matrix per token (prediction_coefs) that mixes the copies, and adds the mix to the copies. correct maps the modalities to one coefficient per copy (correction_coefs, plus one). Each coefficient scales the block’s innovation, which is the activated output minus the active prediction, and the scaled innovation is added to every prediction. coef_clip bounds both coefficient weights during training, as the reference clamps them (_Coefficients).

def projections() -> dict[str, tuple[int, int]]
def model_flops(lengths) -> float

Both router reads, coefficient maps and the prediction’s copy contraction.

def projection_uses() -> dict[str, int]
def parameter_flops(lengths) -> float
def mixing_flops(lengths) -> float
def setup()
def modalities(x)
def predict(stream, train: bool = False)
def correct(predictions, activated, train: bool = False)
def scale_corrected_output(corrected)

Flax module source

class LaurelBlock(
norm_eps: float = 1e-06,
dtype: Dtype | None = None,
precision: PrecisionLike = None,
)

Computes Gemma 3n’s learned augmented residual (LAuReL).

The output is x + post_laurel_norm(linear_right(linear_left(x))).

def model_flops(lengths) -> float
def projections() -> dict[str, tuple[int, int]]

function source

def clamped_predictions(params, clip: float)

params with every AltUp layer’s stored prediction coefficients clamped to +-clip, as the reference clamps prediction_coefs.weight in place before a training forward (modeling_gemma3n.py:1027-1028), so the optimizer updates the clamped weight. Any other leaf, and a tree holding no AltUp layer, is returned as it is.

function source

def rescale_to(x, target)

Return x scaled per token to target’s RMS magnitude.

The mean square of x is floored at MAGNITUDE_EPSILON before its square root (the reference’s new_magnitude). A zero target (a zero pad embedding at a padded slot) gets magnitude zero with a zero gradient (sqrt_or_zero).