dew.nn.mlp
The feed-forward every family builds: GatedMLP, two maps with an
activation between or three with a gate, which owns its maps’ names,
placement, remat names, activation rounding and work count.
| Name | Summary |
|---|---|
GatedMLP | Computes the gated feed-forward down_proj(act(gate_proj(x)) * up_proj(x)). |
geglu_feed_forward | diffusers’ FeedForward: a GEGLU to four times dim and back, biased, its gate and up projections packed as the checkpoint keeps them, the up projection first; approximate is the gate’s tanh form of GELU, else the erf form, and fields any other GatedMLP field. |
GatedMLP
Section titled “GatedMLP”class GatedMLP( activation: GatedActivation | Activation = 'swiglu', use_bias: bool = False, activation_sparsity: float = 0.0, swiglu_limit: float | None = None, init_std: float | None = None, output_init_std: float | None = None, packed: bool = False, packed_order: tuple[str, str] = ('gate_proj', 'up_proj'), linear: Callable[..., nn.Module] = functools.partial(Dense, dot_general=float32_dot_general), dropout_rate: float = 0.0, ngpt: NGPT | None = None, dtype: Dtype | None = None, precision: PrecisionLike = None, in_features: int | None = None, activation_after: bool = False, names: tuple[str, str] = ('up_proj', 'down_proj'), named_axes: tuple[tuple[str, LogicalAxes], ...] = (), gate_multiplier: float = 1.0, down_multiplier: float = 1.0,)Computes the gated feed-forward down_proj(act(gate_proj(x)) * up_proj(x)).
activation picks act: swiglu is silu, geglu is the tanh approximation of
gelu (HF’s gelu_pytorch_tanh), and geglu_exact is the erf form (HF’s gelu).
A Situ is Kimi K3’s SiTU, which transforms both halves
(dew.nn.activations.gated_product). The UNGATED activations, by name, or an
activation function (dew.nn.activations) build the ungated MLP with two
projections.
activation_sparsity is Gemma 3n’s gaussian top-k on the gate
(gaussian_topk). swiglu_limit is the clamp that
GLM-5.3-Flash and DeepSeek V4 apply before the activation: the gate is
capped from above, and the up projection on both sides. Both need a gated
activation. packed keeps the gate and up projections in one
gate_up_proj from init on, as DeepSeek-V4.1’s vision tower stores them,
in packed_order (diffusers’ GEGLU stores the up projection first), and
linear is the projections’ module (Gemma 4’s vision tower clips
theirs). Where the generation’s kernel row measured it faster
(‘gate_up’: ‘joined’), separate gate and up projections run as one GEMM
over their kernels joined per call (dew.nn.joined). The product reads a
packed or joined output in place (packed_gated_product). dropout_rate
drops the activated hidden units before the down projection while
training, as T5’s T5DenseActDense does.
The ungated MLP may read another width than it writes (in_features, which
the count reads), apply its activation after the down projection too
(activation_after, a diffusion time embedding’s or a vision adapter’s),
and store its two projections under the names its checkpoint uses
(names, up then down).
use_bias: bool-
Whether its projections add a bias: none by default, as Llama’s and most published gated feed-forwards (
mlp_bias: false), where Flax’s Dense adds one. linear: Callable[..., nn.Module]-
How each projection is built: a factory of
dew.nn.dense.Dense’s signature, which declares each projection’s axes where it draws it (kernel_axes). ngpt: NGPT | None-
The normalized transformer’s gains (
dew.nn.ngpt.NGPT): the gate and the up projection each take a learned per-feature gain (suv_gate,suv_up) and the gated product is unit-normed before the down projection; None for none. named_axes: tuple[tuple[str, LogicalAxes], ...]-
The logical axes of projections under names of the owner’s own (
names), besideMLP_AXES. gate_multiplier: float-
A gated MLP’s gate projection’s output times this before the activation, and
down_multiplierits down projection’s output: Falcon-H1’smlp_multipliers, each a product rounded once to its operand’s dtype (dew.nn.precision.scaled), as torch’s product of a tensor and a Python float. reads_train: bool-
Whether its call takes
train(ReadsTrain): it drops hidden units while training.
GatedMLP.model_flops
Section titled “GatedMLP.model_flops”def model_flops(lengths) -> floatThe residual-width MLP’s two maps, or three with a learned gate.
GatedMLP.projections
Section titled “GatedMLP.projections”def projections() -> dict[str, tuple[int, int]]The feed-forward’s maps as {name: (inputs, outputs)}: what setup
builds, the count reads and the packed group (_group) concatenates.
GatedMLP.setup
Section titled “GatedMLP.setup”def setup()GatedMLP.projection_groups
Section titled “GatedMLP.projection_groups”def projection_groups() -> tuple[ProjectionGroup, ...]Its group (_group) where it is gated and its variables hold it
(ProjectionSites), which setup reads packed in the members’ place.
geglu_feed_forward
Section titled “geglu_feed_forward”def geglu_feed_forward(dim: int, *, approximate: bool = True, **fields={}) -> GatedMLPdiffusers’ FeedForward: a GEGLU to four times dim and back, biased, its gate and up
projections packed as the checkpoint keeps them, the up projection first; approximate is the
gate’s tanh form of GELU, else the erf form, and fields any other GatedMLP field.