dew.nn.hyper_connections
Manifold-constrained hyper-connections: the residual as hc_mult streams.
GLM-5.3-Flash and DeepSeek V4 carry the residual as hc_mult streams,
[B, S, H, D]. Each sublayer reads one collapse of the streams and writes
back into all of them through a mixing they choose (mHC, Xie et al. 2026,
section 2.2 eq. 8; Glm5NextTextHyperConnection, modeling_glm5_next.py:219-295,
and DeepseekV4HyperConnection, modeling_deepseek_v4.py:867-943):
flat = rmsnorm(streams.reshape(B, S, H*D)) # no weight, fp32mixes = flat @ fn^T # [(2 + H) * H]pre = sigmoid(mixes[:H] * scale[0] + base[:H]) + epspost = 2 sigmoid(mixes[H:2H] * scale[1] + base[H:2H])comb = softmax(mixes[2H:] * scale[2] + base[2H:], over the H x H row) + epscomb = sinkhorn(comb) # hc_sinkhorn_iters alternating normalisationscollapsed = sum_h pre[h] streams[h] # the sublayer's inputstreams' = post[:, None] * sublayer(collapsed)[None, :] + comb^T @ streamsSinkhorn-Knopp normalises columns first, then iters - 1 rounds of rows and
columns, eps in each denominator, in fp32; the result is asymmetric, so it
applies transposed, streams'[k] = sum_j comb[j, k] streams[j]. The final
collapse is GLM’s unweighted mean (modeling_glm5_next.py:298-302) or V4’s
learned collapse shaped like pre (modeling_deepseek_v4.py:946-962), the
record’s head.
DeepSeek-V4.1’s Single-Pass mHC (arXiv 2609.19969, section 2.4.1, eq. 6)
collapses each sublayer’s input by the previous site’s pre, the stack’s
first sublayer reading the first stream alone (V4.1 inference/model.py:968-994,
:1159-1163); under single_pass the stack carries Carried(streams, pre)
and the final norm reads the last site’s collapse (:1268), head ‘carried’.
Parameters are the checkpoints’: fn [(2 + H) H, H D] in torch’s [out, in] layout, base [(2 + H) H] and scale [3] under attn_hc and
ffn_hc; the weighted head’s hc_fn, hc_base, hc_scale under hc_head.
Qwen4-Exp’s gated residual (Qwen4ExpTextGatedResidual,
modeling_qwen4_exp.py:941-969 in transformers 5.16.1; hc_lowrank) reads
and writes the streams without mixing them:
normed = rmsnorm(streams), each stream's own statistics, (1 + w)weights = sigmoid(up(silu(down(normed) / H))) # [B, S, H, D]read = mean_h(weights[h] * normed[h]) # the sublayer's inputstreams'[h] = streams[h] + 2 sigmoid(inject(normed) / H)[h] * sublayer(read)down is [r, H D] and up [H D, r] for the rank r; the model’s head
is the read alone, with no inject, and no norm after it.
| Name | Summary |
|---|---|
HyperConnections | Holds the stream count, the Sinkhorn floor and iterations, the head and the schedule. |
HyperConnections
Section titled “HyperConnections”class HyperConnections( hc_mult: int = 4, hc_eps: float = 1e-06, hc_sinkhorn_iters: int = 20, head: str = 'mean', single_pass: bool = False, hc_lowrank: int | None = None,)Holds the stream count, the Sinkhorn floor and iterations, the head and the schedule.
A model names this on hyper_connections. The field names are the
references’ own config fields. head picks what collapses the streams
before the final norm; ‘carried’ is the last site’s pre, which only
the single_pass schedule carries (the module doc). hc_lowrank makes
the sites Qwen4-Exp’s gated residual of that rank (GatedResidual), its
head ‘gated’.
gated: bool-
Whether the sites are Qwen4-Exp’s gated residual (
GatedResidual).