Skip to content

dew.nn.hyper_connections

Manifold-constrained hyper-connections: the residual as hc_mult streams.

GLM-5.3-Flash and DeepSeek V4 carry the residual as hc_mult streams, [B, S, H, D]. Each sublayer reads one collapse of the streams and writes back into all of them through a mixing they choose (mHC, Xie et al. 2026, section 2.2 eq. 8; Glm5NextTextHyperConnection, modeling_glm5_next.py:219-295, and DeepseekV4HyperConnection, modeling_deepseek_v4.py:867-943):

flat = rmsnorm(streams.reshape(B, S, H*D)) # no weight, fp32
mixes = flat @ fn^T # [(2 + H) * H]
pre = sigmoid(mixes[:H] * scale[0] + base[:H]) + eps
post = 2 sigmoid(mixes[H:2H] * scale[1] + base[H:2H])
comb = softmax(mixes[2H:] * scale[2] + base[2H:], over the H x H row) + eps
comb = sinkhorn(comb) # hc_sinkhorn_iters alternating normalisations
collapsed = sum_h pre[h] streams[h] # the sublayer's input
streams' = post[:, None] * sublayer(collapsed)[None, :] + comb^T @ streams

Sinkhorn-Knopp normalises columns first, then iters - 1 rounds of rows and columns, eps in each denominator, in fp32; the result is asymmetric, so it applies transposed, streams'[k] = sum_j comb[j, k] streams[j]. The final collapse is GLM’s unweighted mean (modeling_glm5_next.py:298-302) or V4’s learned collapse shaped like pre (modeling_deepseek_v4.py:946-962), the record’s head.

DeepSeek-V4.1’s Single-Pass mHC (arXiv 2609.19969, section 2.4.1, eq. 6) collapses each sublayer’s input by the previous site’s pre, the stack’s first sublayer reading the first stream alone (V4.1 inference/model.py:968-994, :1159-1163); under single_pass the stack carries Carried(streams, pre) and the final norm reads the last site’s collapse (:1268), head ‘carried’.

Parameters are the checkpoints’: fn [(2 + H) H, H D] in torch’s [out, in] layout, base [(2 + H) H] and scale [3] under attn_hc and ffn_hc; the weighted head’s hc_fn, hc_base, hc_scale under hc_head.

Qwen4-Exp’s gated residual (Qwen4ExpTextGatedResidual, modeling_qwen4_exp.py:941-969 in transformers 5.16.1; hc_lowrank) reads and writes the streams without mixing them:

normed = rmsnorm(streams), each stream's own statistics, (1 + w)
weights = sigmoid(up(silu(down(normed) / H))) # [B, S, H, D]
read = mean_h(weights[h] * normed[h]) # the sublayer's input
streams'[h] = streams[h] + 2 sigmoid(inject(normed) / H)[h] * sublayer(read)

down is [r, H D] and up [H D, r] for the rank r; the model’s head is the read alone, with no inject, and no norm after it.

NameSummary
HyperConnectionsHolds the stream count, the Sinkhorn floor and iterations, the head and the schedule.

dataclass source

class HyperConnections(
hc_mult: int = 4,
hc_eps: float = 1e-06,
hc_sinkhorn_iters: int = 20,
head: str = 'mean',
single_pass: bool = False,
hc_lowrank: int | None = None,
)

Holds the stream count, the Sinkhorn floor and iterations, the head and the schedule.

A model names this on hyper_connections. The field names are the references’ own config fields. head picks what collapses the streams before the final norm; ‘carried’ is the last site’s pre, which only the single_pass schedule carries (the module doc). hc_lowrank makes the sites Qwen4-Exp’s gated residual of that rank (GatedResidual), its head ‘gated’.

gated: bool

Whether the sites are Qwen4-Exp’s gated residual (GatedResidual).