Cross-layer connections

Context. In a standard pre-norm transformer every layer reads one residual stream and writes its output back into it, so all communication between layers goes through a single running sum. Deep models show the symptoms of that bottleneck: token representations grow more alike with depth, the inputs of adjacent late layers become nearly identical (cosine above 0.97 in Pythia-2.8B), and attention piles onto low-information sink tokens. The fix is to let later layers read earlier layers' values or hidden states directly. A cross-layer shortcut helps when it supplies information the residual stream no longer carries cleanly. The value stream is the safest place for it, since it changes what attention moves without changing where it looks, and making the reads per token, the way attention addresses positions, gives the general form. Value residual (Zhou et al.) is the simplest case, MUDD connections (Xiao et al.) the general one.

Value residual mixes the first layer's values into every layer. With $H_{n-1}$ the input of layer $n$ and $V_1=H_0W_1^V$, ResFormer replaces layer $n$'s values by $$V_n'=\lambda_{n,1}V_1+\lambda_{n,2}\,H_{n-1}W_n^V,\qquad U_n=A_nV_n',$$ so layer $n$'s own attention pattern carries the first layer's values. On Llama-like models of 82M to 468M parameters it matches the baseline's loss with 16% fewer parameters or 20% fewer tokens (single unseeded runs). The ablations show where the shortcut must enter. The same 0.5/0.5 shortcut added to queries, keys or the attention matrix does nothing or hurts (2.742, 2.746, 2.757 against a 2.739 baseline), added to values it gives 2.712, and adding $V_1$ to the attention output with no mixing across positions is far worse (3.137). Where a deep layer attends should depend on its contextual state; a query or key shortcut brings token-level features into those scores, while a value shortcut only changes what the existing pattern moves. A hidden-state shortcut $H_n'=0.5H_0+0.5H_n$ hurts (2.781) but helps when started at zero weight (2.722), so a hidden shortcut is safe only if its weight starts at zero.

A shortcut helps only when its source is information the residual stream no longer carries cleanly. In an 8-layer model, value shortcuts from $V_2$ or later layers give little, because the ordinary residual already carries $H_1$ forward, so $V_2=H_1W_2^V$ is reachable anyway; when the residual chain is started at $H_2$ instead, a shortcut from $V_2$ becomes useful. So the first layer is not special in itself. Deep layers want it most (a single shortcut works best at layer 7 of 8, and learned weights grow toward the top), and the best weights depend on depth and are found by hand. The paper also ties it to attention sinks: a sink token's value norm is tiny, which makes attending to it a cheap no-op, and mixing in $V_1$ gives it a real value; ResFormer flattens the sink, though that account is correlational.

Dynamic dense connections are the general form. MUDDFormer lets each layer read, separately for each of its four inputs (query, key, value, residual), its own per-token mixture of all earlier outputs: $$\bar X_i=\sum_{j=0}^{i}A_{ij}\odot X_j,\qquad A_i=\mathrm{GELU}\big(\mathrm{RMSNorm}(X_i)W_1\big)W_2+a_i,$$ so position $t$ weights source layer $j$ by a small MLP of its current state, which is depth-wise attention with weights from the query side alone. The 405M ablation (Pile perplexity) shows where the gain comes from: Transformer++ 11.68, static dense (DenseFormer, one weight per pair of layers) 11.44, dynamic 11.09, dynamic multiway 10.83, and 10.77 after shifting FFN width toward the upper layers. Dynamic weights are the main ingredient. Removing one stream's dense reads gives 10.89 (query), 10.90 (key), 11.05 (value) and 11.14 (residual): the residual reads matter most overall and, among attention's inputs, the value stream does, as in value residual; the mechanism the paper offers is that attention lets a layer pull an old representation into its values and move it to another position without writing it into its own residual. $W_2$ starts at zero and $a_i$ one-hot on the current layer, so the model starts as an ordinary transformer and learns dense reads only where they help; the paper calls this critical. MUDDFormer-834M matches Transformer++ trained with 1.89 times the compute, and MUDDPythia-2.8B, changing only the connections, matches Pythia-6.9B on 300B tokens; across sizes MUDD trains at 84 to 96% of baseline throughput, since the aggregates are many small memory-bound operations.

def attention(x, v1, lamb):          # lamb: learned scalar per layer, init 0.5
    q, k, v = c_q(x), c_k(x), c_v(x)
    if v1 is None:                   # first layer: its own values are V1
        v1 = v
    v = (1 - lamb) * v + lamb * v1   # value residual
    y = causal_attention(q, k, v)
    return c_proj(y), v1