Conditional Memory via Scalable Lookup

Context. Mixture-of-experts models scale parameters without compute by routing, which is sparsity in computation. Much of text, though, is local and stereotyped (multi-token names, idioms, formulaic phrases), and a transformer has no lookup primitive, so it rebuilds such patterns with its early attention and MLP layers. Engram (Cheng et al., DeepSeek) adds the primitive: it hashes the last two or three tokens into large embedding tables, gates the retrieved vector by the current hidden state, and adds it to the residual stream at a few early layers. Lookup is a second form of sparsity. A table indexed by the last few tokens adds parameters at no FLOP cost, and with the number of idle parameters (those not used for a given token) fixed, the best model gives about a quarter of them to memory and the rest to experts. The speedrun took the simplest form, a hashed bigram added to the residual, which grew into the 84.6M-row table of record #92.

Retrieval, gating and placement. Tokens are first mapped to canonical ids (NFKC, lowercasing), which concentrates counts on fewer n-grams. Each order $n\in\{2,3\}$ gets $K=8$ hash heads, each into its own table; a collision in one head rarely coincides with a collision in another, so the concatenated vector $e_t$ still identifies the n-gram (the hash-embedding idea of Svenstrup et al.). A static vector cannot tell which sense of an ambiguous n-gram is meant, and with hashing it may be another n-gram's vector altogether, so the hidden state $h_t$ gates it and can switch it off: $$\alpha_t=\sigma\Big(\tfrac{1}{\sqrt d}\,\mathrm{RMSNorm}(h_t)^\top\mathrm{RMSNorm}(W_Ke_t)\Big),\qquad \tilde v_t=\alpha_tW_Ve_t.$$ A single module works best at layer 2 (validation 1.770 against 1.808 without Engram), early enough that later layers need not rebuild local patterns, but after one round of attention has given the gate some context. In the ablations, gating and tokenizer compression matter most. Because row indices depend on token ids alone, they are known before the forward pass, so tables can be sharded or kept in host memory with rows prefetched while earlier layers compute (in inference, a 100B-parameter table offloaded this way costs at most 2.8% throughput on an 8B dense backbone).

With idle parameters fixed, memory and experts are complements. The allocation experiment holds total and active parameters fixed and moves idle capacity between routed experts and table slots. Loss is U-shaped in the share $\rho$ given to experts, with the minimum near 0.75 to 0.8 at both compute budgets tested (2e20 and 6e20 FLOPs); at 6e20, moving 20 to 25% of the idle budget into Engram lowers loss by 0.014, and too much memory hurts, since a table cannot do context-dependent computation. Two budgets at one sparsity ratio are little evidence for a scaling law. At a fixed backbone, loss falls linearly in the log of the number of table slots from 0.26M to 10M. At scale, Engram-27B beats a parameter- and FLOP-matched MoE-27B (Pile loss 1.950 against 1.960; one pair of runs). Zeroing the Engram output at inference leaves reading comprehension at 81 to 93% of its score while TriviaQA falls to 29%, so the table holds much of the factual knowledge; the test is post hoc, and the probes behind an "effective depth" reading are correlational.

rows = 15 * vocab_size                      # 192-dim rows in record #83
sign_table = torch.randn(8192, dim).sign()  # fixed +-1 patterns, never trained
prev, cur = ids[:-1], ids[1:]
row = torch.full_like(ids, rows - 1)        # position 0 gets a reserved row
row[1:] = ((36313 * cur) ^ (27191 * prev)) % (rows - 1)
sign_idx = torch.zeros_like(ids)
sign_idx[1:] = (prev ^ cur) % 8192          # a second hash picks the pattern
x0_bigram = bigram_embed(row) * sign_table[sign_idx]
# each block: x[..., :192] += lam[i] * x0_bigram

Why signs help, and how far the table grew. With signs, the bigrams that share a row read it through different fixed sign patterns, an old trick from feature hashing. A toy least-squares fit (mine) shows when it helps. Let Zipf-weighted bigrams share a $d$-dimensional row read through a generic $k$-dimensional subspace. Without signs every bigram in the bucket gets the weighted mean, and the rare ones keep an error about equal to their own target; with signs each sees a different linear map of the row, so at $d=192$, $k=8$ all fit exactly up to $d/k=24$ per row, and beyond that the frequent one stays nearly exact. With full-dimensional or coordinate-aligned readouts, signs do not help. So the sign trick helps when rows are underused (each read sees only a low-dimensional slice of the row), which fits the PR's finding that narrower rows with more slots helped; the author also reports that reducing the row count did not behave as this picture predicts, so it is a hypothesis. Record #92 then scaled the table to 84,602,880 rows of width 768 with bigram and trigram channels, sharded across the GPUs with a row-sparse Adam: about 65 billion parameters beside a 124M-class transformer, with each step reading and updating only the rows it touches. In its table-size sweep (two runs per point) the old 377,280-row size gives 3.315, a quarter of the big table 3.283 to 3.286, and the full table 3.279; the ablation credits the table with 7.05 s, mostly 35.8 millinats converted at 164 ms per millinat. Trigrams, which lost at #62, returned only in the big table.