The capacity wall is not a capacity wall

Entry 002 ended on what seemed like a clear next step. The benchmark put Reverie at ~33% on memorizing 36 arithmetic facts, flat across two orders of magnitude of training data, while a ten-thousand-parameter untrained transformer climbed to 100%. The cause looked obvious: the context-cell pool holds 64 templates, and 36 facts × several sliding 4-windows each is more than 64. The substrate ran out of slots in its first plastic phase and froze. The fix seemed obvious too: let the slots recycle.

So that’s what I built.

Attempt 1 — LRU eviction (ADR 0015)

The change is small. Track the last tick at which each context cell’s template was matched (even partially). When a new pattern arrives unmatched and every cell is claimed, evict the least-recently-matched cell and reuse its slot. About forty lines including the bookkeeping. Default-on, behind a flag so callers can recover the old frozen behavior for reproducibility.

The benchmark with this change applied is worse than the baseline across the board. Task 1 dropped from 36% to 13%. Task 4’s “preserved A” number, where the substrate had previously held onto 57% of its phase-1 knowledge after a phase of B-only training, collapsed to 11%. Task 5 was unchanged at chance. The fix removed the only metric Reverie was winning.

The cause has two interacting parts. The first is structural: with 36 facts in shuffled order, the active working set of templates is always larger than 64. Under LRU, every new window evicts another before the displaced cell has had time to accumulate stable preferences for its template. The pool thrashes. The second is methodological: every test probe observes a 4-token prompt, and with eviction on those probe-time observations are themselves evicting templates we were about to read. The test was erasing what we wanted to measure.

The architectural lesson was sharper than the negative result. The original frozen behavior of ADR 0012 — never let a claimed template go — turned out to be doing structural work that wasn’t named at the time. By refusing to forget, it preserved a stable subset of templates long enough for them to acquire well-formed preferences. The “ceiling” wasn’t really a failure mode; it was accidental consolidation by neglect of any forgetting mechanism. ADR 0015 destroyed that without replacing it.

So I flipped ADR 0015’s default to off, kept it available as an opt-in flag, and added an honest “what didn’t work” section to the ADR itself. The next half of the fix was clear: the substrate needs intentional consolidation.

Attempt 2 — consolidation during dreaming (ADR 0016)

Reverie’s excite() function is the system thinking during quiet periods — sampling tokens from the replay buffer and re-running the observation pipeline at reduced learning rates. ADR 0004 named this replay. The mechanism for consolidation falls out naturally: during each excitation tick, scan the replay buffer for occurrences of each context cell’s template. If the template appears, refresh its last-match tick. Templates that recur in recent experience stay warm; templates whose patterns have aged out of the buffer eventually become eviction candidates.

The implementation is a vectorized numpy comparison over a sliding-window view of the buffer. About thirty lines. The mechanism is correct — a focused unit test confirms that after consolidation runs, rehearsed templates have strictly more recent ticks than unrehearsed ones, which is exactly the property LRU reads.

The benchmark with ADR 0015 + ADR 0016 + 200 dream ticks between training and probe, compared against the frozen baseline:

                                       Frozen      Full (0015+0016)   Transformer
  Task 1, N=2048                       36.1%       20.0%              100.0%
  Task 2, held-out                      3.3%       16.7%               73.3%
  Task 3, cycle                       100.0%      100.0%              100.0%
  Task 4, A after phase 2              62.2%       12.2%               43.3%
  Task 4, B after phase 2               5.6%       16.7%              100.0%
  Task 5, K=512 adaptation              4.4%       17.8%              100.0%

The combined system didn’t beat the frozen baseline. It changed the failure mode. The frozen substrate is rigid: it preserves what it learned first and refuses to learn anything new. The full substrate is plastic: it adapts to new material in a small but real way (the Task 5 curve actually climbs now, where before it was flat at chance), and Task 2’s held-out generalization lifted from 3% to 17%. But it loses old material in roughly equal measure (Task 4 A drops from 62% to 12%). Neither configuration wins on net; together they describe a trade-off the architecture cannot escape with policy changes alone.

What changed in the picture

The two ADRs do exactly what they say. The eviction policy is correct. The consolidation refreshes the right ticks. The tests are green and probe direct invariants of the mechanism. The combination behaves as designed.

It just doesn’t help.

The wall I thought I was breaking — the 64-cell capacity ceiling — wasn’t actually the wall. With 64 cells and 36 facts, the substrate can encode at most 64 distinct K-token windows, full stop. ADR 0015 lets it cycle through that capacity. ADR 0016 lets the cycling prefer rehearsed material over forgotten material. Neither manufactures more capacity. The structural fact remains: the substrate’s encoding is one fact per cell. There is no shared substrate across facts, no compression, no abstraction. The transformer, by contrast, has weights that participate in encoding every fact — its 10,000 parameters are reused across all 36 instances of digit + digit = digit, which is why it generalizes (73% on held-out facts) and why its capacity scales with abstraction rather than count.

This is the problem ADR 0014 named when it introduced slot cells: variable binding, the architectural primitive that would let one cell encode “the operator + between two digits” rather than 36 cells each encoding one specific instance. ADR 0014’s implementation didn’t deliver that — entry 002 measured 3% on held-out facts with slot cells active, which is no generalization at all. With ADRs 0015 + 0016 enabled the same number lifted to 17%, which is some signal that the binding architecture is doing a small amount of work, but it’s still nowhere near the transformer’s 73%. The slot mechanism is undercooked, and the rest of the substrate cannot route around its absence by being clever about caches.

What I’d build next

Three things, ordered by how directly they target the actual wall:

One: redesign slot cells so they generalize, not just compose. This is the explicit purpose ADR 0014 set out for them. The current implementation binds operands as concrete (operand-1, operand-2) tuples, treating each tuple as a fresh key — which is why 1 + 2 = 3 doesn’t transfer to 2 + 1 = 3 even though both bindings involve the same digits. The fix isn’t capacity-related at all. It’s a representation change: the substrate needs cells that learn parameterized functions of role-bound values, not pattern-bound concrete strings. This is the territory of symbolic learning, and it’s hard, but it’s the real wall.

Two: experiment with much larger context-cell pools. With 128 or 256 context cells instead of 64, the working set might actually fit, and the ADR 0015 + 0016 combination might produce different results — possibly lifting Task 1 closer to the transformer’s numbers at the cost of more memory. This is cheap to test; I just didn’t sweep pool size in the original benchmark. The finding could be a useful boundary on where “more capacity” stops mattering.

Three: a non-symbolic task domain. All of entry 002 and entry 003’s tests are arithmetic. That’s a domain where the right answer is a clean function of two operands, which is exactly the kind of structure transformers are built to find via gradient descent. Reverie was originally pitched as a substrate for emergent behavior on observation streams — closer to language modeling or sequence prediction than arithmetic recall. A benchmark on, say, character-level prediction of a structured corpus might shift the comparison meaningfully. Or it might not. Either result is worth knowing.

The honest summary of this entry: I built two ADRs to fix a problem I had identified clearly in entry 002, the ADRs do exactly what I designed them to do, and the problem wasn’t the one I thought it was. The substrate’s capacity ceiling is real but it’s downstream of a representation ceiling, and you can’t get around the second by improving the first.

A research log is the right place to publish this. Negative results that arrive at a sharpened understanding are more useful than positive ones that don’t.

— Kaan, with Claude as collaborator on the build, the experiments, and these notes. ADRs 0015 and 0016 are live with their empirical findings inline. Benchmark code at model/scripts/benchmark.py; run with --variant full --seeds 5 to reproduce the right-hand column above.