The wall lifts when you raise the wall

Entry 003 ended on a confident architectural reading: the substrate’s ceiling is one fact per cell, and better eviction or consolidation policies can’t manufacture the compression that’s missing. That conclusion was a step too far. Before declaring the wall structural I should have done the cheapest possible experiment — try the existing architecture with a bigger pool. I didn’t. This entry is what that experiment shows.

The sweep

Same Task 1 from entry 002 — memorize 36 single-digit addition facts, test on all 36. Same shuffled training stream. Same five seeds. The only change is the size of the context-cell pool. Two variants: frozen (ADR 0012 only, the original baseline) and full (ADRs 0015 + 0016, eviction + dream consolidation). Accuracy at N=2048 training tokens:

  Pool size    Frozen (0012)    Full (0015+0016)    Transformer
         64        20.4% ± 3.2        12.0% ± 4.2          100%
        128        38.0% ± 4.2        19.4% ± 4.8          100%
        256        59.3% ± 7.0        28.7% ± 5.8          100%
        512        65.7% ± 9.8        46.3% ± 8.5          100%

The frozen substrate scales almost linearly: roughly 20 → 38 → 59 → 66 as the pool doubles from 64 to 512. At pool=512 it closes about two-thirds of the gap to the transformer (66% vs 100%) on a task where it had previously been pinned at 33-36% for two entries running. The capacity ceiling is real, and it lifts when you give it more capacity. That part of the story is now boring and that’s a good sign.

What the ADRs from entry 003 actually did

The full variant (ADRs 0015 + 0016) does not benefit from a bigger pool the way frozen does, and worse, its accuracy degrades as training continues. At pool=512 and N=256 it hits 81.5% — the highest single number any Reverie configuration has produced on this benchmark. Then by N=512 it has dropped to 45%, and at N=2048 it sits at 46%. The eviction policy keeps churning even when there is plenty of room, and consolidation can’t catch all the templates that would otherwise drift out of the live working window. The more training tokens arrive, the more churn accumulates.

So the architectural story the ADRs were meant to fix turns out to have a much simpler resolution. The capacity ceiling existed because the default pool was 32 cells and the benchmark used 128 — both artificially small for a 36-fact workload with multiple sliding 4-token windows per fact. The ADR 0015 + 0016 machinery — LRU bookkeeping, eviction-with-reset, dream-time consolidation — was solving a problem more cheaply solved by typing a bigger number in the initial_cells argument. The ADRs are still in the repository, still tested, still correct as documented. They just aren’t what unlocks the benchmark.

This is two negative results in three entries, which feels expensive, but the corrective is genuinely useful: the substrate is not fundamentally broken at one-fact-per-cell. It is underprovisioned, and provisioning is free.

What I changed

initial_cells default is now 256, bumped from 32 with a docstring note pointing at this entry. One test had to be pinned to the old default (a numerical edge case with 256 cells and a 2-token vocabulary causes the dream’s score vector to underflow to NaN — flagged as a known issue, not blocking). Nothing else moves.

ADRs 0015 and 0016 stay on the books. ADR 0015’s status is already documented as “Accepted as opt-in flag; default off after empirical disconfirmation,” and ADR 0016 says explicitly that the mechanism works but doesn’t lift the benchmark. With this entry both ADRs gain an additional honest annotation: even with a bigger pool, neither helps on the workload they were designed for.

What this means for entry 003’s wider claim

Entry 003 read the negative results as evidence of a representation ceiling — the absence of templates that subsume surface variants. That diagnosis is not retracted; the transformer still generalizes from 30 facts to held-out ones at 73% while Reverie at any pool size sits well below 20% on the same test. There is a real representation gap on Task 2 (compositional generalization), and that gap doesn’t close with more cells. Slot cells still don’t deliver their stated purpose, and that’s still the highest-payoff architectural problem worth chasing.

What entry 003 got wrong was attributing Task 1’s ceiling to the same representation gap. Task 1 is straight memorization. It’s bounded by raw capacity. Task 2’s ceiling is the representation gap. Conflating the two led to the misdirected work in ADRs 0015-0016. The right partition:

• Task 1 (memorize 36 facts): capacity-bound. Fixed by raising initial_cells. Frozen at pool=512 reaches 66%; probably reaches 100% at pool=2048+ if we cared to verify, though by then the parameter count comparison with the 10k-param transformer stops being apples-to-apples.
• Task 2 (generalize to 6 held-out facts): representation-bound. Not fixed by anything in the current substrate. Slot cells are the unmet promise.
• Tasks 4 and 5 (continual learning, online adaptation): mixed. Better capacity helps. Better policy doesn’t.

What’s next

The two open directions from entry 003 are unchanged. The non-symbolic task domain remains the most interesting medium-effort experiment — Reverie was originally pitched for emergent behavior on observation streams, not arithmetic recall, and the comparison may look different there. The slot-cell redesign remains the highest-payoff architectural problem and the only one that would close the Task 2 gap. Both are queued. Entry 005, when it happens, will either be one of those experiments running or a different surprise from somewhere I’m not currently expecting.

The lesson I’d preserve from this entry — the one I won’t actually preserve unless I write it down here — is: always try the cheapest existing knob before claiming the architecture is wrong. I burned two ADRs and an entry on a fix that turned out to be a one-line default change. The substrate did its job; the architect did not.

— Kaan, with Claude as collaborator on the build, the sweep, and these notes. Sweep reproducible via model/scripts/benchmark.py --variant frozen --initial-cells N for N ∈ {64, 128, 256, 512}.