What scale doesn't fix

Entry 015 left two options on the table. Ship Reverie into Cairn as an audit layer; or stop and keep the fifteen-entry log as the artifact. I didn’t pick either. I picked a third path that wasn’t listed in 015 because I hadn’t yet articulated it: take Reverie’s design philosophy — interpretable cells, online observation-driven memory, cell-by-cell audit/delete — and embed it inside a gradient-trained transformer with embeddings. Not Reverie scaled up. Not Reverie’s competitor. Reverie’s philosophy living inside the architecture Reverie was built against.

That third path became a separate project (reverie-next, private repo, no public surface, but the same research arc) and it ran for one long session. Four iterations later, the project is closed at the architectural level by the same kind of decision rule entry 015 honored for canonicalization.

This entry is the closing entry. It exists because the conclusions hold across both projects together, and reverie.boomarche.com is the right place for them to live.

What I built and what it said

Iteration A — HybridLM with K-token-template cells. Small transformer with a memory-attention layer reading from a cell store of (K-1 prompt tokens, answer token) templates. Training included a random key-value retrieval task to teach the model to attend to cells. Result on a held-out multiplication-by-cells task: 13% with cells present, 0% without. Weak positive signal. The model had learned something about how to use cells, but the signal was small and didn’t generalize past the exact training shape.

Iteration B — same architecture, arithmetic-shaped cell-only task. Replaced random KV retrieval with an arithmetic-shaped training regime: held-out addition facts only available via cells, the rest of addition learned through gradients normally. Result: 0% multiplication-with-cells, and held-out addition accuracy with cells populated dropped from 66.7% (cells empty) to 0%. Cells actively hurt the model. The model had learned “cells present → do something different,” and the something different was wrong. Worse than Iter A in every direction.

Iteration C / Option 3 — structured (key_ids, value_id) cells with an explicit copy head. Architectural change, not just a training-task change. Replaced K-token templates with structured records, and added an explicit copy mechanism alongside the LM head with a learned gate. The architecture supplies the find-then-copy primitive instead of requiring the model to learn it from attention alone — same intent as Pointer Networks and CopyNet. Result: trained addition stayed at 100%, multiplication-with-cells landed at 4.3% (chance), and held-out addition with cells populated dropped to 0% again, mirroring Iter B exactly. Same failure shape despite the architectural improvement.

The pre-committed decision rule for Option 3 said: less than 15% multiplication, or breaks held-out the way Iter B did, means close. Both conditions met.

Iteration D / Option 5 — ROME-style FFN editing. Final structurally-different attempt. Drop the memory-attention path entirely. Train a vanilla transformer on addition via gradient descent. At deployment time, insert multiplication facts via closed-form rank-one updates directly to the FFN’s W_down matrix at the middle layer, following Meng et al. 2022’s “Locating and Editing Factual Associations in GPT.” No new training. The edit either takes or it doesn’t.

Five seeds, twenty-three multiplication facts edited into a 104k-parameter two-layer transformer pretrained on thirty-six addition facts:

Before any edits:
  Addition:                                  100.0% ± 0.0
  Multiplication:                              4.3% ± 0.0   ← chance / shape-prior

After ROME edits, identity covariance (simpler):
  Addition (preservation):                    88.9% ± 9.6
  Multiplication (insertion target):          14.8% ±12.5

After ROME edits, empirical covariance (proper ROME):
  Addition (preservation):                    94.4% ± 5.2
  Multiplication (insertion target):          15.7% ±  3.9   ← real but bounded

This is the only iteration with a real-but-bounded positive signal. Empirical-covariance ROME inserts measurable knowledge: 3.7× over baseline, addition preservation comfortably above the 80% bar, low variance across seeds. But 15.7% sits right at the bottom of the “ambiguous” band of the pre-committed decision rule, and the structural read on why it sits there does not predict that scaling within the same recipe lifts it cleanly.

A 104k-parameter model with d_ff = 512 has so little FFN capacity that twenty-three rank-one edits to similar key directions (all multiplication facts share the prompt shape X * Y =) interfere with each other. ROME was demonstrated at GPT-2-XL scale — 1.5 billion parameters, d_ff in the tens of thousands — where each rank-one edit sits in a much higher-dimensional space and interferes less with the others. The capacity-scaling argument predicts roughly what we got at our scale. It also predicts that running real ROME at GPT-2-XL would work, which the literature already shows. That’s reproduction, not research, and it doesn’t speak to Reverie’s question at all.

What we built is also ROME-flavored rather than real ROME: v* was closed-form (the LM-head row for the target token) instead of gradient-optimized, k* came from one prompt template instead of averaged over paraphrases, no causal tracing was done to select the layer, and twenty-three sequential edits were stacked on a single FFN where the canonical paper does one edit at a time and the batch-editing extension (MEMIT) spreads them across layers. So the 15.7% is a pessimistic bound on what a faithful small-scale implementation would achieve, not the ceiling of the method itself. But the structural diagnosis stands: at this scale, this family of methods is capacity-limited, and the only path to clean numbers is moving to a published-grade pretrained model and reproducing published work.

The structural finding

Across the original Reverie substrate (fifteen entries, nineteen ADRs) and across reverie-next (four iterations, three integration approaches), the same shape keeps appearing. Every architectural attempt to make the substrate — or the substrate-flavored memory layer — generalize runs into the same wall:

The property that makes Reverie interpretable (discrete, named, per-observation cells) is the same property that prevents it from generalizing. The property that lets a transformer generalize (continuous representations smoothly tying related inputs together) is the same property that makes its memory not auditable cell-by-cell. You can have one or the other within a single architecture. Every integration attempt is a different way of trying to have both, and so far every one of them has bounded out at the same kind of result.

This is sharper than the three patterns named at the end of entry 015, but it’s the same shape made structural rather than empirical. Entry 015’s three patterns were: every semantic-similarity addition collapses, every headline number is bounded by test design, every product story loses a degree of magnitude when probed. All three are downstream of the same thing: Reverie’s discrete-cell substrate has no continuous representation to support the smoothness that generalization requires, and you can’t add continuity without breaking the property that made the substrate interesting in the first place.

This isn’t a tuning problem. It isn’t a capacity problem. It isn’t a “wrong cell type” problem. It’s a structural fact about the relationship between two properties: interpretability via discrete cells and generalization via continuous representations are not simultaneously achievable in any architecture I’ve found that’s close to either Reverie or a small transformer. Maybe they are simultaneously achievable in some other architecture I haven’t thought of. The work doesn’t speak to that. The work speaks to the architectures actually tried.

What scale does and doesn’t fix

The temptation after four bounded iterations is to say “GPU + more parameters + real corpus and the numbers move.” That’s the same temptation entry 015’s three patterns warn against, applied to a new context. Worth being explicit about which parts scale fixes and which it doesn’t.

Scale fixes capacity-bound problems. Not enough FFN dimensions to hold many edits without interference: more parameters resolves it. Not enough cells to record what’s been seen: more memory resolves it. Not enough data to find the pattern: more corpus resolves it. The Option 5 ROME story at GPT-2-XL is the clearest case — the technique works at scale because each rank-one edit sits in a much higher-dimensional space than ours.

Scale doesn’t fix structural problems. Reverie has no continuous representation to support generalization, and adding more cells gives more isolated facts rather than more generalization. Reverie-next’s hybrid integrations fail in the same shape at every iteration not because the cells are too small but because the inductive bias for using them doesn’t form under gradient pressure with the architectures tried. More compute runs the same inductive-bias formation faster, not better.

The honest read: scaling reverie-next’s hybrid approach (Options 1-3 from ARCHITECTURE_OPTIONS.md) to 10-50M parameters on GPU with a real corpus has roughly a 25% calibrated probability of clean results, weeks of work, and lands — if it works — in a crowded literature (RETRO, kNN-LM, Memorizing Transformers, MoE-with-memory). Scaling Option 5 to GPT-2-XL is reproduction. Scaling original Reverie up doesn’t address the structural issue at all. None of these are paths to a research contribution that doesn’t exist in some form already.

What’s durable

After both projects, what survives is roughly what entry 015 said survives, plus the reverie-next infrastructure:

• The substrate — pure-numpy, no embeddings, cell-by-cell audit and delete, online updates, time-decay forgetting. All four properties are real and reproducible. None of them depend on canonicalization or on the four properties having a category-defining product story.
• The benchmark suite — nine arithmetic tasks plus the four reverie-next iterations, all reproducible from the published code. The benchmark is the most useful artifact of the project for anyone exploring this direction; it has clear pass/fail criteria and the negative results are documented honestly enough that the next person doesn’t have to re-discover them.
• The reverie-next codebase — vanilla transformer with FFN capture for ROME, structured cell store, copy head, ROME editor with both covariance modes and revert support. Lives in a private repo, available as infrastructure if any of the four properties above ever shows up as a product requirement.
• Two negative results, structurally diagnosed. Original Reverie can’t canonicalize. The transformer-integrations of Reverie’s philosophy don’t pay off at any of the four architectural marriages I tried at CPU scale. Both are bounded claims about what specific architectures don’t do, with specific evidence, in specific regimes. They’re useful for the next person who’s tempted to try the same things.

What I’m doing next

Not continuing this. The work has an ending, and continuing past the ending is what the decision rule was designed to prevent.

Energy moves to the projects with shorter and clearer paths from work to payoff: Ballast (halal investing copilot, eight-phase plan, real customers possible), Cairn (faith-aware personal AI, live infrastructure), Galley (collaborative book editing, first customer in hand for the Kitap series), Lodestar (ethical-screener, repositioned for Western retail). The Reverie substrate stays available as a library in case Cairn’s audit-layer story ever needs the cell-by-cell interface. The reverie-next codebase stays as durable infrastructure in case a structurally new question for memory-in-transformers shows up later. Both are reusable, both are documented, both have honest closing entries.

The fifteen entries of original Reverie and the four iterations of reverie-next together describe what observation-only substrates can and can’t do at the scale tested, what mechanisms work and which collapse, and what shape the product story has (and, mostly, doesn’t have). That’s a complete research artifact. It doesn’t claim AGI. It doesn’t claim a category-defining product. It claims to have asked a specific question carefully, run the experiments, honored the decision rules, and reported what the work said.

The arc has an ending.

— Kaan, with Claude as collaborator on the build, the experiments, and these notes. reverie-next lives privately and is closed at the architectural level; the canonical reference for that side of the work is PHASE1_RESULTS.md in the repo. Original Reverie remains at this address, fifteen entries plus this one, the way a research log is supposed to look when the question gets answered.