A tiny transformer as a mirror

A friend looked at the architecture and said this looks a bit like a transformer. The surface case is real: each cell holds a vector of weights over tokens, the way an attention head holds weights over positions; context cells with N-token templates rhyme with key-matching; slot cells with wildcards loosely echo the query side of attention. None of the mechanism is transformer-like — there’s no softmax over QKV, no stacked layers, no backprop, and Reverie maintains state across ticks while a transformer is stateless per forward pass — but the shape of “what each unit cares about” is similar enough that the analogy is worth taking seriously.

He proposed a benchmark. Take a tiny transformer, untrained — no pretraining of any kind, just random weights — and have it consume the same observation stream Reverie does. See what each architecture extracts from identical data.

So I built one. About ten thousand parameters, single transformer block, single attention head, ReLU FFN, trained from scratch on the same token streams. Then ran three tasks on both.

What the benchmark says

Task 1 — memorize 36 single-digit addition facts. Both models see 1+1=2, 1+2=3, …, 8+1=9 (every a+b=c with a,b ≥ 1 and a+b ≤ 9), shuffled. After N tokens of training, can it answer when prompted a + b =?

  N tokens             Reverie         Transformer
        64         32.8% ± 5.7         27.2% ± 5.0
       128         33.9% ± 5.3         37.8% ± 8.2
       256         32.8% ± 3.6         73.9% ±11.9
       512         35.0% ± 6.4         95.0% ± 4.1
      1024         33.3% ± 7.1         95.0% ± 9.7
      2048         36.1% ± 3.9        100.0% ± 0.0

The transformer climbs from chance to perfect across two orders of magnitude of training data. Reverie sits at ~33% from the first sample to the last. This is not noise; it’s a ceiling. The substrate’s context cells (ADR 0012) are responsible for exact-window memorization, and they each claim at most one template. The default population has 64 of them. Thirty-six facts × multiple sliding windows per fact exhausts the capacity. More observations don’t help, because the cells that could memorize the remaining facts have already been spent on the first batch.

A sharper way to put it: Reverie’s memorization is bounded by its allocation, not by its data.

Task 2 — generalize from 30 facts to 6 held out. Train on 30 of the 36 facts, test on the 6 the model never saw.

  Reverie (with slot template):    3.3% ± 7.5
  Transformer:                    73.3% ± 14.9

This was the result I was least prepared for. A ten-thousand-parameter untrained transformer, after seeing about two thousand tokens, generalizes addition. It produces the correct sum for facts it was never shown three times out of four. The variance is the held-out facts varying across seeds; the mean is unambiguous.

Reverie does not generalize. Three percent on six held-out facts is one correct answer across five seeds, which is within the rate you’d get by coincidence. ADR 0014 added slot cells specifically to give the substrate variable binding — the architectural primitive needed for compositional generalization. The slot template (*, +, *, =) should let the system learn what + does across operand pairs. The empirical answer is that it doesn’t, at least not at this configuration. The bindings (1, 2) and (5, 2) remain distinct dictionary keys in practice, exactly as the ceiling described in entry 001 predicted.

Task 3 — anticipate a cycle. Train on A B C D A B C D …. Given A, predict B.

  N tokens             Reverie         Transformer
        32        100.0% ± 0.0        100.0% ± 0.0
       64         100.0% ± 0.0        100.0% ± 0.0
      128         100.0% ± 0.0        100.0% ± 0.0
      256         100.0% ± 0.0        100.0% ± 0.0
      512         100.0% ± 0.0        100.0% ± 0.0

Both architectures solve this from the smallest training size. Reverie’s STDP-built directional connections (ADR 0008) plus predictive cells (ADR 0009) handle short cycles cleanly. Attention does too. There is no daylight between them here.

What changed in my picture

Entry 001 ended on a ceiling: the substrate memorizes but does not generalize abstract operations. That was true. What I didn’t yet have was a comparator. The benchmark provides one, and the comparison reframes what each of the substrate’s pieces actually buys.

The findings I’d preserve from this:

1. Cyclic anticipation is not where the action is. Both architectures are saturated on it. STDP and predictive cells are doing real work in Reverie, but they’re not what distinguishes the substrate — a tiny transformer learns the same patterns from the same data with no temporal primitives at all.

2. Reverie’s memorization is allocation-bound. The flat 33% ceiling on 36 facts has a specific architectural cause and a specific fix: grow the context-cell pool, or let context cells be reclaimable, or both. This was visible in principle from ADR 0012 but had never been measured against a baseline that didn’t share the constraint.

3. Slot cells did not yet deliver their purpose. Three percent on held-out generalization is, empirically, no generalization. ADR 0014 named the goal precisely; the implementation does not yet meet it. That’s worth flagging here rather than discovering it three iterations later. Whatever variable-binding looks like in a substrate like this, the current shape isn’t it.

4. Untrained transformer ≠ trivial baseline. Going in, I expected a randomly-initialized transformer to be near-chance on everything — its whole point is the trained weights. That intuition was wrong for this scale. Ten thousand parameters and two thousand tokens are enough to extract the addition function. The architecture itself, with backprop, is doing more of the work than the pretrained-weights frame gives it credit for.

What I’d build next

Not differently — in addition. The substrate’s most defensible remaining case is the one it was originally proposed for: online, streaming, no-backprop, observation-driven learning where what’s learned is interpretable cell-by-cell and where the system never has to be retrained from scratch. The benchmark didn’t probe any of those properties because the tasks were small and offline. A more honest test of Reverie’s actual claim would be:

• A non-stationary stream where the distribution shifts mid-training, and the question is how fast does each architecture adapt.
• A continual-learning probe: train on facts A, then on facts B, then re-test A. The transformer will forget; the substrate’s cell-by-cell memory may not.
• An interpretability probe: after training, can a human read what each cell encodes? For Reverie this is by construction; for the transformer it requires probing tooling that doesn’t exist at this scale.

Those are the tests where I’d expect the substrate to earn its keep. The current benchmark answers a different question — is the architecture itself sample-efficient on small symbolic tasks compared to a vanilla transformer — and the answer for the symbolic tasks here is no, except on the trivial one. The honest record of that is this entry.

The friend’s intuition was sharper than the analogy. I owe him a coffee.

— Kaan, with Claude as collaborator on the build and these notes. Benchmark code lives at model/scripts/benchmark.py; the transformer at model/scripts/tiny_transformer.py. Both runnable from the model/ directory with five seeds via --seeds 5.