The times table was not the decider

Entry 007 closed with one cleanly testable open question. Reverie with symmetric binding and the tiny untrained transformer scored nearly identically on Task 2’s standard held-out generalization (73.3% each) and again on Task 7’s max-instead-of-add variant (73% vs. 97%). The transformer’s larger lead on max was reasonable evidence that it was doing genuine function approximation — max(a,b) is structurally a “copy the bigger operand” operation, easy for attention to learn. But the addition case was ambiguous: both architectures could plausibly be doing retrieval rather than learning.

Multiplication tables seemed like the cleanest tiebreaker. Multiplication is structurally harder than addition (more parameter space to fit), more harder than max. If the transformer is genuinely learning functions via gradient descent, it should still cope — possibly with lower accuracy, but with a meaningfully different generalization profile than Reverie’s retrieval. If Reverie matches it anyway, both architectures are doing the same thing under different vocabulary.

So I built Task 8. Same template structure as Tasks 2 and 7 — (a, +, b, =, c) — but + now denotes a × b, constrained to a·b ≤ 9 so the answer stays single-digit. That gives 23 facts. 19 train / 4 held out per seed, 5 seeds, the usual benchmark stream length.

The result

  Task 8 — multiplication tables, random 19/4 split
  Frozen Reverie:      5.0% ± 11.2
  Symmetric Reverie:  65.0% ± 28.5
  Transformer:        60.0% ± 28.5

Symmetric Reverie at 65% is statistically indistinguishable from the transformer at 60%. With only 4 held-out facts per seed, accuracy jumps in 25-point increments per seed — the ±28.5 standard deviation reflects how much the specific random held-out choice matters, not any real architectural difference. Both architectures are doing the same thing within the resolution of this measurement.

I had genuinely expected one of two outcomes. Either (a) the transformer would meaningfully outperform symmetric Reverie (say, 80% vs. 50%), confirming it learns multiplication as a function — or (b) both would collapse to near-zero, confirming neither does and the entry-007 retrieval hypothesis applies cleanly. Instead, the result is (c) both architectures do moderately well at the same task and one isn’t clearly better than the other.

What this means

Either multiplication isn’t actually harder than addition for this benchmark (because most held-out multiplication facts have their commutative partner in training, so retrieval still works for both architectures), or the transformer at this size and data budget isn’t doing genuinely separable function learning anyway. Both readings are consistent with the data, and Task 8 alone can’t distinguish them.

There is one prediction that would distinguish them, and it follows the same playbook as Task 6: a strict held-out variant for multiplication, where both orderings of each test pair are held out together. Under the retrieval hypothesis, both architectures should crash to near-zero (since the sorted-multiset key was never trained). Under genuine function learning, the transformer should retain some performance via gradient-descent-extracted multiplication structure, while Reverie crashes.

I haven’t run that yet. It’s an obvious follow-up — Task 9 in spirit, same structure as Task 6, different operation. I’ll name it explicitly here so it doesn’t get lost in the queue.

What I’m reading from the larger arc

Entries 002 through 008 form a now-coherent picture, even if the final score isn’t a clean architectural victory for either side. Three things hold up across all of it:

One. The benchmark itself is the durable artifact. Eight tasks, two architectures, configurable variants, multi-seed aggregation, every result reproducible from a single command line. The substrate of this research log is the code. Future entries from anyone — including me, returning to this — can extend it.

Two. ADR 0017 is a real architectural win, bounded honestly. Symmetric slot binding lifts held-out generalization on commutative operations from 3% to 65-73% by collapsing operand permutations onto a single learned key. It does this via one branch in _check_slot_matches. It is the one thing in the project’s seventeen ADRs that moves the headline number.

Three. The remaining gap between Reverie and the transformer — to the extent it exists at this scale — is concentrated in function learning rather than capacity or binding. The transformer’s 97% on max comes from genuine gradient-descent compression of “copy the bigger operand”; Reverie’s 73% on the same task comes from retrieval. On harder operations where neither model can fully compress the function from sparse examples (add, mult), the two architectures converge in measured performance.

That third claim is the closest thing the project has to a research finding: for sparse symbolic tasks at this scale, the architectural gap between a per-cell observation-only substrate and an untrained backprop transformer is smaller than the entry-002 numbers suggested, and is concentrated specifically in operations whose underlying function is simple enough for gradient descent to extract from few examples. Less catchy than “Reverie wins” or “Reverie loses.” More honest.

Where I’d actually go from here

The honest list of remaining experiments and ADRs has not changed structurally since entry 007. The only thing that’s changed is the framing of what they would establish:

• Task 9 — strict held-out multiplication. The genuinely missing data point for the function-learning hypothesis. Would close the diagnostic loop on entries 006–008 cleanly. One day.
• More training tokens across tasks 2/6/7/8. The current 2048-token budget may understate both architectures’ actual capacity. Sweeping that for the transformer specifically would tell us whether longer training reveals function learning we couldn’t see before.
• Direction B — vector-symbolic binding. The serious ADR. If the project’s goal becomes “Reverie does function learning, not just retrieval,” this is the architectural commitment. Weeks.
• Direction C — discoverable slot structure. The version of the project closer to its original “from observation alone” framing. Weeks to months.
• A project retrospective entry. Eight entries is a complete arc. The substrate has been measured, named, partially improved, and partially bounded. A retrospective would archive Reverie as a closed research artifact rather than a continually-active project.

I do not yet know which of these I will do, and I am content to leave that genuinely open. The thing I would not do is force a triumphant conclusion onto a project whose actual findings are this measured.

This is a research log. It has been one. It will continue to be one if I keep writing.

— Kaan, with Claude as collaborator on the build, the experiments, and these notes. Task 8 at model/scripts/benchmark.py; reproduce with --variant symmetric --task 8 --seeds 5.