The symmetric win, bounded

Entry 006 had two named limits on its result. The 73.3% on Task 2’s held-out generalization was partly an architectural property and partly the easy structure of the random 30/6 split — most randomly held-out arithmetic facts have their commutative partner trained, and symmetric binding can retrieve them perfectly under the sorted key. The second concern was whether the symmetric-binding mechanism was quietly arithmetic-specific: maybe + happened to be exactly the kind of operation where this fix worked, and other commutative operations would behave differently.

This entry is the two cheap follow-up tests entry 006 named, with the answers they produce.

Test one — strict held-out

Task 2’s standard form holds out 6 random facts of 36 per seed. Most of those 6 are arithmetic pairs a+b=c whose commutative partner b+a=c is still in the training set. With symmetric slot binding, the sorted-multiset key for that pair is already trained — the architecture’s “generalization” is mostly a retrieval under a key it already learned.

Task 6 — the new strict variant — fixes that. For each held-out pair, both orderings are held out together. The architecture has never seen the multiset, only commute-disjoint pairs. If the substrate (or the transformer) is genuinely learning addition, this should still work. If both are doing key-lookup with extra steps, both should fall to chance.

  Task 6 — strict held-out (5 seeds)
  Symmetric Reverie:    3.3% ± 7.5
  Frozen Reverie:      13.3% ± 13.9
  Transformer:          0.0% ± 0.0

Symmetric Reverie at 3.3% is chance. Frozen Reverie at 13.3% is faintly above chance, probably because context cells with partial-match patterns occasionally fire for digits that happen to be the right answer. The transformer at 0.0% across all five seeds with zero variance is striking: not just failure, but systematic failure — the model is confidently predicting wrong answers, presumably because its learned function reflects the frequency distribution of the training set rather than the structure of addition.

The reading I want to commit to from this:

Neither architecture learns the addition function from 30 sparse examples. The 73% from entry 006 was almost entirely the easy-split crutch — the held-out facts whose commute partner was trained. The substrate’s apparent generalization was a more accurate retrieval, not an inference of structure. The transformer’s apparent generalization was the same thing in different vocabulary: it interpolates among trained examples but doesn’t extrapolate to operand pairs it never saw a member of.

This isn’t a retraction of entry 006. ADR 0017 still does exactly what it says it does — collapses operand permutations onto a single key — and that’s a real architectural property worth having. But what it gives you is commutativity retrieval, not arithmetic comprehension. The substrate has not learned addition. It has learned that when the wildcards are interchangeable, training on one ordering applies to the other.

Test two — a different commutative operation

The second concern was whether the mechanism was quietly addition-specific. Task 7 is the same template (*, +, *, =) and the same vocabulary, but the operator + now denotes max(a, b) instead of addition. The substrate has no semantic grasp of what + means; the input is just five-token sequences with the same shape and different answer tokens. Since max is also commutative, symmetric binding should work the same way.

  Task 7 — max(a,b), random 30/6 split (5 seeds)
  Symmetric Reverie:   73.3% ± 14.9
  Frozen Reverie:      10.0% ± 9.1
  Transformer:         96.7% ± 7.5

Symmetric Reverie lifts from 10% to 73% on max — the exact same magnitude as the lift on addition. The mechanism isn’t arithmetic-specific; it’s commutativity-specific, which is what ADR 0017 claims. Worth keeping.

What’s striking is the transformer at 96.7% — substantially higher than its own 73% on addition. max(a, b) is a structurally simpler function than a + b: the answer is one of the two operands, copied directly, rather than a third value derived through positional arithmetic. The transformer can express max as something like “select operand-2 if operand-2 > operand-1 else operand-1” — a function that’s natural for attention to learn even from 30 examples and that generalizes to held-out operand pairs with high accuracy.

This is the comparison I wanted to find back in entry 002: a setting where the substrates’ relative behaviors diverge meaningfully. They do diverge here — not in the direction I hoped (Reverie winning), but in a more interesting one. The transformer’s lead over Reverie is now a function of how compressible the underlying operation is: when the function is structurally easy (max), the transformer’s parameter sharing gives it a large edge over Reverie’s per-key storage. When the function is structurally harder (+), the gap narrows because gradient descent over 10,000 parameters and 30 examples isn’t enough to fully learn it either.

What this changes about the project’s claims

Both tests are useful for tightening what entry 006 was allowed to assert.

The claim that survives: ADR 0017’s symmetric slot binding is sound, correct, and general across commutative operations. It is the first architectural change in the project’s history to lift the benchmark on the property entries 002–005 said the substrate was missing. None of that has been retracted.

The claim that doesn’t survive in its original form: Reverie now generalizes “the way the transformer does.” It doesn’t. The transformer at 73% on Task 2 and 97% on Task 7 is learning a function — imperfectly for addition, well for max — and that function applies to operand pairs it never saw. Symmetric Reverie at 73% on both tasks is doing a structural shortcut: retrieving the commutative partner that was trained. When the shortcut is removed (Task 6), the substrate has nothing left.

So the cleaner architectural reading is: ADR 0017 fixes the symmetry missing from ADR 0014’s binding. It does not fix the function-learning missing from the substrate as a whole. Both are missing. ADR 0017 only addresses one of them.

What’s next, again

The list from entry 006 is largely intact, with one item closed and the framing of the rest sharpened:

• B. Vector-symbolic binding. Still parked. Would give the substrate genuine function approximation via algebraic operations on token vectors. Weeks. Probably the next ADR worth attempting if function-learning is the goal.
• C. Discoverable slot structure. Still parked, and now arguably the more honest direction. A substrate that learns which wildcards are interchangeable from observation statistics would be closer to the project’s original framing of “from observation alone.” Weeks to months.

Two cheaper experiments that became thinkable only after this entry’s results:

• More training tokens for Task 6. I trained for 2048 tokens, which the transformer needed to hit 100% on Task 1. Maybe the strict held-out test would lift if either architecture saw 20× more data. The current 0% / 3% / 13% results may be partly a data-budget property, not purely a structural one.
• A simpler symbolic task where the transformer demonstrably learns a function. Multiplication tables would be a natural one. If the transformer learns the times table cleanly from sparse examples and Reverie does not, that nails down the function-learning gap precisely.

Either could be entry 008. Or this is the point to step away — the project now has a real, bounded architectural win (ADR 0017), an honest characterization of what it does and doesn’t fix, and a clear queue of follow-ups. Seven entries, fifteen accepted ADRs, one mechanism that actually moves the headline number, and a written record of every wrong turn that got there.

I am genuinely unsure whether to keep going. Either answer is fine.

— Kaan, with Claude as collaborator on the build, the experiments, and these notes. Tasks 6 and 7 are at model/scripts/benchmark.py; reproduce with --variant symmetric --task 6 --seeds 5 and --task 7 --seeds 5.