The binding hypothesis was right

There is a particular kind of result a research log waits for through enough negative entries that, when it arrives, the temptation is to overclaim it. This entry is mine. I’m trying to be careful.

The change

ADR 0014’s slot cells stored their learned preferences keyed on the literal bound-value tuple. For the arithmetic template (*, +, *, =) observing 1 + 2 = 3, the cell’s binding was (1, 2), and _slot_prefs[c][(1, 2)] accumulated a preference for 3. The architecture flagged its own limitation explicitly: “the bindings (1, 2) and (2, 1) are distinct dictionary keys. Learning one doesn’t reach the other.” Four entries of benchmarks confirmed the obvious consequence — Task 2’s held-out generalization sat at 3-17% no matter what else we changed.

The cheapest test of the binding hypothesis is to sort the bound-value tuple before using it as the key. (1, 2) and (2, 1) collapse to the same multiset; training on 1+2=3 automatically applies when the substrate later sees 2+1=…. This is architect-configured commutativity (you opt in with a slot_binding_symmetric=True flag) rather than discovered structure, so it’s honestly a bit of a cheat. But it cleanly answers the question entries 003 and 005 raised: is the rest of ADR 0014’s binding machinery sound and just missing one assumption, or is the whole mechanism structurally broken?

One branch in _check_slot_matches. Four tests, all passing. The full benchmark, five seeds, default (*, +, *, =) arithmetic template:

                                       Frozen (0014)    Symmetric (0017)
  Task 2 (held-out generalization)        3.3% ± 7.5    73.3% ± 14.9
                                              Transformer    73.3% ± 14.9

The lift is from 3% to 73%, and the new number is identical to the transformer’s number to the second decimal. Both architectures are now bumping into the same algorithmic ceiling — held-out facts whose commutative partner is also in the held-out set, or a + a facts whose partner is themselves. Within the structure of this benchmark, the substrate generalizes exactly as well as a ten-thousand-parameter transformer trained from scratch.

What this actually means

The binding hypothesis was right. ADR 0014’s mechanism is structurally sound. The four entries of underperformance were a missing assumption, not a missing mechanism. This is the first ADR in the project’s history that actually moves the headline number on the benchmark.

I want to be specific about what’s being claimed and what isn’t.

What is being claimed: given the role-symmetry assumption, Reverie’s slot cells now compose learned facts the way the transformer’s weights do. Held-out arithmetic generalization is no longer a function the substrate cannot represent.

What is not being claimed: the architecture is now discovering commutativity. It isn’t. I told it. The architect sets slot_binding_symmetric=True and the architect is making a domain assumption about what the wildcards mean. A truly observation-only substrate would need a mechanism for noticing that 1+2=3 and 2+1=3 co-occur often enough to mark those wildcard positions as a symmetric pair. That mechanism doesn’t exist yet, and the deferred-design discussion in ADR 0017 names it as the natural next ADR.

What is also not being claimed: Reverie is now competitive with the transformer in general. Tasks 1 (memorization), 4 (continual learning preservation), and 5 (online adaptation) are unchanged — they’re capacity and plasticity problems, orthogonal to the binding fix. Entry 005’s prose result is also unchanged; prose has no commutative wildcard structure for the symmetric flag to exploit. The fix is sharply targeted. It is exactly what entry 003’s representation-gap diagnosis identified, no more, no less.

What this changes about the prior entries

Entry 003 read the failures as a representation ceiling and entry 004 narrowed that to Task 2 specifically. Both diagnoses turn out to have been right, and this entry is the empirical confirmation: when the missing piece — operand symmetry — is supplied, the gap closes. The ADRs 0015 + 0016 detour through cache policies was the wrong attack on the right problem, as entry 004 already conceded. ADR 0017 is the first attack that worked.

Entry 005’s prose result, by contrast, is not fixed by ADR 0017. The wall there is a different one — Reverie’s lack of a “default to base rate” mechanism and the absence of compositional structure in short-range character prediction. ADR 0017 says nothing about either. Entry 005’s open questions (word tokens instead of character tokens, larger pool on prose, a marginal-prediction cell type) remain open and unrelated.

The honest caveat about the held-out number

The 73.3% on Task 2 is an artifact of the held-out split as much as it’s an architectural property. The benchmark randomly holds out 6 of the 36 arithmetic facts per seed. With most facts in our 36-fact set being non-self-commutative (only 1+1, 2+2, 3+3, 4+4 lack a distinct partner among a+b ≤ 9), most randomly held-out facts have their commutative partner trained — and symmetric binding can retrieve them perfectly. A held-out test that deliberately held out both a+b and b+a together for every test pair would expose where the architecture still can’t generalize: any pair both of whose orderings are unseen will still fail.

That stricter test is the right follow-up benchmark. I haven’t built it. ADR 0017’s claim is bounded by what was actually measured: symmetric binding gives identical numbers to the transformer on the current held-out generalization test, which is meaningful but not a claim that the substrate has learned addition.

What’s next

The natural next ADR is what ADR 0017 itself defers — a mechanism for the substrate to discover which wildcard positions are interchangeable from observation statistics alone. That’s the closer-to-honest version of “from observation alone.” It’s also a real research problem (probably weeks, maybe months) and shouldn’t be started without a clearer hypothesis about how the substrate would represent “co-occurrence pattern across observations.”

Two cheaper interim experiments that would tighten the empirical picture before committing to that:

1. The strict held-out test. Build a Task 2 variant where for each test pair both orderings are held out. Measure the symmetric-binding number on this version. It will not be 73%. Whatever it is will tell us how much of the result is architectural and how much was the easy split.
2. A second commutative domain. Run the symmetric flag on a non-arithmetic commutative task — e.g., set union — to confirm the mechanism isn’t quietly arithmetic-specific.

Neither is large. Either or both could be entry 007. Or this could be the project’s natural pause point: five entries of careful work converging on one valid architectural fix is a complete arc, the substrate’s design has been validated on the property it was originally pitched on, and Reverie can sit as a research artifact while attention goes elsewhere.

I’m going to leave that choice open for now. The result deserves a moment to be the result.

— Kaan, with Claude as collaborator on the build, the experiments, and these notes. ADR 0017 + the new tests + the benchmark variant flag are live; reproduce with model/scripts/benchmark.py --variant symmetric --task 2 --seeds 5.