Decision 0012: Context cells with n-gram templates

Date: 2026-05-24 Status: Accepted Builds on: ADRs 0001–0011

Context

Through ADR 0011 the architecture has eleven mechanisms working together: cell types, the asking protocol, connections (now multi-lag and directional), replay during excitation, decay, normalization, stagnation, predictive cells, and spontaneous output. The arithmetic experiment was the original test case from the architectural sketch, and we’ve now confirmed three times — at ADRs 0008, 0009, and 0010 — that it doesn’t crack with the substrate as it stood. Each of those ADRs explained why: multi-tick STDP encodes marginal associations per lag, not conjunctive ones. The system cannot represent “the specific 4-token sequence [1, +, 1, =]” as distinct from “the marginal fact that 1 and + and = co-occurred sometime.”

This ADR introduces conjunctive context binding through the simplest mechanism that fits Reverie’s architecture: a fourth cell type, CONTEXT cells, each of which locks onto a specific K-token sequence (an “n-gram template”) and activates only when that exact sequence has just been observed.

It is critical to be honest about scope. This ADR enables memorization of specific sequences — [1, +, 1, =] → 2, [2, +, 1, =] → 3, and so on. After enough training, the arithmetic experiment should produce dominant outputs of the trained answers. What this ADR does not enable is generalization — the system cannot infer 5 + 3 = ? from facts it has memorized. True generalization across operand pairs requires variable binding, which is a different architectural primitive that the substrate has no native form of.

The honest motivation: memorization is a real architectural step that demonstrates conjunctive binding, validates the n-gram approach, and surfaces what would be needed for generalization (an open problem deferred).

Decision

Fourth cell type — CONTEXT

Add a new integer constant CONTEXT = 3 alongside the existing types. The cell-type allocation now splits four ways. With default context_cell_fraction = 0.2, roughly one cell in five is a context cell. The remainder splits among mimicking, hebbian, and predictive in their existing proportions.

Recent-tokens window

The system maintains a bounded list of recent observed token indices:

• _recent_observed_tokens: list, max length context_window_size. Default size K = 4 (sufficient for 5-element arithmetic sequences like [op1, +, op2, =, ans] where the context is the first 4 and the answer is the 5th).

This window is updated at the start of each observe() call. It captures what was observed before the current token — the context the current observation comes after.

Template lifecycle

Each context cell has a single “template” — a list of K token indices (or None if unset). Lifecycle:

• At allocation: template is None (no template claimed).
• On observation: for each unclaimed context cell, with some probability the current window becomes its template — if the window is “interesting” (has at least one non-None entry and isn’t a duplicate of an already-claimed template).
• Once claimed: the template is fixed for that cell’s lifetime.

Claiming is one-at-a-time per observation: at most one new template per observed token, to spread coverage across observations rather than greedily claiming all cells on the first interesting window. This is a simple round-robin: the next free context cell claims the current window.

Activation and learning

When observe(token) is called:

1. Compute the current context window (the K tokens observed before this one).
2. For each claimed context cell, check whether its template exactly matches the current window. If yes, the cell is “activated by context.”
3. Activated cells receive a strong activation boost — they’re the cells the system has learned represent this specific context.
4. Apply the existing observation pipeline (preference learning, etc.). The boost means activated context cells reinforce pref[observed_token] strongly — exactly the conjunctive learning we want.

Match must be exact in v0.11. Approximate matching (one wildcard slot, weighted partial match) is a candidate for a future ADR but not in scope here.

Generation

Context cells contribute to generate() the same way other cells do — activations @ preferences. When a context cell is activated (its template matched), its high activation × strong pref[next_token] pulls the output distribution toward the token it has learned follows that specific context.

Defaults

• context_cell_fraction = 0.2 — 20% of cells are context cells.
• context_window_size = 4 — large enough for 5-element arithmetic facts.
• Setting context_cell_fraction = 0.0 disables context cells entirely (revert to v0.10 four-type-with-no-context).

What this fixes and what it does not

Fixes

• Memorization of specific n-grams. Train on [1, +, 1, =] → 2 and similar facts; the system will produce dominant 2 output when given [1, +, 1, =] as recent context.
• Conjunctive binding as a primitive: context cells represent specific multi-token configurations, not marginal associations. This is the missing primitive flagged by the previous three ADRs.

Does not fix

• Generalization. Each context cell binds exactly one window. There is no mechanism for “context-with-a-slot” or “operand role.” 5 + 3 = ? works only if exactly [5, +, 3, =] was a training sequence.
• Symbolic transformation. Even with all binary-sum facts memorized, the system can’t combine two operands into an answer the way arithmetic actually does. It just retrieves stored associations.
• Capacity scaling. Context cells are finite. After all are claimed, new patterns cannot be memorized without growth.

These are the honest limits. ADR 0012 demonstrates that the architecture can learn specific symbolic patterns, but the more interesting question — whether it can ever generalize them — remains open.

Stability and architectural considerations

Memory cost

Each context cell stores one template (K integers). For K=4 and 32 context cells: 128 integers. Plus the normal preferences[c] array. Negligible.

Activation magnitudes

Context cells with matching templates get a substantial activation boost (the constant CONTEXT_MATCH_ACTIVATION_BOOST, default 3.0). This means they tend to win the lateral-inhibition field on the tick they activate. The existing activation cap (2 · (baseline + saturation) = 8.0) keeps things bounded.

Template duplication

Two context cells could theoretically claim the same template if observed multiple times. To avoid wasted slots, the claim logic rejects windows that exactly match any already-claimed template.

Interaction with other mechanisms

Context cells use the normal _preferences array for predictions and participate in connection updates like any cell. They are not exempt from normalization, decay, or stagnation tracking. The novelty is purely in their activation rule and the template state.

Behavioral predictions

These claims have tests in tests/test_context_cells.py:

1. Default population includes context cells at ~20% of the pool.
2. With context_cell_fraction = 0, no context cells exist — backward compatibility.
3. A context cell claims an n-gram template on observation when the recent window is full.
4. Re-observing the same n-gram activates the claimed cell strongly.
5. The arithmetic experiment shows measurable preference for trained answers — [1, +, 1, =] query produces a meaningfully higher fraction of 2 than other digits.
6. All prior 87 tests continue to pass under the default settings.

Open questions deferred

• Approximate matching. A context cell might activate (weakly) for partial matches. This would enable some smoothing across contexts but blur the “this exact sequence” semantic.
• Variable-binding slots. Cells that bind to roles rather than concrete tokens, e.g., a cell whose template is [OPERAND, +, OPERAND, =] and that learns to predict the sum of whatever operands fill the slots. This is the next architectural step for true arithmetic, and it’s hard. Probably a future ADR with significant design work.
• Template forgetting. Once a template is claimed, it’s fixed. A future ADR may allow cells to “release” stale templates and learn new ones.
• Growth of context cells. Capacity-driven growth (ADR 0001) doesn’t currently consider context cells specifically. Under heavy n-gram load, we might want to grow them preferentially.