The hybrid works

The arithmetic benchmark through entry 011 was always a head-to-head measurement: transformer vs. Reverie, which one wins on each task. The cleaner question — the one entries 010 and 011 kept circling around without quite asking — was whether Reverie’s strengths are useful as a component of something larger rather than as a competitor. This entry runs that experiment.

The setup

Twelve-token shared vocabulary: digits 1-9, +, *, =. The operator distinguishes the two domains.

Phase 1 — A TinyTransformer (the same one we’ve been comparing against, ~10k parameters, untrained) is trained on the 36 addition facts using + as the operator until it hits 100% accuracy. Then frozen — gradients are disabled, no further updates ever.

Phase 2 — A Reverie substrate observes a stream of 23 multiplication facts using * as the operator. The transformer never sees these observations. Reverie never sees the addition facts.

Phase 3 — Three predictors are evaluated on both domains:

• The transformer alone (knows addition, oblivious to multiplication)
• Reverie alone (knows multiplication, oblivious to addition)
• A hybrid that combines both via a gate using Reverie’s own confidence as the routing signal: when Reverie’s top valid-answer probability exceeds a threshold the substrate has expertise here and its prediction is used; otherwise the transformer’s prediction is used.

What happened

At 5 seeds:

  Predictor                Acc on addition     Acc on multiplication
  Transformer only           100.0% ± 0.0           4.3% ± 0.0
  Reverie only                10.0% ± 5.8         100.0% ± 0.0
  Hybrid                     100.0% ± 0.0         100.0% ± 0.0

The hybrid covers both domains at 100% where each component alone covers exactly one. The transformer’s failure on multiplication (4.3%) is total — it produces a confident, wrong answer for every multiplication query, because the vocabulary token * was in its training vocab but never appeared in any training sequence. Reverie’s failure on addition (10%) is the natural failure of a system that simply never saw those facts. The hybrid, gated by Reverie’s confidence, routes each query to the architecture that actually learned it.

This isn’t an architectural breakthrough on either Reverie or the transformer in isolation — it’s the demonstration that the two together have a property neither has alone.

What’s actually happening in the gate

The first three combinators I tried didn’t work:

1. Static weighted average — drowned Reverie’s correct-but-spread predictions under transformer’s confidently-wrong-on-multiplication predictions.
2. Confidence-weighted with sharpening — sharpening Reverie made it dominate addition queries it didn’t know.
3. Entropy gate over valid tokens — the transformer is equally peaked on queries it learned and queries it’s confidently wrong about. Entropy doesn’t correlate with correctness for it.

The fourth one worked: use Reverie’s confidence as the gate signal. The transformer is fluently confident on both in-distribution and out-of-distribution queries; its outputs don’t tell us when it’s guessing. Reverie’s outputs do — its mass over the valid-answer subspace only concentrates when a slot cell has actually bound to the prompt’s window. On unobserved queries Reverie’s distribution spreads toward uniform. So the gate is just: if Reverie’s top valid-token probability is above a threshold (0.4 in this script — well above the uniform baseline of 1/9), Reverie has expertise and its prediction is used. Otherwise the transformer’s.

This is itself a use-case for Reverie’s interpretability. The transformer’s confidence is uncalibrated to its own knowledge. Reverie’s confidence — because it comes from a cell-by-cell substrate where you can literally inspect which cells fired — is meaningfully tied to whether the substrate has learned the query. That makes Reverie a usable gating signal for hybrid systems even when its own predictions wouldn’t be competitive standalone.

What this looks like for Cairn

The architecture maps directly onto the personal-AI use case I’ve had on the back of the project notes through all eleven prior entries. A frozen pretrained transformer — Claude, or any model with general language capability — handles fluent generation and reasoning. A Reverie substrate observes user-specific facts continuously after deployment: preferences, schedules, names, the things the generic transformer doesn’t and can’t know. At each generation step both contribute. The transformer never gets retrained on personal data — it stays generic and replaceable. Reverie is the user-specific layer, fully interpretable cell-by-cell, deletable cell-by-cell if needed.

The properties this gives Cairn that neither component alone has:

• Generic + personalized. The transformer’s pretrained knowledge plus a continually-updated personal substrate.
• Auditable memory. Every personal observation lives in a specific cell you can inspect. Unlike RAG’s vector database — which is queryable but not interpretable per-entry — every Reverie cell has a readable role.
• No fine-tuning ever. Personalization happens online, without backprop. Privacy implications are real: personal data never enters the transformer’s weights.
• Deletable on request. Reverie cells can be inspected, edited, or zeroed individually. “Forget that I said X” is a specific action against a specific cell, not a model retrain.
• Resilient to model swaps. Reverie is decoupled from the transformer. Upgrade the LLM to a newer Claude; Reverie’s substrate carries forward unchanged.

That’s a real product story, not just a research story. It’s also distinct from existing retrieval-augmented systems: RAG’s memory is a trained encoder + vector database, neither part of which is interpretable cell-by-cell. Reverie’s memory is.

The honest scope of the result

A few things this experiment does NOT claim:

• The hybrid doesn’t beat the transformer at language modeling, reasoning, or any task transformers natively do well. It just extends the transformer’s coverage to a domain Reverie has observed. The reasoning ceiling is still whatever the transformer can do.
• The gate works because the prompts are operator-distinguished. With identical prompts that have different correct answers under different conventions, no gate based on either model’s outputs alone can route correctly. We had this exact failure in the first iteration, fixed it by separating the operators.
• Reverie has to be the “novel knowledge” component. Swapping roles — Reverie pretrained, transformer learning online — wouldn’t work, because transformers can’t learn online without backprop and parameter updates.
• The threshold gating value (0.4) was tuned empirically. A more principled approach would learn the threshold from a small validation set, or use a per-token probability ratio rather than an absolute cutoff. Out of scope here.

What this changes about closing the project

Entry 010 closed the project. Entry 011 closed it again, with less certainty. Entry 012 isn’t going to close it cleanly because the natural next step — wiring this hybrid into Cairn as the actual memory layer of a real personal AI — is a thing that wants to happen, not a thing I have to force.

What stays true: Reverie alone is not a competitive replacement for transformers. The eleven prior entries established that clearly and bound the claim from multiple angles. What this entry adds: Reverie as a component in a hybrid system has unique properties (interpretable memory, online learning, no-backprop personalization) that the transformer alone doesn’t have, and the integration is genuinely simple — a confidence-gated routing function on top of two distributions.

The research log Reverie has been is complete. The product Reverie might enable is just starting.

— Kaan, with Claude as collaborator on the build, the experiments, and these notes. The hybrid benchmark is at model/scripts/hybrid_benchmark.py; reproduce with --seeds 5. The natural next step is wiring this into ~/cairn as the personal-memory layer; that’s its own project.