The Cairn integration test

Every benchmark in the project’s history through entry 012 measured Reverie head-to-head against a ten-thousand-parameter untrained transformer trained from scratch on the same observation stream. That was the fair comparison for the question “what does the architecture buy you per observation, holding pretraining and parameter count constant.” It was not, and never claimed to be, a test of whether Reverie is useful as a component of a system whose language layer is a real pretrained LLM.

Entry 012 demonstrated the hybrid pattern with the small transformer: token-level distribution mixing, gated by Reverie’s confidence. That mechanism doesn’t translate to Claude — Claude is an API that returns finished text; there’s no per-token decoding step where Reverie could intercept. So before any Cairn wiring, the genuinely informative experiment is whether Reverie’s interpretable cell-by-cell memory provides useful retrieval context for Claude in the natural shape Claude is actually called.

This entry runs that experiment.

Setup

A small set of synthetic facts about a fictional user named Yusuf (the real user’s name was kept out to avoid Claude’s pretraining contaminating the test). Ten facts, mixing structured identifiers (KvK numbers, IBANs) with narrative facts (where he prays, what he drinks, who his wife is). The Anthropic SDK calls Claude Sonnet 4.6.

Three retrieval conditions per query:

1. Naive Claude — no extra context, just the question. Claude is instructed to answer “unknown” if the context doesn’t contain the information.
2. RAG — all ten facts dumped into the prompt as raw text. The upper bound: Claude has perfect access to the corpus.
3. Reverie templates — a Reverie substrate observes each fact several times; at query time, the substrate’s context cells are inspected, the templates whose tokens overlap most with the query are retrieved, and those templates are surfaced to Claude as context. The actual hybrid.

Eight queries, all of which require user-specific information that Claude cannot know from pretraining.

What happened

Starting with default Reverie config (K=4 context-window, top-5 template retrieval), and iterating on parameters as the result demanded:

  Predictor                          Accuracy (8 queries, 3 seeds)
  Claude alone (no context)                            0.0%
  Claude + RAG (all facts dumped)                    100.0%
  Claude + Reverie templates, K=4                     12.5%
  Claude + Reverie templates, K=8                     62.5%
  Claude + Reverie templates, K=12                    75.0%
  Claude + Reverie templates, K=16, top_k=15          87.5%

There are two thresholds the experiment exposed.

The first is K, the context-window size of each context cell. At Reverie’s default K=4, each context cell encodes a 4-token window of observed text. Most facts in the test are 6-10 tokens long. So at K=4 the substrate’s cells encode fragments of facts — Yusuf 's KvK number is in one cell, KvK number is 84112503 in another, and the keyword KvK in the query retrieves a cell whose 4-token template doesn’t contain the answer value. Claude correctly answers “the context mentions KvK but doesn’t provide the number.” That’s 12.5% accuracy — basically Reverie returning one query (tafsir) whose answer happened to be in the same 4-token window as its question keyword.

Lifting K to 8 brings most facts into single-cell coverage and lifts accuracy to 62.5%. K=12 lifts it further to 75%. Beyond that, K stops helping — the remaining failures aren’t about template length.

The second is the retrieval mechanism. The default surfaces the top-5 cells whose templates have the highest keyword overlap with the query. The remaining failures at K=16 are all queries whose natural phrasing uses synonyms of the words actually stored in the cells. “In which city does Yusuf live?” doesn’t keyword-overlap with “Yusuf’s home is in Utrecht near the canal.” Increasing top_k to 15 (retrieve nearly all cells with any keyword overlap, let Claude sort it out) lifts accuracy to 87.5%. The single remaining failure across all settings is the city ↔ home synonym pair — the keyword retrieval simply doesn’t know they’re related.

What this means for the Cairn wiring

Three honest readings, in order of how much they constrain the integration:

Reverie’s interpretability is real, but its retrieval quality is below RAG’s at this scale. A naive RAG implementation (dump all observed text, let Claude sort) hits 100% on this test. Reverie’s cell-by-cell retrieval with keyword overlap hits 87.5%. The 12.5-point gap is exactly the “semantic synonym” problem dense vector embeddings solve in standard RAG. So for raw retrieval quality, RAG wins.

Reverie wins on a different axis Cairn actually cares about: auditability. Each context cell holds an inspectable K-token template — a human can read every cell and verify what’s stored. RAG’s typical vector database is opaque: you can search it, but you can’t read it. For a personal-memory layer where the user might want to ask “what does Cairn remember about me” or “delete what you remember about X,” the cell-by-cell substrate has a property RAG fundamentally doesn’t. The question is whether that’s worth a 10-15 point accuracy gap.

The right Cairn integration probably isn’t pure Reverie or pure RAG. It’s both. Use Reverie as the inspectable interface to the memory (every observation lands in specific cells; the user can audit and delete cell-by-cell); use dense embeddings as the retrieval mechanism on top of the cell texts (so synonym matching works). Reverie becomes the storage and identity layer; vector search becomes the access layer. Each plays to its strength.

That’s a more sophisticated architecture than entry 012’s “two predictors plus a gate” pattern. It also makes more sense for a deployed product that has to actually work for users who type natural-language questions, not benchmark queries with controlled vocabulary.

What I’d actually build in Cairn now

Given the result, the minimal-viable Cairn memory layer is:

1. A Reverie substrate per user that observes journal/reality entries as they’re created. Cells claim templates over the user’s text. The substrate is the durable, inspectable record of what the user has told the system.
2. A simple text embedding index on the same observed text (separate from Reverie). When the briefing builder needs context, it embeddings-search the recent entries for relevance to today’s calendar/emails/etc., and pulls the top-N as supplementary briefing context.
3. Cell-level audit endpoints on the Reverie side: list all claimed cells, show what each one encodes, allow deletion of specific cells. This is the user-facing “what does Cairn remember about me” surface.
4. Briefing prompts get a “what Cairn has noticed” section drawn from the embedding-search results, scoped to whatever feels relevant.

This is a real, scoped, ~2-3 day product feature, not a research experiment. The Reverie substrate is doing real work (durable observation, interpretable storage, user-controllable deletion) — but it’s not doing the retrieval. That’s by design now, not against the substrate’s properties.

What this changes about whether the project is over

For the third or fourth time in this log, I’d been about to declare the research-log Reverie complete and move attention to the product Reverie. Each time, a question I should have asked sooner turned out to be small enough to test in one focused session, and each time the test changed the answer to “what is the integration actually shaped like” in a meaningful way.

The pattern now: every time I think I’m done, the next entry shifts the framing rather than confirms the conclusion. Entry 010 said “this is where I’m stopping.” Entry 011 said “with less certainty.” Entry 012 reopened it. Entry 013 says: the Cairn integration shape is now clear enough that the product work can start; the research log is still doing its job, and might keep doing it.

The research artifact and the product opportunity have always been the same code base, the same substrate, the same set of mechanisms. The entries trace what that substrate can and can’t do, what it’s worth as a research artifact, and what shape it takes when paired with real systems. Thirteen entries in, the answer to “what is Reverie?” has more sides than it had after entry 002, and most of them have been useful.

I’m closing this one without claiming it’s the last. The Cairn integration is the next thing to actually build. That will probably produce entry 014 of its own, or it won’t — and either outcome would be the right shape for a research log.

— Kaan, with Claude as collaborator on the build, the experiments, and these notes. The Claude integration test is at model/scripts/claude_hybrid_test.py; reproduce with --seeds 3 --show-answers. Uses Claude Sonnet 4.6 via the Anthropic SDK and reads the API key inline from macOS Keychain via bmsecret.