Prose instead of arithmetic

The arithmetic benchmark in entries 002–004 had a fair criticism baked into it: arithmetic is exactly the kind of task transformers are built to find via gradient descent. The architecture was being graded on the home turf of its opponent. Several of those entries flagged a non-symbolic comparison as the obvious counter-test — Reverie’s pitch is observation streams with emergent structure, not closed-form symbolic functions. Maybe prose is where the substrate’s design actually pays off.

So I ran it.

Setup

The corpus is the markdown body of entry 001 (the substrate post) — 7,761 characters, 70 unique. Self-contained in the repo, structured enough to have real next-character patterns, no copyright or alignment-data issues. Split 80/20: train on the first 6,208 chars, test on the last 1,553. The task: at each position in the test segment, take the preceding 4 characters and predict the next one. Top-1 accuracy.

Both models see the identical stream. The transformer is the same one from entry 002 — ten-thousand-parameter single-block causal architecture, untrained, vocab scaled to the 70 chars. Reverie runs with a 1024-cell pool (informed by entry 004’s finding that capacity matters), no slot template, default everything else.

One subtle methodology point worth naming: Reverie’s generate() is stochastic — it samples from an activation-weighted distribution. For prose prediction I ran two variants of the probe. Argmax computes the deterministic most-probable next character; sample takes the mode of 60 samples. The difference matters more than I expected.

Results

  Baseline (always predict most-frequent char): 15.1%

  Reverie (pool=1024, argmax):  17.0% ± 0.0
  Reverie (pool=1024, sample):  11.1% ± 0.6
  Transformer:                  41.5% ± 1.2

The first thing to notice is that Reverie’s sample-mode probe scores below the trivial baseline of always predicting space. The mode of 60 stochastic samples is dominated by whichever cell happens to have the highest activation-weighted preference, which on average is not the frequency-dominant character. This is interesting in its own right but it’s not really fair to the architecture — it’s mostly a measurement artifact of the probe.

The argmax probe is what tells the actual story. Reverie predicts the right character 17% of the time on the held-out tail. That’s two percentage points above always-guessing-space, which is to say barely above the most embarrassing possible baseline. The transformer, with the same observations, scores 41.5%. The gap is bigger here than on arithmetic.

I had hoped — partly out of architectural sympathy, partly out of the stated rationale of the project — that prose would shift the comparison meaningfully. It didn’t. If anything, it sharpens the original finding: the substrate’s encoding scheme doesn’t reliably extract short-range character structure either.

Why this is interesting anyway

Two observations from these numbers that are worth keeping.

The argmax-vs-sample gap is 6 points, and it’s all on Reverie’s side. This is a real property of the substrate: its prediction distribution and its modal sample are not the same thing. The deterministic “what’s most likely?” gets the most-frequent character right when the context doesn’t disambiguate; the sampled mode wanders. For a research log, this means future entries should probably benchmark argmax — it’s the architecture’s best foot forward, and the sampling probe added confusion without adding signal.

The architecture has no “default to base rate” mechanism. When no context cell’s template matches the current window (the common case on novel prose), Reverie’s activation pattern is determined by whichever non-context cells happen to have high preferences. There’s no fallback to “well, when I have no idea, predict the thing I see most often.” Transformers get this for free from their softmax — when context is uninformative, the marginal distribution dominates. Reverie’s cell-by-cell architecture doesn’t have an analogous mechanism, and that absence is plausibly a meaningful chunk of why it underperforms here.

That second observation isn’t actionable yet — fixing it would mean some kind of “marginal prediction cell” that always votes for the empirical token distribution — but it’s the kind of architectural-gap-naming that entries are useful for. The gap is in the architecture, not in the test.

What this rules out

Going into this experiment I genuinely didn’t know which way it would go. There were three reasonable outcomes:

1. Reverie wins on prose, loses on arithmetic. The substrate’s pitch was correct; the arithmetic benchmark was unfair.
2. Both perform comparably on prose; transformer dominates on arithmetic. The substrate has its niche, just not in symbolic compression.
3. Reverie underperforms on both. The substrate’s wall is general.

The result is #3. The architecture’s failure to extract structure isn’t specific to the symbolic task it was originally graded on; it generalizes to the workload it was originally pitched for. That’s the negative-result reading.

I want to caveat this honestly: it’s one corpus, one configuration, one pool size, one context-window setting. The space of “structured non-symbolic streams” is large, and there are surely configurations under which Reverie would do better. But the cheap test answered the cheap question. The substrate as it stands does not particularly favor prose over arithmetic, and on both it sits well below a ten-thousand-parameter untrained transformer.

What’s still open

The slot-cell redesign from entry 003 remains the only architectural change in the queue that could plausibly close the representation gap. Entry 004’s lesson — try the cheapest knob first — applies here too: before declaring the slot-cell architecture broken and embarking on weeks of design work, the right move is probably one more round of cheap experiments. Things I haven’t yet ruled out as the actual bottleneck on this prose task:

• Word-level rather than character-level tokens. Reverie was designed for discrete tokens, and characters may be too fine-grained. A word-token stream would have fewer distinct patterns per training byte and might let context cells claim more useful templates.
• Bigger pool. Pool=1024 was a guess. Entry 004 showed the relationship between pool size and accuracy is almost linear on arithmetic; possibly the same applies here.
• A “marginal prediction” cell type. A small architectural change — one cell whose preferences are pinned to the empirical token frequency — would lift Reverie’s floor to at least baseline without touching the structure-learning mechanisms.

Any of those could become entry 006. But the broader headline is now firmer: across two domains and four entries of careful comparison, Reverie’s architecture as it currently stands does not reach the structure-extraction performance of a ten-thousand-parameter untrained transformer. The next interesting thing this project can do is either find the configuration where it does — or accept the negative result formally and write the project’s retrospective.

— Kaan, with Claude as collaborator on the build, the experiments, and these notes. Prose benchmark code at model/scripts/prose_benchmark.py; rerun with --seeds 5 to tighten the bars further.