Decision 0004: Replay during internal excitation

Date: 2026-05-23 Status: Accepted Supersedes: none Builds on: ADRs 0001, 0002, 0003

Context

After ADR 0003, internal excitation (excite()) does activation decay + connection-mediated propagation + noise. That’s a useful mechanism — it lets activation patterns persist briefly across ticks — but it isn’t what the architecture’s original sketch called for. The sketch says the system, between observations, replays its own past inputs to itself (recent ones weighted more) and the resulting internal cycles drive the system toward stagnation.

This ADR adds that replay mechanism. After it lands, excite() no longer just lets activations decay; it actively re-injects a past observation into the system on each tick — running the same activation, inhibition, and connection-update machinery as a real observation, but at reduced learning rates (“consolidation, not first-time learning”).

This is what makes the project’s name Reverie literal. Between observations, the system is dreaming — playing back what it has seen, in recency-weighted random order.

Decision

A buffer of past observations

Every call to observe(token) appends the token’s index to a bounded buffer (default size replay_buffer_size = 256). Once the buffer is full, the oldest entry rolls off.

Recency-biased sampling

When excite() runs, it picks one past observation from the buffer with geometric-decay weights:

weight[i] = replay_recency_bias ^ (n - 1 - i)

where i indexes positions in the buffer (most recent has the highest weight). With replay_recency_bias = 0 only the most recent observation is ever replayed; with replay_recency_bias = 1 sampling is uniform over the buffer. Default 0.95 gives meaningful recency without ignoring older context.

Replay = soft observation

A replay tick runs the same machinery as observe():

1. Snapshot mimicking-cell bets.
2. Compute preference-driven activation for the replayed token.
3. Apply lateral inhibition.
4. Apply the learning rules — Hebbian reinforcement for Hebbian cells, predict-then-correct for mimicking cells.
5. Update connections via centered Hebbian co-activation.

The only differences:

• Learning rates are scaled by replay_learning_factor (default 0.1). Replay reinforces existing patterns at one-tenth the strength of a real observation. This is the “consolidation, not first-time learning” property.
• No token registration. Replay only ever picks from tokens already in the vocabulary; nothing new gets added.
• No buffer append. Replay doesn’t feed itself.
• No growth check. Capacity-driven growth is a response to new input, not to rehearsal.

Connection-mediated propagation is folded into the activation formula on replay ticks, so connected cells inherit some activation from peers even before the inhibition stage:

propagated = C @ activation_prev
raw_activation = baseline + sat·tanh(pref/sat) + propagation_scale · propagated + noise
activation = max(0, raw_activation - inhibition_scale · mean(raw_activation))

Empty-buffer fallback

If excite() is called before any observation, or if replay_buffer_size = 0, the system falls back to the v0.2 excitation dynamics — pure decay + propagation + noise. This preserves backward compatibility and supports disabling replay via the parameter.

Disabling replay

Setting replay_learning_factor = 0 makes replay activation-only (refreshes the activation field, updates connections through co-activation if connection_learning_rate > 0, but doesn’t move preferences). Setting replay_buffer_size = 0 disables replay entirely, returning to v0.2 excitation.

Why fold replay into excite() rather than make it a separate method

A separate replay() method would be cleaner in some sense, but it would shift coordination cost onto callers. The original architecture sketch says internal excitation is replay — they’re the same thing, not parallel processes. Code structure should reflect that. Calling excite() after a sequence of observe() calls should produce the dreaming dynamics; callers shouldn’t have to know about a second method.

Anticipated consequences

This is the change that actually makes the project alive between observations. Three predictions worth recording:

1. The frequency skew during excitation will sharpen further. Under ADR 0003 we saw skew amplify from a 6:1 input ratio to a ~40:1 output ratio after long excitation, driven by connection reinforcement. With replay running and reinforcing the dominant token’s assembly, the effect compounds.

2. The system will exhibit content-dependent dynamics during quiescence. What the system “thinks about” in excitation depends on what it has recently seen. Two systems with different observation histories should produce different excitation trajectories even given identical initial conditions.

3. Stagnation becomes a meaningful concept. Currently excitation just decays toward zero. With replay, the system continually re-energizes itself, but the content of that energy stabilizes over time as it cycles through its recent buffer. Detecting that stabilization is the next architectural problem (deferred to ADR 0005).

Stability constraints

Three failure modes discovered during implementation, each worth recording.

Replay can compound preference drift

Because replay does the same learning that observation does (at reduced rate), the content of recent observations gets reinforced beyond their frequency-based weight. If recent observations are unrepresentative (e.g., a short imbalanced burst), replay will overemphasize them. With replay_learning_factor = 0.1 and a buffer of 256, a burst of 50 observations of token X gets ~50 extra reinforcement events from replay before it rolls off — meaningful but not catastrophic.

If this turns out to bias the system too strongly in practice, options:

• Lower replay_learning_factor.
• Increase replay_buffer_size.
• Sample with anti-recency bias instead (replay older observations more, like rehearsal of long-term memory). A future ADR could make this a configurable strategy.

Connection growth accelerates under replay

Replay produces additional connection updates per real observation (one per excite() call between observations). If a system observes 100 tokens and runs 1000 excitations between them, connections accumulate 10× faster than under v0.2. The connection_decay and connection_max parameters keep this bounded but the effective time constant of connection learning shortens.

Activation runaway via propagation + connection accumulation (discovered)

ADR 0003 claimed that propagation_scale · connection_max < 1 keeps activations bounded. That bound is correct for the contribution from a single connection edge, but it ignores the per-row sum of the connection matrix. As connections accumulate — especially under ADR 0004’s replay-driven extra updates — the matrix-vector product (C @ activations) can have entries on the order of n_cells · connection_max · max_activation. The natural-looking constraint isn’t actually sufficient.

When activations include the propagation term and the propagation term grows linearly with the population, you get a positive-feedback loop: high activations → large propagation → larger activations next tick → … → NaN within a few hundred ticks.

The fix is a hard cap on raw_activation after the propagation add, set at 2 · (baseline + activation_saturation) — twice the natural ceiling from the preference saturation. Propagation can amplify activations into a meaningful “above natural” regime but cannot run away. This was discovered when the v0.3 test suite hit Probabilities contain NaN on long excitation runs and is now permanently encoded in _compute_activations.

Recency bias dominates the output distribution (architectural finding)

This isn’t a failure mode — it’s the intended behavior, just worth flagging because it surprised the test suite. With replay_recency_bias = 0.95 and a sequential observation burst (“30 frequent, then 5 rare”), the most recent 5 observations (all “rare”) get sampled disproportionately during replay. The output distribution then reflects that recency bias, not the underlying observation frequency.

This is the working-memory effect: between observations, the system thinks about what it has recently seen, not what it has seen most often. Both are aspects of memory; recency dominates in this regime. To test frequency-preservation properties cleanly, observations need to be interleaved so the recency tail doesn’t bias the buffer.

The architectural takeaway: ADR 0004 changes the meaning of “output during excitation.” Before this ADR, output during excitation reflected accumulated preference. After this ADR, output during excitation reflects replay sampling, which is jointly determined by preference, recency, and assembly dynamics. The system’s “voice” during quiescence is now content-dependent in a way it wasn’t before.

Behavioral predictions to be verified

These claims have corresponding tests in tests/test_replay.py:

1. Replay reinforces recently observed tokens. After many excitations following a burst of one token, that token’s cell preferences should grow further (proportional to replay_learning_factor).
2. Recency bias affects which token gets reinforced. With low replay_recency_bias, the most recent token dominates replay. With high bias, older tokens get sampled more.
3. Disabling replay restores v0.2 behavior. With replay_buffer_size = 0 or replay_learning_factor = 0 (and connection_learning_rate = 0), excitation should not change preferences.
4. The frequency-skew preservation from ADR 0003 is unchanged or improved. Replay should not break the property that observed-frequency dictates output distribution.
5. All earlier tests still pass. Vocabulary tracking, cell-type behavior, connection mechanics — none affected by the replay layer.

Open questions deferred

• Stagnation detection. When has the system “thought enough”? Detecting that internal cycles have stabilized is the natural next step after replay. (ADR 0005.)
• Replay sampling strategies beyond geometric-recency. Anti-recency (rehearsal of older memory), assembly-driven (replay the token whose assembly is currently most active), or thematic clustering. To be revisited if simple geometric recency is insufficient.
• Output generation during excitation. Currently generate() and excite() are independent. A future ADR may auto-generate output per tick, or define when the system “speaks” during dreaming.
• The interaction between replay-driven connection growth and the architectural intent of connection_decay. Replay accelerates accumulation; decay slows it. The balance is currently default values, not derived; a future ADR may make decay rate adaptive.