webLTM proves the mechanism — a block that loops over its own latent state with no chain-of-thought slot — on 153K parameters that only add numbers. This demo proves the mechanism retrofits onto an actual pretrained language model: take a real 30-layer transformer, freeze 29 of those layers, replace the 10 in the middle with one shared, looped layer, and lightly fine-tune only that one layer on real English.
Prelude and coda are the base model's own weights, byte-for-byte, run once each. The only thing that changed is one layer, looped.
The core starts as a deep-copy of the original layer 15, then gets looped
r times per forward pass and re-trained with randomized depth
(r sampled 1–12 every step) — the same trick webLTM uses, so one
set of weights has to work across a range of thinking budgets rather than
memorizing a single loop count. No KV-cache: every generated token recomputes
the whole sequence through prelude + core×r + coda, which is why this demo
is slow (see §03) — that's real transformer compute, not a shortcut.
Only the core layer's 3.54M parameters have gradients; everything else stays exactly as SmolLM2 shipped it. Loss and a same-prompt sample, logged every 50 steps:
| step | train loss | val loss (r=2) | val loss (r=10) | val loss (r=12) |
|---|---|---|---|---|
| 0 | 12.85 | 6.29 | 13.31 | 14.34 |
| 100 | 5.44 | 4.82 | 5.88 | 6.30 |
| 300 | 5.21 | 4.56 | 4.98 | 5.06 |
| 499 | 4.85 | 4.42 | 4.88 | 5.05 |
Real words, real (if shaky) English syntax, by the end — capitalization, punctuation, plausible nouns for "history of a city" (planned, block, square, built, 1800). Not fluent. 500 steps on a CPU, fine-tuning 3.4% of the parameters, was never going to produce a chatbot.
❗ One honest miss, stated plainly: in webLTM's original addition task, more loop iterations clearly helped on harder problems. Here they don't — held-out loss is lowest at r=2 and gets steadily worse through r=10 and r=12, the opposite of the finding this whole project set out to demonstrate. The likely reason: 500 steps fine-tuning only one layer isn't enough to teach it to do useful new work on iterations 3–12 — it mostly learned to be a good single pass and a mediocre one when forced to loop further. A real budget (more steps, more of the stack unfrozen) would very plausibly close this gap; this run didn't have one. The architecture is real. The result training bought in 26 minutes on a CPU is a starting point, not a finished model.
The model and tokenizer load from Hugging Face and run entirely in this tab via onnxruntime-web (WASM, no GPU needed) — set how many times the core layer loops, and watch it generate.