Sixteen tokens for a whole library: what Memorilla's compression actually buys

Sixteen tokens is shorter than one sentence. Stanford's SNAP group says that is enough to carry an entire document collection into a frozen language model: on NarrativeQA, 67,385 tokens of book collapse into 16, and the model answers about the book anyway. The paper, Memorilla: Latent Semantic Memory for LLMs, went to arXiv on 15 October 2026 and reached revision three on 3 November. The r/singularity thread that circulated it on 6 November was mostly reacting to the reduction rather than the method, which is the usual order of things and exactly backwards.

The memory sits outside the model

A frozen encoder, Qwen3-Embedding-4B by default, turns each incoming session or chapter into a single embedding, and the store becomes nothing more than a pile of those vectors, the same object a dense retrieval index already holds. At query time, sixteen learned slots attend over the question and the whole store at once, and their output is written into the decoder's input embedding space as if it were tokens the user had typed. The decoder is untouched stock Qwen3-8B; only the memory module trains, in three stages that move from Wikipedia paraphrasing to mixed question answering and summarisation, then to each benchmark's training split. Earlier this autumn QwenGram took a different route by teaching a model to transfer n-gram memory instead of compressing the store.

Sixteen is a setting, not a law. The paper fixes K at 16 for the supervised runs after an ablation showed personalisation benchmarks losing ground at smaller counts, and the TextWorld agent runs on eight.

The numbers, minus the framing

Every method in the comparison answers through the same frozen Qwen3-8B, which makes the table unusually clean:

Benchmark	Memorilla	Best competitor
PersonaMem-v2 (accuracy)	44.60	30.90, xRAG
FactKG (accuracy)	84.33	80.43, full context
PersonalizationV4 (accuracy)	66.65	67.88, MemoRAG
NarrativeQA (ROUGE-L)	19.31	20.51, MemoRAG
PubMedQA (accuracy)	59.00	69.00, LLMLingua-2

The headline win is PersonaMem-v2. Memorilla scores 44.60 against 30.90 for the strongest retrieval baseline, while full context manages 27.92 after being handed 1,974 times more document tokens. That is not a marginal improvement in which text reaches the model; it is a different answer. On FactKG the gap runs the other way from intuition: 84.33 with sixteen tokens against 80.43 when the model reads every fact, so the gain is not more facts but a different arrangement of them.

Where the budget runs out

Biomedical QA is the honest failure. PubMedQA gives Memorilla 59.00 against 67.00 for plain full context and 69.00 for LLMLingua-2, an eight-point deficit that no amount of token arithmetic hides. NarrativeQA lands at 19.31 ROUGE-L behind MemoRAG's 20.51, and the project page concedes the pattern: retrieval wins on questions whose answer appears verbatim in the text, the who and which questions, while the compressed memory wins on why and how questions that need facts assembled across chapters.

The authors state the mechanism plainly in their limitations section. "Memorilla can lose fine-grained factual details, likely because the document encoder was never trained for memory compression." The site says the same in fewer words: the fixed token budget can lose details such as exact names. Sixteen tokens is a lossy channel by construction, and where the missing detail is the whole answer, it fails.

The asterisks the authors put there themselves

Three qualifications matter more than the headline. First, the 49.68 mean that tops the comparison combines accuracy, ROUGE-L and exact match on a shared 0–100 scale, so it ranks systems rather than measuring one capability. Second, PersonaMem-v2 and PersonalizationV4 receive additional benchmark-specific fine-tuning, so the two personalisation benchmarks, one of them Memorilla's largest win, are not evaluated on quite the same footing as the rest. Third, the design is not portable by default. The project page notes that because memory is written straight into the model's input embeddings, Memorilla needs open weights and a memory module trained for each LLM, and every experiment pairs a Qwen encoder with a Qwen decoder. A Llama or Mistral deployment is a retraining job, not a swap.

Latency is a different claim from speed

The 27x figure is real and narrow. On one H100-80GB at batch size 1 with 512 documents, end-to-end latency is 0.74 s against 19.9 s for full context, which needs YaRN to fit the collection at all; retrieval was not timed. The paper is careful about what this means, and its appendix is where the useful sentence lives: growing the collection by 512x decreases Memorilla's end-to-end latency by 1.9 ms while full-context decoding grows by 19,184 ms. The benefit is a serving cost that stays flat, not a faster single query, because decoding dominates in both cases. Training is the other hidden cost, run in bf16 on four H100-80GB cards with the encoder and decoder frozen throughout.

A frozen agent that stops circling

The agent experiment is the part that reads like a capability result rather than a compression result. In TextWorld, a frozen Qwen3-4B explores up to 65 rooms in 50 turns, and eight memory tokens replace a growing chat log. Average success rises from 18.6 percent with full-history in-context learning to 24.2 percent, but the largest behavioural change is termination: the in-context baseline burns through its 4,096-token budget on 71 percent of rollouts against 58 percent for Memorilla. The agent stopped re-entering rooms it had already searched; that is the failure a memory system exists to prevent, and in this game suite it did.

What ships, and what does not

There is code, and there are weights. The repository is Apache-2.0, last pushed on 6 November 2026, with 487 stars, 39 forks and no open issues. Five trained memory adapters sit on Hugging Face as memory.pt files, one each for the three training stages and the two personalisation recipes, all trained against stock Qwen3-8B. Adoption is still a rounding error: 54 downloads and 7 likes as of this writing, which is what a month-old research release looks like.

Nothing here is a product, an API or a hosted memory service, and no third party has reproduced the eight-benchmark table. The venue is a COLM workshop on context beyond the window, not a main track. Read the sixteen tokens as a claim about plumbing, and a strong one: what a frozen model can be made to see is a design decision, not a context-window purchase. Whether the plumbing knows which detail you will need at 2 a.m. is a different question, and PubMedQA already answered it once.