All News
quantizationlocal-llmapple-siliconinkling-2-minikld

Inkling 2 Mini in 5.4 GB: a real file behind an inherited 12x KL headline

A one-author quantization repo put Inkling 2 Mini in 5.4 GB and topped Hacker News. Its 12x KL claim comes from Gemma 4 E2B, and equal-size KL is disputed.

Vlad MakarovVlad Makarovreviewed and published
2 min read
Inkling 2 Mini in 5.4 GB: a real file behind an inherited 12x KL headline

A one-author repo published on GitHub on 31 October spent the weekend near the top of Hacker News on the strength of a file size. Its post, "Inkling-2-Mini in 5.4 GB Memory", points at the lookatmyquant README, where Thinking Machines' looped 26.4B MoE is compressed to 5.38 GB for his Apple Silicon runtime. Two numbers got argued about: the 5.4 GB is real, the 12x is not about Inkling.

What the build actually measures

The 5.4 GB file is 1.65 bits per text weight against 18.89 GB of BF16. On his harness it scores 174 of 200 on GSM8K and 117 of 164 on strict HumanEval, against 193 and 152 for the teacher. Its KL from that teacher is 0.0864 on English, chat and code, 0.2773 across thirteen languages. The model card puts calibration at 35 percent teacher rollouts, 10 percent GSM8K items and 55 percent TextMix, "strictly disjoint from test". The method is proximal quantization-aware training on four B200s, not a new format: the weights were taught to survive the codes.

The 12x belongs to Gemma

That ratio is measured on Gemma 4 E2B, on the project's method page: a compiled GGUF at 3.39 GB with KL 0.012 against llama.cpp's Q4_K_M at 3.41 GB and 0.160. The page credits the gap to Gemma's 2.35-billion-weight lookup table. That is a structural property of one small model. Nothing in the Inkling table claims 12x, and commenters noticed. "The headline claim of this repo (12X better) doesn't even apply to Inkling-2-Mini," wrote one, in a thread also arguing about speed and whether Inkling 2 Mini was worth running at home.

Equal size is where the argument breaks

The 5.4 GB card compares benchmarks, not KL at matched file size. It claims the build outscores 7.76 GB UD-IQ1_S and 9.14 GB UD-Q2_K_XL "at 41% smaller size". A commenter who reran it got the opposite ordering: 0.0895 against 0.0808 for UD-IQ1_S on English, chat and code, and 0.287 against 0.124 across thirteen languages. His English figure lands near the card's own 0.0864 and 0.2773, so both sides measured the same quantity and drew opposite conclusions.

What is not published

The README lists its own omissions. The drafter as trained is 1.8 GB at sixteen bits and is not released; only the 4-bit file ships. The dense weights behind the LookAtMyQuant 2 benchmarks and its per-problem results are absent too. The method page excludes Google's QAT Gemma file from its equal-size table because Google trained further after quantizing, so its KL includes that training, an asymmetry that applies equally to a distilled student next to stock GGUFs. KL is only as stable as the text it averages over, and the page gives no per-source breakdown for Inkling. No third-party replication exists yet.

Related Articles

Scroll down

to load the next article