Prism ML put a 27B model in 5.9 GB. Stock llama.cpp returns garbage

Prism ML published Bonsai 2 27B on Hugging Face on September 17, a rebuild of Qwen3.8-27B whose language weights are restricted to the balanced ternary values -1, 0 and +1. The arithmetic works out to 27.36 billion parameters resident in 5.95 GB, the base architecture untouched, the licence Apache 2.0, and a vendor table claiming 98.2 percent of the full-precision model's benchmark average. It also will not run on the llama.cpp you have installed.

No high-precision escape hatches

Ternary g128 means each weight carries one of three values and shares a single FP16 scale factor with 127 neighbours, for a true 1.71 bits per weight and 1.72 across the model. Embeddings, attention projections, MLP projections and the LM head are all ternary. Only 26.2M parameters, 0.0976 percent of the language model, stay above it: the recurrent state path of the linear-attention layers and the normalization weights. The weights are stored in a blockwise Hadamard-rotated basis, block 1024 with fixed signs, and the matching transform is applied to activations at runtime. The rotation is folded into the stored weights offline, so it costs neither bits nor weight traffic, and the pack declares its rotation as metadata.

The rest of the model is inherited whole: 24.35B parameters across 64 language blocks, 2.54B of embedding and output head, 0.46B of vision tower in 27 blocks, with hybrid attention at roughly 75 percent linear and 25 percent full, plus SwiGLU, RoPE and RMSNorm. Context is 262K tokens. Two GGUF packings carry it, both needing custom kernels: PTQ1_0 packs trits densely at 1.75 bits per weight and 5.95 GB, PQ2_0 gives each trit a 2-bit slot at 2.13 bits and 7.21 GB. The vision tower ships separately as a 0.63 GB Q8_0 mmproj pack, and an MLX 2-bit companion exists for Apple Silicon.

Every number in this table was measured by the seller

Prism ML evaluated on H100 hardware with EvalScope and vLLM, in thinking mode, across 14 benchmarks in six skill categories. No third party has re-run any of it, and no inference provider serves it.

Variant	True bits/weight	Footprint	Thinking avg	vs FP16
Qwen3.8-27B FP16	16.0	54 GB	86.32	100%
UD-Q4_K_XL ("4-bit")	5.2	17.6 GB	85.18	98.7%
IQ2_XXS ("2-bit")	2.8	9.4 GB	72.59	84.1%
Bonsai 2 27B	1.72	5.9 GB	84.78	98.2%

At 1.72 bits per weight the model averages 84.78, which the card puts within 0.4 points of the "4-bit" UD-Q4_K_XL build at a third of that build's footprint. The more useful comparison is IQ2_XXS, a "2-bit" build that is really 2.8 bits per weight at 9.4 GB. It averages 72.59, and the loss is not spread evenly: 57.5 on AIME26 and 56.4 on LiveCodeBench while still hitting 88.93 on MMLU-Redux, a pattern the card describes as "why casual testing misses the collapse". Bonsai 2 holds 95.83 and 90.07 on those two rows. Category by category, math slips from 97.06 to 96.57 and coding edges up from 89.07 to 89.42, while knowledge and reasoning falls 5.69 points to 79.86 and vision falls 5.17 to 66.19. The damage concentrates where the questions are hardest.

Two packings, no strict ordering

Throughput is a trade rather than a ranking. PTQ1_0 moves less weight per step but pays arithmetic to unpack dense trits, so it wins decode on the Ada-class cards and the 72-watt L4: 32.1 tokens a second against 29.8 there, and 91.1 against 81.2 on an RTX 4090. PQ2_0 wins batch-one decode on H100, A100 and the Blackwell parts, and prompt processing everywhere: 129.9 tokens a second on an RTX 5090 with a 3893-token-a-second prefill at 1.95 joules per token, and 113.9 on H100 SXM. On an Apple M5 Pro the model decodes at 28.1 tokens a second, streaming roughly 204 GB/s of weights at 27.5 W on the GPU rail, 34.1 W across CPU and GPU, against 300 to 455 W of board power on the NVIDIA cards. The card's own Limitations section concedes that returning the footprint advantage as latency on every target remains engineering work.

The loader is the real barrier

The plainest warning is a tooling one: "These files need our llama.cpp build." Stock llama.cpp rejects PQ2_0 and PTQ1_0 as unknown types, and it loads Q2_0 without any warning and produces garbage, because it has no Hadamard activation runtime. The kernels live in the PrismML-Eng llama.cpp fork for CUDA and Metal, with a separate MLX fork for Apple Silicon, and the Bonsai-demo repository is named the source of truth for running the models, with pinned binaries. Generation defaults follow the base model: temperature 1.0, top_p 0.95 and top_k 20 for thinking, 0.7 and 0.80 for instruct. Default reasoning effort is xhigh, medium is offered as the shorter option, and low is not supported, behaving close to xhigh.

What the released material does not claim

Three claims travelling with the launch are not in it. "98 percent of GPT-4 Turbo's reasoning at 40 percent lower VRAM", "12 percent faster inference than Qwen 3.8", and a "dynamic tensor slicing" or "runtime-adaptive tensor partitioning" mechanism appear in neither the model card, the linked whitepaper repository, nor the launch thread. The card benchmarks Bonsai 2 only against builds of its own base model at matched settings, and no slicing machinery exists in the release, because the representation is fixed ternary g128 with an offline-folded Hadamard rotation. Treat all three as unverified.

Adoption, and what the download counter means

The GGUF repository shows a trailing-month download counter just over 406,000, a figure describing one month of traffic rather than cumulative installs, and the collection carrying it lists 93 upvotes. The r/LocalLLaMA launch thread by u/xenovatech, posted September 17 at 21:05 UTC, sat at roughly 1,280 points and more than 250 comments, with a critical thread titled "Ternary Bonsai is a headless chicken" alongside it the next day. This lands in a busy stretch for 27B derivatives: Swift-Qwen3.8-27B changed how long the model thinks, K2 Horizon 7B traded size for a 27B-class row, and Tencent's Hy4 made a similar argument about patched forks and tooling gaps. Humanlike-chat variants of the same base model change what it says; this one changes how little it costs to hold.

What would settle it

Three things would help. A third-party re-run of the 14 benchmarks on a neutral harness, an independent check of the AIME26 and LiveCodeBench rows where conventional 2-bit builds collapse, and a stock llama.cpp that loads the file without producing garbage. The weights are downloadable today; the demonstration is a fork that most users do not yet have.