Forty mini PCs, USB cables and 14 tokens per second: a 320B model on a $5,000 rack

A hobbyist posting as u/Environmental_Hour66 has put a rack of 40 Intel N150 mini PCs behind GLM-5.3-Flash and reports 14 tokens per second at 1k context. The nodes talk over USB 3.2, not Ethernet. His figures: about $5,000 to build, roughly 740 W at the wall under peak load, and a 320B model resident entirely in system RAM. It is one person's hardware, and 14 tokens per second is not a fast number.

The interconnect is the experiment

Each node carries 12 GB of DDR5-4800, which the post puts at a physical ceiling of about 33 GB/s. The wiring is the odd part: 10 Gbps USB 3.2 through the debug-bridge path, in a tree of 1, 3, 9 and 27. A 1 Gbps Ethernet attempt topped out near 7 tokens per second. Nothing off the shelf splits a model this finely over USB, u/discosoc noted, and the builder confirmed a model wrote him a custom runtime from scratch.

The card says 320B total parameters, 18B active, so the memory math matters. Forty boxes at 12 GB is 480 GB pooled before overhead. The post says the model runs "at native precision with room left for 8x1M context windows" but never names that precision. Its safetensors total 321.3B across BF16, FP8 and F32 (the card rounds), putting the weight file between roughly 320 GB and 640 GB. Fitting it with headroom means most of the file is already 8-bit, and the post does not say so.

Three throughputs, one of them awkward

The post reports three throughputs:

• 14 tokens/s single stream at 1k context, 12 tokens/s at 1M
• about 40 tokens/s with decode batched
• about 85 tokens/s prefill

That last line is the problem. Filling a million-token prompt at 85 tokens per second takes over three hours before the first answer token exists, as u/discosoc noted. Batching turns 14 into 40, which suits queue work, not chat.

Cheap memory, not cheap speed

The $5,000 includes the 40 PCs, which he says cost about $85 each last December. That money also buys a decent used GPU workstation, so the rack wins on memory capacity and loses on latency. For scale, Revolut's self-hosted GLM deployment holds a 754B model on 56 B200 GPUs and treats 200 tokens per second as ordinary.

Power decides whether this is thrift. At 740 W peak, the rack draws about 18.5 W a node, frugal for 480 GB of addressable memory, though 17.8 kWh a day if it ran flat out.

The builder says he is compute-limited, not bandwidth-bound: 350 GB/s of a practical 650 GB/s, his estimate. That reframes the open question: not whether speculative decoding helps, since he is already adding it, but whether the N150's iGPUs or its DRAM bus gives out first.