Update, October 16, 7:55 PM: the daily lane is NVFP4, and it is now the faster one. RadixArk's NVFP4 build of the same model runs on the same eight Sparks with 4× the context (512K; a 514K-token needle passes). With MTP guessing three tokens ahead instead of one it does 35.7 / 29.9 / 25.8 tok/s single stream (code / prose / explanation) against FP8's 29.9 / 26.9 / 23.1, and 61.7 tok/s at four users against 40.1. The cost is about 2.3× the divergence from BF16. Side by side below; recipe dgx-spark-8x/glm-5.3-nvfp4-tp8. The FP8 lane stays on disk as the fidelity reference.
October 17, 4:11 PM: the daily lane is now SGLang. Same checkpoint, stock SGLang v0.5.22, after an eleven-arm campaign: 36.2 / 28.7 / 23.7 tok/s, 59.5 at four users, needles to 514K, and the Hermes tool chain 30% faster than this lane. This vLLM recipe stays the 1M-context and fastest-prefill lane. SGLang vs vLLM, the memory bug, and the campaign; recipe dgx-spark-8x/glm-5.3-nvfp4-sglang-tp8.
GLM-5.3 744B runs on all eight Sparks of GhettoRak; first with Z.ai's own FP8 weights. Nothing is re-quantized. It is the most faithful copy of the model that exists below BF16 (third-party KLD 0.0223), and it only fits because eight GB10s give it 88 GiB per node. One request at a time it decodes at 23–30 tok/s. Recipe: J-M-Recipes · dgx-spark-8x/glm-5.3-fp8-tp8.
The recipe card, generated from the recipe's measured numbers. Amber = not measured yet.
Why eight nodes
GLM-5.3's official FP8 checkpoint is 703.7 GiB. Four Sparks have about 486 GiB between them, so a 4-node build has to use a 3–4-bit quant. Eight Sparks have about 973 GiB, which leaves room for the real thing: 90.96 GiB per node for the weights and the speculative-decoding (MTP) head, and 9–11 GiB per node for the KV cache.
The trade is context. The 4-bit builds leave ~36 GiB of KV per node and reach 512K; FP8 leaves one ~131K request.
Quality is the other side: on the same comparator (malaiwah/quant-fidelity-suite), FP8 measures KLD 0.0223
against BF16, while the EXL3 and NVFP4 builds that fit a vLLM lane measure 0.045–0.075 (unsloth's Q4 GGUF gets 0.0275, but that is a llama.cpp lane).
The setup
| Weights | zai-org/GLM-5.3 @ ba32f8c1, as shipped. Pulled once, copied to the other seven over the fabric (~13 min), 143/143 files SHA-256-matched to the Hub on every node. |
|---|---|
| Runtime | ciprianveg's gb10-glm-5.2:v19-vision image (a vLLM 0.11.2 fork with B12X sparse-attention kernels for GB10), pinned by digest. TP8, one rank per Spark, no Ray. |
| Launcher | 0xMagnus's FP8 TP8 launcher, pointed at the official weights, with MTP turned back on (adaptive depth 2/4/5) and an 8-thread weight loader. Every change is marked in the script. |
| Fabric | Each Spark's two ~100G PCIe functions on one 200G cable, both used by NCCL. 8-node all-reduce: 20.1 GB/s at 1 GiB, zero switch errors. |
| Defaults | reasoning_effort: low, 131,072 max context, 4 requests at once, bearer key. |
Results
| Test | Result |
|---|---|
| One request, code prompt | 29.9 tok/s (29.4–30.1) |
| One request, prose email | 26.9 tok/s (25.7–26.9) |
| One request, technical explanation | 23.1 tok/s (22.9–23.6) |
| Four requests at once | 40.1 tok/s total, 11.5–14.2 each |
| Find a code hidden in 35K / 70K tokens | pass / pass, in 40 s / 58 s (~1.2K tok/s prefill at 70K) |
| Tool call and tool-result round | pass on both boots |
| Wall power, decode + 70K prefill | 169–189 W per Spark, ~1.5 kW for the rack |
Method: temperature 0, reasoning low, 800 new tokens, streamed; one warm-up request, then the median of three. Decode speed is measured from first to last token. For comparison, 0xMagnus measured 22.8 tok/s with MTP on the uncensored FP8 build on the same image class.
What the optimization hour found
- Boot was the only easy win. Every node reads all 704 GiB to keep its 88, on a single CPU thread. The head took 957 s just to load weights. Eight loader threads cut that to 567 s, and the whole cold boot went from 27.5 to 13.2 minutes (the compile cache from the first boot helped too). Decode did not move.
- The custom NCCL build was already on. The image preloads NCCL 2.30.4 itself, so there was nothing to A/B.
- Light Foundry's biggest lever is not in this image. Their RoCE all-reduce setting took Int4-Int8Mix prose from 28.5 to 38 tok/s; v19 has no such switch. On FP8 it would help less anyway: decode here is mostly reading weights from each node's 273 GB/s memory. A small-message all-reduce probe read 81 µs at 12 KB, which I estimate at ~13–14 ms of an 80–150 ms decode step.
- MTP is already tuned by itself. It guesses 2 tokens ahead on prose (about 2.2 accepted per step) and 4–5 on code (4–5 accepted).
NVFP4: the daily lane
RadixArk/GLM-5.3-NVFP4 @ 6e389189 quantizes the routed experts to 4 bits (ModelOpt NVFP4) and leaves attention,
shared experts and the MTP layer in bf16, so it is 433 GiB rather than 704. That frees ~34 GiB per node, and nearly all of it goes to KV.
Same image, same cluster, same tests; 0xMagnus's NVFP4 settings (memory 0.80, 512K), except MTP guesses three tokens ahead instead of one (why).
| NVFP4 (daily) | FP8 (reference) | |
|---|---|---|
| Weights + MTP per node | 57.1 GiB | 91.0 GiB |
| Context / KV pool | 512K / 720,384 tokens | 131K / 150–182K |
| One request: code / prose / explanation | 35.7 / 29.9 / 25.8 tok/s | 29.9 / 26.9 / 23.1 tok/s |
| Four requests at once | 61.7 total (17.1–22.7 each) | 40.1 total (11.5–14.2 each) |
| Deepest needle passed | 513,814 tokens in 414 s (~1.2K tok/s) | 69,979 tokens |
| Needles 35K / 70K / 140K / 280K | pass in 38 / 50 / 102 / 215 s | pass / pass / — / — |
Reasoning high, 1,500-token budget | partial answer | empty answer |
| Divergence from BF16 (third-party KLD) | 0.0511 | 0.0223 |
| Cold boot | 17.5 min | 13.2 min |
Same scripts as the FP8 runs: temperature 0, reasoning low, 800 new tokens, median of three after a warm-up. Needles are one run per length at depth 0.5 — retrieval, not long-context reasoning. KLD figures are malaiwah/quant-fidelity-suite stored-weight measurements, not ours.
The NVFP4 recipe card. Amber = not measured yet.
Why NVFP4 for daily use: it now wins on speed, context and concurrency. The cost is fidelity, and it is real (2.3× the divergence). When the exact model matters, the FP8 lane is one relaunch away; both checkpoints stay on every node.
Why the 4-bit build was slower at first
On its first boot, with 0xMagnus's settings, NVFP4 was 7–12% slower than FP8 on one request. That looked backwards for a checkpoint that is about 40% smaller. Reading both checkpoints' tensor headers explains it. RadixArk only shrinks the routed experts. Attention and the shared experts stay in bf16, twice the bytes of FP8, and every token reads them. So for one token, each node reads 8.2 GiB with NVFP4 and 6.2 GiB with FP8.
The experts are where NVFP4 is cheap, and a decode step reads more experts when it checks more tokens at once. MTP (the model's built-in guesser) was set to guess one token ahead, so each step checked at most two. Around three tokens per step, NVFP4 starts reading less than FP8. The guesses were also good: the single guess was kept most of the time. So I swept the depth, one boot per setting, everything else frozen:
| MTP guesses ahead | code | prose | explanation | 4 users | tokens kept per step |
|---|---|---|---|---|---|
| 1 (as shipped) | 26.3 | 24.9 | 23.8 | 59.1 | 1.7–1.9 |
| 3 | 35.0 | 28.9 | 25.3 | 61.6 | 2.9 |
| 3, second boot | 35.7 | 29.9 | 25.8 | 61.7 | — |
| 4 | 36.2 | 27.9 | 25.4 | 58.9 | 2.6–3.0 |
| up to 5, adaptive (FP8's setting) | 35.0 | 28.9 | 25.9 | 59.1 | 2.2–2.4 |
tok/s, same scripts as above. KV pool (721–723K tokens) and free memory on the head (9.7–10.9 GB) did not change between settings.
Three wins. The first guess is kept 89–97% of the time, the second 72–89%, the third 58–74%. Past that, guesses 4 and 5 were kept too rarely to pay for the extra work, and the adaptive setting kept backing itself down. One number in the launcher was worth more than the whole first hour of tuning on the FP8 lane.
Where the memory goes
Every Spark has 121.7 GiB of unified memory that the CPU and GPU share. vLLM does not size the KV cache from the context length you ask for.
It claims a fixed share of memory (--gpu-memory-utilization × 121.7 GiB), loads its slice of the weights, runs one test forward pass
to see what activations, CUDA graphs and communication buffers need, and turns everything left inside the claim into KV cache. The
context length is then just a cap that has to fit inside that pool.
Per node, from each lane's boot log (FP8: first boot). The orange slice and the grey remainder are derived by subtraction.
| Per node | Official FP8 | RadixArk NVFP4 |
|---|---|---|
| Memory claimed by vLLM | 0.86 × 121.7 = 104.7 GiB | 0.80 × 121.7 = 97.4 GiB |
| Weights + MTP head (measured) | 90.96 GiB | 57.13 GiB |
| KV cache (measured) | 8.82 GiB (10.85 on the second boot) | 37.82 GiB |
| Activations, CUDA graphs, buffers (derived) | ≈ 4.9 GiB | ≈ 2.4 GiB |
| Left outside the claim (derived) | 17.0 GiB | 24.3 GiB |
| Actually free on the head while serving | ~6.8–8 GB | ~8.5–8.9 GB |
| KV pool | 149,824 tokens (181,632 second boot) | 720,384 tokens |
| Cost per token (pool ÷ tokens) | ~63 KB | ~56 KB |
- Every node holds the whole cache. With decode-context parallelism off (it fails on GLM-5.3 on this stack), the compressed MLA cache is copied onto all eight ranks, not split between them. A 720K-token pool means 720K tokens on each node. More nodes only help context by shrinking each node's share of the weights.
- That is why the weight format decides the context. NVFP4 frees 33.8 GiB of weights per node, and almost all of it lands in the KV cache: 4.3× the space and 4.8× the tokens. The claim is even smaller (0.80 vs 0.86) because 0xMagnus saw NVFP4's kernel autotune run a node out of memory at 0.86.
- FP8 cannot simply claim more. At 0.80 FP8 leaves no room for KV at all (0xMagnus measured −0.42 GiB), and above ~0.87 vLLM refuses to start because the memory is not free at launch. 0.86 is about the only setting that works.
- The memory outside the claim is not spare. Of the 17–24 GiB, only ~7–9 GB is actually free while serving; the OS, Docker, the vLLM processes' CPU-side memory and network buffers use the rest. On GB10, file cache also eats the same pool, which is why the launcher drops caches every minute during boot.
- Per token: the compressed MLA cache alone is 656 bytes per token per layer, or 51 KB across 78 layers. The measured cost is 56–63 KB, most likely because the sparse-attention indexer stores its own per-token keys, plus allocator overhead. I have not worked out why the two lanes differ by ~7 KB per token. The pool also moves 10–20% from boot to boot at the same settings (FP8: 8.82 vs 10.85 GiB).
Watch out for
- Reasoning
highwith a small budget returns nothing. With 1,500 max tokens the model spent all of them thinking and returned an empty answer. That happened on the official weights, so it is not an artifact of the uncensored build where 0xMagnus first saw it. Uselow, or givehigh16K+ tokens. Onlylowandhighare honoured; anything else meansmax. - Memory is tight. The head node has about 7 GB of host memory free while serving, so 128K retrieval is untested and the memory setting should not go up.
- Not yet measured: a teacher-forced check that MTP on and off give the same model output on this stack. The weights are byte-identical to the release and MTP verifies every token against the full model, so I expect no change, but the recipe stays experimental until it is measured.
Next
- A served teacher-forced comparison of NVFP4 against the FP8 lane on this cluster, so the fidelity gap is ours, not just a third-party stored-weight number.
- Pre-sharded weights (
sharded_state) to get the boot under 5 minutes. - Route the lab gateway and Hermes to the NVFP4 lane (needs James).
Credits
RadixArk (the NVFP4 build), 0xMagnus (FP8 and NVFP4 TP8 launchers, memory limits, the reasoning traps), ciprianveg (gb10-vllm and the v19 image), wrldsuksgo2mars and the b12x stack (SM121 sparse-attention kernels), local-inference-lab (the vLLM fork), CosmicRaisins (adaptive MTP), Light Foundry (public 8× GB10 GLM-5.3 numbers), eugr (spark-vllm-docker), malaiwah (quant-fidelity-suite), and Z.ai for GLM-5.3 and its FP8 release.