J&M Labs Blog by Milo

Building the future, locally

← Back to Home

Full GLM-5.3 on eight DGX Sparks: official FP8 and NVFP4

Update, October 16, 7:55 PM: the daily lane is NVFP4, and it is now the faster one. RadixArk's NVFP4 build of the same model runs on the same eight Sparks with 4× the context (512K; a 514K-token needle passes). With MTP guessing three tokens ahead instead of one it does 35.7 / 29.9 / 25.8 tok/s single stream (code / prose / explanation) against FP8's 29.9 / 26.9 / 23.1, and 61.7 tok/s at four users against 40.1. The cost is about 2.3× the divergence from BF16. Side by side below; recipe dgx-spark-8x/glm-5.3-nvfp4-tp8. The FP8 lane stays on disk as the fidelity reference.

October 17, 4:11 PM: the daily lane is now SGLang. Same checkpoint, stock SGLang v0.5.22, after an eleven-arm campaign: 36.2 / 28.7 / 23.7 tok/s, 59.5 at four users, needles to 514K, and the Hermes tool chain 30% faster than this lane. This vLLM recipe stays the 1M-context and fastest-prefill lane. SGLang vs vLLM, the memory bug, and the campaign; recipe dgx-spark-8x/glm-5.3-nvfp4-sglang-tp8.

GLM-5.3 744B runs on all eight Sparks of GhettoRak; first with Z.ai's own FP8 weights. Nothing is re-quantized. It is the most faithful copy of the model that exists below BF16 (third-party KLD 0.0223), and it only fits because eight GB10s give it 88 GiB per node. One request at a time it decodes at 23–30 tok/s. Recipe: J-M-Recipes · dgx-spark-8x/glm-5.3-fp8-tp8.

8 × GB10
tensor-parallel 8, one CRS804, 200G RoCE, stock clocks
704 GiB
official FP8, every file SHA-256 checked on every node
30 / 27 / 23
tok/s code / prose / explanation, one request
131K
context, 150–182K-token KV pool
13 min
cold boot (27.5 min stock)
Recipe card: GLM-5.3 744B FP8 on 8x DGX Spark, C1 26.9 tok/s prose, C4 40.1 tok/s aggregate, prefill 1.2K tok/s

The recipe card, generated from the recipe's measured numbers. Amber = not measured yet.

Why eight nodes

GLM-5.3's official FP8 checkpoint is 703.7 GiB. Four Sparks have about 486 GiB between them, so a 4-node build has to use a 3–4-bit quant. Eight Sparks have about 973 GiB, which leaves room for the real thing: 90.96 GiB per node for the weights and the speculative-decoding (MTP) head, and 9–11 GiB per node for the KV cache.

The trade is context. The 4-bit builds leave ~36 GiB of KV per node and reach 512K; FP8 leaves one ~131K request. Quality is the other side: on the same comparator (malaiwah/quant-fidelity-suite), FP8 measures KLD 0.0223 against BF16, while the EXL3 and NVFP4 builds that fit a vLLM lane measure 0.045–0.075 (unsloth's Q4 GGUF gets 0.0275, but that is a llama.cpp lane).

The setup

Weightszai-org/GLM-5.3 @ ba32f8c1, as shipped. Pulled once, copied to the other seven over the fabric (~13 min), 143/143 files SHA-256-matched to the Hub on every node.
Runtimeciprianveg's gb10-glm-5.2:v19-vision image (a vLLM 0.11.2 fork with B12X sparse-attention kernels for GB10), pinned by digest. TP8, one rank per Spark, no Ray.
Launcher0xMagnus's FP8 TP8 launcher, pointed at the official weights, with MTP turned back on (adaptive depth 2/4/5) and an 8-thread weight loader. Every change is marked in the script.
FabricEach Spark's two ~100G PCIe functions on one 200G cable, both used by NCCL. 8-node all-reduce: 20.1 GB/s at 1 GiB, zero switch errors.
Defaultsreasoning_effort: low, 131,072 max context, 4 requests at once, bearer key.

Results

TestResult
One request, code prompt29.9 tok/s (29.4–30.1)
One request, prose email26.9 tok/s (25.7–26.9)
One request, technical explanation23.1 tok/s (22.9–23.6)
Four requests at once40.1 tok/s total, 11.5–14.2 each
Find a code hidden in 35K / 70K tokenspass / pass, in 40 s / 58 s (~1.2K tok/s prefill at 70K)
Tool call and tool-result roundpass on both boots
Wall power, decode + 70K prefill169–189 W per Spark, ~1.5 kW for the rack

Method: temperature 0, reasoning low, 800 new tokens, streamed; one warm-up request, then the median of three. Decode speed is measured from first to last token. For comparison, 0xMagnus measured 22.8 tok/s with MTP on the uncensored FP8 build on the same image class.

What the optimization hour found

NVFP4: the daily lane

RadixArk/GLM-5.3-NVFP4 @ 6e389189 quantizes the routed experts to 4 bits (ModelOpt NVFP4) and leaves attention, shared experts and the MTP layer in bf16, so it is 433 GiB rather than 704. That frees ~34 GiB per node, and nearly all of it goes to KV. Same image, same cluster, same tests; 0xMagnus's NVFP4 settings (memory 0.80, 512K), except MTP guesses three tokens ahead instead of one (why).

NVFP4 (daily)FP8 (reference)
Weights + MTP per node57.1 GiB91.0 GiB
Context / KV pool512K / 720,384 tokens131K / 150–182K
One request: code / prose / explanation35.7 / 29.9 / 25.8 tok/s29.9 / 26.9 / 23.1 tok/s
Four requests at once61.7 total (17.1–22.7 each)40.1 total (11.5–14.2 each)
Deepest needle passed513,814 tokens in 414 s (~1.2K tok/s)69,979 tokens
Needles 35K / 70K / 140K / 280Kpass in 38 / 50 / 102 / 215 spass / pass / — / —
Reasoning high, 1,500-token budgetpartial answerempty answer
Divergence from BF16 (third-party KLD)0.05110.0223
Cold boot17.5 min13.2 min

Same scripts as the FP8 runs: temperature 0, reasoning low, 800 new tokens, median of three after a warm-up. Needles are one run per length at depth 0.5 — retrieval, not long-context reasoning. KLD figures are malaiwah/quant-fidelity-suite stored-weight measurements, not ours.

Recipe card: GLM-5.3 744B NVFP4 on 8x DGX Spark, C1 29.9 tok/s prose, C4 61.7 tok/s aggregate, prefill 1.2K tok/s, MTP k=3

The NVFP4 recipe card. Amber = not measured yet.

Why NVFP4 for daily use: it now wins on speed, context and concurrency. The cost is fidelity, and it is real (2.3× the divergence). When the exact model matters, the FP8 lane is one relaunch away; both checkpoints stay on every node.

Why the 4-bit build was slower at first

On its first boot, with 0xMagnus's settings, NVFP4 was 7–12% slower than FP8 on one request. That looked backwards for a checkpoint that is about 40% smaller. Reading both checkpoints' tensor headers explains it. RadixArk only shrinks the routed experts. Attention and the shared experts stay in bf16, twice the bytes of FP8, and every token reads them. So for one token, each node reads 8.2 GiB with NVFP4 and 6.2 GiB with FP8.

The experts are where NVFP4 is cheap, and a decode step reads more experts when it checks more tokens at once. MTP (the model's built-in guesser) was set to guess one token ahead, so each step checked at most two. Around three tokens per step, NVFP4 starts reading less than FP8. The guesses were also good: the single guess was kept most of the time. So I swept the depth, one boot per setting, everything else frozen:

MTP guesses aheadcodeproseexplanation4 userstokens kept per step
1 (as shipped)26.324.923.859.11.7–1.9
335.028.925.361.62.9
3, second boot35.729.925.861.7—
436.227.925.458.92.6–3.0
up to 5, adaptive (FP8's setting)35.028.925.959.12.2–2.4

tok/s, same scripts as above. KV pool (721–723K tokens) and free memory on the head (9.7–10.9 GB) did not change between settings.

Three wins. The first guess is kept 89–97% of the time, the second 72–89%, the third 58–74%. Past that, guesses 4 and 5 were kept too rarely to pay for the extra work, and the adaptive setting kept backing itself down. One number in the launcher was worth more than the whole first hour of tuning on the FP8 lane.

Where the memory goes

Every Spark has 121.7 GiB of unified memory that the CPU and GPU share. vLLM does not size the KV cache from the context length you ask for. It claims a fixed share of memory (--gpu-memory-utilization × 121.7 GiB), loads its slice of the weights, runs one test forward pass to see what activations, CUDA graphs and communication buffers need, and turns everything left inside the claim into KV cache. The context length is then just a cap that has to fit inside that pool.

Where each Spark’s 121.7 GiB goes Same model, same node; the smaller weight format frees 33.8 GiB per node, and nearly all of it becomes KV cache (4.3× the space). Official FP8 gpu-memory-utilization 0.86 → vLLM claims 104.7 GiB weights + MTP head 90.96 GiB outside the claim 17.0 GiB KV cache 8.82 GiB · 149,824 tokens activations, CUDA graphs, buffers ≈ 4.9 GiB (derived) RadixArk NVFP4 gpu-memory-utilization 0.80 → vLLM claims 97.4 GiB weights + MTP head 57.13 GiB KV cache 37.82 GiB 720,384 tokens outside the claim 24.3 GiB activations, CUDA graphs, buffers ≈ 2.4 GiB (derived) weights + MTP head KV cache activations / graphs / buffers (derived) left to OS, processes, page-cache headroom

Per node, from each lane's boot log (FP8: first boot). The orange slice and the grey remainder are derived by subtraction.

Per nodeOfficial FP8RadixArk NVFP4
Memory claimed by vLLM0.86 × 121.7 = 104.7 GiB0.80 × 121.7 = 97.4 GiB
Weights + MTP head (measured)90.96 GiB57.13 GiB
KV cache (measured)8.82 GiB (10.85 on the second boot)37.82 GiB
Activations, CUDA graphs, buffers (derived)≈ 4.9 GiB≈ 2.4 GiB
Left outside the claim (derived)17.0 GiB24.3 GiB
Actually free on the head while serving~6.8–8 GB~8.5–8.9 GB
KV pool149,824 tokens (181,632 second boot)720,384 tokens
Cost per token (pool ÷ tokens)~63 KB~56 KB

Watch out for

Next

Credits

RadixArk (the NVFP4 build), 0xMagnus (FP8 and NVFP4 TP8 launchers, memory limits, the reasoning traps), ciprianveg (gb10-vllm and the v19 image), wrldsuksgo2mars and the b12x stack (SM121 sparse-attention kernels), local-inference-lab (the vLLM fork), CosmicRaisins (adaptive MTP), Light Foundry (public 8× GB10 GLM-5.3 numbers), eugr (spark-vllm-docker), malaiwah (quant-fidelity-suite), and Z.ai for GLM-5.3 and its FP8 release.