The first half of this post explains how TensorFold fits the model on one GPU. That part has not changed. Sections 7 and 8 hold the new results.
GLM-5.3 is a 744B-parameter mixture-of-experts model: 78 layers, the first three dense, the other 75 each holding 256 small "expert" networks plus one shared expert. For every token, a router in each layer picks just 8 of the 256. NVIDIA's NVFP4 export packs it into 433 GiB (465 GB). The GB300's HBM is 250.7 GiB. So the model is nearly twice too big for the GPU, and the only reason it can run on one Station at all is the 494 GiB of Grace memory welded to the GPU by NVLink-C2C. We have been here before: our slot-cache recipe on vLLM serves the same checkpoint at 54.7 tok/s for one user. TensorFold attacks the same problem with the same basic idea, written from scratch in Zig with its own CUDA kernels, and it handles more than one user much better.
Everything that every token touches stays on the GPU: attention, the three dense layers, the shared experts, the output head and the MTP draft head (whose experts the engine cuts from bf16 to 4-bit at load). That is about 35 GiB as stored. The routed experts are the bulk of the model. Each one is 20.25 MiB (three NVFP4 matrices plus their FP8 scales), 256 of them per layer, 5.06 GiB a layer, 380 GiB in all. All of them live in pinned Grace memory. While the server ran, /proc/meminfo showed 379.8 GiB of shared memory, which matches that arithmetic to within a tenth of a GiB.
The GPU keeps a ring for each MoE layer: 128 slots, each able to hold one expert. That is 9,600 slots, roughly 190 GiB if each slot holds an expert as stored. The startup line says it plainly: 0 experts pinned in HBM, a ring of 128 slots a MoE layer, 4 streams of 32768 tokens. Nothing is permanently resident. The ring is a cache, and Grace keeps the only full copy.
Figure 1. Sizes come from the engine's startup log, the checkpoint's safetensors headers and /proc/meminfo during the October 13 run. "≈190 GiB" for the ring is my arithmetic (9,600 × 20.25 MiB); the engine reports one combined figure, 245.07 GiB.
A decode round walks all 78 layers. At each of the 75 MoE layers, the router looks at every row in the round (one token per conversation, or several when drafting) and picks 8 experts per row. For each pick, the engine checks that layer's ring:
That is why half a layer's experts is enough to get good speed. Routing is not uniform: some experts get picked far more than others, and consecutive tokens tend to reuse some of the same experts. A ring holding the 128 a layer has used recently will catch most picks. It can never catch all of them, so every miss is paid for in C2C time.
Figure 2. The copy kernels and the "Grace experts staged" message are taken from the binary itself. The engine does not report a ring hit rate, so the green/pink mix is illustrative, not measured.
The binary also contains kernels named tf_offload_claim_predict and tf_offload_copy_predicted. From the names, the engine seems to start some copies before the router asks for them. I have not seen the source for this CUDA family (it is not in the public tree yet), so take that as a reading of symbol names, not a description of the algorithm.
GLM-5.3 ships with an MTP ("multi-token prediction") head: one small extra layer trained to guess the next few tokens. TensorFold uses it to propose a chain of guesses (about three a round on average tonight), then runs the full model over the whole chain in one pass and keeps the guesses up to the first one the model disagrees with, plus the model's own next token.
A drafted round costs more than a plain one, because more rows go through every layer. Tonight a plain round took about 21.7 ms and a drafted round about 38.7 ms. But a drafted round committed 2.65 tokens on average, and the extra rows mostly hit the same experts the first row already fetched. So one stream goes from 46 tok/s to 68 tok/s on a 1,024-token prose reply.
Figure 3. Round counts, draft counts and rates come from the server's per-request log lines and streamed timings for the 1,024-token prose prompt; the other three prompts committed 2.5–2.7 tokens a round.
The important part is the rule that drafts may not change the output. Batched math normally does change the output: a GPU summing four rows at once can split the work differently from one row alone, and floating-point rounding then differs in the last bit, which eventually flips a token. TensorFold's exact-decoding rule forbids that. A verify row uses the same group order, the same split of the long dot products and the same rounding it would use alone, and router ties break by a stable expert order. The draft only decides how many rows go into a pass; the model still chooses every token. Each finished request logs an output fingerprint, and the drafted and plain replies matched on all six prompt pairs I tried.
With --parallel 4, TensorFold does not run four conversations side by side. It runs one round at a time with every live conversation's rows in it. That matters a lot for an offloaded model. Within a layer, the experts four conversations need overlap with each other, so the round fetches their union once, and every row that wanted an expert reads the same copy. Four conversations need fewer than four times the misses, and the fixed cost of walking 78 layers is paid once per round instead of four times.
The same exactness rule holds across streams: each conversation's reply must equal what it would have been alone. I checked this with identical prompts (four copies of the same prompt matched the solo reply) and with different prompts (each stream's fingerprint matched across repeats, and the prose and code streams matched their solo runs).
On my first run, with Ash's flags (32K × 4), four different conversations gave 80 tok/s in total and 21–25 tok/s each. Each user is slower than alone, but the box does more work. With 16 streams the total goes to 83 tok/s (section 7, Figure 4). Our vLLM daily cannot do this: it serves one conversation at a time, and an older vLLM setup with 4 sequences fell to 26 tok/s in total, because four unrelated prompts thrashed a slot cache sized for one user.
Prompts are a different story. An 8,350-token prompt took 6.0 s to its first token, about 1,400 tokens a second. That is where vLLM is clearly faster (section 7, Figure 5).
October 13, 9:04–9:15 PM CDT. James's Station, idle except for this test. Binary sha256 8e72be0a…f806 (tensorfold-native 1.0.8, chip nvidia-sm103), checkpoint GLM-5.3-NVFP4, flags as in Ash's notes: --context 32768 --parallel 4, greedy, thinking on. The model loaded in 187.8 s. Every reply below was still in its thinking phase when it hit max_tokens, so these are reasoning-text speeds.
| Case | Ash's notes | My run, October 13 |
|---|---|---|
| One stream, drafted (default) | 76 tok/s prose, 71 code (256 tokens) | 64.0–68.4 tok/s decode over 1,024 tokens (4 prompts); 62–65 tok/s including first token over 256 |
One stream, "draft": false | 48 tok/s | 44.3–46.2 tok/s (4 prompts) |
| Drafted equals plain? | "the same, token for token" | Yes: 6 of 6 prompt pairs, by full text (256) and fingerprint (1,024) |
| Two streams, total | 84 tok/s | 73.6 tok/s, different prompts (104 with the same prompt twice) |
| Four streams, total | 100 tok/s | 79.9 tok/s, different prompts (134 with the same prompt four times) |
| First token, ~1K-token prompt | about 0.8 s | 0.87 s (1,041 tokens) |
| First token, ~8K-token prompt | about 5.2 s | 6.0 s (8,350 tokens) |
| Four ~1K prompts sent together | all first tokens in about 1.9 s | 1.1, 2.2, 3.3, 4.5 s |
| MTP drafts kept | not stated | 53–61% of drafted tokens on the 1,024-token runs |
My numbers come in 10–20% under Ash's. In the first version of this post I said that different prompts could explain it. For one stream, that is wrong. I then sent Ash's own smoke request (the refrigerator prompt, 256 tokens, timed start to end of the curl call, as his smoke.sh does) eight times:
| Run | tok/s | Rounds |
|---|---|---|
| Ash's prompt, cold | 62.2 | 96 |
| Ash's prompt, runs 2–5 | 63.1–63.2 | 96 |
| A different prompt, then Ash's prompt again | 62.8 | 96 |
| Ash's notes, same prompt | 76 | not stated |
I cannot check two things from my side: how Ash timed 76 tok/s (end-to-end, or the server's own decode rate), and whether he timed the same binary. The binary's timestamp is eight minutes before his notes. Only Ash can answer these two. For four streams, the prompt mix does matter: four different prompts gave 80 tok/s and four copies of one prompt gave 134. His 100 is between the two.
The "same prompt" rows show how much shared routing helps. Four identical conversations route identically, so the round copies each expert once for all four. Real traffic is closer to the different-prompts number.
On October 13–14 I ran both engines on the Station with the same script, one engine at a time, with the box otherwise idle. Each engine ran at the setup it would use in real work:
| TensorFold | vLLM, our slot-cache recipe | |
|---|---|---|
| Context and users | 128K × 4 streams (prefill, agents); 32K × 16 streams (concurrency curve) | 256K, 1 sequence at a time |
| Expert slots in HBM | 104 per layer = 7,800 (startup log) | 7,360 |
| Drafts | MTP, on | MTP, 2 tokens |
| Prompt cache | none on this build ("no prompt cache" in the startup log) | prefix cache on |
The slot counts are close, so the expert cache is not what separates the two here.
Each step sends 1 to 16 conversations at the same time. Each one asks for a 512-token reply. There are 8 prompt types, one warm-up pass, and then two scored passes.
Figure 4. Total tokens divided by wall time, mean of the two scored passes. "First token" is the median. vLLM at 8 and 16 needs a different container (more sequences, less context), which I did not get to before the campaign stopped. The October 13 version of this figure used vLLM numbers from an older harness; this one replaces it.
| Streams | TensorFold 32K × 16: total / each / first token | TensorFold 128K × 4: total / each | vLLM 1 seq: total / first token |
|---|---|---|---|
| 1 | 61.3 / 63.8 / 0.34 s | 61.3 / 63.8 | 57.5 / 0.53 s |
| 2 | 65.0 / 34.6 / 0.59 s | 65.0 / 34.9 | 54.4 / 5.0 s |
| 4 | 69.7 / 18.6 / 1.1 s | 68.9 / 18.2 | 53.6 / 14.5 s |
| 8 | 74.8 / 10.5 / 2.0 s | 68.2 / 17.9 (4 wait in a queue) | not run |
| 16 | 82.9 / 5.4 / 3.7 s | — | not run |
Two things to read here. First, TensorFold's total keeps going up as users are added, and every user gets a first token in a few seconds. vLLM's total stays flat, because it decodes one reply at about 56–62 tok/s while the others wait. Second, each TensorFold user gets less: 5.4 tok/s at 16 streams. TensorFold is a good fit for many slow conversations. It is not a fast single lane.
One request at a time, with a new prompt each time, so no cache can help either engine.
Figure 5. Time to first token, mean of two runs (four at 89K on TensorFold). TensorFold ran at 128K × 4, so it cannot take a 212K prompt.
TensorFold reads about 1,300 tokens a second up to 44K tokens, then slows to about 1,140 at 123K. vLLM reads about 3,500 tokens a second from 5K tokens all the way to 212K. At 1.4K tokens the two are equal. For a chat with short messages this does not matter. For an agent that sends 50K tokens of history, it matters a lot. Section 8 shows how much.
--prompt-cache-gib 16 set. Every request in its log shows cached=0.James asked for the agent benchmarks that Nous uses for Hermes. "Hermes Bench" itself is not public, so I used the closest thing that is: SkillsBench, run with the Harbor harness and its Hermes Agent adapter, at reasoning effort "high". That is how Nous runs it for the Hermes Index. This is not Hermes Bench.
Figure 6. TensorFold token counts are its Prometheus counters after the whole SkillsBench run. The vLLM figure is the change in its prefix-cache counters over the first ~6 minutes of its run (89,967 prompt tokens, 59,776 hits); I did not get a whole-run figure before the stop.
| TensorFold | vLLM recipe | |
|---|---|---|
| Status | Complete: 20 tasks, 11:46 PM to 4:59 AM | Partial: 4 of 20 finished when James stopped the campaign |
| Score | 4.83 of 20 | 0.83 of the 4 that finished |
| Passed | exceltable-in-ppt, pddl-tpp-planning, powerlifting-coef-calc, suricata-custom-exfil (1.0 each); dialogue-parser (0.833) | dialogue-parser (0.833) |
| Ran out of time | 7 of 20 | 0 of 4 |
| Could not run on ARM | 2 (fix-build-google-auto, seismic-phase-picking) | not reached |
The 4 tasks that vLLM finished are the first 4 in alphabetical order. On those 4, both engines scored 0.833 in total. TensorFold ran out of time on 3 of them. vLLM finished all 3 and failed the checks. So on this sample neither engine is smarter. One is faster at agent turns. Four tasks is far too few for any claim about quality, and I make none.
The cause of the timeouts is clear from the counters. Over its SkillsBench run, TensorFold read 14.74 million prompt tokens and wrote 0.44 million. That is 34 tokens read for each token written. With no prompt cache and prefill at about 1,300 tok/s, the GPU spends most of each agent turn reading history it has read before. MTP drafts still worked: 59% of drafted tokens were kept.
--prompt-cache-gib); this build reports "no prompt cache" for this model. If Ash's newer builds add it, the agent result could change a lot, and I will run it again.A side finding: Harbor 0.24's Hermes adapter does not work with current Hermes. It calls hermes version, which is now hermes --version, and a new HERMES_HOME needs hermes pm repair before the CLI runs. Harbor's own token counts for the Hermes agent are also zero, so I used the servers' counters. I patched the adapter locally. The fix is a candidate for a small upstream PR.
From Ash's notes, with the box-specific device UUID and paths removed. The GB300 has to be free (the engine takes nearly all of its HBM and about 410 GB of host memory), CUDA 13.2's cuBLASLt has to be on the library path, and the GB300 should be chosen by UUID, because CUDA's device order can differ from nvidia-smi's and the Station's display GPU must not be touched.
export CUDA_VISIBLE_DEVICES=<GB300 UUID from nvidia-smi -L>
export LD_LIBRARY_PATH=/usr/local/cuda-13.2/lib64
./tensorfold-native serve /models/GLM-5.3-NVFP4 \
--host 127.0.0.1 --port 8080 --context 32768 --parallel 4
curl -s 127.0.0.1:8080/health
curl -s 127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" \
-d '{"model":"glm","temperature":0,"max_tokens":256,
"messages":[{"role":"user",
"content":"Explain how a refrigerator moves heat out of its cabinet, step by step."}]}'
Add "stream": true to stream. The reply's thinking arrives in reasoning_content before content; give it enough max_tokens or it will finish while still thinking (the server warns you). Add "draft": false to decode one token at a time and see the same reply more slowly. /metrics exposes Prometheus counters, including mtp_drafted_total and mtp_accepted_total. It listens only on 127.0.0.1.
tput.py) for both engines and the servers' Prometheus counters. One engine ran at a time. The campaign stopped at 7:28 AM CDT on October 14 to give Ash a slot on the GPU; vLLM SkillsBench (16 tasks) and vLLM at 8–16 users were not run.capabilities --json); the copy-kernel names (the binary's symbols).