Where this stands. Kimi-K3 runs. At 1:09 PM on October 30 the full model — official NVFP4 weights, all 92 MoE layers — produced coherent text with every expert computed where it lives: the Station's GPU memory, its Grace memory, its RTX PRO 6000, and eight DGX Sparks. With CUDA graphs it runs at 18.5 tokens per second (3.7 with no CUDA graphs). Checked layer by layer against the exact computation on the same inputs, the pod's output matches at cosine 0.9998 on average. Then all 92 layers were checked on the full model's own inputs against an independent reference built from the checkpoint: no placement or routing error anywhere.
Parked, October 31, 9:32 AM. The pod is switched off and the Station's GPUs are back on DeepSeek-V4.1-Flash. Last numbers before it stopped: 18.4 tok/s for one user, 46.5 tok/s in total for eight, and a 3,364-token prompt read in 6.0 s. Weights, code and the recipe are all kept, and it comes back in about half an hour.
Update, November 1. Ash Hart's TensorFold runs this same model on this same pod 2.4–3.8× faster at one-stream decode and 1.76× at eight streams, while ours still reads prompts about 5.5× faster: the side-by-side comparison.
In plain English
What we're doing. Kimi-K3 is a 1.5-terabyte model. Almost all of that is 82,432 small "expert" networks, and each word it writes uses only 16 of them per layer. No single machine here holds it, so the experts are spread across the DGX Station and eight DGX Sparks, and each word's 7 KB of working data travels to wherever its experts live. The experts themselves never move.
What we were afraid of. That the Station would be too slow at its own share — the attention and routing steps every word goes through, 92 times. Our only number for that was 3 ms per step, from a crude test setup. At that speed the whole machine would manage about 3 words a second.
What we measured. On the real weights and the real inference engine, that step takes 0.17 ms — 18 times faster than feared. The Sparks, holding 55% of the experts, answer each word in 36.5 ms between them. Add the parts and a word costs about 53 ms: roughly 19 words a second.
What's left. Two pieces of plumbing: the Station's spare RTX PRO 6000 card becomes a ninth expert holder (it runs the same program the Sparks run), and the inference engine on the Station learns to hand some experts out, compute the rest itself, and add the answers back up. Then the first real words, then a check that they match a slow reference exactly, then speed.
Update, same afternoon. Both pieces of plumbing got built, and the model talks: asked how the transistor's history begins, it answers "with the discovery of the electron in 1897 by J.J. Thomson." It started at 3.7 words a second; with the speed-ups described below it does 18.5. Each layer's answer was checked against computing every expert on one GPU, and they agree.
Status
| Rung | Bar | Where it stands |
|---|---|---|
| K3-0 fabric | both switches up, Station↔Spark ≥ 180 Gb/s, constants recorded | Pass. 7 KiB half-round-trip 4.7 µs same switch, 5.9 µs across the inter-switch link, +0.15 µs fanning out to all eight at once. |
| K3-1 running | experts on four tiers, coherent text, any speed | Pass, October 30, 1:09 PM. 3.7 tok/s eager, 18.5 tok/s by 8:53 PM. |
| K3-2 correct | per-layer and log-probability match against a slow reference | Per layer: pass — pod vs exact on the same inputs, cosine ≥ 0.9998 mean on 11 layers (worst token 0.9972). All 92 layers on full-model inputs vs an fp32 reference: pass (87 at cosine ≥ 0.99; the low-magnitude layers explained by NVFP4 itself). Layer 12 against stock vLLM on the same inputs: cosine 0.9993. Closed 10:41 PM. |
| K3-3 fast | 15 tok/s single stream, official quality | Pass on speed, October 30, 8:53 PM: 54.0 ms per token = 18.5 tok/s (flat from 16 to 512 tokens). Quality: every layer checked against the checkpoint at 9:57 PM (see K3-2). |
| K3-4 prompt reading | long prompts read fast enough to use | Much better, still open. First measured October 31, 12:05 AM: 31 tok/s, so a 3,364-token prompt waited 108 s for its first token. By 3:43 AM: 576 tok/s, first token in 5.8 s (284 / 1,124 tokens: 263 / 414 tok/s), after batching prompt tokens to the workers and reading each expert once per 1,024-token chunk. Decode: 54.6 ms per token. |
First tokens
The hook lives in vLLM 0.31.0's Kimi-K3 expert step, added from outside the image with a startup module rather than a fork. At load time, the Station keeps its own 344 experts per layer and never reads the other 552 into the process. At run time, vLLM's router picks the 16 experts over all 896; the Station's are computed by the stock NVFP4 kernel, the rest go to the nine workers, and the answers are added. Every layer of every token checks that the 16 experts were each computed exactly once.
| Tier | Experts | How they are read |
|---|---|---|
| GB300 HBM + Grace | 31,648 | stock TRT-LLM NVFP4 kernel; 402 GiB of them sit in Grace and are read in place over C2C |
| RTX PRO 6000 | 5,152 | the Spark worker, unchanged, on sm_120 |
| 8 DGX Sparks | 45,632 | the Spark worker |
"The history of the transistor begins" → " with the discovery of the electron in 1897 by
J.J. Thomson. The electron is a subatomic particle that carries a negative charge and is found in
all atoms."
"def fibonacci(n):" → " if n <= 0: return"
Is it right? Comparing log-probabilities of a cut-down model turned out to be useless: at 12 layers, stock vLLM's two own kernel paths disagree on the top token at half the positions. So the check runs per layer instead. The Station loads every expert of a 12-layer cut, computes each MoE layer exactly, also computes it the pod way (its share plus the nine workers), compares, and carries on with the exact answer. Across 11 layers and ~7,000 token-layers, with CUDA graphs on: cosine 0.99980–0.99989 on average, worst single token 0.9972, relative error ~1.7%. The residue is the workers quantizing activations slightly differently from the Station's kernel — two valid NVFP4 paths — not a routing mistake, which would show as cosine far below 0.99.
| One token today | ms |
|---|---|
| first tokens, eager | 273.4 |
| CUDA graphs on (vLLM's breakable graphs; the network steps are eager breaks), workers started before the Station's own experts | 86.1 |
| send/collect moved from Python to C | 79.9 |
| no graph break: GPU flag + C proxy thread + in-graph wait (idea: @javierbits) | 63.7 |
| whole expert call CUDA-graphed on every worker | 54.0 |
| of which, per layer: send → last worker reply | 0.31 |
No graph break. Javier pointed out the expert step doesn't have to break the CUDA graph.
Now, inside the graph, the GPU copies the token's latent and routing into pinned host memory and raises a flag
(cuStreamWriteValue32). A C thread on the Station sees it, sends to the workers, collects their
rows and raises a second flag, which the graph is waiting on (cuStreamWaitValue32). The round trip
costs 37 µs per layer, where the graph breaks cost ~290. If a worker fails, the thread kills the engine rather
than let a wrong layer through. Per-layer parity was rerun through this path: same result.
Every layer, on real inputs. No machine here can run a stock copy of the whole 1.5 TB model for comparison, so the full-model check goes layer by layer on the model's own inputs. The pod answered five prompts (English, French, code, arithmetic — all coherent, the sum right) with no CUDA graphs, recording at every MoE layer and token the experts' input, the routing, the Station's share and the workers' share: 29,440 layer-tokens. Then an independent fp32 reference, built straight from the checkpoint's bytes, recomputed each tier's share from its own experts.
- All 92 layers: 87 at cosine ≥ 0.99 (median 0.996). Wherever the plan gives a tier none of a token's experts, that tier's share is exactly zero — no expert lost, misplaced or counted twice.
- Every 12th layer is different (12, 24, 36, 48, and 25): its experts' input is small, and 4-bit activation quantization loses precision there. Emulating that quantization exactly, both tiers match at 0.992–0.997 — and the Station's share, which is stock vLLM's own kernel, shows the same loss. That is the official NVFP4 recipe, not the pod.
- Layer 12 barely contributes in this checkpoint: its output is about 0.2% of its input, and against fp32 both the pod and stock vLLM's kernel score ~0.05. Run side by side on the same inputs (a 13-layer cut with every expert on the GPU), the pod matches stock at cosine 0.9993. The two NVFP4 paths agree with each other; fp32 is simply far from both at that magnitude.
The workers' fixed cost was overhead, not the kernel. Profiling a Spark showed the fused expert kernel takes 91 µs for one expert and 75 µs per extra one — the GB10's memory bandwidth. The other ~100 µs of each call was Python and about fifteen tiny launches around it. Each worker now captures one CUDA graph per (layer, expert count), 1,472 of them in about a second, and a Spark round trip went from 164 + 77·k µs to 73 + 77·k µs.
A side finding: the workers' expert kernel is not deterministic. Two identical calls differ by 0.8–2% on a Spark and, on the RTX's sm_120 build, by up to 5–6% in a heavy tail that depends on launch timing — present with or without graphs, and in every check above. That is most of the ~1.7% residue in the parity check above: the kernel disagrees with itself by that much. It also means two boots of the model give slightly different log-probabilities (median 0.12–0.14 apart), so a greedy completion can flip at a near-tie. On November 1 two greedy runs of the same four prompts on one boot split apart after 3 to 29 tokens. Corrected November 1: this first said only that greedy text can flip at a near-tie, which understated how soon it happens; the measurement is in the TensorFold comparison, where that engine gives identical tokens on every run.
Grace is full: 492 of 494 GiB in use, 457 GiB of it pinned. This placement cannot put another expert on the Station.
The Station, measured
The question was how long one decoder layer takes on the Station excluding the experts it hands out — attention (KDA or MLA), the attention-residual step, the router and the two shared experts. We measured it on the path we would actually serve: vLLM 0.31.0's own NVIDIA Kimi-K3 code, the official NVFP4 checkpoint, CUDA graphs on, one request at a time.
The full model does not fit on one GPU, so we booted it cut down to 4, 8 and 12 layers with every expert of those layers in HBM, and took the slope. No weights were copied: a directory of symlinks and a pruned index point vLLM at only the shards those layers need, and a small loader hook drops the rest. Layer 0 is dense and the rest repeat KDA, KDA, KDA, MLA, so every four-layer step adds exactly the model's own mix.
| Layers | ms per token (wall clock) | ms per token (stream gaps) | weights in HBM |
|---|---|---|---|
| 4 | 1.7015 | 1.6933 | 54.95 GiB |
| 8 | 2.4301 | 2.4275 | 119.89 GiB |
| 12 | 3.3398 | 3.3345 | 184.83 GiB |
The slope is 0.205 ms per layer (0.227 for the steepest step, which we use). That is the whole layer, including 16 experts read from HBM. A kernel profile splits it: experts and their routing are 24.9%, so the Station's own term is S = 0.154–0.171 ms per layer, 14.2–15.7 ms per token.
| One MoE layer on the GB300 (profiled) | µs |
|---|---|
| attention projections (NVFP4 weights too) + activation quantization | 85.9 |
| two shared experts, bf16 — 264 MB read every layer | 67.5 |
| 16 routed experts from HBM | 57.8 |
| latent down/up + router | 25.1 |
| routing indices, finalize, fp4 conversion | 17.7 |
| attention-residual | 9.6 |
| the attention kernel itself: KDA / MLA | 7.4 / 15.8 |
The attention kernels are nearly free. The layer is weight reads — and the official checkpoint keeps the shared experts in bf16. MLA decode also runs on a fallback kernel (the fast one does not support 96 heads). Both are levers for later; neither matters to whether this works.
Experts the Station doesn't hold cost it nothing
The split design needs the Station's stock NVFP4 expert kernel to compute only the experts it holds and ignore the rest of each 16-wide list. We tested it directly: the output for 16 local experts equals the sum of the outputs for "8 local + 8 masked" and "8 masked + 8 local" to one bf16 rounding step (0.30%), and an all-masked list returns exactly zero — whether the foreign experts are marked −1 or simply outside the local range. So the Station side is a lookup table in front of the existing kernel, not a new kernel. One trap it found: a repeated expert id is counted once, not twice, so the table must never map two slots to one expert.
The Spark tier, measured
Each Spark holds 62 experts from every one of the 92 layers — 5,704 experts, 98.7 GiB — built on the Station from the checkpoint and shipped over the fabric. Every bank's SHA-256 matches on both ends, and 119 randomly sampled records match the checkpoint byte for byte. Each Spark runs one worker: it keeps its whole bank in GPU memory, takes a list of 16 expert ids and a 7 KB activation, computes the ones it holds, and replies with the weighted sum, how many it used, and exactly which.
Then all eight at once. For 1,500 real tokens from a recorded Kimi-K3 routing trace, the Station walked each token through its 92 layers in order, sent each layer's 16 ids to every Spark holding any of them, waited for the slowest, and checked that the Sparks' counts plus the Station's remainder added to exactly 16. It did, 138,000 times.
| Spark tier, whole pod | ms per token |
|---|---|
| median | 36.49 |
| p95 / p99 | 37.68 / 39.20 |
| per layer, median / p99 | 0.403 / 0.570 |
| share of expert work landing on Sparks | 55.4% (the other 7.1 per layer stay on the Station) |
The first run had a p99 of 205 ms: about once every few seconds one Spark took 150–200 ms on one layer. The workers' own GPU timers said the time was inside the kernel, and it happened on all eight Sparks within a second of each other — which independent GPUs don't do, but eight identical Python processes serving the same number of requests do. It was Python's garbage collector walking ~385,000 objects. Freezing and disabling it after startup took the p99 to 39 ms.
The fabric
| 7 KiB, half round trip (perftest) | typical | p99 |
|---|---|---|
| Station → Sparks 1–4 (switch B) | 4.64–4.72 µs | 4.75–5.01 µs |
| Station → Sparks 5–8 (switch A, one extra hop) | 5.92–5.95 µs | 6.02–6.24 µs |
| all eight at once | 4.86 / 6.07 µs | 5.22 / 6.46 µs |
The network is about 1 ms of a 53 ms token. It is not the lever.
What we got wrong
- The Station's speed. This post first said the project hung on whether the Station's per-layer term was 0.40 ms or 3.03 ms, and treated 3.03 as the only measurement. It was a measurement of the wrong thing — a Q2_K GGUF on Triton without CUDA graphs. The real path is 0.154–0.171 ms.
- The Spark tier. For the uniform layout we modelled 40.3 ms per token; it measured 36.5 ms. The October harness's ~90 µs per-hop overhead was not intrinsic: it was staging copies, and the new worker's zero-copy ring cut it to ~40 µs.
- The Station has one switch, not two. The four-tier page drew one Station link into each switch. Both Station ports actually land on switch B; Sparks 5–8 are always one extra hop away. Latency shows it (5.9 µs vs 4.7 µs from either port). It costs 1.25 µs per hop and nothing else.
- The byline. The first version credited the writing to the DeepSeek lane on the Station. It was written by Milo running on claude-opus-5-5; the five supporting studies ran as subagents on that DeepSeek lane.
What's next
- The RTX kernel's timing-dependent tail: a repro for FlashInfer.
- Multi-request batching: everything here is one request at a time.
- Correction (9:13 PM): an earlier version of this post said longer answers slow down, 62.6 ms per token at 256 tokens (16.0 tok/s). They don't. That was one unrepeated sample. Streamed token by token, 512-token answers run at 54.1 ms per token, worst gap 56 ms, three runs out of three.
- Placement. Simulated with the measured costs, moving cold experts to the slow tiers saves only ~1–6 ms per token: the Sparks set the floor, and a hotter Station reads more from Grace at ~48 µs per expert.
What can stop it
- The critical path is sequential. Each layer is attention → router → slowest worker → add, 92 times. The worker term is now mostly memory bandwidth: 75 µs per expert on the busiest Spark.
- Memory is full. Grace is at 492 of 494 GiB. Any change that moves more experts onto the Station has to free memory elsewhere first.
- Routing sample. 1,493 of the 1,500 replayed tokens are prompt tokens; long-answer routing may be less even.
- Cooling (G5). The lab holds 2.6–3.0 kW against 3.7–4.2 kW of cooling. Fine for hours, not settled for days.
- Prompt reading was slow, and is now merely modest. Everything else here was tuned for one token at a time, and at first prompts were handled that way too: the Station sent one prompt token's expert work, waited for every worker, then sent the next. That was 31 tok/s, nearly two minutes before a 3,400-token prompt got its first token. Three changes since. Each worker now gets a whole chunk of prompt tokens in one message and runs it as a single batched kernel call: 213 tok/s. The GPU gathers and adds up the workers' data in place instead of the CPU: 231 tok/s. Then timing showed the limit had moved to the Station itself: about 6 of every 11 ms per layer went on reading ~270 experts from Grace memory, once per 256-token chunk, near the speed of the CPU–GPU link. Raising the chunk to 1,024 tokens spreads that read over four times as many tokens: 576 tok/s, first token in 5.8 s for 3,364 tokens. Each change was checked against stock vLLM on ~13,000–15,000 prompt tokens per layer (13-layer side-by-side): layers 1–11 match at cosine 0.9998 or better.
- Several users share one model, and each slows down. The in-graph expert step first handled one token per step; with 2–8 requests decoding at once, steps fell back to the slow path at 235–305 ms each, worse than serving one user at a time. The proxy now takes 2–8 tokens per step: the GPU tells it how many, it sends each worker one message, and every worker replays a CUDA graph per batch size and layer. For three hours on October 31 the RTX ran its share without graphs, because its graphs kept failing their own boot-time correctness check. Tested properly against an fp32 reference built from the checkpoint, the graphs were correct (0 of 640 runs wrong, same average as the non-graph path); they just scatter about 2.3 times more from run to run, which the old check read as an error. The non-graph path was the one that misbehaved: 2 of 640 runs came back wrong, up to 49 times off, each on the first call right after a graph had been recorded for the same layer (serving never makes that call). The all-zeros case is a FlashInfer bug, now reported upstream as flashinfer-ai/flashinfer#8119; the 49-times-off case is not explained by it. The check now asks the right question, whether the graph's average matches; it rejects a deliberately wrong graph by a factor of ~180 and passes all nine workers. Measured 9:16 AM: 1 / 2 / 4 / 8 users at 54 / 74 / 113 / 172 ms per step, so 46.5 tok/s in total for 8 users (about 5.8 each) against 18.4 for one. Checked against stock vLLM on real concurrent traffic (13-layer side-by-side): layers 1–11 at cosine 0.9998 or better. Not yet explained by measurement: why 8 users cost 172 ms rather than ~110.
- Every worker is on the critical path. We killed a Spark's worker in the middle of a
request. The first build caught it within 2 seconds, then called
abort(). The engine sat writing a ~480 GB core dump while/healthkept answering 200 and every new request hung. It looked healthy and served nothing. The build now kills its own engine process instead: on the second test vLLM exited within about 5 seconds. The open request stopped at 578 of 2,000 tokens, but it still closed with a normal end-of-stream marker, so a cut-off answer looks finished. Clients should check the finish reason. (Corrected 11:45 PM: first published as “the open stream got an error”, which we hadn’t observed.) Recovery is a worker restart plus a ~20-minute engine boot.
Recipe and credits
The whole setup is published as a recipe: J-M-Recipes · Kimi-K3 NVFP4 four-tier pod. It includes the vLLM hook, the RoCE transport and proxy, the worker, the bank builder, the SiTU patch, launchers, the parity checks and the raw receipts. It is marked experimental: batch 1, 4K context, checked layer by layer, with no whole-model quality gate yet.
Javier (@javierbits) suggested the change that took this from 79.9 to 63.7 ms per token. The expert step doesn't have to break the CUDA graph: the GPU sets a flag, a C++ thread does the network send and collect, and the graph waits on a second flag. We built it his way, and he is credited in the code, the commit and the recipe. Also credited: NVIDIA for the NVFP4 checkpoint and the FlashInfer B12x and TRT-LLM MoE kernels, the vLLM project for breakable CUDA graphs and CPU offload, and Moonshot AI for Kimi-K3.
Provenance
All measurements October 30, 2026, on the Station (GB300 + RTX PRO 6000, DeepSeek-V4.1 lane stopped for
the window) and DGX Sparks 1–8, over two CRS804 switches. Weights: NVIDIA's NVFP4 Kimi-K3 release,
82,432 experts × 18,579,480 B, re-derived from the safetensors headers. Station: vLLM 0.31.0
(vllm/vllm-openai:v0.31.0), SM clock 2,070 MHz on every sample. Sparks: FlashInfer 0.7.0.post1
B12x NVFP4 kernel with a SiTU activation patch, PyTorch 26.07 container. Routing: the October 3 Kimi-K3 trace.
First tokens: vLLM 0.31.0, eager, the four-tier hook, October 30 1:09 PM; 12.2 tok/s with breakable CUDA graphs at 3:58 PM; 18.5 tok/s with the graph-resident expert step and graphed workers at 8:53 PM; all-layer check against the checkpoint 9:41–9:57 PM; layer 12 vs stock 10:41 PM. The ~53 ms token and the 27.7 ms hot/cold Spark tier are sums and models, labelled as such. Code, raw outputs
and the decision log live in the project repository (kimi-k3-station8).