J&M Labs Blog by Milo

Building the future, locally

← Back to Home

Kimi K3 NVFP4 on Station, and 8 Sparks

Where this stands. Kimi-K3 runs. At 1:09 PM on October 30 the full model — official NVFP4 weights, all 92 MoE layers — produced coherent text with every expert computed where it lives: the Station's GPU memory, its Grace memory, its RTX PRO 6000, and eight DGX Sparks. With CUDA graphs it runs at 18.5 tokens per second (3.7 with no CUDA graphs). Checked layer by layer against the exact computation on the same inputs, the pod's output matches at cosine 0.9998 on average. Then all 92 layers were checked on the full model's own inputs against an independent reference built from the checkpoint: no placement or routing error anywhere.

Parked, October 31, 9:32 AM. The pod is switched off and the Station's GPUs are back on DeepSeek-V4.1-Flash. Last numbers before it stopped: 18.4 tok/s for one user, 46.5 tok/s in total for eight, and a 3,364-token prompt read in 6.0 s. Weights, code and the recipe are all kept, and it comes back in about half an hour.

Update, November 1. Ash Hart's TensorFold runs this same model on this same pod 2.4–3.8× faster at one-stream decode and 1.76× at eight streams, while ours still reads prompts about 5.5× faster: the side-by-side comparison.

0.17 ms
Station attention + router per layer, measured (was guessed at 3.03)
36.5 ms
all eight Sparks per token, 1,500 real tokens
18.5 tok/s
the full model today, batch 1; flat from 16 to 512 tokens
790 GiB
expert banks built, shipped, hash-checked on the Sparks
244 µs
one Spark hop, one expert, round trip
4 tiers
first tokens with every expert in its home tier, 1:09 PM Oct 30

In plain English

What we're doing. Kimi-K3 is a 1.5-terabyte model. Almost all of that is 82,432 small "expert" networks, and each word it writes uses only 16 of them per layer. No single machine here holds it, so the experts are spread across the DGX Station and eight DGX Sparks, and each word's 7 KB of working data travels to wherever its experts live. The experts themselves never move.

What we were afraid of. That the Station would be too slow at its own share — the attention and routing steps every word goes through, 92 times. Our only number for that was 3 ms per step, from a crude test setup. At that speed the whole machine would manage about 3 words a second.

What we measured. On the real weights and the real inference engine, that step takes 0.17 ms — 18 times faster than feared. The Sparks, holding 55% of the experts, answer each word in 36.5 ms between them. Add the parts and a word costs about 53 ms: roughly 19 words a second.

What's left. Two pieces of plumbing: the Station's spare RTX PRO 6000 card becomes a ninth expert holder (it runs the same program the Sparks run), and the inference engine on the Station learns to hand some experts out, compute the rest itself, and add the answers back up. Then the first real words, then a check that they match a slow reference exactly, then speed.

Update, same afternoon. Both pieces of plumbing got built, and the model talks: asked how the transistor's history begins, it answers "with the discovery of the electron in 1897 by J.J. Thomson." It started at 3.7 words a second; with the speed-ups described below it does 18.5. Each layer's answer was checked against computing every expert on one GPU, and they agree.

Status

Where the Kimi-K3 pod stands: 18.5 tokens per second on all four tiers, every layer checked against the checkpoint A status board of ten rows. Gates G1 through G4 have passed or been measured: the Station's per-layer attention path is 0.154 to 0.171 milliseconds, the stock NVFP4 kernel skips masked expert ids exactly, a Spark hop costs 73 plus 77 microseconds per expert after graphing the worker, and the Station has its own 20 amp circuit. G5, cooling for multi-day runs, is open. Track A, the Spark tier, is done at 36.5 milliseconds per token. Track B, the Station measurements, is done. The RTX PRO 6000 is the ninth expert worker, the vLLM hook is built, the first tokens came out coherent, and with CUDA graphs, a graph-resident expert step and graphed workers it runs at 18.5 tokens per second. All 92 layers check out against an fp32 reference built from the checkpoint; layers 1 to 12 also match stock vLLM on identical inputs. Kimi-K3: four tiers, 18.5 tok/s, every layer checked What is measured, what is open, what gets built next · October 30, 2026 G1 Station attention + router per layer 0.154–0.171 ms (needed ≤ 0.35) PASS G2 Stock NVFP4 kernel skips masked ids −1 and foreign ids add exactly 0 PASS G3 Station → Spark hop, k = 1…16 73 + 77·k µs (graphed worker; was 164 + 77·k) DONE G4 Station on its own 20 A circuit K3 at 88% of it PASS G5 Lab cooling for multi-day runs mini-split, or night-only runs OPEN A·B Spark tier + Station measurements 36.5 ms/token Spark tier; fabric constants DONE C1 RTX PRO 6000 as a 9th expert worker 5,152 experts, sm_120, verified DONE C2 vLLM hook: mask, dispatch, join eager; Grace holds 402 GiB read in place DONE K3-1 First tokens; then 15 tok/s (K3-3) 18.5 tok/s batch 1 (3.7 eager at first) PASS K3-2 Every layer vs an fp32 checkpoint reference 92 layers vs fp32; layers 1–12 vs stock; closed PASS Memory closes only with all four tiers: Sparks 45,632 experts + RTX ~5,152 + Station HBM and Grace for the rest of the 82,432.
Gates are the measurements that could have stopped the project; tracks are the work. G5 (lab cooling for multi-day runs) is the only open gate, and it does not block first tokens.
RungBarWhere it stands
K3-0 fabricboth switches up, Station↔Spark ≥ 180 Gb/s, constants recordedPass. 7 KiB half-round-trip 4.7 µs same switch, 5.9 µs across the inter-switch link, +0.15 µs fanning out to all eight at once.
K3-1 runningexperts on four tiers, coherent text, any speedPass, October 30, 1:09 PM. 3.7 tok/s eager, 18.5 tok/s by 8:53 PM.
K3-2 correctper-layer and log-probability match against a slow referencePer layer: pass — pod vs exact on the same inputs, cosine ≥ 0.9998 mean on 11 layers (worst token 0.9972). All 92 layers on full-model inputs vs an fp32 reference: pass (87 at cosine ≥ 0.99; the low-magnitude layers explained by NVFP4 itself). Layer 12 against stock vLLM on the same inputs: cosine 0.9993. Closed 10:41 PM.
K3-3 fast15 tok/s single stream, official qualityPass on speed, October 30, 8:53 PM: 54.0 ms per token = 18.5 tok/s (flat from 16 to 512 tokens). Quality: every layer checked against the checkpoint at 9:57 PM (see K3-2).
K3-4 prompt readinglong prompts read fast enough to useMuch better, still open. First measured October 31, 12:05 AM: 31 tok/s, so a 3,364-token prompt waited 108 s for its first token. By 3:43 AM: 576 tok/s, first token in 5.8 s (284 / 1,124 tokens: 263 / 414 tok/s), after batching prompt tokens to the workers and reading each expert once per 1,024-token chunk. Decode: 54.6 ms per token.

First tokens

The hook lives in vLLM 0.31.0's Kimi-K3 expert step, added from outside the image with a startup module rather than a fork. At load time, the Station keeps its own 344 experts per layer and never reads the other 552 into the process. At run time, vLLM's router picks the 16 experts over all 896; the Station's are computed by the stock NVFP4 kernel, the rest go to the nine workers, and the answers are added. Every layer of every token checks that the 16 experts were each computed exactly once.

TierExpertsHow they are read
GB300 HBM + Grace31,648stock TRT-LLM NVFP4 kernel; 402 GiB of them sit in Grace and are read in place over C2C
RTX PRO 60005,152the Spark worker, unchanged, on sm_120
8 DGX Sparks45,632the Spark worker
"The history of the transistor begins" → " with the discovery of the electron in 1897 by
   J.J. Thomson. The electron is a subatomic particle that carries a negative charge and is found in
   all atoms."
"def fibonacci(n):" → " if n <= 0: return"

Is it right? Comparing log-probabilities of a cut-down model turned out to be useless: at 12 layers, stock vLLM's two own kernel paths disagree on the top token at half the positions. So the check runs per layer instead. The Station loads every expert of a 12-layer cut, computes each MoE layer exactly, also computes it the pod way (its share plus the nine workers), compares, and carries on with the exact answer. Across 11 layers and ~7,000 token-layers, with CUDA graphs on: cosine 0.99980–0.99989 on average, worst single token 0.9972, relative error ~1.7%. The residue is the workers quantizing activations slightly differently from the Station's kernel — two valid NVFP4 paths — not a routing mistake, which would show as cosine far below 0.99.

One token todayms
first tokens, eager273.4
CUDA graphs on (vLLM's breakable graphs; the network steps are eager breaks), workers started before the Station's own experts86.1
send/collect moved from Python to C79.9
no graph break: GPU flag + C proxy thread + in-graph wait (idea: @javierbits)63.7
whole expert call CUDA-graphed on every worker54.0
of which, per layer: send → last worker reply0.31

No graph break. Javier pointed out the expert step doesn't have to break the CUDA graph. Now, inside the graph, the GPU copies the token's latent and routing into pinned host memory and raises a flag (cuStreamWriteValue32). A C thread on the Station sees it, sends to the workers, collects their rows and raises a second flag, which the graph is waiting on (cuStreamWaitValue32). The round trip costs 37 µs per layer, where the graph breaks cost ~290. If a worker fails, the thread kills the engine rather than let a wrong layer through. Per-layer parity was rerun through this path: same result.

Every layer, on real inputs. No machine here can run a stock copy of the whole 1.5 TB model for comparison, so the full-model check goes layer by layer on the model's own inputs. The pod answered five prompts (English, French, code, arithmetic — all coherent, the sum right) with no CUDA graphs, recording at every MoE layer and token the experts' input, the routing, the Station's share and the workers' share: 29,440 layer-tokens. Then an independent fp32 reference, built straight from the checkpoint's bytes, recomputed each tier's share from its own experts.

The workers' fixed cost was overhead, not the kernel. Profiling a Spark showed the fused expert kernel takes 91 µs for one expert and 75 µs per extra one — the GB10's memory bandwidth. The other ~100 µs of each call was Python and about fifteen tiny launches around it. Each worker now captures one CUDA graph per (layer, expert count), 1,472 of them in about a second, and a Spark round trip went from 164 + 77·k µs to 73 + 77·k µs.

A side finding: the workers' expert kernel is not deterministic. Two identical calls differ by 0.8–2% on a Spark and, on the RTX's sm_120 build, by up to 5–6% in a heavy tail that depends on launch timing — present with or without graphs, and in every check above. That is most of the ~1.7% residue in the parity check above: the kernel disagrees with itself by that much. It also means two boots of the model give slightly different log-probabilities (median 0.12–0.14 apart), so a greedy completion can flip at a near-tie. On November 1 two greedy runs of the same four prompts on one boot split apart after 3 to 29 tokens. Corrected November 1: this first said only that greedy text can flip at a near-tie, which understated how soon it happens; the measurement is in the TensorFold comparison, where that engine gives identical tokens on every run.

Grace is full: 492 of 494 GiB in use, 457 GiB of it pinned. This placement cannot put another expert on the Station.

The Station, measured

The question was how long one decoder layer takes on the Station excluding the experts it hands out — attention (KDA or MLA), the attention-residual step, the router and the two shared experts. We measured it on the path we would actually serve: vLLM 0.31.0's own NVIDIA Kimi-K3 code, the official NVFP4 checkpoint, CUDA graphs on, one request at a time.

The full model does not fit on one GPU, so we booted it cut down to 4, 8 and 12 layers with every expert of those layers in HBM, and took the slope. No weights were copied: a directory of symlinks and a pruned index point vLLM at only the shards those layers need, and a small loader hook drops the rest. Layer 0 is dense and the rest repeat KDA, KDA, KDA, MLA, so every four-layer step adds exactly the model's own mix.

Layersms per token (wall clock)ms per token (stream gaps)weights in HBM
41.70151.693354.95 GiB
82.43012.4275119.89 GiB
123.33983.3345184.83 GiB

The slope is 0.205 ms per layer (0.227 for the steepest step, which we use). That is the whole layer, including 16 experts read from HBM. A kernel profile splits it: experts and their routing are 24.9%, so the Station's own term is S = 0.154–0.171 ms per layer, 14.2–15.7 ms per token.

One MoE layer on the GB300 (profiled)µs
attention projections (NVFP4 weights too) + activation quantization85.9
two shared experts, bf16 — 264 MB read every layer67.5
16 routed experts from HBM57.8
latent down/up + router25.1
routing indices, finalize, fp4 conversion17.7
attention-residual9.6
the attention kernel itself: KDA / MLA7.4 / 15.8

The attention kernels are nearly free. The layer is weight reads — and the official checkpoint keeps the shared experts in bf16. MLA decode also runs on a fallback kernel (the fast one does not support 96 heads). Both are levers for later; neither matters to whether this works.

One Kimi-K3 token: October's estimate was 308 milliseconds; measured today it is about 53, under the 66.7 millisecond line for 15 tokens per second Three horizontal bars on a 0 to 320 millisecond axis. October estimate: 308.3 ms, of which 278.8 is the Station's own attention and router path at 3.03 ms per layer. Measured today: 0.85 fixed plus 15.7 Station plus 36.5 Spark tier equals 53.1 ms, 18.8 tokens per second. Modelled next with a hot and cold expert map: 44.3 ms. A dashed line marks 66.7 ms, 15 tokens per second. The Station was never the problem. The guess about it was. ms per Kimi-K3 token · 92 MoE layers · 16 experts per layer · batch 1 October estimate S = 3.03 ms/layer, from a Q2_K GGUF on Triton eager Station 278.8 ms 308.3 ms → 3.2 tok/s Measured today S = 0.171 ms/layer + Spark tier 36.5 ms, uniform banks 53.1 ms → 18.8 tok/s Modelled next same Station; Sparks hold the coldest experts (27.7 ms) 44.3 ms → 22.6 tok/s 15 tok/s = 66.7 ms 0 40 80 120 160 200 240 280 320 fixed (lm_head, layer 0, sampling) Station attention + router, 92 × S expert path, measured expert path, modelled Today's Station-side routed experts (7.1 per layer, HBM / RTX / Grace) are assumed to finish inside the Spark wait, and the join is not yet measured. Neither lower bar is a measured token; the middle one is built from measured parts.
The same token three ways. October's bar is a guess built on a crude proxy; the middle bar is built from today's measured parts; the bottom bar swaps in the modelled hot/cold expert map.

Experts the Station doesn't hold cost it nothing

The split design needs the Station's stock NVFP4 expert kernel to compute only the experts it holds and ignore the rest of each 16-wide list. We tested it directly: the output for 16 local experts equals the sum of the outputs for "8 local + 8 masked" and "8 masked + 8 local" to one bf16 rounding step (0.30%), and an all-masked list returns exactly zero — whether the foreign experts are marked −1 or simply outside the local range. So the Station side is a lookup table in front of the existing kernel, not a new kernel. One trap it found: a repeated expert id is counted once, not twice, so the table must never map two slots to one expert.

The Spark tier, measured

Each Spark holds 62 experts from every one of the 92 layers — 5,704 experts, 98.7 GiB — built on the Station from the checkpoint and shipped over the fabric. Every bank's SHA-256 matches on both ends, and 119 randomly sampled records match the checkpoint byte for byte. Each Spark runs one worker: it keeps its whole bank in GPU memory, takes a list of 16 expert ids and a 7 KB activation, computes the ones it holds, and replies with the weighted sum, how many it used, and exactly which.

Station to one Spark, end to end: about 164 microseconds plus 77 per expert the Spark computes, measured on the real bank A line chart of round-trip time against k, the number of experts the Spark computes for one request, from 1 to 16. Measured medians rise from 243.5 microseconds at k equals 1 to 1,387 at k equals 16 and sit on the line 164 plus 76.7 times k. October's harness measured 269 at k equals 1 and 438 at k equals 3, both above today's line. A bar on the right splits the k equals 1 round trip into 12 microseconds of wire and client, 40 of worker overhead, and 191 of GPU kernel. A Spark hop is the GB10's memory bandwidth, plus 164 µs Station → Spark → Station, 7,424 B each way, real NVFP4 bank, 400 requests per point, every reply checked 0 400 800 1200 1600 1 2 4 8 12 16 k = experts this Spark computes for one (token, layer) round trip, µs 1387 µs k=8: 773 µs k=1: 243.5 µs ○ October harness: 269 µs (k=1), 438 (k=3) p50 = 164 + 76.7·k µs (bars: p50→p99) One k=1 hop: 243.5 µs median 191.4 µs GPU kernel (median) ~115 fixed + one 18.6 MB expert 39.2 µs worker outside kernel worker median − kernel median 12.9 µs wire + Station client round trip median − worker median Measured 2026-10-30, Station mlx5_0 → spark1. Hollow red circles: the October g2a harness on the same hardware (pinned staging, 16 KiB sends).
One Spark, Station to Spark and back, on the real bank. The kernel streams one 18.6 MB expert at about 248 GB/s — the GB10's memory ceiling — so each extra expert costs ~77 µs. The fixed part is the next thing to shrink.

Then all eight at once. For 1,500 real tokens from a recorded Kimi-K3 routing trace, the Station walked each token through its 92 layers in order, sent each layer's 16 ids to every Spark holding any of them, waited for the slowest, and checked that the Sparks' counts plus the Station's remainder added to exactly 16. It did, 138,000 times.

Spark tier, whole podms per token
median36.49
p95 / p9937.68 / 39.20
per layer, median / p990.403 / 0.570
share of expert work landing on Sparks55.4% (the other 7.1 per layer stay on the Station)

The first run had a p99 of 205 ms: about once every few seconds one Spark took 150–200 ms on one layer. The workers' own GPU timers said the time was inside the kernel, and it happened on all eight Sparks within a second of each other — which independent GPUs don't do, but eight identical Python processes serving the same number of requests do. It was Python's garbage collector walking ~385,000 objects. Freezing and disabling it after startup took the p99 to 39 ms.

The fabric

7 KiB, half round trip (perftest)typicalp99
Station → Sparks 1–4 (switch B)4.64–4.72 µs4.75–5.01 µs
Station → Sparks 5–8 (switch A, one extra hop)5.92–5.95 µs6.02–6.24 µs
all eight at once4.86 / 6.07 µs5.22 / 6.46 µs

The network is about 1 ms of a 53 ms token. It is not the lever.

What we got wrong

What's next

  1. The RTX kernel's timing-dependent tail: a repro for FlashInfer.
  2. Multi-request batching: everything here is one request at a time.
  3. Correction (9:13 PM): an earlier version of this post said longer answers slow down, 62.6 ms per token at 256 tokens (16.0 tok/s). They don't. That was one unrepeated sample. Streamed token by token, 512-token answers run at 54.1 ms per token, worst gap 56 ms, three runs out of three.
  4. Placement. Simulated with the measured costs, moving cold experts to the slow tiers saves only ~1–6 ms per token: the Sparks set the floor, and a hotter Station reads more from Grace at ~48 µs per expert.

What can stop it

Recipe and credits

The whole setup is published as a recipe: J-M-Recipes · Kimi-K3 NVFP4 four-tier pod. It includes the vLLM hook, the RoCE transport and proxy, the worker, the bank builder, the SiTU patch, launchers, the parity checks and the raw receipts. It is marked experimental: batch 1, 4K context, checked layer by layer, with no whole-model quality gate yet.

Javier (@javierbits) suggested the change that took this from 79.9 to 63.7 ms per token. The expert step doesn't have to break the CUDA graph: the GPU sets a flag, a C++ thread does the network send and collect, and the graph waits on a second flag. We built it his way, and he is credited in the code, the commit and the recipe. Also credited: NVIDIA for the NVFP4 checkpoint and the FlashInfer B12x and TRT-LLM MoE kernels, the vLLM project for breakable CUDA graphs and CPU offload, and Moonshot AI for Kimi-K3.

Provenance

All measurements October 30, 2026, on the Station (GB300 + RTX PRO 6000, DeepSeek-V4.1 lane stopped for the window) and DGX Sparks 1–8, over two CRS804 switches. Weights: NVIDIA's NVFP4 Kimi-K3 release, 82,432 experts × 18,579,480 B, re-derived from the safetensors headers. Station: vLLM 0.31.0 (vllm/vllm-openai:v0.31.0), SM clock 2,070 MHz on every sample. Sparks: FlashInfer 0.7.0.post1 B12x NVFP4 kernel with a SiTU activation patch, PyTorch 26.07 container. Routing: the October 3 Kimi-K3 trace. First tokens: vLLM 0.31.0, eager, the four-tier hook, October 30 1:09 PM; 12.2 tok/s with breakable CUDA graphs at 3:58 PM; 18.5 tok/s with the graph-resident expert step and graphed workers at 8:53 PM; all-layer check against the checkpoint 9:41–9:57 PM; layer 12 vs stock 10:41 PM. The ~53 ms token and the 27.7 ms hot/cold Spark tier are sums and models, labelled as such. Code, raw outputs and the decision log live in the project repository (kimi-k3-station8).