J&M Labs Blog by Milo

Building the future, locally

← Back to Home

GhettoRak's second switch: Kimi-K3 in four tiers

Kimi-K3 in NVIDIA's NVFP4 release is 1,499.2 GiB, and 1,426.4 GiB of that is 82,432 routed experts at 18,579,480 bytes each. It does not fit any box here. Three tiers — Station, Grace, eight Sparks — closed with 4.88 GiB of slack, which is no slack at all once one reservation is wrong. Four tiers close with 58.6 GiB: the GB300 keeps attention and the hottest experts, Grace holds a warm band, the Station's RTX PRO 6000 computes a middle band, and all eight Sparks hold the rest at 100 GiB each and run it themselves.

Update, October 30, 2026: the execution plan is published — Kimi K3 NVFP4 on Station, and 8 Sparks. It reorders the next steps below. The Spark tier needs none of the Station's GPUs, so it starts now while the DeepSeek lane keeps serving, and the Station work moves into one 2.5-hour measurement window that buys the number which actually decides the outcome: the Station's own attention time per layer. At the only value ever measured (3.03 ms) this design is a 3.2 tok/s machine; at 0.40 ms it is 15.1.

1,485 GiB
expert memory, four tiers
58.6 GiB
slack, up from 4.88
100 GiB
per Spark, decided today
18.6 MB
one expert, never on the wire
269 µs
remote-expert hop
1,347 µs
same expert from NVMe

Next steps, in dependency order

#ActionNeeds
1End AP2 on the StationJames's call. Returns 254 GiB of GB300 HBM, 66 GiB of the RTX 6000 and 366 GiB of pinned Grace.
2E0 fabric build: switch B, second power feed, 4+4 splitSwitch, its own feed, 1 m breakout measured; Lane B down 90–120 min.
3E1–E3: topology A/B, Station link, per-call constantsE0. E2 and E3 also need AP2 ended.
4E4 sidecar: K3 experts on the RTX 6000AP2's code path, K3 experts, an sm_120 SiTU kernel. Pass: cosine ≥ 0.999, ≤ 150 µs per expert call.
5E5 first tokens: one layer, then 92E4 plus a one-layer harness; all eight Sparks and the whole pod.
6E6 parity, then speedE5. Log-probability envelope first; tok/s only after parity.

Status

RungBarWhere it stands
K3-0 fabricSwitch built, Station↔Spark ≥ 180 Gb/s, constants recordedSwitch arrives tomorrow; the Station is mid-campaign.
K3-1 runningExperts on four tiers, coherent text, any speedNever attempted end to end. Pieces measured: 269 µs for one expert round trip between Sparks, 438 µs for three, 175 µs inside a local Spark bank, ~314 µs a layer for a 92-layer loop.
K3-2 correctPer-layer and log-probability match against a slow referenceNot started.
K3-3 fast15 tok/s single stream, official qualityNot proven. The Station's own 279 ms/token term is the risk, not the network.

The ledger

Every byte is from the safetensors headers, not a model card: 82,432 experts × 18,579,480 B is 1,426.36 GiB, and the text non-expert set is 72.03 GiB. Each tier pays its own reservations first; those reservations are assumptions, unchanged since October 3, and the first thing to re-measure when an engine exists.

Kimi-K3 NVFP4 expert placement across four memory tiers: Station HBM 165.4 GiB, Grace 430.0 GiB, RTX PRO 6000 sidecar 89.6 GiB, eight Sparks 800.0 GiB, for a total of 1,485 GiB against a 1,426.4 GiB expert set A single horizontal bar split into four tier segments that sum to 1,485.0 GiB of expert budget. The model's 82,432 experts occupy 1,426.4 GiB, marked by a dashed line at 96 percent of the bar, leaving 58.6 GiB of slack at the right end. Below, three boxes explain the delta from the previous plan: plus 5,177 experts from the RTX PRO 6000, minus 2,072 experts from budgeting Sparks at 100 GiB instead of the 104.5 GiB ceiling, equal to 3,387 experts of slack where there were 282. Where 82,432 experts live 1,485.0 GiB of expert budget · 1,426.4 GiB of experts · 58.6 GiB slack GB300 HBM 165.4 Grace host 430.0 GiB read in place 8 × DGX Spark 800.0 GiB 5,779 experts each, computed locally RTX PRO 6000 sidecar · 89.6 GiB 82,432 experts end here 58.6 GiB slack Previous plan: 282 experts of slack (4.88 GiB). This one: 3,387 (58.6 GiB). +5,177 experts RTX PRO 6000 · 95.6 GiB, counted now −2,072 experts Sparks at 100 GiB, not the 104.5 ceiling = 3,387 experts of slack was 282 · every tier closes
Each segment is the expert budget left in that tier after its own reservations; the dashed line is the 1,426.36 GiB expert set. The slack is drawn at the end for legibility — the planner may leave it anywhere.
TierCapacitySubtracted firstExperts
GB300 HBM250.69 GiB (256,703 MiB)non-expert 72.03 + KV 0.84 + KDA 0.42 + workspace 12.09,558 · 165.40 GiB
Grace446.03 GiB obtainablehost 12.0 + staging 4.024,852 · 430.03 GiB
RTX PRO 600095.59 GiB (97,887 MiB)context + ring + display 6.05,177 · 89.59 GiB
8 × Spark104.49 GiB ceiling eachOS 11.19 + worker 6.0, then budget 10046,232 · 800.00 GiB
Total1,485.02 GiB budgeted85,819 slots for 82,432 experts

That is 3,387 spare experts, or 58.6 GiB. The slack absorbs the assumptions instead of resting on them: 128K context instead of 32K costs 2.53 GiB and leaves 56.1 GiB, loading vision costs 0.83 GiB, and a 20 GiB host overhead still closes. Lowering the Sparks to 95 GiB also closes, with 1,075 spare; 90 GiB does not, at −1,237. So 100 GiB is the floor with room. Shutting down desktop software buys nothing measurable: the Sparks run no display manager, and the Station's GNOME, Xorg and mutter total about 0.5 GiB of host RAM.

What crosses the wire

The fabric moves activations, not weights. One expert is 18.58 MB; one activation is about 7 KiB, so a dispatch is roughly 2,600× cheaper than shipping the expert. Sixteen dispatches per layer, 92 layers per token, and every expert stays where it is.

Four-tier Kimi-K3 topology: the GB300 Station holds attention, KV and 9,558 hot experts, Grace holds 24,852 warm experts, the RTX PRO 6000 sidecar computes 5,177 experts, and eight Sparks hold 46,232 experts reached over two CRS804 switches Left: the Station box with three memory tiers inside it. Right: two CRS804 switches joined by a 400G inter-switch link, each feeding four Sparks. Two 200G links were planned from the Station, one to each switch; as built, both land on switch B. Per layer and per token, 16 activation round trips cross the fabric at about 7 KiB each; the 18.6 MB expert never moves. GB300 Station attention · KDA · router · shared · KV, every layer GB300 HBM · 250.7 GiB 9,558 hot experts · 165.4 GiB non-expert 72.0 GiB + KV + KDA state local compute, no wire Grace host · 446 GiB 24,852 warm experts · 430.0 GiB read in place over C2C · 350 GB/s bandwidth-metered, not free RTX PRO 6000 · 95.6 GiB 5,177 experts · 89.6 GiB computed here, rows come back the sidecar pattern already running on this Station for DeepSeek-V4.1 CRS804-B · 4 Spark legs @ 200G Spark 1 5,779 experts Spark 2 5,779 experts Spark 3 5,779 experts Spark 4 5,779 experts CRS804-A · 4 Spark legs @ 200G Spark 5 5,779 experts Spark 6 5,779 experts Spark 7 5,779 experts Spark 8 5,779 experts 400G inter-switch 200G (as planned) 200G on the wire: 7 KiB of activation out and back, per expert dispatch — 16 per layer the expert's 18.58 MB never moves Per token: 92 layers × 16 dispatches. Weights stay where they are; tokens move. All eight Sparks are down for the K3 window.
Each Spark holds 5,779 experts and runs them itself; the Station ships activations out and gets results back, which is why those links carry arrowheads at both ends. Every Spark leg is 200G. The Station links are drawn as planned, one into each switch. Corrected October 30: as built, both Station ports land on switch B, so Sparks 5–8 are one inter-switch hop away — 5.9 µs against 4.7 µs for Sparks 1–4, measured from either port. See the measured fabric constants.

Port arithmetic still forces 4 + 4. Four cages per CRS804: one cage on switch B for the Station's two legs (planned as one per switch; both landed on B), one inter-switch link, two cages left per switch for Sparks. The 400G inter-switch link carries no K3 traffic — the pod talks Station to Spark, one hop on each switch.

The miss tax

Four teams have shipped NVMe expert streaming for K3: SGLang's SSD Expert Pack (17–22.7% hit rate, 22–27 GB of disk traffic per token), llama.cpp's bit-exact streaming PR, Pulsar at 0.7 tok/s on one Spark, SparkLab's NVFP4 repack at 0.16 tok/s. All of it works, and all of it is storage-bound. That is why the disk is not a tier here.

Cost of one 18.58 MB expert by where it is fetched from: 1,347 microseconds from Station NVMe, 269 from a remote Spark across the fabric, 175 from a local Spark bank, 93 from the RTX PRO 6000 sidecar, and 53 from Grace over C2C Log-scale horizontal bars. NVMe is 25 times a Grace read and five times a remote Spark hop. Bars marked est are arithmetic on measured bandwidth; the rest are measured on this hardware. NVMe is the failure path, not a tier. What one expert costs, by where it lives one expert = 18.58 MB · log scale · the token waits for whichever tier answers Station NVMe · 13.8 GB/s 1,347 µs Remote Spark · fabric 269 µs Local Spark bank 175 µs RTX 6000 sidecar 93 µs Grace over C2C 53 µs est 10 µs 100 µs 1,000 µs NVMe is 25× a Grace read and 5× a remote Spark hop. It is the failure path, not a tier.
One expert, five places to fetch it from. Remote Spark and local bank are measured on this rack; the 93 µs sidecar call is measured on a DGX Station GB300 for DeepSeek-V4.1 at one row, not for Kimi-K3; the 53 µs is arithmetic on a measured 350 GB/s C2C path.

At 13.8 GB/s the Station's RAID0 takes 1,347 µs for one expert. A tenth of the routing on disk is about 200 ms per token — three times the whole budget for 15 tok/s. The RTX 6000 answers in 93 µs because it reads its own HBM and only the answer rows cross PCIe, the same pattern AP2 already runs here for DeepSeek; a published same-session A/B on a DGX Station GB300 measured 1.97× at concurrency 1 and 2.83× at 16 over host memory. NVMe stays as the failure path: if a tier is missing, the model reads from the checkpoint it already has and gets slower instead of stopping.

Grace is the other meter. It is 3.4 TB/s of HBM against 350 GB/s of C2C, so half the expert mass warm costs 425 µs a layer — about 39 ms per token before attention. Placement has to put the hottest mass in HBM and in the 6000's own HBM first.

Not this

What can stop it

Provenance

Device figures read live on October 28, 2026: B300 256,703 MiB and RTX PRO 6000 97,887 MiB from nvidia-smi at 17:15 CDT, Grace from free -b. The byte census (82,432 × 18,579,480 B; 72.03 GiB text non-expert) is from the safetensors headers. The 269 / 438 / 175 / 314 µs and the 279 ms/token term are October 3 measurements. Grace's 53 µs and NVMe's 1,347 µs are arithmetic on measured 350 GB/s and 13.8 GB/s, and say so on the chart. The sidecar's 93 µs and 1.97× / 2.83× are measured on a DGX Station GB300 for DeepSeek-V4.1, not Kimi-K3. Every reservation in the ledger is an assumption, and says so.