One GB300 Station + one RTX PRO 6000 vs four RTX PRO 6000s: same DeepSeek-V4.1-Flash, their harness
Fraser Price published DeepSeek-4.1-Flash-1017-BRRR: the same 510 GB DeepSeek-V4.1-Flash-1017 checkpoint we serve, on four RTX PRO 6000 Blackwell Max-Q cards (TP4, Engram tables on NVMe, hand-written fused decode kernels on top of Local Inference Lab's b12x/vLLM), with a C1 decode table that reads 703 tok/s on a 2K coding prompt. Our lane is one DGX Station GB300 with one RTX PRO 6000 Max-Q as an expert sidecar (recipe).
The rule here is that you don't quote a peer's number against your own until you have run their harness on your lane. So we ported their bench.py (stdlib, same prompt, same two-pass method, same columns) and ran it on our production launch, bracketed with controls, in one window on the night of October 25. We also tested the two ideas their recipe carries that we had never tried: block verification for the DSpark drafts, and drafting more than 5 tokens per step.
Short version: on their harness they beat us 1.5× at C1 (not the 2.4× a naive read of the two tables gives), we are level at C4, slightly ahead at C16 aggregate, and 2.6× faster on prefill. About a sixth of their C1 edge is a prompt-class choice (k=7 pays on padded codegen and loses on real traffic); the rest is step time, and that splits between our sidecar hop under high-acceptance load and launch overhead their persistent fused kernel removes. Block verification is a real but small win for us (+1–5% C1 by class, no class loses, free) and is now a promotion candidate. Drafting 9 tokens is a 15–20% loss on every class we serve, and the per-position acceptance curves show exactly why.
Side by side, their harness
Their numbers are from their model card (4× RTX PRO 6000 Max-Q at 325 W, Threadripper 9960X, 64 GB RAM, Samsung 990 PRO, adaptive verification off, 5 rounds, cool-to-70 °C between rounds). Ours are from the port (peerbench.py, 2 rounds, no cool-down wait, hottest GPU logged) on the production sidecar launch, arm A of Card BR2; arm A2 (a second control boot after the k=9 arm) agreed within 0–3% on every cell. Thinking off, temperature 1.0 / top_p 1.0 as their bench sends, natural EOS up to 2,048 generated tokens, nonce per stream to defeat the prefix cache. TG = decode tok/s per stream while all streams decode; PP = prefill tok/s from a max_tokens=1 pass; accept = 1 + accepted/drafts and, in parentheses, accepted ÷ draft tokens.
| prompt | conc | TTFT p50 theirs | ours | PP p50 theirs | ours | TG p50 theirs | ours | TG total theirs | ours | accept theirs | ours |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 2K | 1 | 172 ms | 152 ms | 12,452 | 14,279 | 703 | 466 | 706 | 467 | 5.94 (71%) | 5.13 (83%) |
| 2K | 4 | 489 ms | 264 ms | 4,470 | 8,137 | 329 | 321 | 1,320 | 1,287 | 5.94 | 5.15 |
| 2K | 8 | 829 ms | 408 ms | 2,589 | 5,259 | 231 | 213 | 1,847 | 1,697 | 5.97 | 3.76 |
| 2K | 16 | 1,333 ms | 643 ms | 1,610 | 3,357 | 166 | 169 | 2,648 | 2,719 | 5.96 | 3.70 |
| 16K | 1 | 972 ms | 349 ms | 16,953 | 47,297 | 684 | 471 | 691 | 470 | 5.99 | 5.21 |
| 16K | 4 | 2,362 ms | 1,025 ms | 7,199 | 16,356 | 321 | 319 | 1,281 | 1,277 | 5.95 | 5.05 |
| 16K | 8 | 4,089 ms | 1,617 ms | 4,072 | 10,271 | 226 | 208 | 1,806 | 1,665 | 5.99 | 3.76 |
| 16K | 16 | 7,624 ms | 2,781 ms | 2,167 | 5,944 | 158 | 170 | 2,522 | 2,715 | 5.98 | 3.70 |
| 64K | 1 | 3,677 ms | 1,419 ms | 17,852 | 46,246 | 675 | 453 | 669 | 454 | 5.97 | 5.05 |
| 64K | 4 | 9,165 ms | 3,597 ms | 7,435 | 18,847 | 312 | 316 | 1,243 | 1,266 | 6.02 | 5.13 |
| 64K | 8 | 16,396 ms | 6,098 ms | 4,051 | 10,860 | 210 | 209 | 1,686 | 1,675 | 5.95 | 3.81 |
| 64K | 16 | 30,893 ms | 10,890 ms | 2,131 | 6,045 | 131 | 165 | 2,096 | 2,653 | 5.98 | 3.72 |
Three things the table says without any decomposition:
- C1 is theirs, by 1.5× at every prompt size (703/466, 684/471, 675/453).
- Prefill and TTFT are ours, by 2.3–2.8× at 16K and 64K. They cap batched tokens at 4,096 per step and throttle their own prefill; we run 8,192 with the #58132 bounded replay. At 64K the first token arrives in 1.4 s here and 3.7 s there.
- Above C4 the two boxes converge. At C16 our aggregate is 3–27% higher; at C8 theirs is 1–9% higher. Our k-schedule drops to 3 draft tokens above 4 live sequences, which is why our
acceptcolumn falls from 5.1 to 3.7 at C8 — that is a configuration choice, not a hardware limit.
Not shown: context. Their 4 cards hold 1M context for 16 sequences; our lane holds 4,400,013 KV tokens at 1M context with the 48 GiB host prefix tier underneath it, and our 1M-prompt first token is 33 s.
Where the C1 gap actually comes from
Decompose the 2K/C1 cell. Tokens per second = (tokens per verify step) × (steps per second).
| theirs (4×6000, k=7, block) | ours, k=5 | ours, k=9 + block | |
|---|---|---|---|
| TG p50 | 703 | 466 | 468 |
| tokens per verify step | 5.94 | 5.13 | 5.70 |
| steps per second | 118 | 91 | 82 |
| step time | 8.45 ms | 11.0 ms | 12.2 ms |
So 16% of the gap is more tokens per step (k=7 vs 5, on a prompt where the deeper positions get accepted), and 30% is a faster step. Two mechanisms explain the step time.
Our step slows down as acceptance goes up; theirs does not. On our real mixed-traffic prompt pool the same lane decodes at ~2.1 accepted tokens per step and the step is ~9.8 ms. On their padded codegen prompt, 5.1 tokens per step, the step is 11.0 ms; at k=9 it is 12.2 ms. This is the expert-union cost we have written about before: each accepted token routes to its own 6 of 384 experts per layer, so a 6-token verify batch touches ~22 unique experts per layer and a 9-token batch ~27. On our lane 295 of those experts per layer live in GB300 HBM and 89 are computed on the RTX PRO 6000; every layer waits for the sidecar's answer over PCIe before it can continue, forty times per step, and the wait grows with the cold share of the union. They have no cold tier at all: all 384 experts live in HBM across four cards, each card reading a quarter of each expert in parallel (4 × ~1.8 TB/s, roughly our single GB300's 8 TB/s), and their only cross-device sync is the TP all-reduce, which they fused and overlapped.
Their fused persistent decoder removes the launch churn we measured. Our decode-step profile on this lane found ~3.5 ms per step spent in 1,317 small kernel launches — routing, finalize, norms, compare/where — on a ~14 ms step. Their decoder.cu (83 KB, compiled for sm_120a) runs everything after attention as one persistent kernel per step: output projection, hyper-connections, router, shared and routed experts, and both all-reduces chunk-overlapped with the matmuls still in flight. That is about the 2.5 ms in the table. The kernel itself does not port (TP4 shapes baked in, FP8 GEMMs where our experts are MXFP4), but the idea — a persistent decode kernel for small M — is exactly what the staged upstream MegaKernel work (#56255 Mega-MHC, #56266 Mega-Gate, #56344 Attention MegaKernel) is chasing for this model.
The rest of their recipe is specific to their box and not a lever here: phased_all_reduce.cu and vocab-parallel sampling (TP4 collectives; TP1 has none), skinny.cu (FP8), and Engram-on-NVMe via io_uring (our Engram row gather measures 0.04 ms of the step from pinned Grace memory; their disk tier exists to fit 2 × 98 GB of tables on four cards).
Card BR1: block verification (same k, more kept)
Their spec config is num_speculative_tokens 7, draft_sample_method probabilistic, rejection_sample_method block. Probabilistic drafting was already in our launch. Block verification (Sun et al., Block Verification Accelerates Speculative Decoding) checks the draft block jointly instead of left-to-right-and-stop-at-the-first-reject, keeping the output distribution identical. Two facts decided how to measure it: the key is not in vLLM's compute hash, so it boots as a hash hit (528 s); and in vLLM's rejection sampler it is applied only on the non-greedy path (USE_BLOCK_VERIFICATION and not is_greedy), so at temperature 0 it is a no-op — every speed cell had to sample, and we kept a temperature-0 window purely as a parity check.
Bracketed A → B → A2 on the production sidecar launch, one key changed, temperature 1.0 / top_p 1.0 / seeded, 40 real transcripts per class, forced 256 tokens, r1/r2:
| class | C1 A / A2 / B | B ÷ mean(A, A2) | C8 A / A2 / B | B ÷ mean |
|---|---|---|---|---|
| prose | 266.7 / 270.2 / 278.3 | 1.037 | 972 / 979 / 983 | 1.008 |
| structured | 302.2 / 300.8 / 305.3 | 1.013 | 1,060 / 1,025 / 1,068 | 1.025 |
| code | 352.8 / 353.0 / 357.3 | 1.013 | 1,174 / 1,160 / 1,221 | 1.047 |
| shell_ops | 374.0 / 385.5 / 378.1 | 0.996 | 1,194 / 1,246 / 1,320 | 1.082 |
| tool_json | 333.4 / 326.4 / 347.6 | 1.054 | 1,184 / 1,156 / 1,203 | 1.028 |
Tokens per verify step over the window: A 3.20, B 3.26, A2 3.19 (+2.0%). Mixed-traffic C1 (natural length, 67 real prompts): 212.05 / 212.77 / 212.65 — a wash. Temperature-0 window: B within the controls' own r1/r2 spread on every class, as the kernel predicts.
Fidelity, since the acceptance rule changed: teacher-forced top-1 flips B-vs-A 2.17% and B-vs-A2 2.18% against an A-vs-A2 floor of 2.17% (i.e. indistinguishable from re-booting the same config); 0 flips at reference-confident positions on agent/tool documents; gauntlet 12/12 on all three arms; tool-call emission per replay pass 7/12 vs the controls' 6–8; KV pool unchanged at 4,400,013 tokens; 0 sidecar timeouts; 0 foreign clients on any arm (the lane is firewalled to loopback while dark).
Verdict: the pre-registered gate set returned FAIL on two instrumentation defects (an order-sensitive container-env comparison — Docker returns the same 44 variables in varying order — and a temperature-0 noise bar tighter than the class's own r1/r2 spread); re-scored PASS with both fixes, both verdicts kept in the ledger. Honest reading: +1–5% at C1 by class, +1–8% at C8, no class loses, zero cost. It is the one lever we know that raises tokens per verify step without touching more experts. Promotion candidate for the sidecar recipe; not promoted yet.
Card BR2: drafting 9 tokens (and why their k=7 works for them)
The checkpoint's DSpark head was trained with dspark_block_size = 5. vLLM's DSpark speculator emits N = num_speculative_tokens query positions (the anchor plus N−1 noise tokens) in one parallel pass with no engine cap, so k=7 and k=9 boot — and, it turns out, boot as hash hits too (528 s; num_speculative_tokens is not in the autotune hash on this image, which retires a piece of our own lore). Positions beyond 5 are extrapolation for the drafter. We ran A → K9 → A2, where the controls are the production launch (k=5, no block verification) and K9 is block verification plus k=9 — so the K9 rows measure the pair against production, not k alone — same cells, plus the per-position acceptance histogram vLLM exports (vllm:spec_decode_num_accepted_tokens_per_pos).
K9 ÷ mean(A, A2), temperature 1.0:
| prose | structured | code | shell_ops | tool_json | mixed C1 | |
|---|---|---|---|---|---|---|
| C1 | 0.81 | 0.86 | 0.87 | 0.92 | 0.85 | 0.79 |
| C8 | 0.90 | 0.89 | 0.93 | 0.93 | 0.88 | — |
A loss on every class. The curves say why. Per-position acceptance at C1 (fraction of verify steps in which the draft at position p was accepted), and the marginal rate (position p given p−1):
| p1 | p2 | p3 | p4 | p5 | p6 | p7 | p8 | p9 | |
|---|---|---|---|---|---|---|---|---|---|
| prose, k=5 | .717 | .476 | .310 | .205 | .145 | ||||
| prose, k=9 | .612 | .379 | .235 | .147 | .097 | .063 | .036 | .020 | .011 |
| prose marginal, k=9 | .61 | .62 | .62 | .62 | .66 | .65 | .57 | .57 | .54 |
| code, k=5 | .823 | .653 | .536 | .445 | .371 | ||||
| code, k=9 | .719 | .548 | .428 | .322 | .255 | .179 | .108 | .064 | .037 |
| code marginal, k=9 | .72 | .76 | .78 | .75 | .79 | .70 | .60 | .60 | .57 |
| shell_ops marginal, k=9 | .76 | .81 | .80 | .78 | .82 | .72 | .63 | .53 | .51 |
Two effects stack. The marginal acceptance holds at 0.62–0.85 through position 5 and drops to 0.51–0.63 from position 7, where the drafter is guessing at offsets it never trained on. And the longer block hurts the early positions too: position-1 acceptance on prose falls from 0.72 to 0.61, so tokens per step at k=9 (2.60 prose, 3.66 code) ends up below k=5 (2.85, 3.83) on our real transcripts — while the verify step pays for nine tokens' worth of expert union (12.2 ms vs 11.0). On their padded codegen prompt the deep positions do get accepted: K9 on the peer harness reached 5.7–6.05 tokens per step and matched or beat k=5 at C1 (468 vs 466 at 2K; 489 vs 453 at 64K). That is the whole story of their k=7: it is the right setting for the prompt their table shows and the wrong one for the traffic we serve. We keep k=5.
What transfers, what does not
- Block verification: adopted. Promoted to the serving sidecar lane as v27 on 2026-10-26 04:08 CDT (one key, hash-hit boot, measured above). But see the addendum: it only pays where drafting is already probabilistic.
- k>5: no, for an agent lane. The per-position histogram is now a standing instrument; any future drafter (or a different traffic class) gets re-read off it rather than swept.
- Persistent fused decode kernel: the lever worth engineering, via the upstream MegaKernel PRs rather than their
sm_120abuild. - Their fidelity screen (full-vocab KLD over 1,024 forced tokens, two runs per concurrency, all-vs-all so the noise floor is in the table) is cheaper than our teacher-forced-flip pipeline and worth adding.
- Their two prompt-side patches — a message holding only tool results must not open a new turn (otherwise reasoning inside a tool loop is dropped), and parallel tool results sorted by call order before multimodal parsing — are agent-traffic fidelity bugs we have not yet checked on our image. Desk follow-up.
- b12x is archived; its development moved into FlashInfer (their README is now a migration guide). Our sidecar's kernel base is a dead branch; anything from that lineage reaches us through FlashInfer from here.
Addendum, 2026-10-26: the GB300-only lane, and the key under the key
The sidecar lane above was already drafting with draft_sample_method: probabilistic when block verification landed. Our GB300-only recipe (no second GPU, experts streamed from Grace) drafted greedy. Two more bracketed cards on that lane, same cells and gates, same day:
Card BR3 — block verification alone on greedy drafting: a null. Accepted tokens per verify step B ÷ control = 0.999 (3.116 vs 3.118). Mixed C1 −0.2%. The pre-registered verdict was FAIL on one structured-text cell (0.973 against a −2% bar), but the headline is that nothing moved: a one-hot draft leaves block verification nothing to reweight, so it degenerates to standard verification.
Card BR4 — probabilistic drafting, then probabilistic + block (A → P → PB → A2).
| vs mean(A, A2), temp 1.0 | P (probabilistic) | PB (+ block) |
|---|---|---|
| accepted tokens / verify step | +10.3% (3.438 vs 3.12) | +12.0% (3.491) |
| C1 tok/s: prose / structured / code / shell / tool_json | +10 / +11 / +5 / +11 / +16% | +11 / +14 / +8 / +13 / +15% |
| C8 aggregate | +9 / +8 / +10 / +13 / +19% | +13 / +12 / +13 / +16 / +17% |
| acceptance by draft position 1…5 | .79 .59 .44 .35 .28 (control .70 .50 .38 .30 .25) | .79 .60 .46 .36 .29 |
| TF top-1 flips / margin flips / gauntlet | 1.10–1.13% (floor 1.14%) / 0 / 12/12 | 1.10% / 0 / 12/12 |
| KV tokens | 2,481,694 vs 2,495,694 (−0.56%: the bf16 draft-logits cache is allocated before KV profiling) | |
P passed every gate and is now that recipe's reference (v28-prob). PB adds +1.5% tokens per step over P — the same size as block's gain on the sidecar lane — but failed two temperature-0 parity gates in that card. Two more brackets of block-on-probabilistic followed (BR5, and BR6 with a redesigned replay gate: 8 passes per arm against a pooled-control bar, after the 2-pass version proved to false-fail about a third of the time). Neither passed clean: BR5 fell on the replay-turns gate (the one that was then redesigned), and BR6 on shell_ops (temp-1 −5.0% against its spread bar, missed by 0.1 tok/s; temp-0 parity −3.3%). shell_ops was also BR4's parity trip (+3.6%). Block's gain on that lane sat at +1.4–2.5% tokens per step — inside the lane's own per-class noise — and we retired it there for cause on 2026-10-26. The sidecar lane's v27 stands: different schedule, drafter already probabilistic, one bracket that passed on re-score after two instrument defects (documented in its bundle).
The lesson is a correction to the sentence at the top of this post. The idea we took from the BRRR config was block verification; the key that mattered for a greedy-drafting lane was the one next to it. With a temperature-1 target, the drafter's samples match the target distribution far better than its argmax does — acceptance rises 0.04–0.09 at every position — and that is worth ten percent before block verification does anything. Same verify shape, same expert union, no retune. If your DSpark lane drafts greedy and serves sampled traffic, turn probabilistic drafting on first — and bracket block verification on your own lane rather than copying the key; on one of our two lanes it was worth +2% and on the other it never cleared the gates.
Method notes and receipts
Box: DGX Station GB300 (GB300, 494 GiB Grace, 250.7 GiB HBM visible) + RTX PRO 6000 Blackwell Max-Q, vLLM nightly af7f9488 (pinned by digest), original-el8's M3 hook, RonanLabsAI's sidecar protocol, our 295/89 → rowmap-mixed-ar-v1 map, --kv-cache-memory 8.5 GiB, 48 GiB host prefix tier, k-schedule [[1,4,5],[5,8,3],[9,32,3]]. Every arm: hash-hit boot 528–529 s, KV 4,400,013 tokens, fail-stop sidecar watchdog in enforce mode, 0 timeouts, the lane firewalled to loopback while dark, production pair restored and re-verified between and after cards. Both cards' runners, launchers, verdict scripts, the stdlib port of their bench and all receipts: ~/dsv41/m3/br1/ on the Station (results-cardBR1/, results-cardBR2/, VERDICT-rescore.txt in each). Peer harness rounds: 2 (theirs 5). Their bench's prompt_tokens_details cache-bust check is not exposed on our server, so the port falls back to the prefix-cache hit counter in /metrics.
Credit: Fraser Price (recipe, kernels, bench, Apache-2.0); Local Inference Lab (b12x, vLLM fork); RonanLabsAI and original-el8 (the sidecar lane we run); vLLM. Block verification: Sun et al. The night's numbers were produced by Milo (Claude Fable 5.1) with two Claude Opus 5.5 desk workers authoring the runners; James Meadlock owns the lane and the decisions.