Benchmarks
Benchmark scorecards, methodology notes, throughput reports, and non-comparable result boundaries.
Apple Silicon 512 GB measurements now have their own index: M3 Ultra 512 Benchmarks. GB300 Station numbers live under DGX Station GB300.
- We tried Christopher Owen's prefill yield on our GB300: real, small, and not promoted November 17, 2026
The one Spark idea that ports to our GB300 sidecar lane: Christopher Owen's prefill yield. It cut a short chat's first token behind a 413K-token prompt by about 40% on two boots and cost nothing we could measure, but the row we had promised to judge it on did not reproduce, so we rolled it back. - We moved our four-Spark lane to Christopher Owen's candidate 60: 19% faster decode, prompts read up to twice as fast November 16, 2026
Same four DGX Sparks, same night, same tests: Christopher Owen's spark-ds41f candidate 60 in our shape against the r6x build it replaces. Faster on every row, same answers, and what we chose not to adopt. - NVIDIA's NVFP4 DeepSeek V4.1 Flash on four DGX Sparks: same speed, lower quality November 2, 2026
We ran niklaslenz's speed and quality tests on Mia's TensorFold recipe with NVIDIA's NVFP4 checkpoint, the same recipe on the original weights, and our two original-quant recipes (SGLang, vLLM). NVFP4 is as fast as the original weights on the same engine (within 3%) and 4.1 points lower on the full MMLU-Pro. Our recipes prefill 1.5-1.8x faster. - Testing DGX Spark Kernels November 2, 2026
We installed and tested DGX Spark kernels 7.0.0-1019 and 7.0.0-1024 on four Sparks. 1024 had 0 resets in 6 h of soak and the same speed as 6.17; 1019 works only with NVIDIA's kho=off fix. Plus our full Spark setup: October firmware, 0.5 GB display memory and headless mode gave +2.0 GiB per Spark. - Kimi-K3 on the same pod, two engines: why TensorFold decodes 2.4โ3.8ร faster November 1, 2026
Ash Hart's TensorFold and our vLLM four-tier pod ran the same Kimi-K3 NVFP4 checkpoint on the same Station-plus-eight-Sparks hardware. On the same prompts TensorFold decodes one stream at 44.4โ69.7 tok/s to our 18.4, and 74.4 to our 42.2 at eight streams. Four reasons, with diagrams. Ours still reads prompts about 5.5ร faster. - Hermes Agent embedding/reranking on Mac tested November 1, 2026
llama.cpp vs oMLX vs MLX for Hermes Agent's Qwen3 embedder and reranker on an M4 Max and an M5 Max: llama.cpp for embeddings, a batched reranker, 3ร faster llama.cpp on the M5 with a current build, and a 4B reranker that did not help. - Our roadmap: making one GB300 faster for a single user October 27, 2026 ยท updated 8:44 PM CDT
Where our one-GB300 DeepSeek-V4.1-Flash setup is behind a 4ร RTX PRO 6000 box, what we have ruled out, and the order we will try fixes in. Plain English, with the numbers. - One afternoon against the 703 gap: two banks, one kill, one dud October 27, 2026 ยท updated 5:36 PM CDT
We ran four measurements against the plan in "Why one GB300 doesn't beat 4ร RTX PRO 6000 at single-stream decode". One lever banked and promoted, one killed by its own bar, one instrument failed, one reframing finding. Here is what changed, what didn't, and what comes next. - Why one GB300 doesn't beat 4ร RTX PRO 6000 at single-stream decode โ and what it would take October 27, 2026 ยท updated 2:25 PM CDT
Fraser Price's 703 tok/s (padded codegen, thinking off, k=7, 4ร RTX PRO 6000 Max-Q) is real and reproduced; our GB300 + RTX PRO 6000 sidecar lane does 466 on the same harness. An nsys trace of the exact production config shows the GPU 98.6% busy, so launch overhead is not the cost: 1,035 kernels under 10 ยตs take 4.54 ms of the 11.2 ms step. Portable work is worth roughly +15โ20% (modeled); matching that codegen number needs an sm_103 fusion program. On prose we are level. - Prefill > Decode: the token volume says so, the tok/s benchmarks don't October 26, 2026 ยท updated 6:24 AM CDT
90 days of agent traffic: 13.96 B prompt tokens in, 75.6 M out (185:1). The GB300 DeepSeek-V4.1-Flash lane, 7 days: 53:1 with a 91.6% prefix-cache hit rate and still 4.5 prefilled tokens per token decoded. Prefill is the volume, the KV footprint and ~90% of the bill; decode owns per-request wall time (3.5 s vs 0.24 s) only because the cache carries the rest. The tok/s fixation measures the bounded term. - One GB300 Station + one RTX PRO 6000 vs four RTX PRO 6000s: same DeepSeek-V4.1-Flash, their harness October 26, 2026 ยท updated 3:36 AM CDT
On Fraser Price's own bench.py, four RTX PRO 6000 Max-Q cards beat our one GB300 Station + one RTX PRO 6000 sidecar lane 1.5ร at C1 on DeepSeek-V4.1-Flash; we are level at C4, slightly ahead at C16 aggregate, and 2.6ร faster on prefill. Block verification is a real but small win (+1โ5% C1 by class, no class loses, free); drafting 9 tokens is a 15โ20% loss on every class we serve. - NVIDIA Flare on Two DGX Sparks: GLM-5.3-Flash TP2, Tuned October 21, 2026
Flare TP2 on Sparks 5โ6, with diagrams: defaults 37 tok/s one user and no drafting at four users; 8 seats + capacity 48 + 3 drafts at fixed width โ 78 tok/s at four, 107 at eight. 1.7ร our vLLM pair's cold prefill, 6.1M-token KV, 570K needle, GSM8K-200 96.5%. - NVIDIA Flare for agents: two rough edges, measured October 21, 2026
Two agent-facing rough edges from the four-way run: tool calls arrive all at once (2 deltas in the final ~54 ms vs SGLang's 9 over ~500 ms), and the strict-JSON session-title request gets an instant HTTP 400 on reasoning_effort "none" โ 8/8 across two runs, so sessions never get titled. The agent chain itself still ties. - DeepSeek V4.1 Flash on four DGX Sparks: four engines, one winner October 21, 2026
Four engines, one model, four Sparks. Winner: the vLLM fork. SGLang is the daily pick for shared use, Flare holds 5.8ร the KV cache, TensorFold is the fastest single stream on a 3.5-bit quant. - Why Four DGX Sparks Talk Slowly in the Middle: NCCL Tuning in Plain English October 26, 2026
A thread on X showed two NCCL settings that fix a mid-size all-reduce hole on four DGX Sparks. We reproduced it on ours: a 1 MiB exchange went from 1,154 ยตs to 137 ยตs (1.36 to 11.5 GB/s) with the 256 MiB peak unchanged at ~22 GB/s. Both of our serving recipes already carried the fix; what changed is our fabric gate, which only looked at big messages. Also measured: the Spark's 10 GbE Realtek port drops packets only under an 850 kpps UDP flood; every TCP test ran 9.41 Gbit/s with zero retransmits, so it stays stock. - Same Model, Three Harnesses: DeepSeek Harness vs Claude Code vs Hermes on DeepSeek-V4.1-Flash-1017 October 20, 2026
Same DeepSeek-V4.1-Flash-1017 endpoint, three harnesses, 120 Terminal-Bench 2.0 trials: Pass@1 Hermes 75%, Claude Code 75%, DSH 70% โ not distinguishable. Tokens and wall time are where they differ: DSH 1.23ร Hermes prompt tokens and fastest lanes, Claude Code 0.74ร and fewest output tokens, with a vLLM prefix-cache caveat on its speed. - DeepSeek Flash 1017: TensorFold vs SGLang vs vLLM on 4 DGX Sparks October 19, 2026
Third lane on the same four Sparks: a vLLM fork (Christopher Owen's spark-ds41f r6x, TileLang kernels, local build), not stock vLLM, which cannot serve V4.1 on SM121 yet. It leads SGLang at every concurrency (92 vs 85 tok/s at C1, 287 vs 263 at C6), prefills ~6K tok/s cold at 93K and reads a 300K needle in 55 s; TensorFold EXL3 still wins one stream at 100. Why Tony's 48.5 was a kernel-family gap, and nine tuning arms that all lost to Owen's defaults. - DeepSeek Flash 1017: TensorFold vs SGLang on 4 DGX Sparks October 18, 2026
Same model, same day, same switch: LibertAI's 3.5 bpw EXL3 pack on TensorFold (Sparks 1โ4) against the native MXFP4/FP8 checkpoint on SGLang via knapcio v2.4 (Sparks 5โ8). EXL3 wins single-stream decode by 18% (100 vs 85 tok/s); native wins prefill 1.6โ2ร and the Hermes 20-hop agent gauntlet by 26%, and leads from four streams up. Quality gate identical, 71/75 both. Now extended to a 3-way with a tuned vLLM fork lane; since October 19 Lane B is stopped (kept) because that lane runs on its nodes. - Watt-hours per solved task: Hermes Agent on Terminal-Bench 2.0, GLM-5.3 on eight DGX Sparks October 18, 2026
First scored run of a wall-plug energy benchmark for agents: Hermes Agent v0.22.2 driving GLM-5.3 744B NVFP4 on eight DGX Sparks through a frozen 20-task Terminal-Bench 2.0 slice. Pass@1 12/20, 487 Wh per solved task gross, 148 Wh net of a 559 W loaded-idle rack, 7.3 h at c=1. Why failures cost as much as solves, the four Harbor patches it took, and what the number is not. - 4ร RTX PRO 6000 vs One GB300 Station: DeepSeek-V4.1-Flash-1017 on the Same Grid October 17, 2026
Same 45-case grid on two single-box builds of the native 763B checkpoint: 0xSero’s (sybil-solutions) 4ร RTX PRO 6000 (SGLang, DSpark k=5, Engram on NVMe) against one GB300 Station (vLLM, UVA expert offload, pin-hot hook). Station leads single-stream decode 1.25ร and prefill 3.4ร with TTFT about 3ร lower; the four-GPU box leads 0.86ร at C4 and 0.56ร at C6–C8, where the Station’s k-schedule drops to one draft token — flat k=5 recovers most of that. - GLM-5.3 on eight DGX Sparks: SGLang vs vLLM, and a unified-memory bug October 17, 2026
Same 744B model, same eight GhettoRak Sparks, second engine. A 13-arm SGLang v0.5.22 campaign against the vLLM lane: 4โ16% slower single-stream, faster at four users with 2-step EAGLE, and every long prompt hung until one env var bounded the sparse indexer's scratch buffer. The bug, the fix, and what refuses to boot on SM121. - Full GLM-5.3 on eight DGX Sparks: official FP8 and NVFP4 October 16, 2026
GLM-5.3 744B tensor-parallel 8 across all eight GhettoRak Sparks, two ways. Daily lane: RadixArk NVFP4, 512K context, 25.8โ35.7 tok/s single-stream once MTP guesses three tokens ahead, 61.7 at 4 users. Fidelity reference: Z.ai's official FP8, 131K, 23.1โ29.9 tok/s, 2.3ร closer to BF16. Why the 4-bit build first lost, and recipes for both. - DeepSeek-V4.1-Flash on four DGX Sparks: stock vs abliterated October 13, 2026
DeepSeek-V4.1-Flash serves from Sparks 3โ6 (TP4, 1M context). Abliterated weights cost no speed and no accuracy on math, tools or long context; GPQA-Diamond with thinking on drops from 84.8% to 80.3% (p = 0.08, one run). - I went looking for M5 kernel fruit. Apple already ate it. September 29, 2026
MLX already routes ~98% of M5 prefill compute through NAX kernels; 4-bit GEMM within 3โ15% of fp16 at ~57 TFLOPS. No dense-kernel fruit left; headroom is attention at long context and the serving layer. - DeepSeek-V4-Flash-Vision-Exp on One GB300: Native FP4, SGLang, DSpark, 1M Context September 3, 2026
Single DGX Station TP=1, no requantization: 2,420 tok/s at C32, exact recall to 810K, zero repetition at C64. Recipe + harness on GitHub. - Vision-Exp thinking levels on 2ร DGX Spark September 1, 2026
Matched off/low/high/max thinking probes. Effort labels are not a ladder. - NVFP4 vs EXL3: GLM-5.3-Flash on Two DGX Sparks August 29, 2026
Matched cold-prefill / decode / concurrency probes of two 4-bit quants on the same Spark pair. - GLM Flash 5.3 Testing โ Dual DGX Spark August 27, 2026
Day-0 SGLang bring-up of GLM-5.3-Flash NVFP4: GB10 memory trap, 20-hop gauntlet, 24.7 tok/s prose. - DeepSeek Flash 0731 Testing: weights vs thinking level August 26, 2026
2ร2 A/B on dual Sparks: official vs ablit weights, low vs high thinking. Neither explained the tangents. - Local LLM Testing โ Terminal-Bench 2, June 2026 June 22, 2026
A narrowed benchmark page focused on one measurement regime: Terminal-Bench core, terminus-2, native tool calling, and comparable local model results. - GLM-5.2: Terminal-Bench Benchmark โ MLX vs GGUF June 22, 2026
GLM-5.2 benchmark results across local serving stacks, including the Terminal-Bench score, timeout behavior, and what the numbers mean for agent routing. - DS4-F Under Three Lights: Tool Discipline, Throughput, and Hermes Fit June 25, 2026
A three-part read on the live DeepSeek V4 Flash route: tool discipline, throughput, and whether the endpoint is a good Hermes fit. - The Tool-Calling Benchmark: 9 Models, Local vs Cloud April 12, 2026
Seven models, same 20 prompts, deterministic scoring. The question: how does a locally-run 397B parameter model compare to the top cloud models on agentic tool calling? The answer was surprising. - MiniMax M2.7 vs Qwen3.5-397B vs Claude Sonnet 4.6: Tool Calling on Apple Silicon April 12, 2026
Three models, same benchmark. Two run locally on a Mac Studio M3 Ultra. One is Claude Sonnet 4.6 via API. How close can local get to cloud on agentic tool calling? - Making an Agentic Benchmark Modeled on Doing Agentic Benchmarks April 12, 2026
Most benchmarks are single-shot snapshots that rot the moment you change hardware or models. Milo-Bench fixes this with frozen test cases, deterministic scoring, and a SQLite results DB that accumulates runs over time. 27 tests across 6 categories, open source. - Speculative Decoding on 512GB Mac Studio: Does the 4B Draft Model Actually Help? April 12, 2026
Long reasoning tasks: +58% speedup. Large-context tool calls: -88%, catastrophic. The answer depends entirely on what you are asking the model to do.