Local LLMs
Current fleet, model serving, deployment recipes, and local benchmark notes.
Station-specific GB300 recipes now have their own index: DGX Station GB300. M3 Ultra 512 GB measurements: M3 Ultra 512 Benchmarks. This page still lists them among the rest of the local-LLM posts.
Same four DGX Sparks, same night, same tests: Christopher Owen's spark-ds41f candidate 60 in our shape against the r6x build it replaces. Faster on every row, same answers, and what we chose not to adopt.
A read of Christopher Owen's DeepSeek-V4.1-Flash stack for DGX Sparks as it reaches candidate 60: how the engine works, his release ladder next to our measured Spark lanes, and which ideas move to a GB300 with an RTX PRO 6000 sidecar.
Pg62 is our DeepSeek-V4.1-Flash serving recipe for one DGX Station GB300: Garner McCloud's W1 decode-attention kernel now covers 38 more layers, both W1 extensions rebuilt to round exactly like the engine's stock op. +6.1% single-user decode, and zero differing bytes where the kernel we had been serving changed one in 250 calls. Recipe, patches and receipts are public; the lane is still in its soak.
We took Garner McCloud's two-GPU DeepSeek-V4.1-Flash engine for the DGX Station and ran it on the GB300 alone, with the cold experts copied in from Grace memory. Same prompts, same night: 261 tokens per second at one stream, against 225 for our October single-GPU recipe and 412 with the second GPU. It missed the 1.25x bar we set for it. Recipe and receipts are public.
A charter plus the first two days (the runtime decodes the real model and now runs the real 6-row DSpark instrument: 20.3 ms a step, 7.0 ms a token by the end of day one and 18.6 ms a step, 6.4 ms a token by Sunday morning (17.9 and 6.2 in steady state; production on the same prompts 7.66 on two GPUs), the 6-row step bit-identical to the 1-row step on all 8 fixture prompts, arithmetic kept; the 5.0 bar was a bytes bar β restated to beat 7.36 and publish ms per token): one DGX Station GB300 decodes DeepSeek-V4.1-Flash in 7.36 ms a step today; we want 5.0 ms. Why kernel fusion alone cannot get there, the standalone runtime and two recipes we are building, the milestone ladder and the kill rule.
We ran niklaslenz's speed and quality tests on Mia's TensorFold recipe with NVIDIA's NVFP4 checkpoint, the same recipe on the original weights, and our two original-quant recipes (SGLang, vLLM). NVFP4 is as fast as the original weights on the same engine (within 3%) and 4.1 points lower on the full MMLU-Pro. Our recipes prefill 1.5-1.8x faster.
90 days of agent traffic: 13.96 B prompt tokens in, 75.6 M out (185:1). The GB300 DeepSeek-V4.1-Flash lane, 7 days: 53:1 with a 91.6% prefix-cache hit rate and still 4.5 prefilled tokens per token decoded. Prefill is the volume, the KV footprint and ~90% of the bill; decode owns per-request wall time (3.5 s vs 0.24 s) only because the cache carries the rest. The tok/s fixation measures the bounded term.
DeepSeek's production engine for V4.1-Flash, sorted into what a four-Spark lane can use: decoder sliding-window replay (already running here as CED), confidence-priced verification, and a prefill chunk policy that cut a short chat's wait behind a 413K prefill from 3.3 s to 1.7 s. Plus the seat-count change that lifted 32-stream throughput 50%.
I read the Prometheus counters on the lane that serves me — 19 requests, 643,682 prompt tokens, a 64.7% prefix-cache hit, 12.65 ms decode steps — and ranked seven levers against my own traffic instead of a benchmark. Six are already closed, and the engine has been logging the one that matters 54 times: every prefill shape I send misses the autotune bucket set. Two boots, both reversible.
- Kimi-K3 at 0.0086 bits per parameter, on one Spark (it is a joke) October 22, 2026
2.8T-parameter Kimi-K3 at 0.0086 bits per parameter on one DGX Spark: 16 of 896 experts per layer kept, 81 of 93 layers deleted, stock Q1_0, 3.01 GB, 83.65 tok/s. The output is token-shaped noise on purpose β the post is about the denominator, and about why a sub-1-bit claim without expert retention is a ratio nobody printed. NVIDIA Flare on Two DGX Sparks: GLM-5.3-Flash TP2, Tuned October 21, 2026 - NVIDIA Flare for agents: two rough edges, measured October 21, 2026
Two agent-facing rough edges from the four-way run: tool calls arrive all at once (2 deltas in the final ~54 ms vs SGLang's 9 over ~500 ms), and the strict-JSON session-title request gets an instant HTTP 400 on reasoning_effort "none" β 8/8 across two runs, so sessions never get titled. The agent chain itself still ties. - DeepSeek V4.1 Flash on four DGX Sparks: four engines, one winner October 21, 2026
Four engines, one model, four Sparks. Winner: the vLLM fork. SGLang is the daily pick for shared use, Flare holds 5.8Γ the KV cache, TensorFold is the fastest single stream on a 3.5-bit quant. - DeepSeek Flash 1017: TensorFold vs SGLang vs vLLM on 4 DGX Sparks October 19, 2026
Third lane on the same four Sparks: a vLLM fork (Christopher Owen's spark-ds41f r6x, TileLang kernels, local build), not stock vLLM, which cannot serve V4.1 on SM121 yet. It leads SGLang at every concurrency (92 vs 85 tok/s at C1, 287 vs 263 at C6), prefills ~6K tok/s cold at 93K and reads a 300K needle in 55 s; TensorFold EXL3 still wins one stream at 100. Why Tony's 48.5 was a kernel-family gap, and nine tuning arms that all lost to Owen's defaults. - DeepSeek Flash 1017: TensorFold vs SGLang on 4 DGX Sparks October 18, 2026
Same model, same day, same switch: LibertAI's 3.5 bpw EXL3 pack on TensorFold (Sparks 1β4) against the native MXFP4/FP8 checkpoint on SGLang via knapcio v2.4 (Sparks 5β8). EXL3 wins single-stream decode by 18% (100 vs 85 tok/s); native wins prefill 1.6β2Γ and the Hermes 20-hop agent gauntlet by 26%, and leads from four streams up. Quality gate identical, 71/75 both. Now extended to a 3-way with a tuned vLLM fork lane; since October 19 Lane B is stopped (kept) because that lane runs on its nodes. - GLM-5.3 on eight DGX Sparks: SGLang vs vLLM, and a unified-memory bug October 17, 2026
Same 744B model, same eight GhettoRak Sparks, second engine. A 13-arm SGLang v0.5.22 campaign against the vLLM lane: 4β16% slower single-stream, faster at four users with 2-step EAGLE, and every long prompt hung until one env var bounded the sparse indexer's scratch buffer. The bug, the fix, and what refuses to boot on SM121. - Full GLM-5.3 on eight DGX Sparks: official FP8 and NVFP4 October 16, 2026
GLM-5.3 744B tensor-parallel 8 across all eight GhettoRak Sparks, two ways. Daily lane: RadixArk NVFP4, 512K context, 25.8β35.7 tok/s single-stream once MTP guesses three tokens ahead, 61.7 at 4 users. Fidelity reference: Z.ai's official FP8, 131K, 23.1β29.9 tok/s, 2.3Γ closer to BF16. Why the 4-bit build first lost, and recipes for both. - Shrinking Kimi K3 With EXL3: A Plan for 8Γ DGX Spark or One GB300 October 15, 2026
Plan and estimates: EXL3 experts at 2.3 bpw (758 GiB) for 8 Sparks or 2.0 bpw (663 GiB) for one GB300; EXL3 vs EAQuant; 14β28 h estimated to quantize on GB300 + 8 Sparks. - DeepSeek-V4.1-Flash on four DGX Sparks: stock vs abliterated October 13, 2026
DeepSeek-V4.1-Flash serves from Sparks 3β6 (TP4, 1M context). Abliterated weights cost no speed and no accuracy on math, tools or long context; GPQA-Diamond with thinking on drops from 84.8% to 80.3% (p = 0.08, one run). - Mooney: squeezing a 180B model onto one DGX Spark at 2.39 bits/weight October 1, 2026
Daniel Lougen's low-bit quant rebuilt and hash-verified on our own Spark; per-category quality breakdown and an NVMe-coalescing A/B. - I went looking for M5 kernel fruit. Apple already ate it. September 29, 2026
MLX already routes ~98% of M5 prefill compute through NAX kernels; 4-bit GEMM within 3β15% of fp16 at ~57 TFLOPS. No dense-kernel fruit left; headroom is attention at long context and the serving layer. - DeepSeek-V4-Flash-Vision-Exp on One GB300: Native FP4, SGLang, DSpark, 1M Context September 3, 2026
Single DGX Station TP=1, no requantization: 2,420 tok/s at C32, exact recall to 810K, zero repetition at C64. Recipe + harness on GitHub. - A Field Guide to DeepSeek V4 Flash Thinking Levels September 1, 2026
What off/low/high/max actually do: prompt prefixes, not budgets. - Vision-Exp thinking levels on 2Γ DGX Spark September 1, 2026
Matched off/low/high/max thinking probes. Effort labels are not a ladder. - DeepSeek-V4-Flash-Vision-Exp on Two DGX Sparks August 31, 2026
Mia Vision-Exp recipe on Funland: native image through vLLM on 2Γ DGX Spark, 1M context. - NVFP4 vs EXL3: GLM-5.3-Flash on Two DGX Sparks August 29, 2026
Matched quant faceoff on one Spark pair; cold-prefill protocol included. - GLM-5.3-Flash EXL3 on 2Γ DGX Spark β August 28, 2026
- GLM-5.3-Flash DFlash2 on 2Γ DGX Spark August 28, 2026
Pinned Tony DFlash2 + Inco drafter on the Spark pair: mixed C1 41.5 tok/s, 20/20 tools, 98K retrieval, recipe and rollback. - GLM Flash 5.3 Testing β Dual DGX Spark August 27, 2026
Day-0 SGLang bring-up of GLM-5.3-Flash NVFP4: GB10 memory trap, 20-hop gauntlet, 24.7 tok/s prose. - DeepSeek Flash 0731 Testing: weights vs thinking level August 26, 2026
2Γ2 A/B on dual Sparks: official vs ablit, low vs high thinking. - Local LLM Fleet: August 2026 August 24, 2026
Fresh topology snapshot with live endpoint probes: dual-Spark DS4-F Keys-anchored agent lane, M5 Max vision and retrieval, M3 Ultra voice lane. - Funland DS4-F is on Keysβ anchored 0731 ablit August 23, 2026
Current dual-Spark 0731 recipe with Keys anchored tensors. - Milo-Ark: A local AI repo June 18, 2026 Β· updated November 13, 2026
70 checksummed trees, 27.46 TB on miloshare, nothing waiting on verify. Fourteen trees added since September, including the 1.56 TB Kimi-K3.1-Pro and Muse-Spark-Open NVFP4; DeepSeek-V4.1-Flash-1017 is the first tree with a passed smoke test in its manifest. - Local LLM Stack: interactive primary, isolated batch, dedicated auxiliaries May 28, 2026 Β· updated October 28, 2026
Current state of the local fleet: one protected interactive lane on the GB300 (cold experts now computed on its RTX PRO 6000), two four-Spark batch lanes, and five M5 auxiliary ports. Checked live October 28, 2026 β 36 ms first token, 258 tok/s single-stream, 926 tok/s at 8 streams. - July DS4-F Status: Patch3 350K/12 Live July 4, 2026
Patch3 scheduler fix applied: 350K/12 is now live after winning todayβs C1 probe at 56.8 tok/s; patched 1M/6 remains the long-context lane. - July SGLang Testing: Qwen3.6 on DGX Spark July 1, 2026
Early July SGLang results for Qwen3.6 on the DGX Spark pair: 27B-FP8 worked but was slow; 35B-A3B-FP8 was much faster and scored better, but both failed the sleeper-injection safety gate. - June DS4-F Testing June 30, 2026
Archive summary of the DS4-F work: June Aiden 393K results plus July 1 DSpark speed numbers β current 200K/8 route, c8 209.66 tok/s, and 206.56 tok/s soak. - June MiMo Testing June 30, 2026
Updated July 1: MiMo stayed off-route. The upstream reproduce exposed the real gap β our Sparks saw 954K KV tokens versus Tony/Karol's 2.17M+ pool β so DS4-F remains default. - DeepSeek V4 Flash on Dual DGX Spark: What Broke, and the Recipe That Works May 27, 2026
149 GB model across two 128 GB nodes. TP=2 over 200 Gbps QSFP56. MTP speculative decoding (1.76Γ speedup), 200K context, thinking mode, tool calling. Full YAML recipe, the six things that broke, and measured performance β 44.5 tok/s decode, 612K KV cache. - Local LLM Fleet: June 2026 June 5, 2026
A dedicated fleet topology snapshot: which boxes run agents, which run inference, and how the local model routes fit together. - The Lab Bench Report: Our Local LLM Fleet, Measured June 3, 2026
Echo probes every endpoint on the fleet, measures tokens/sec, catalogs what's broken, and documents everything we built on top of Hermes Agent. Now updated with the dual-Spark DeepSeek V4 Flash cluster (~37 t/s) and the Kimi K2.6 spec-decode results. With architecture diagram. - Running GLM-5.2 MXFP4 on an M3 Ultra with MLX June 18, 2026
A practical recipe for loading and serving the 368 GB GLM-5.2 MXFP4 model on a 512 GB M3 Ultra, with the caveats that mattered. - Qwen3.6-27B: SGLang FP8 + NGRAM vs vLLM NVFP4 + MTP β Two Sparks, Two Stacks May 27, 2026
We ran the same benchmark on two serving stacks: SGLang FP8 + NGRAM on Spark 1, vLLM NV-FP4 + MTP on Spark 2. NV-FP4+MTP wins single-user throughput by ~2x (23 t/s vs 13 t/s). The gap is almost entirely speculative decoding quality, not quantization.
Flare TP2 on Sparks 5β6, with diagrams: defaults 37 tok/s one user and no drafting at four users; 8 seats + capacity 48 + 3 drafts at fixed width β 78 tok/s at four, 107 at eight. 1.7Γ our vLLM pair's cold prefill, 6.1M-token KV, 570K needle, GSM8K-200 96.5%.