Local LLMs

Current fleet, model serving, deployment recipes, and local benchmark notes.

Station-specific GB300 recipes now have their own index: DGX Station GB300. M3 Ultra 512 GB measurements: M3 Ultra 512 Benchmarks. This page still lists them among the rest of the local-LLM posts.

• DeepSeek-V4-Flash-Vision-Exp on One GB300: Native FP4, SGLang, DSpark, 1M Context September 3, 2026
  Single DGX Station TP=1, no requantization: 2,420 tok/s at C32, exact recall to 810K, zero repetition at C64. Recipe + harness on GitHub.
• A Field Guide to DeepSeek V4 Flash Thinking Levels September 1, 2026
  What off/low/high/max actually do: prompt prefixes, not budgets.
• Vision-Exp thinking levels on 2× DGX Spark September 1, 2026
  Matched off/low/high/max thinking probes. Effort labels are not a ladder.
• DeepSeek-V4-Flash-Vision-Exp on Two DGX Sparks August 31, 2026
  Mia Vision-Exp recipe on Funland: native image through vLLM on 2× DGX Spark, 1M context.
• NVFP4 vs EXL3: GLM-5.3-Flash on Two DGX Sparks August 29, 2026
  Matched quant faceoff on one Spark pair; cold-prefill protocol included.
• GLM-5.3-Flash EXL3 on 2× DGX Spark — August 28, 2026
• GLM-5.3-Flash DFlash2 on 2× DGX Spark August 28, 2026
  Pinned Tony DFlash2 + Inco drafter on the Spark pair: mixed C1 41.5 tok/s, 20/20 tools, 98K retrieval, recipe and rollback.
• GLM Flash 5.3 Testing — Dual DGX Spark August 27, 2026
  Day-0 SGLang bring-up of GLM-5.3-Flash NVFP4: GB10 memory trap, 20-hop gauntlet, 24.7 tok/s prose.
• DeepSeek Flash 0731 Testing: weights vs thinking level August 26, 2026
  2×2 A/B on dual Sparks: official vs ablit, low vs high thinking.
• Local LLM Fleet: August 2026 August 24, 2026
  Fresh topology snapshot with live endpoint probes: dual-Spark DS4-F Keys-anchored agent lane, M5 Max vision and retrieval, M3 Ultra voice lane.
• Funland DS4-F is on Keys’ anchored 0731 ablit August 23, 2026
  Current dual-Spark 0731 recipe with Keys anchored tensors.
• Milo-Ark: A local AI repo June 18, 2026
  Pinned Hugging Face snapshots with checksums and provenance on miloshare.
• Local LLM Stack: Current Architecture and Benchmarks May 28, 2026 · updated September 3, 2026
  DeepSeek-V4-Flash-Vision-Exp is the single model everywhere: Reachy, the agent family, and (soon) Stackchan all route to it on the GB300. The Spark pair mirrors it as backup, M3 Ultra is the open test bench. Retire + speed-migration analysis on select/embed/rerank. Fresh measure 173 tok/s single-stream, 2,420 tok/s at C32.
• July DS4-F Status: Patch3 350K/12 Live July 4, 2026
  Patch3 scheduler fix applied: 350K/12 is now live after winning today’s C1 probe at 56.8 tok/s; patched 1M/6 remains the long-context lane.
• July SGLang Testing: Qwen3.6 on DGX Spark July 1, 2026
  Early July SGLang results for Qwen3.6 on the DGX Spark pair: 27B-FP8 worked but was slow; 35B-A3B-FP8 was much faster and scored better, but both failed the sleeper-injection safety gate.
• June DS4-F Testing June 30, 2026
  Archive summary of the DS4-F work: June Aiden 393K results plus July 1 DSpark speed numbers — current 200K/8 route, c8 209.66 tok/s, and 206.56 tok/s soak.
• June MiMo Testing June 30, 2026
  Updated July 1: MiMo stayed off-route. The upstream reproduce exposed the real gap — our Sparks saw 954K KV tokens versus Tony/Karol's 2.17M+ pool — so DS4-F remains default.
• DeepSeek V4 Flash on Dual DGX Spark: What Broke, and the Recipe That Works May 27, 2026
  149 GB model across two 128 GB nodes. TP=2 over 200 Gbps QSFP56. MTP speculative decoding (1.76× speedup), 200K context, thinking mode, tool calling. Full YAML recipe, the six things that broke, and measured performance — 44.5 tok/s decode, 612K KV cache.
• Local LLM Fleet: June 2026 June 5, 2026
  A dedicated fleet topology snapshot: which boxes run agents, which run inference, and how the local model routes fit together.
• The Lab Bench Report: Our Local LLM Fleet, Measured June 3, 2026
  Echo probes every endpoint on the fleet, measures tokens/sec, catalogs what's broken, and documents everything we built on top of Hermes Agent. Now updated with the dual-Spark DeepSeek V4 Flash cluster (~37 t/s) and the Kimi K2.6 spec-decode results. With architecture diagram.
• Running GLM-5.2 MXFP4 on an M3 Ultra with MLX June 18, 2026
  A practical recipe for loading and serving the 368 GB GLM-5.2 MXFP4 model on a 512 GB M3 Ultra, with the caveats that mattered.
• Qwen3.6-27B: SGLang FP8 + NGRAM vs vLLM NVFP4 + MTP — Two Sparks, Two Stacks May 27, 2026
  We ran the same benchmark on two serving stacks: SGLang FP8 + NGRAM on Spark 1, vLLM NV-FP4 + MTP on Spark 2. NV-FP4+MTP wins single-user throughput by ~2x (23 t/s vs 13 t/s). The gap is almost entirely speculative decoding quality, not quantization.