October 28, 2026 · James Meadlock & Milo (revision, probes and numbers on session model deepseek-v4.1-flash-uva via custom; reasoning setting not recorded) · living architecture note

Local LLM Stack: interactive primary, isolated batch, dedicated auxiliaries

Created · Updated

CURRENT CALL Split the fleet by workload shape, and protect the fast machine. One interactive lane on the GB300 serves conversation and tool loops. Four Sparks take everything document-sized. A second four-Spark lane (vLLM on .55:8002) now carries its own batch work. Five M5 ports do the small specialist jobs. Nothing about this week's numbers argues for sharing the interactive lane with batch work — the interference measurement below still costs a 226 ms reply its 13×.
First token36 msmedian, idle, 10 samples
Single stream258 tok/s400-token completion
8 streams926 tok/saggregate, 148 per stream
Spark lanes2 of 2 up.51–.54 SGLang · .55–.58 vLLM
Fleet as checked October 28, 2026 listeners and health probes, read live; not a benchmark GB300 Station · interactive · UP :30006 · dsv41-flash-uva · vLLM · 1M ctx · cold experts on the RTX 6000 36 ms first token 4× DGX Spark .51–.54 · batch · UP :8888 · deepseek-v4.1-flash · SGLang TP=4 · 1M ctx 27 h uptime 4× DGX Spark .55–.58 · batch B · UP :8002 · deepseek-v4.1-flash-spark-vllm · vLLM TP=4 · 1M ctx 4 ranks M5 Max .18 · auxiliaries · UP :8002 embed · :8003 rerank · :8011 / :8016 catalog · :8021 gated 5 ports

Every tier probed from the M4 on October 22, 2026 between 11:00 and 11:15 CDT. The station's :30001 and :30003 lanes refused connections, as did .55:8002.

The architecture in one paragraph

The stack is deliberately asymmetric. The fastest box is not asked to do every job; it is protected for the jobs where latency changes the experience — conversation, tool loops, robots, family agents. Batch work is large and can wait, so it lives on separate silicon where a 134k-token prefill hurts nobody. Small specialists — embedding, reranking, selection, vision — are cheaper and more predictable as dedicated services than as a general model that may reload state. Experiments do not get a new box; they borrow a production lane inside an agreed window and restore afterwards.

The governing rule is unchanged: the fastest machine is protected, not saturated. Fleet utilisation is secondary to interactive reliability.

What is serving, checked live

HostEndpointRoleState today
GB300 Station192.168.1.9:30006
dsv41-flash-uva
Interactive text primary. vLLM, 1,048,576-token context: 295 hot experts per layer in HBM, the 89 coldest computed on the RTX PRO 6000 over PCIe, Engram in CPU memory.Up. 36 ms median first token, 258 tok/s single stream, 926 tok/s aggregate at 8 streams (three reps: 900–943).
GB300 Station:30001 · :30003GLM-5.3-Flash lane and the older DSF lane.Both refused connections. Swapped out, not deleted.
4× DGX Spark
.51–.54
192.168.1.51:8888
deepseek-v4.1-flash
Batch and overflow. SGLang, TP=4, 1M context, RoCE fabric, container dsv41-head on image dsv41-4x-spark:canary-roce.Up 27 hours, healthy. The far end of the CRS804 fabric.
4× DGX Spark
.55–.58
.55:8002Second batch lane. vLLM, TP=4, 1M context, RoCE fabric.Up. .55:8002 lists deepseek-v4.1-flash-spark-vllm at 1,048,576 tokens; .56–.58 are ranks and serve no HTTP.
M5 Max .18:8002 · :8003Embedding 8B (4,096-dim) and reranker 4B. Single-model services.Up, both return 200.
M5 Max .18:8011 · :8016 · :8021Vision, selection, and a key-gated 27B lane.Up. :8021 correctly returns 401 without a key.

The M5 ports are a catalog, not one model each

This is the detail that changes how you call them. :8011 and :8016 both answer /v1/models with the same six-model catalog — Qwen3.5-4B, a Qwen3.8-27B abliterated oQ4e build, gemma-4-26b-a4b, Qwen3-VL-30B-A3B, Qwen3-Embedding-8B and Qwen3-Reranker-4B — because they are multi-model servers reading one model directory, not one pinned process per port. A request can pay a model load inside a port; a port name is not a promise about what is resident. Only :8002 and :8003 are genuinely single-model.

Interactive lane: today's numbers

Ten-sample probes against the live :30006 endpoint, all from the M4 during business hours on a shared lane. Percentages and rates are defined below; nothing here is a leaderboard entry.

MeasureTodayHow it was taken
First token, idle36 msmedian of 10 streaming requests, 48-token cap, first streamed delta (reasoning included)
Short turn complete162 msmedian of 10 × 40-token turns, reasoning on
Single-stream decode258 tok/sone 400-token completion, completion tokens ÷ wall
8-stream aggregate926 tok/s8 concurrent 400-token completions, 148 tok/s per stream; three reps 900–926–943. The same instrument read 1,172 on October 22, and the recipe changed in between (block verification, the k-schedule at five or more streams), so this is a number to chase, not a verdict
Context window1,048,576max_model_len from the model list
GOTCHA Reasoning tokens are part of the completion budget. A 48-token request came back with reasoning deltas and zero content — and those deltas arrive under delta.reasoning, not delta.reasoning_content, so a client that reads only content sees an empty stream. A 400-token request produced 177 tokens of which 50 were reasoning; asking for reasoning_effort=low produced 59 reasoning tokens, not fewer. Any caller that caps max_tokens tightly will occasionally get an empty answer and a finish_reason of length.

Why long prefills stay off the interactive lane

This is the measurement the whole architecture rests on, and it was taken on October 20, 2026 — it has not been re-run today, so it stays dated rather than restated as current.

Condition on GB300 :30006 (October 20, dsv41-flash-uva, vLLM)Measured result
Short interactive turn, idle, clean samples53–66 ms first token; 65 ms median; complete 40-token turn 226 ms
Long nonce-prefixed request (cold prefill)133,647 prompt tokens in 7.37 s — 18,134 tok/s
Short turn dispatched 0.6 s into that prefill3.46 s complete (15.3× idle); first token at 679 ms
Short turn dispatched 4.4 s in2.94 s complete (13.0× idle); first token at 931 ms
Short turn arriving as the prefill ends271 ms (1.2×)
Short turns after the long request completed270, 293, 217 ms — full recovery

The practical size of the blast radius: a 134k-token prompt is about 7.4 seconds of prefill, and any conversation sharing the lane turns a 226 ms reply into roughly 2.9–3.5 seconds. At 1M tokens the wait becomes tens of seconds. That is not a throughput concern, it is a user-visible one — which is why extraction, compression, triage, kanban decomposition and curator work live on the Sparks. The Spark cluster is not idle redundancy; absorbing prefills is its day job.

Current agent routing

ClientNormal routeLocal auxiliariesFallback
Echo / Forge gatewayStation :30006Spark lane for extraction and compression; M5 :8016 for skills and titles; M5 :8021 for auxiliary visionSpark lane .51–.54
Milo / Desktopdsv41-flash-uva — this note was written and measured on that laneM5 :8016 for skills and profile descriptionsCloud profile fallback
Family minisCloud Sonnet 5.5 via the Nous inference APINone activeCloud
ReachyBroken. The running brain process (pid 8899) still launches with --model_name dsf-vision-exp --responses_api_base_url http://192.168.1.9:30003/v1, and :30003 refuses connections.None — no s2s fallbackNone. Voice turns retry until the brain is repointed.

Routing is allowed to differ per client. A local fleet is a set of capabilities, not a rule that every agent share a route, and profiles migrate deliberately rather than all at once. Reachy is the standing example of the cost: its launch script was never repointed after :30003 was swapped out, and the process has been restarting against a dead lane since.

Operating rules

Method and provenance