This is the current DeepSeek V4 Flash setup I would actually run behind Hermes: Aiden's production-v2 container, B12X MoE enabled, 393K context, and the stable deepseek-v4-flash model alias. The point is not to win a 1M-context screenshot. The point is to drive agents without making every normal turn pay the 1M scheduling penalty.
Short version: the 393K Aiden profile is live on the dual Spark cluster, tool calling works, long context works, and c6 short concurrency is healthy. Terminal-Bench is a separate problem; the first TB2 run failed because the model would not obey Terminus-2's strict JSON completion protocol, not because the endpoint failed.
| Setting | Value | Why |
|---|---|---|
| Image | aidendle94/sparkrun-vllm-ds4-gb10:production-v2 | Aiden v2 includes the prefix-cache eviction fix and B12X path used in the 1M community recipe. |
| Served name | deepseek-v4-flash | Keep Hermes and existing clients stable. Test aliases are for tests. |
| Max model length | 393216 | Enough for Hermes system prompt, skills, tool traces, and long turns without the 1M profile's concurrency hit. |
| Scheduler | max_num_seqs=6, max_num_batched_tokens=8192 | The measured middle ground for interactive agent use. |
| Memory | gpu_memory_utilization=0.82, FP8 KV, block size 256 | Keeps KV room without pushing GB10 unified memory to the edge. |
| Reasoning/tool mode | DeepSeek V4 tokenizer, reasoning parser, tool parser, auto tool choice | Hermes can use native OpenAI tool calls; thinking can be disabled per request with chat_template_kwargs. |
The 1M profile launches and works. It is also the wrong default for Hermes. At 1M context, you buy a larger headline number and pay for it in scheduling/concurrency. Hermes usually needs a big working window, but it also needs several agents and subagents to move at once.
| Profile | Context | c3 short aggregate | c6 short aggregate | 32K-ish TTFT | Operational read |
|---|---|---|---|---|---|
| May baseline | 128K | ~54.9 tok/s | ~81.4 tok/s | 25.92s | Stable, but worse long-prompt TTFT and lower c6 throughput. |
| Aiden 1M | 1,000,000 | ~71.7 tok/s | ~68.5 tok/s | 18.72s | Works, but c6 fell behind the mid-context profile. |
| Aiden Hermes | 393,216 | ~80.2 tok/s | ~124.3 tok/s | 13.20s | Best default agent-driver profile in the selection sweep. |
Selection sweep provenance: preserved June 22 setup run notes from the dual Spark cluster. The live verification numbers below were re-run while writing this post.
The host names and addresses are local to my fleet. Replace them with your own. The shape matters more than the names: worker first, head second, same container image, same model path, same NCCL interface settings.
docker run -d --name vllm_aiden --gpus all --ipc host --net host --privileged -v /home/milo/models:/models -v /home/milo/.cache/huggingface:/cache/huggingface -v /home/milo/.cache/ds4-aiden:/cache -v /dev/infiniband:/dev/infiniband aidendle94/sparkrun-vllm-ds4-gb10:production-v2 sleep infinity
export PATH="/opt/env/bin:/opt/env/nvvm/bin:/opt/env/targets/sbsa-linux/nvvm/bin:$PATH" export HF_HOME=/cache/huggingface export HF_HUB_OFFLINE=1 export TRANSFORMERS_OFFLINE=1 export VLLM_CACHE_ROOT=/cache/vllm export VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 export VLLM_USE_B12X_MOE=1 export VLLM_SPARSE_INDEXER_MAX_LOGITS_MB=256 export VLLM_NCCL_SO_PATH=/opt/env/lib/python3.12/site-packages/nvidia/nccl/lib/libnccl.so.2 export NCCL_NET=IB export NCCL_IB_DISABLE=0 export NCCL_SOCKET_IFNAME=bond0 export NCCL_IB_HCA=rocep1s0f0,rocep1s0f1 export NCCL_IB_GID_INDEX=0 export NCCL_CROSS_NIC=0 export NCCL_CUMEM_ENABLE=0 export NCCL_IGNORE_CPU_AFFINITY=1 export GLOO_SOCKET_IFNAME=bond0 export TP_SOCKET_IFNAME=bond0 export MN_IF_NAME=bond0 export UCX_NET_DEVICES=bond0 export OMPI_MCA_btl_tcp_if_include=bond0 export OMP_NUM_THREADS=8
exec /usr/local/bin/dsv4-vllm-entrypoint serve /models/DeepSeek-V4-Flash --served-model-name deepseek-v4-flash --host 0.0.0.0 --port 8000 --trust-remote-code --tensor-parallel-size 2 --pipeline-parallel-size 1 --kv-cache-dtype fp8 --block-size 256 --max-model-len 393216 --max-num-seqs 6 --max-num-batched-tokens 8192 --gpu-memory-utilization 0.82 --enable-prefix-caching --speculative-config '{"method":"mtp","num_speculative_tokens":2}' --tokenizer-mode deepseek_v4 --distributed-executor-backend mp --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --reasoning-config '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}' --override-generation-config '{"temperature":0.6,"top_p":0.95}' --default-chat-template-kwargs.thinking=true --default-chat-template-kwargs.reasoning_effort=high --enable-flashinfer-autotune --nnodes 2 --node-rank 0 --master-addr 10.0.0.1 --master-port 25000
# Same command, but rank 1 and headless: --nnodes 2 --node-rank 1 --master-addr 10.0.0.1 --master-port 25000 --headless
The live endpoint was already running from a manual docker exec launch when I hardened it, so I installed the systemd scaffold without starting it. That avoids duplicating or killing the current vLLM process. The units will supervise the profile after reboot/login.
| Node | User unit | Wrapper | Container policy |
|---|---|---|---|
| Spark1 head | ds4-aiden-hermes-head.service | ~/vllm-scripts/run-aiden-hermes-head-service.sh | vllm_aiden: unless-stopped |
| Spark2 worker | ds4-aiden-hermes-worker.service | ~/vllm-scripts/run-aiden-hermes-worker-service.sh | vllm_aiden: unless-stopped |
Both milo users have linger enabled. The wrappers refresh the container's /tmp/aiden-hermes-*.sh from the host copy before starting vLLM, then keep docker exec in the foreground so systemd can restart it on failure.
These are operational probes, not leaderboard scores. They measure the endpoint as Hermes sees it over the OpenAI-compatible API with chat_template_kwargs: {"thinking": false} unless noted.
| Probe | Result | Latency | Notes |
|---|---|---|---|
| Plain chat | OK | 0.188s | No reasoning field when thinking is disabled. |
| Native tool call | add({"a": 19, "b": 23}) | 1.629s | Returned a real OpenAI tool_calls[] entry, not prose. |
| Long context sentinel | Hermes long-context sentinel abc123 | 0.609s | 60,030 prompt tokens. Prefix cache likely helped this repeat run. |
| c6 short smoke | 6/6 OK | median 0.365s | Six simultaneous exact-response requests. |
| Workload | Prompt tokens | Output tokens | TTFT | Elapsed | Decode after TTFT | Wall tok/s |
|---|---|---|---|---|---|---|
| Short c1 | 31 | 256 | 1.214s | 7.074s | 43.68 tok/s | 36.19 tok/s |
| 23K prompt c1 | 23,419 | 114 | 11.082s | 14.066s | 38.2 tok/s | 8.1 tok/s |
| 94K prompt c1 | 93,619 | 64 | 36.048s | 37.712s | 38.46 tok/s | 1.7 tok/s |
| c3 concurrent short decode | 21 each | 426 total | median 1.953s | 7.62s wall | per-request mixed | 55.91 aggregate tok/s |
| c6 concurrent short decode | 21 each | 864 total | median 0.237s | 8.222s wall | per-request mixed | 105.09 aggregate tok/s |
The short c1 decode number is the one to compare with the older DS4-F bring-up post: 43.68 tok/s after TTFT. The c6 aggregate number is the more relevant Hermes number: 105.09 output tok/s across six simultaneous short agent-like requests.
I would not publish a Terminal-Bench 2 score for this profile yet. The endpoint was healthy during the first run: no server-side InternalServerError, RateLimitError, or BadRequestError. The failure was protocol fit. Terminus-2 requires strict JSON-only responses with task_complete: true; the Hermes-tuned profile kept doing useful shell work but never marked tasks complete, and it wrapped JSON in Markdown or extra text.
That is a benchmark-agent mismatch, not a serving-stack failure. The fix is a separate benchmark profile with strict JSON-only behavior, then a small pilot before committing to another 80-task run. Do not mix that diagnostic failure into a public score table.
~/.hermes/config.yaml.bak-ds4-aiden-20260622-180028.systemctl --user disable ds4-aiden-hermes-head.service on Spark1 and systemctl --user disable ds4-aiden-hermes-worker.service on Spark2.drop_caches. Use cache drops only as part of a controlled teardown/relaunch on GB10 unified memory.Measured on June 22, 2026 against the live Spark1 API endpoint. The API advertised deepseek-v4-flash with max_model_len=393216. Streaming throughput probes used Python's stdlib HTTP client and OpenAI-compatible /v1/chat/completions.