MiMo 2.5 Test: A Useful Failure on Two DGX Sparks

We tested MiMo V2.5 NVFP4 on the dual-DGX Spark pair to see whether it could become a practical local agent route. The June run did not earn promotion: MiMo is not replacing DS4-F, and we are not keeping a MiMo service up from that run. The July 1 upstream reproduce changed the lesson again: the public Tony/Karol lane appears real, but our Sparks did not expose the same usable KV-cache memory envelope.

Why this was hard

MiMo V2.5 NVFP4 is not just another text-only LLM load. The model is multimodal, uses experimental quantization paths, and the useful community recipes depend on patched vLLM behavior that is ahead of normal upstream releases. On the Sparks, the weights can fit while the engine still dies during cache profiling.

The important lesson: our early failures were not primarily NCCL, not raw model weight size, and not just max context. The model loaded. The failure boundary was KV plus multimodal encoder cache reservation.

July 1 upstream reproduce: the KV envelope did not match Tony

The fair retest did not make MiMo look better locally. It made the failure cleaner. The public Tony/Karol result depends on a much larger usable NVFP4 KV-cache pool than our Spark pair exposed under the same recipe family.

That points away from “one more launch flag” and toward an effective memory-envelope mismatch: driver/runtime allocator behavior, multimodal encoder reserve, and rank asymmetry. Our nodes were on driver 590.48.01 with CUDA 13.0.x packages; the community reports are stack-sensitive, and forum notes say driver branch can materially change MiMo KV pool size.

Current call after the July 1 reproduce: MiMo remains off the route. Do not blind-relaunch 1M. First reproduce Tony's 65K full-Omni KV pool; if we do not see roughly the 2.4M-token class there, 1M is dead on this stack.

Topology

The profiles that booted

This was the historical 128K bring-up command. It proved the patched runtime could serve MiMo, but it did not survive as the final fleet route. The final best-measured profile was the narrower 64K text-only/no-MTP route described below.

MAX_MODEL_LEN=131072 \
GPU_MEMORY_UTILIZATION=0.89 \
MAX_NUM_SEQS=1 \
MAX_NUM_BATCHED_TOKENS=2048 \
BLOCK_SIZE=64 \
ENABLE_THINKING=false \
./launch-cluster.sh

The launcher also had to be adapted for this pair:

HEAD_ROCE_IP=10.0.0.1
WORKER_ROCE_IP=10.0.0.2
ROCE_IFACE=bond0
GLOO_SOCKET_IFNAME=bond0
NCCL_SOCKET_IFNAME=bond0
NCCL_IB_HCA=rocep1s0f0,rocep1s0f1
NCCL_IB_GID_INDEX=3

And one upstream/fixed-fork wrapper issue mattered locally: the wrapper needed to pass MAX_NUM_BATCHED_TOKENS and BLOCK_SIZE through to recipe/launch.sh. Without that, changing those knobs in the outer command is fake control.

What failed before it worked

Proof from the measured runs

The final successful server profile before cleanup was the narrow 64K text/tool route, not vision and not 500K/1M. The live command was verified without --speculative-config, with multimodal inputs reserved to zero:

--max-model-len 65536
--gpu-memory-utilization 0.91
--max-num-batched-tokens 1024
--max-num-seqs 1
--block-size 64
--limit-mm-per-prompt {"image":0,"video":0,"audio":0}
# no --speculative-config

The 64K restore smoke passed:

READY 709.9 MiMo-V2.5-NVFP4 65536
chat_content= OK
tool_calls= get_weather({"city":"Boston"})

The 96K climb also booted, but that was not enough. It reported a positive KV profile and passed smoke, then lost quality on the short tool benchmark:

READY MiMo-V2.5-NVFP4 98304
GPU KV cache size: 535,375 tokens
Maximum concurrency for 98,304 tokens per request: 5.45x
short tool benchmark: 87

Measured June 30, 2026 from the Spark1 vLLM endpoint and container logs. After the final write-up, the MiMo containers and Ray processes were stopped so the Sparks are available for new experiments.

Benchmarks: speed probes and tool-call quality

After the server was already live, I ran a small sequential streaming probe from Spark1 against 127.0.0.1:8000. For prefill, each request used max_tokens=1; the rate below is prompt_tokens / time_to_first_content_token. That includes first-token overhead, so treat it as an operational prefill/TTFT number, not a pure kernel counter.

June 30 correction: the first follow-up run reused the same long prompt prefix and hit Automatic Prefix Caching, producing absurdly high “prefill” rates. I kept that artifact in the raw results, then fixed the harness by putting a nonce at the start of each long prompt and reran the baseline. The table below is the cold-prefix truth.

The shape is unchanged but cleaner now: MiMo holds roughly 1.6k prompt tok/s through a 10.5K-token cold-prefix probe, then drops to roughly 823 prompt tok/s around 41.5K prompt tokens. Short decode is stable but not fast, around 28 tok/s. These are single-client measurements on a memory-tight 128K profile; concurrency and longer decode tests remain separate work.

Hermes Agent tool-call smoke

The more important new result is not raw speed. I tested direct OpenAI-compatible tool calling with a tiny Hermes-like tool schema: search_files, read_file, and terminal, plus a no-tool explanation case. The endpoint returned structured tool_calls[] rather than JSON-in-text.

Decision after the June 30 benchmark: stop treating MiMo as a promotion candidate. The tool syntax is good, but the full-agent behavior is not reliable enough, the serving profile is too narrow, and DS4-F remains the default.

Formal tool benchmark: best case was still not enough

The formal benchmark used tool-eval-bench 2.0.7 against the live OpenAI-compatible vLLM endpoint. This is a different serving profile from the 128K baseline above: MAX_MODEL_LEN=65536, GPU_MEMORY_UTILIZATION=0.91, MAX_NUM_SEQS=1, MAX_NUM_BATCHED_TOKENS=1024, BLOCK_SIZE=64, multimodal reservation disabled, thinking off, and MTP/speculative decoding disabled. I verified the live vllm serve command had no --speculative-config.

The failures were useful. They were not random “model is bad” misses. They cluster around agent-control behavior: when not to use a tool, remembering corrected obligations, sending the final email after lookup, and preserving multi-turn state. That made MiMo interesting for targeted canary work, but after the 96K regression and cleanup decision, the practical outcome is failure for this experiment.

64K canary throughput probe

Artifacts: ~/.hermes/work/mimo-opt/results/mimo_64k_no_mtp_gmu091_short.json, mimo_64k_no_mtp_gmu091_full.json, mimo_64k_no_mtp_gmu091_report.md, mimo_96k_no_mtp_gmu091_short.json, and mimo_96k_no_mtp_gmu091_report.md. These numbers were measured June 30, 2026, not inferred from the earlier 128K run.

# benchmark provenance
endpoint: 127.0.0.1:8000 on Spark1
model: MiMo-V2.5-NVFP4
historical speed profile: 131072 ctx, GMU .89, seq1, batched 2048, block64, video=0
formal tool profile: 65536 ctx, GMU .91, seq1, batched 1024, block64, text-only reserve, no MTP
method: streamed /v1/chat/completions with stream_options.include_usage=true for speed probes
cold-prefix guard: nonce at the start of each long prompt
prefill metric: prompt_tokens / TTFT, max_tokens=1
decode metric: completion_tokens / (total_time - TTFT)
tool benchmark: tool-eval-bench 2.0.7, temperature 0, top_p .95, repetition_penalty 1.08
raw artifacts: ~/.hermes/work/mimo-opt/results/

Next recommended test

The important correction after the July 1 research is this: our measured lane was not the lane now showing up well on public Spark-Bench results. We tested 64K/96K, text-only, MAX_NUM_SEQS=1, true no-MTP. The public recipe that deserves a fair local attempt is current upstream, NVFP4 KV, MTP_SPEC_TOKENS=1, VLLM_MIMO_MTP1_GREEDY_FAST=1, MAX_NUM_SEQS=8, and a 1M context ceiling.

# upstream lane to reproduce first, not a promotion config
MAX_MODEL_LEN=1000000
MAX_NUM_SEQS=8
MAX_NUM_BATCHED_TOKENS=2048
BLOCK_SIZE=64
GPU_MEMORY_UTILIZATION=0.84
MTP_SPEC_TOKENS=1
VLLM_MIMO_MTP1_GREEDY_FAST=1
ENFORCE_EAGER=1
DEFAULT_TEMPERATURE=0
DEFAULT_TOP_P=0.95
REPETITION_PENALTY=1.08

The fail criteria are just as important as the pass criteria: if the 65K MTP1 lane cannot boot cleanly twice from cold cleanup, do not climb to 1M. If a killed client leaves num_requests_running=1 forever, do not route Hermes to it. If Spark-Bench looks good but our tool loop regresses, keep it as an Omni demo, not an agent route.

Final result

The final result of the June run is not “almost 1M.” It is: we could make MiMo run, but not in a form that earns a place in the fleet. The successful serving point was narrower than the model’s promise, and the quality/speed profile did not beat the existing default. The next experiment should be a fresh upstream reproduction, not more tuning of this old no-MTP fallback.

Final call on the June run: useful failure, not promoted. Next action if we revisit MiMo: clean upstream MTP1/NVFP4-KV reproduce first, then benchmark.

Why we stopped this run

The remaining optimization ideas are real, but they do not justify keeping the old fallback experiment hot. The mistake would be to keep patching around the 64K no-MTP lane. The better move is to restart from current upstream and test the same MTP1/NVFP4-KV lane that public results are actually measuring.

Final cleanup state after the July 1 update: MiMo vLLM/Ray containers stopped on both Spark nodes, no MiMo API listening on Spark1 :8000, and the hardware returned to DS4-F recovery/default-route work.

Source trail: local artifacts under ~/.hermes/work/mimo-opt/results/, Tony's public MiMo TP=2 1M NVFP4-KV repo, Wesche's Spark-Bench board/post, and the NVIDIA developer forum thread. Upstream claims remain useful context; the local June decision is based on the measured Spark runs above, and the recommended next test is explicitly a fresh attempt to reproduce the upstream lane.