July DS4-F Status: Patch3 350K/12 Live
This page is the current DS4-F operating note. It tracks what profile is live on the dual DGX Spark route, which trials produced real measurements, and what not to over-claim from the DSpark chase.
July 4 call: DS4-F is now running the Tech2Wild/Tony Patch3-style profile at 350K/12. That profile gave the best single-client C1 result in today’s trial: 56.8 tok/s, beating patched 1M/6 at 54.4 tok/s and the old live 1M/6 baseline at 46.8 tok/s. 1M/6 remains the long-context lane; 200K/16 remains the older high-concurrency fallback.
Running now350K/12
Best C1 today56.8 tok/s
Draft acceptance0.626
KV pool975,835
Long-context lane1M/6
Current running profile
| Item | Current value |
|---|
| Status | live route 350K/12, selected because James asked to keep whichever profile produced the best C1 performance. |
| Model | deepseek-v4-flash-dspark |
| Endpoint | Spark head OpenAI-compatible endpoint, port :8888 |
| Runtime profile | MAX_MODEL_LEN=350000, MAX_NUM_SEQS=12, MAX_NUM_BATCHED_TOKENS=8192, GPU_MEMORY_UTILIZATION=0.80, MTP_NUM_TOKENS=5, MAX_CUDAGRAPH_CAPTURE_SIZE=12 |
| Patch set | Patch3 scheduler guard bind-mounted; DSpark draft sampling set to probabilistic; old server-side repetition_penalty=1.05 override removed; VLLM_USE_FLASHINFER_SAMPLER=1. |
| Fresh live proof | /v1/models max_model_len=350000; exact smoke returned DS4F_350K12_LIVE_OK; repeated C1 benchmark best was 56.8 tok/s. |
| Startup proof | GPU KV cache size: 975,835 tokens; Maximum concurrency for 350,000 tokens per request: 2.79x; Application startup complete. |
July 4 Patch3 trial
The interesting change from Tech2Wild’s post was not simply “MTP5” — our live lane already used five speculative tokens. The useful change was the scheduler-level Patch3 guard plus probabilistic DSpark draft sampling and removal of the repetition-penalty override.
| Profile | C1 result | Acceptance | Decision |
|---|
Old live 1M/6 | 46.8 tok/s | 0.483 | Baseline only. This was the pre-Patch3 path. |
Patched 1M/6 | 54.4 tok/s | 0.591 | Works and remains the long-context lane. Startup proof: max_model_len=1048576, KV 1,665,106, full-context concurrency 1.59x. |
Patched 350K/12 | 56.8 tok/s | 0.626 | selected live Best C1 in this trial, with more agent-friendly slot count. |
Gotcha we hit: for MTP5, setting the cudagraph capture cap too low crashed startup with No valid cudagraph sizes after rounding to multiple of 6. Setting MAX_CUDAGRAPH_CAPTURE_SIZE=12 fixed the launch. vLLM logs may later truncate the effective max capture size to 8, but the larger configured value is what let startup pass.
Profile history and fallback map
| Profile | Status | Observed result | Operational read |
|---|
350K/12 | running now | Patch3 profile booted, smoked, and repeated C1 best at 56.8 tok/s. | Current best-C1 agent lane. |
1M/6 | long-context lane | Patch3 profile booted and produced 54.4 tok/s C1; previous isolated live probes showed c1/c4/c6 viability. | Use when the million-token ceiling matters more than the small C1 edge. |
200K/16 | older fallback | Previously booted, smoked, and benchmarked with strong C16 behavior. | Still the known short/mid-context high-concurrency fallback if the Patch3 lane misbehaves. |
1M/12 | failed bind | Earlier clean retry had enough KV but failed after warmup in the shared-memory / broadcast path. | Do not retry unchanged. Any new try needs the Patch3/new-compose path and bounded rollback gates. |
1.5M/12 | failed bind | Earlier run loaded weights and profiled KV, then never bound the API. | Not a production candidate; also beyond the model’s 1,048,576 YaRN-calibrated ceiling. |
What changed in the live files
| Change | Why it matters |
|---|
| Patch3 scheduler guard | Prevents DSpark spec placeholders from being resized on chunked prefill chunks, the root fix for cold long-prompt garble / prompt echo. |
draft_sample_method=probabilistic | Matches the Tech2Wild/Tony recipe and improved acceptance in today’s route trial. |
Removed --override-generation-config | Drops the old server-side repetition_penalty=1.05, which is a DSpark spec-decode crash risk rather than a garble fix. |
MAX_CUDAGRAPH_CAPTURE_SIZE=12 | Needed for MTP5 startup on this image; lower sizing produced the cudagraph rounding failure above. |
VLLM_USE_FLASHINFER_SAMPLER=1 | Part of the Jul 4 serving recipe now persisted in the default env. |
What not to over-claim
| Claim | Reality |
|---|
| “350K/12 is globally faster than every other profile.” | No. It won today’s C1 probe. C4/C8/C12 curves still need fresh measurement before making wider throughput claims. |
| “1M/6 is obsolete.” | No. It is still the million-token lane. The 350K route is a better default when short agent turns and C1 speed matter more. |
| “MTP5 was the new discovery.” | No. We already had MTP5. The root change was Patch3 + probabilistic draft + no repetition override. |
| “1M/12 works now.” | Not proven. The old unchanged 1M/12 path failed; a Patch3 1M/12 retry remains a separate experiment. |
Next tests
| Next step | Why | Pass condition |
|---|
Run c4/c8/c12 on patched 350K/12 | C1 picked the default, but fleet behavior depends on concurrency. | All streams complete, queue returns idle, no garble, and acceptance stays in-family. |
Keep patched 1M/6 ready | The 350K lane gives up the 1M ceiling. | Documented env switch and smoke path remain boring. |
Only retry 1M/12 with changed variables | Unchanged retries already point at SHM/startup behavior, not raw KV. | Bind, exact smoke, and queue-idle proof under a bounded startup window. |
Provenance
| Item | Value |
|---|
| Jul 4 live endpoint | http://192.168.1.11:8888/v1, model deepseek-v4-flash-dspark, max_model_len=350000. |
| Smoke marker | DS4F_350K12_LIVE_OK from an OpenAI-compatible chat request issued after the route switch. |
| Bench method | benchmarks/keys-concurrency/bench_concurrent.py http://127.0.0.1:8888 1, best of two measured rounds after warmup. |
| Runtime | vllm-dspark-runtime:dspark-nvfp4-stage-c; DSpark TP=2 across the Spark pair; B12X MoE backend confirmed in logs. |
| Working directory | /home/milo/ds4-f-wesche-keys-c12 |
| Persisted files | .env.dspark and docker-compose.dspark.yml now carry the patched 350K/12 defaults on Spark1 and Spark2. |
| Earlier reports retained for history | /home/milo/ds4-f-wesche-keys-c12/benchmarks/20260703-live-1m6-isolated-c1-c4-c6-20260703-094345.txt
/home/milo/ds4-f-wesche-keys-c12/benchmarks/20260702-live-1m6-c6-bench-20260703-092526.txt |