Speculative Decoding by Traffic: v14 Wins the Matched Replay
Round 7: the advantage survives matched requests and reversed boot order
The missing comparison is finished. I ran the same recorded-history requests on both schedules, with three independent boot pairs and the middle pair reversed. v14 led in every pair. The recommendation stays narrow: keep v14 for this four-worker recorded-history workload; do not turn a replay into a claim about how many production agents the machine can serve.
| Measurement | v14 | v13 | Comparison |
|---|---|---|---|
| Initial-namespace replay | 231.7 tok/s | 196.2 tok/s | +18.1% mean paired throughput |
| Identical-request repeat | 260.4 tok/s | 217.8 tok/s | +19.5% mean paired throughput |
| Twelve-task suite wall time | 11.78 s | 14.79 s | 20.3% shorter, from profile means |
| Task successes | 36/36 attempts | 36/36 attempts | Twelve unique tasks, three repetitions each |
The three initial-replay gains were 18.7%, 17.5%, and 18.2%; repeat gains were 20.3%, 18.6%, and 19.7%. These are descriptive paired results, not a significance test. The initial namespace already reuses prefixes within a sequence; the second pass repeats identical requests rather than continuing a live session.
The tradeoff is first-token latency. Per-run median TTFT was 0.356–0.357 s for v14 versus 0.326–0.330 s for v13 initially, and 0.224–0.226 s versus 0.187–0.191 s on repeats. Higher aggregate output rate did not mean a faster first token.
The replay contains twenty recorded context sequences, fifteen serial turns each, four workers, temperature 0, seed 42, reasoning low, and a 400-output-token cap. It is a flattened history proxy: its tool calls are not executed and generated answers do not become the next request's history. The separate tool suite does execute bounded tools and feed results back: four file chains, four code repairs, and four structured-answer tasks.
Audit: all 3,600 replay captures and 72 task attempts are present. Request hashes match within pairs; raw SSE token usage reconciles with capture rows and recorded server deltas. All logged completion requests are accounted for by the harness and came from loopback. The frozen grader reproduced task correctness; recorded sandbox and code-oracle checks passed. No final task, schema, protocol, capture, or budget failures were observed; intermediate failed tool evaluations were retained, not retried as new scores.
What remains unproven: broad quality equivalence and production capacity. Excluding tool-call IDs, 279–290 of 300 paired response records differed per comparison. Each profile reached the replay output cap on 199/1,800 responses. The twelve repeated tool tasks are a small competence check, not 72 independent tasks. Raw private history and completions remain private, so that replay cannot be independently reproduced from the public bundle alone.
Collection finished September 15, 2026 at 12:24 CDT. Both preserved reference containers are stopped; no other serving lane was restored. Full per-run ranges, TTFT tails, task results, audit details and source provenance: Round 7 public-safe evidence bundle. The existing v14 configuration is unchanged.
Historical v13 prose-knee results
Why one number couldn't win
Two days ago the main post ran a speculation-depth sweep and found something inconvenient. With the drafter guessing five tokens at a time (k=5), agent text — shell, code, tool JSON — decodes a third faster than with k=1, because those tokens are predictable and the drafter is usually right. But at eight or sixteen concurrent requests, k=1 beat k=5 by 23–33% on every point. No single k won both. We shipped k=5 because our lanes are agent lanes, and wrote the multi-stream loss down as the price.
The reason is the one this hardware keeps teaching us. Each verify step on our lane costs experts: a five-token draft window touches 3.7× the unique experts of a single token, and a quarter of those experts live in Grace memory across NVLink-C2C. At one stream, that cost is paid for by the tokens the draft wins. At sixteen streams, the batch already fills the step with real tokens — the GPU is busy, the expert traffic is already high — and speculative drafts are mostly just more experts to fetch for guesses the batch didn't need. The draft that is a bargain when the machine is idle is a tax when it's full.
So the fix isn't a better k. It's a k that knows how busy the machine is.
The schedule
vLLM's speculative config has a field for exactly this, num_speculative_tokens_per_batch_size: a list of (range_start, range_end, k) triples, inclusive, keyed on the number of sequences running in the step. We set:
--speculative-config '{"method":"dspark","num_speculative_tokens":5,
"num_speculative_tokens_per_batch_size":[[1,2,5],[3,16,1]]}'Five tokens at one or two streams, one token at three through sixteen. That is the entire diff from v12 to v13. The drafter, verifier, weights and kernels are untouched; the schedule only changes how many draft tokens are proposed each step, and the target verifier remains in place. That design intent is not a substitute for measuring protocol correctness and useful task quality on the implemented serving path.
Figure 1. Left of the dashed line the schedule is k=5 and the bars are identical. Right of it the schedule is k=1 and every point gains 26–35%. The C16 candidate measured 430.8 and 427.1 in the two pairs; controls 317.6 and 320.0.
How it was measured
Same protocol as everything else in this series, because the lane's baseline drifts a few percent day to day and a single before/after is worthless here. Each pair is: boot the v12 reference container, run the six-point concurrency sweep (knee.sh, two runs per point, ~1% spread) and the single-stream category fixture; stop it; boot the candidate with only the schedule added; run the same two instruments; stop it. Then do the whole thing again. The bar, set before the run: single-stream within ±1.5% of control and C8/C16 at least +15%, on both pairs.
Both pairs cleared it with room. The candidate's own numbers agree across pairs to within 1% at all six points, which is as tight as this instrument gets. One flag for honesty's sake: the tool-JSON fixture class came in 7% lower than control in both pairs (146.8 and 146.5 vs 157.6 and 158.7) with acceptance identical. That fixture runs single-stream, where the schedule is k=5, so the schedule shouldn't touch it; and across today's four control boots that class ranged 145–159, so it is inside its own spread. Consistent sign though. It's noted in the results README rather than explained away.
A small pleasant surprise: this schedule reused vLLM's cached FlashInfer autotune. We had budgeted 80 minutes for a fresh MoE kernel tune; the candidate booted in 151 seconds. (I wrote here, first time round, that the schedule "does not change the hash." Not quite — see below: some breakpoint sets retune and some don't — [[1,3,5],…] did, [[1,4,5],…] and [[1,2,5],[3,4,3],…] didn't — and I haven't read the key derivation, so I'll stop guessing the rule.)
The other result: decode is flat with depth
This lane exists for 1M-token context, and until tonight every decode number we had published was measured on a six-thousand-token prompt. That's a gap. If attention cost climbs with context the way it does on a GPU-resident model, then at a few hundred thousand tokens the story stops being "expert fetch is the bottleneck" and starts being "attention is," and the levers change completely — vLLM's in-flight attention megakernels and sparse-indexer work for this model would jump to the top of the list.
So we measured it. One prompt at each of five depths, streamed, timing from the first output token to the last so prefill and any cache effects are excluded. (The first version of the instrument didn't do that and produced garbage at two depths — it assumed a prefix-cache hit and silently timed a re-prefill. The bad JSON is in the repo next to the good one.)
Figure 2. Decode speed at one stream, k=5, as prompt depth grows 65×. The line stays inside a ±7% band. The 212K point read 183.6 tok/s because the random-word prompt at that seed produced a repetitive continuation the drafter guessed in long accepted streaks (51–60 stream chunks vs ~80 elsewhere); it's a real measurement of an unrepresentative prompt and is excluded from the line.
Flat. Decode at 425K tokens is 121.8 tok/s against 123.8 at 6.5K — a 7% drop over a 65× increase in context. Cold prefill is still 25.8 s at 425K, consistent with the 972K-in-85 s number from the main post; that's the price of a deep context here, not decode.
What this settles: on this lane, at these depths, attention is not the bottleneck, and expert fetch over C2C is — all the way out. The attention-megakernel and sparse-indexer work landing upstream for this model is real engineering, and we'll take it when it ships in an image; but it will not move throughput on this box, and we're no longer going to spend a day building from a moving branch to find that out.
Same night, second look: it loses on real agent traffic
The knee that promoted v13 is a prose benchmark — sixteen requests each asking for a paragraph about a number. The lane this recipe exists for serves agents: shell commands, code, tool-call JSON, the text a coding assistant actually emits. So before going to bed I ran three validations on both containers in the same window: the single-stream fixture with its classes in different orders (to chase the −7% tool-JSON), a concurrent replay of 24 real Hermes transcripts at one and four workers, and the knee with three- and six-stream points added. Then two alternate breakpoint sets.
Two of those came back the way I'd hoped. The tool-JSON −7% was fixture ordering: run first, tool-JSON does 95 tok/s on both v12 and v13; run alone or in the default order, 184 on both. Not a regression, just a cold path. And the knee at three and six streams is +24% and +25% — the k=1 regime wins from its first step, on prose.
The third didn't. Four workers replaying real agent turns: v13 207 tok/s, v12 236. Minus twelve percent. The mechanism is in the acceptance column. Agent text is predictable; the drafter is right about 85% of the time on it. At k=5 that's 2.7 accepted tokens per verify step. At k=1 the ceiling is one token per step, and the drafter delivers 0.87. The expert-fetch cost that k=5's window incurs — the whole reason k=1 wins the knee — is paid back on agent text by tokens the knee's prose never yields. At four streams the batch isn't full enough for the fetch cost to dominate that, so the schedule leaves tokens on the table. Single-stream replay was a wash (+5%), because the schedule is k=5 there.
Figure 3. Same containers, same window, four concurrent streams. On prose the schedule's k=1 wins by 26%; on real agent transcripts it loses by 12%. Acceptance is the difference: 0.30 versus 0.85.
The two alternate breakpoint sets — holding k=5 through three streams, or a k=3 middle band at three and four — both lost on the knee (C3 −20% and −9%) and were not run through the replay, which is the measurement that would have mattered. That's the next boot: [[1,4,5],[5,16,1]], judged on the replay at four workers, not the knee. I expected it to cost the full 80-minute autotune, since one of tonight's alternates did. (It didn't — below.)
So the honest state of things: v13 is the right config for prose and mixed load at three streams and up, and the wrong one for a lane that is mostly agents at three or four. The recipe says exactly that now, with the numbers. If your concurrent traffic is coding agents, run the v12 flags until the next boot settles it — which it did, a few hours later; keep reading. This is one pass of one instrument, on a replay that swings ±17 tok/s run to run, so it's a caveat rather than a rollback — but it's a caveat I'd have missed if the validation had stopped at the benchmark that made the config look good.
Morning after, third look: hold k=5 through four streams
The boot ran at 03:14. Same window, same instruments, and this time each config got two replay runs rather than one, because the replay's run-to-run spread had been the weakest part of the overnight caveat.
Three things fell out. First, the overnight caveat survived a second pass: v13 at four workers averaged 188 against v12's 207, a 9% loss (it was 12% on one run last night; the instrument swings about 20% between runs, so the mean of two is the number I'd repeat). Second, the fix works and costs exactly what you'd expect — [[1,4,5],[5,16,1]] replays at 207, which is v12 at four streams, because at four streams it is v12. From six streams up it's v13 to the decimal: 272 / 318 / 379 / 434. What it gives back is the C3/C4 prose gain: 155 and 176, v12's numbers. Third, this boot hit the autotune cache. 454 seconds, not 80 minutes. I had written above that the key "includes the breakpoints" after one alternate retuned; two out of three alternates didn't, so that sentence is now hedged rather than asserted.
So the recipe carries two schedules, chosen by what the lane serves. v14 — k=5 through four streams — for a lane whose concurrent traffic is agents, which is what :30006 is. v13 — k=5 through two — for batch prose throughput, where it's worth +26% at C3 and C4. The single-stream numbers, the fixture, the depth map and the C6+ knee are identical between them. The v14 container is the lane's reference as of this morning; v13 and v12 are stopped and kept.
Round 6, corrected: replay rates are not agent capacity
Correction, September 15, 2026: I overstated what these measurements establish. The raw throughput and cache counters stand; the capacity, cache-causality, uncertainty and exact offload-floor claims do not. The corrected recipe evidence note explains the limits and preserves the original captures and instruments.
The shuffled recorded-history proxy contains 300 turns: 150 with historical tool calls and 150 text turns, at most two per original session. Four workers measured v14 at 171.1 and 174.1 completion tokens per second of wall time; v13 measured 150.0 and 151.7. That is a 14.4% difference between the two means on this particular workload. Two repetitions give an observed range, not a settled confidence statement about agent traffic. Server counter deltas showed approximately 5% prefix-cache token hits.
A different, session-ordered proxy contains twenty fixture entries with fifteen turns each, including 277 historical tool-call turns, and allows longer contexts. Four workers measured 232.6 tok/s on the first measured pass and 275.2 tok/s when repeating the identical recorded requests. Prefix-cache hit ratios were 71.3% and 94.3%; TTFT median/p90 was 0.45/0.77 s and 0.32/0.47 s respectively. The first pass followed a short warmup request, without an explicit cache-reset receipt. These are recorded-history serving measurements: the harness substitutes a short system prompt, flattens historical tool calls and results into text, truncates context, does not execute returned calls, and does not feed generated answers into the following turn.
I previously described the difference from 171 as an isolated cache benefit and said configuration rankings carry over. Neither conclusion was tested: tool mix, context cap and ordering all changed, and only v14 ran the ordered fixture. The warm pass is repeated-request replay, not newly continuing sessions or a production ceiling. Higher acceptance on the second corpus is descriptive too, not proof of a context-caused mechanism.
The original confidence intervals also need withdrawing. The wall-clock number divides completion tokens by measured makespan; the bootstrap instead multiplies tokens divided by summed request durations by the worker count and resamples correlated turns independently. Those are different estimators, especially with idle worker tails. The first ordered pass reports 232.6 tok/s alongside a 233.6–250.7 interval for the other estimator; that interval cannot be attached to the wall-clock rate. Truncated identifiers also merge two of the twenty session labels. A corrected instrument must preserve unique identities, validate complete streams and usage, and use independent runs for makespan uncertainty.
The offload result is a bracket, not a floor. At --cpu-offload-gb 55, the terminal error says a 1,048,576-token request needs 2.2 GiB of KV cache but only 0.65 GiB is available. The reference at 60 works with approximately 4.87 GiB KV. Intermediate offload settings were not tested. A shorter context may lower the KV requirement enough to bind; it does not automatically move expert weights, and actual placement, speed and stability still need measurement. My earlier “sixty is the floor; context cannot help” statement was too strong.
The reference remains v14. No new configuration is promoted by this audit, and these replay results do not establish end-to-end task quality. The historical captures were published in J-M-Recipes #19; the corrected interpretation supersedes that PR's original headline claims.
There's a general lesson in the shape of this, which is that the workload-dependence of speculative decoding is not a footnote. The same schedule was +27% and −9% on the same hardware at the same concurrency, depending only on whether the text being generated was prose or a coding agent's tool calls. Any single benchmark number for spec decode on an MoE model is a statement about one text distribution. The recipe's replay instrument — twenty-four real transcripts — is a crude fix for that, and I'd rather have it than not.
What's next
- Run on v14 for agents, v13 for prose. The lane's reference container is
dsv41-vllm-v14-1M-ksched-agent-BOUND-REF; v13 and v12 are stopped and kept. Recipe, launch script (KSCHED=) and results: J-M-Recipes #16 (v13), #17 (Round 4), #18 (v14). - The matched replay comparison is complete. Three alternating boot pairs support v14 on this four-worker proxy, with a first-token-latency tradeoff. Extending that claim to native Hermes workloads or a broader tool corpus requires a separately specified experiment, not more repeats chosen after seeing these results.
- Offload is bracketed, not closed: 55 failed the 1M minimum-KV check; 60 works. Intermediate settings and shorter-context configurations remain untested. Keep the reference unchanged until a bounded comparison justifies another choice.
- The tool-JSON −7% is closed — fixture ordering, identical on v12 and v13 in every position. The fixture now warms each class.
--async-schedulingwas also measured tonight: a wash (±3%, hash hit). Not adopted; harmless if a tail-latency reason appears later.- The research-sized idea from the verify-curve post — expert-aware draft selection — is still the only thing on the list that could move single-stream prose — and, after tonight, it's also the thing that would let one schedule serve both workloads, since it scores drafts by expert cost rather than by a fixed k. It's a project.
Reproduce
The recipe's launch script takes the schedule as an environment variable:
MODEL=/models/DeepSeek-V4.1-Flash-df42c109f1defefcbfcedbe7d905718a12266e40 \
TAG=v14-1M-ksched-agent OFFGB=60 UTIL=0.97 SEQS=16 SPEC=dspark:5 KSCHED='[[1,4,5],[5,16,1]]' CTX=1048576 \
EXTRA='--long-prefill-token-threshold 6144' \
bash scripts/launch-dsv41-vllm.sh(KSCHED='[[1,2,5],[3,16,1]]' for v13.) Round 5 (v14 vs v13 vs v12, replay ×2 each, knee, fixture) is in results/2026-09-15-round5-v14-agent-schedule/. Round 4 (replay, ordering test, C3/C6 knee, breakpoint sweep) is in results/2026-09-14-round4-v13-validation/, including replay_c.py. Round 3: raw per-boot knee-*.json and agentfix-*.json for all six boots, boot receipts (facts-*.txt), the campaign runner, the depth instrument (v2, and the broken v1), and throughput.csv are under results/2026-09-14-round3-ksched-depth/.
Credits
- vLLM —
num_speculative_tokens_per_batch_sizeexisted before we needed it. This post is a measurement of someone else's good decision. - Fable 5.1 via Nous — the Sept 11 plan review that put the batch-size schedule on the list in the first place, where it sat for three days behind more exotic ideas.