One stray 4-byte read: how the split-KV crash on one GB300 was actually a buffer that ended on a page boundary
Correction, 5:51 PM CDT: the faulting read is in the speculative rejection sampler, not the MoE weight buffers — those only pushed the buffer to a segment boundary. Full fix and numbers in the release post.
Follow-up to Splitting attention on one GB300: a 3.8% single-stream win, and the crash that blocks it, which left this crash as the open bug.
The short version
Yesterday's post left an open bug: splitting decode attention across the GPU made one user 3.8% faster, but the engine died with a CUDA illegal-memory-access whenever the server also handled 8 and 32 users. Splitting was off on the step that died, so it looked like state left behind by the split kernel.
It wasn't the split kernel at all. It was a 4-byte read one slot past a 512-byte tensor in vLLM's speculative rejection sampler (draft_sampled[logit_idx + 1]), and it only faults when that tensor ends a 2 MiB block of PyTorch memory and the next page happens to be unmapped. Our build only changed the layout of GPU memory: together with the MoE weight-scale-transform buffers ahead of it, that pushed the tensor to a segment edge — so a read that had always been harmless started landing in a hole. The MoE buffers are bystanders; the fix is upstream vLLM #60945.
How it was narrowed MEASURED
Each step is a card, and each has receipts.
| Card | Question | Result |
|---|---|---|
| W6 | Is the crash from our rebuilt FlashMLA library? | Split-off with the library loaded: 32-user cells passed, 2,584–2,611 tok/s |
| W8 | Is single-user splitting stable? | Yes: three single-user passes plus a standalone 32-user run, 2,600.9 tok/s, zero errors |
| W9 | Is it set at boot? | No: 32 users first, then 16, then 32 at 64K, then a single-user soak, then 16, then 32 — all clean |
| W10 | Is it the cell order (C8 then C32)? | No: eight cells in that order, each a separate benchmark process, all clean |
| W11 | Is it one process running every cell? | Yes — reproduced on the first arm, and again with splitting OFF |
What the fault actually is MEASURED
Both W11 arms died with the same GPU error (Xid 31, FAULT_PDE, ACCESS_TYPE_VIRT_READ) at an address whose low 29 bits are identical to every earlier crash (0x1F200000, 2 MiB-aligned):
- The address is the end of the segment: exactly 0 bytes past the end of a 2 MiB PyTorch segment.
- The read that lands there is
draft_sampled[logit_idx + 1]in vLLM's speculative rejection sampler (vllm/v1/worker/gpu/spec_decode/rejection_sampler_utils.py:435). Its guard is sized for the configured 5 draft steps, but our k-schedule runs 3 at 9–32 requests, so the last row reads one 4-byte slot past a 512-byte tensor. It only faults when that tensor lands at a segment end. - The last block in the segment is 253,952 bytes (163,072 requested) and ends precisely at the faulting address. The allocator trace tied the segment to the MoE weight-scale-transform buffers allocated by
_transpose_sf_for_utccpandtransform_weights_for_mega_moeduring weight finalization. This post first named those as the cause; they are bystanders that only pushed the sampler's tensor to the boundary. - The following page is an unmapped CUDA reservation, so a read there is a page fault, not a wrong value.
- The fix is upstream: vLLM #60945 (merged 2026-10-12) corrects the sampler guard.
Splitting off, same mount, same fault. So: not the split kernel, not the second library, not the MoE weight transforms — an out-of-bounds read in the speculative rejection sampler, exposed by where our build places memory. Fixed by vLLM #60945.
Why it only shows with our build loaded
Our library adds a few megabytes of allocations at start-up. That shifts the layout enough that the MoE transform buffers fill out a 2 MiB segment and the sampler's 512-byte draft_sampled tensor lands at its tail. Before, the tensor had another allocation behind it, so the 4-byte over-read hit valid memory and nobody noticed. With our build, it hits the boundary. The bug is older than the speed work — it is in vLLM's rejection sampler, fixed upstream by #60945 — and the speed work only made it visible.
What happened next RESOLVED
As first published, this section planned DeepGEMM upstream fixes (#480, #451, #462 against pin e1f418c2) and page slack for the MoE transform buffers. That targeted the bystander and is superseded.
- Backported vLLM #60945 (merged upstream 2026-10-12) onto our pinned copy of
rejection_sampler_utils.py: sha2562d43d513, was42c5e8a0. - Re-ran W11's failing cell shape on the fixed build: clean, 0 new Xid lines.
- compute-sanitizer: 20 invalid reads unpatched → 0 patched. Kernel A/B: 100/100 bit-identical.
Split-KV shipped for single-user traffic as h2-REF. Full fix and numbers: the release post.
Reproduce
~/dsv41/m3/att2e/ (Card W11), receipts under results-cardW11/S6/owner.txt, S7/owner.txt and maptrace/*.snap.pickle. Production reference remains dsv41-vllm-1017-h1-REF on :30006.