Christopher Owen's spark-ds41f at candidate 60: how it works, how it compares to our recipes, and what a GB300 can borrow
In plain English
- What it is. spark-ds41f is Christopher Owen's recipe and engine for running DeepSeek-V4.1-Flash, a model too big for one DGX Spark, across three to eight of them. He does not tune someone else's engine. He writes the engine: 689 patches on a vLLM fork, GPU kernels written for the Spark's chip, and his own network code.
- What I read. His candidate-60 and c61 branches, and the code itself. I applied his 689 patches to the base he pins and got the exact source tree his build expects, then read the parts described below.
- Candidate 60 is a prompt-reading build. On his ring of four Sparks it reads at up to 9,915 tok/s and still does 9,064 tok/s on a 910K-token prompt. Writing speed did not change. c61, the next branch, goes after writing speed for one user with two changes; it has no results yet.
- How it compares to ours. Our four-Spark vLLM recipe is his engine, at the release he called r6x (October 18), with our own settings on top. He has shipped four releases since. By his benches, candidate 60 reads a long prompt 66% faster than r6x and decodes 12% faster for one user. We have not run it yet.
- Where ours was better, and may not be any more. On October 26 we measured his 1025 release against our settings on the same four Sparks. His kernels were 4.5% faster for one user. Our settings answered a chat that arrived during a long prompt in 1.52 s; his took 9.09 s. His next two releases went straight at that weakness. We need a rematch before claiming either way.
- What a GB300 can borrow. His kernels are built for the Spark's GPU and do not run on the GB300. Five ideas are worth porting. His kernel files name the RTX PRO 6000's GPU family as a target, so his expert kernel is a real candidate for the sidecar. His way of testing is usable today.
- Bottom line. When candidate 60 is published, build it on Sparks 5–8, run our battery on it in his shape and in ours, and promote whichever wins the mixed-traffic rows.
1. Where candidate 60 stands
The latest release is spark-ds41f-1107 (November 7). Releases are named for the day they are promoted. Between releases Owen builds numbered candidates. A passing candidate can be frozen as a GitHub pre-release for others to test; the last frozen one is candidate 54 (November 11).
Candidate 60 lives on the repository's candidate-60 and c60-prefill branches, both already merged into main. The record holds its patch series, build inputs and receipts. It qualified on his switched four-Spark and three-Spark testbeds, and a prefill sweep on his switchless ring finished later the same day. It was not yet tagged as a pre-release when I checked (commit fcc97650). The next branch, c61-decode, has code and a test spec but no results.
What "read the code" means here. Owen publishes patches, not a source tree. I checked out the vLLM commit he pins (19f2c20e from Local Inference Lab), applied candidate 60's 689 patches in order, and compared the result with the tree hash his build checks. It matched: afcaac35. So the files quoted below are the ones his image is built from. The c61 changes are six files mounted over that tree; I read their diff in full.
2. How it works
Figure 1. The four-Spark shape. Memory figures are per node from Owen's release notes (71.67 GiB of weights and a 10.5 GiB KV pool per rank). Collective timings are from his collective-policy experiment.
The split. The checkpoint is about 475 GiB and a Spark has 128 GB, so each layer is cut across the nodes (tensor parallelism). At four nodes each Spark holds 16 of the 64 attention heads and a quarter of every expert's width. Small layers are copied to every node so they need no network round. The weights are DeepSeek's, loaded as shipped; nothing is re-quantized.
The network. Every layer has two points where the four partial answers are added together. That is 80 sums per step, so the sum has to be nearly free. Owen's sparknet does it in one RoCE round trip: 13–19 µs against 75–86 µs for NCCL in his test. The sum is also exact in a specific way (tilelang/collectives.py): the four parts are added in 32-bit floats in rank order and rounded once, so a row gets the same bits whatever the message size or which node it sits on.
Figure 2. One decode step. Step rate and tokens per step are candidate 60's on Owen's switched four-Spark testbed (35.74 steps/s, 2.328 tokens/step, 82.91 tok/s). The bytes argument is from his docs/fundamentals.md.
Decode is about bytes. A decode step for one user is six rows: the real token and five drafts. That is so little arithmetic per weight byte (about 12 operations) that the GPU spends the step waiting on memory, not computing. A step is also a chain of about 1,850 small kernels, replayed as CUDA graphs. His decode work follows from those two facts:
- Cache prefetch (
l2_prefetch.py). While slow serial work runs, a side stream pulls the next projections' weights into the GPU's 24 MB cache. His note in the file: 49.36 ms to 47.45 ms per cycle at an eight-row check. - Chained launches (
tilelang/pdl.py). Each kernel starts while the one before it is still finishing, and waits before touching memory. The gaps between kernels were 3.5% of a one-user step. - A smaller word list. The output table is stored in an exact 12-bit form, and an 8-bit copy rules out most words before the exact pass (
tilelang/vocab.py). - A draft budget. Each step,
adaptive_verification.pypicks how many draft rows to check by maximising expected tokens per unit of step cost. Draft rows whose chance of surviving falls under a threshold are masked and skip the expert layers. - Copy drafts (
copy_drafts.py). If the end of a request's context matches text it has already seen, the continuation of that earlier text replaces the drafter's guess. Checking still commits only the model's own tokens, so drafts change speed and never the text.
Prefill is about compute. A prompt chunk is thousands of rows, so the tensor cores are the limit. Two mechanisms carry most of it:
- Sequence parallelism (
sp_prefill.py). In ordinary tensor parallelism every node repeats the per-row work of a layer on every token. For prompt reading, each node instead owns a quarter of the rows for that work, and the rows are gathered back only where an operation needs all of them. - Row skipping (
ced.py). The model is built so that its upper layers need only each request's last window of rows during prompt reading. The engine packs those rows and skips the rest.
The scheduler. One long prompt is read at a time, in chunks. Two patches stop it from blocking everyone else:
- Late admission (
late_admission.py). Normally the engine plans step N+1 as soon as step N starts, so a request arriving during N waits for N+2. Here a CUDA event fires when step N reaches a chosen layer (30 in his recipe), and N+1 is planned then. Arrivals in between join N+1. No tensor is read or written, so outputs cannot change. - Prefill yield (
scheduler.py). A long prompt's chunk gives up its step to prompts that finish within one chunk. On very deep contexts with other users decoding, chunks also shrink: 4,096 tokens from 256K to 512K, 2,048 beyond.
Correct first. This is the part I would copy before any kernel. At temperature 0 a request's output does not depend on what else is in the batch, because every kernel adds in a fixed order. The MoE kernel file says it plainly: neither kernel uses atomics on results, so a token's output does not depend on the rest of the batch. Each candidate boots twice and must produce identical tokens and log-probabilities both times. A change counts as faster only if it beats the build's own repeat-to-repeat spread. Failed ideas are kept in a "tried and not adopted" list with their numbers.
3. What candidate 60 changed
Candidate 60 is six patches on top of candidate 55's series, about 9,200 added lines. Each patch header states its rationale, its quality impact and its tests.
| Patch | What it does | His evidence |
|---|---|---|
| 0811 | Faster prompt tokenizing: a cache of text pieces and parallel encoding split at newlines. Tool schemas can be loaded mid-session without breaking the cached prompt prefix. | token equivalence on 64K–256K prompts |
| 0812 | Prompt-cache retention: which cached blocks to keep under memory pressure. Repeat images are hashed once. Token lists cross between processes without a copy. | a multi-turn hit rate of at least 98.5% under eviction pressure; repeat image load 31.27 ms to 1.66 ms |
| 0813 | Kernels for the step that scores and picks old tokens to attend to, and the front of the attention block. | 1.29–1.47× at 16 and 256 rows, bit-exact; compute-sanitizer clean |
| 0814 | Faster packing of the word list and expert scale codes at load; small speed-ups in sampling and draft checking. | outputs match the unpatched kernels |
| 0815 | The model's n-gram lookup tables sit on disk with a small cache. A prefill used to flush decode's hot rows out of it. A Clock-Pro policy now shields them. | hot-row hit rate 0.19 to 1.00 with a prefill interleaved; ASan and UBSan clean |
| 0816 | Hardening of the sequence-parallel side streams, and one cache-prefetch default (15/12/12 MB) for every shape. | determinism unchanged on all shapes |
One negative result is recorded with it. Caching the packed weights on disk to speed up boot was slower on the Spark than recomputing them on the GPU, so it was removed.
Figure 3. Candidate 60 on Owen's switchless ring of four, two boots, cold Python-source prompts; x is the actual token count. The 1107 line is from his release suite with different text, so the gap between the lines is not a matched gain.
- Peak 9,915 tok/s at a 121,796-token prompt. He set 10,000 as the target; one boot read 9,928 and no size reached it.
- Flat from 16K to 262K, within 1.4% of the peak.
- Long prompts hold up. 9,064 tok/s at 910,469 tokens, 101 s in all.
- Short prompts are start-up bound. A 523-token prompt answers in 0.27 s.
- Ring and switch agree. 9,915 on the ring against 9,924 on his switched testbed at 131K.
- Decode is unchanged. 82.91 tok/s for one user against candidate 59's 82.94, and identical output on every determinism check.
4. The c61 decode branch
His own profile says the routed experts are about 65% of a one-user step, and one-user decode has sat near 83 tok/s since 1107. The c61 record ranks twenty open ideas by expected one-user gain. The branch implements the top two as six files mounted over candidate 60.
Figure 4. Left: derived from the constants in c61's adaptive_verification.py (draft cost ratio 0.55, single-tile row scale 0.42) at the recipe's row cost of 0.035. The candidate-60 bars are its values once its online calibration has fitted; before that it uses the recipe's fixed 0.2 threshold. Right: sizes and timings are from Owen's c61 record.
Change 1: price a draft row honestly. In candidate 60 a step's cost is modelled as 1 + 0.035 per row. That treats drafting as free and every row as equally expensive. The c61 file adds two constants for a step that fits in one 16-row tile, which is every one-user step:
- drafting costs 0.55 of a step and is already spent by the time rows are chosen;
- a row inside the tile costs 0.42 of the old price.
The arithmetic that follows: one more row costs 0.9% of a step instead of 3.5%, and a draft is worth checking if its chance of surviving is above 1.8% instead of 6.3%. So more drafts get checked. The same change starts the confidence estimate from five fitted per-position offsets (−0.52, −0.64, −0.96, −0.85, −0.74) instead of zero, so a fresh boot does not under-draft while it learns. His note on the underlying pull request gives +6.2% for one user at steady state.
Change 2: screen the drafter's word list. The drafter picks its five tokens one after another, and each pick is a full pass over the output table plus a small correction that depends on the previous pick. Candidate 60 already screens the main model's pick with the 8-bit copy. c61 does the same for the drafter: one screening pass gives a proven upper bound for every word, the correction is added to the bounds, and only words whose bound could beat the current best get the exact arithmetic. The picks are identical by construction. It is switched on only when a single request is decoding. His estimate is 0.45–0.75 ms a step.
The branch's test spec compares candidate 60 with c61 on the ring: 150 s of one-user traffic, 90 s at 16 users, and a greedy determinism record. No receipts were in the repository when I read it. The stated goal is 100 tok/s for one user, from 79.9 on his prose bench. The two changes above do not add up to that by his own estimates; the rest of the list (expert-layer pruning for doomed drafts, kernel fusions) is what would have to carry it.
5. His release ladder
Figure 5. Owen's numbers, not ours. Releases r6x to 1107 are from his README's release comparisons (four-node ring). Candidates are from his experiment records: c58 on the ring, c59 and c60 on his switched testbed. One-user figures for the candidates are from his c1-decisions bench, a different instrument from the releases' single-stream suite.
- Prefill moved most. 5,982 tok/s at r6x, 8,657 at 1107, 9,924 at candidate 60. Candidate 60 is about 15% above 1107 at 131K.
- Chats during a long prompt. The first token of a short request arriving beside a 131K prefill, with three streams running: 12.36 s on 1025, 3.30 s on 1030 (prefill yield), 1.02 s on 1107.
- One fix worth knowing. Candidate 54's notes describe a race on the ring that made temperature-0 output differ about one boot in three during that round. Release 1107 has the same code order, though its own gate saw no failing boot in four. The fix is in candidate 54 and later.
6. How it compares to our recipes
Our four-Spark recipes are in J&M Recipes. Two of them serve this model on Sparks, and one is built on Owen's work.
| spark-ds41f (Owen) | Our vLLM lane (Sparks 5–8) | Our SGLang lane (Sparks 1–4, daily) | |
|---|---|---|---|
| Engine | his own: vLLM fork + 689 patches, TileLang kernels, sparknet | his r6x image, built locally | knapcio's SGLang profile v2.5 |
| Version | 1107 released; candidate 60 in progress | r6x (October 18) + his late-admission patch as a mount + sparknet 0.5.0 | v2.5 with vision |
| Seats and chunking | 16 seats, 16,384-token batch budget, late admission at layer 30, prefill yield | 32 seats, 8,192 budget, chunk threshold 3,856, late admission at layer 20 | the profile's defaults |
| Cabling | ring or triangle with no switch; 3–8 through a switch since 1107 | one switch, 4 KiB pages | same switch |
| What the author does | writes kernels and scheduler patches, one measured change at a time | we pick engines, set the shape for our traffic, and publish receipts and failed arms | |
| Quality bar | bit-identical temperature-0 output across boots and batches; LongBench, RULER, HumanEval+, tool and vision evals at release | teacher-forced log-probability gate against a kept control, qeval 71/75, 1M needle, a 20-hop agent run | |
So this is not engine against engine. On the vLLM lane the engine is his, and what is ours is the shape and the evidence. Two things follow.
We are four releases behind. Our lane still runs the gold bar in Figure 5. The last time we measured a newer release was October 26, when we built 1025 and ran it as he ships it next to our lane and our daily SGLang lane. Same instrument, medians of three; the two vLLM arms ran on Sparks 5–8 and the SGLang lane on Sparks 1–4:
| Our battery, October 26 (thinking off, temperature 0) | Owen 1025, his shape | Our vLLM lane (r6x, our shape) | Our SGLang lane |
|---|---|---|---|
| One user, per-stream decode (tok/s) | 110.4 | 105.6 | 98.9 |
| 16 users, total (tok/s) | 475.1 | 481.6 | 444.2 |
| Cold prefill at 32K / 131K (tok/s) | 6,393 / 6,406 | 5,467 / 5,262 | 6,368 / 6,449 |
| 1M-token cold prefill (s) | 218.3 | 260.0 | 239.3 |
| Short chat arriving during a cold long prefill, first token (s) | 9.09 | 1.52 | 2.98 |
| Turn on a cached 68K document behind a 413K prefill, first token (s) | 66.54 | 2.20 | 65.83 |
| qeval, of 75 | 71 | 71 | 71 |
Read it as two results. His newer kernels were faster: +4.5% for one user and +17% to +22% on prefill, with the same quality score. Our shape was far better when traffic is mixed, which is what guests and agents on this rack produce. More seats, smaller prefill chunks and earlier admission are settings, not kernels.
That table is now stale on his side. Prefill yield (1030), a scheduler change for deep contexts (1107) and candidate 60's cache-retention policy all aim at the two rows where our shape won. His own figure for a short request beside a 131K prefill went from 12.36 s to 1.02 s across those releases. We have not measured any of them. The honest status is: our shape won in October against a release he has since replaced twice.
Three smaller points:
- Our cabling is now one of his shapes. When we adopted r6x there was no switched shape, so we wrote a local one and launched around his topology check. Since 1107 he qualifies three to eight Sparks through a switch. A rebuild can start from his
tp4-switchedshape instead of our hand-made one. His recipe table lists that shape at 64 KiB pages while his switched testbed logs 4 KiB like ours, so page size is the first thing to check. - The KV pool is the same size. Both lanes give the KV pool 10.5 GiB per node. Our recipe quotes the engine's startup line for r6x (8,580,566 tokens). His release notes say that line undercounted and now reads 11,554,122 tokens, 11 full 1M contexts. I have not re-derived the count.
- He publishes no container images. Each release is a local build from his commit, about an hour on one Spark for us.
7. What is useful for the GB300 and the RTX PRO 6000
Our GB300 lane is a different machine and a different engine: one Station GPU holding the hot experts, an RTX PRO 6000 in the same box computing the cold ones, Garner McCloud's engine under our policy (the Pg62 recipe). In its promotion window it decoded at 7.2 ms a step and 439 tok/s for one user on our general-prompt set. Nothing of Owen's installs there. Some of it is still worth the work.
Figure 6. Sorting by what it would take. Nothing in the first two columns has been measured on the Station; each is a candidate for a pre-registered card.
Port to the GB300 lane
- Short requests jump the queue. Late admission and prefill yield are scheduler changes in vLLM's Python, not kernels. On our Sparks, his late-admission patch cut a guest turn behind a 413K cold prefill from 1.885 s to 1.196 s. The Station lane is vLLM too, on a different tree, so this is a port and not a mount. It matters there because guests share that lane with long agent prompts.
- The 8-bit word-list screen. The last layer multiplies by a 129,280-word table. He keeps an 8-bit copy, uses it with a provable error bound to rule out almost every word, and runs the exact arithmetic on the finalists only, so the answer is bit-identical. On our Station lane in October that head cost 0.42 ms of an 11.2 ms step: two full reads of the table, already as fast as memory allows. That was the v27 lane and has not been re-measured on Pg62, so the size is unknown, but it is one of the few step costs we had marked as having no cheap fix.
- Skip the experts for drafts that will not survive. He scores each draft row's chance of being accepted and lets rows under 0.2 skip the expert layers. On the Station every extra row can pull in experts that live on the RTX. Fewer live rows means fewer distinct experts per step. Whether it beats what the lane already does is a measurement, not a guess.
- Price draft rows the c61 way. The change is under 200 lines of scheduler Python and two constants. The idea is general: once drafting is paid for, the question is only what one more row costs. His constants are fitted to the Spark's 16-row tile and would have to be re-fitted from our own step timings. On the Station an extra row can also mean an extra trip to the RTX, so the fitted price there may be higher, not lower.
- Host-side fast paths. The tokenizer cache, the repeat-image fast path and the prompt-cache retention policy (patches 0811 and 0812) are engine-side Python. They help the time before the first token on long or repeated prompts. They will not move the decode step, which on that lane is GPU-bound.
The RTX PRO 6000 sidecar
This is the one place where his kernels might run with little change. The RTX PRO 6000 is SM120; the Spark's GB10 is SM121. His kernel files say so in their first line: tilelang/gemm.py is the "block-32 FP8 linear for SM120 / SM121" and tilelang/moe.py is "MoE dispatch and combine for SM120 / SM121". Both are built on the same block-scaled tensor-core instruction.
What the sidecar does all day is what his routed-expert kernel does: FP8 activations against FP4 expert weights, a handful of rows at a time. For steps of 64 tokens or fewer our sidecar already runs a kernel of that class with no atomics on the data. Above 64 tokens it falls back to an older path that accumulates with atomic adds, which is one reason that lane's output is not bit-identical from run to run. His combine sums each token's expert outputs in slot order in 32-bit floats, with no atomics on results, at every row count. That makes it a candidate for the large-batch path. It is untested on SM120: it needs a build, a parity gate against the current path, and the 32-stream stress that has caught our own kernels before.
His memory tricks (coded expert scales, the 12-bit pack) would let more experts sit on either GPU. We measured that lever directly on November 10: going from 255 to 275 hot experts per layer brought the first token of a 61K prompt 11–12% sooner and left one-user decode at 0.98× of the control. So they are a prefill and capacity idea on this lane, not a decode one.
Stays on the Sparks
The RoCE collectives, the ring cabling and the padding for odd node counts solve a problem one GPU does not have. The cache-prefetch windows are tuned to the GB10's 24 MiB L2. The disk reader for the model's n-gram memory tables, and candidate 60's guard on it, are for tables on NVMe; ours sit in the Station's host RAM. The 64 KiB kernel and memory saver are GB10 unified-memory fixes.
Usable today: the method
- Two boots per candidate, and a verdict of same, better or worse against the build's own repeat spread. We already grade against control spread; his version is stricter and automatic.
- compute-sanitizer (memcheck, racecheck, initcheck) on every custom kernel before it serves. A split attention kernel of ours passed its unit tests and then died with an illegal address at 32 streams.
- His measured warning on fusing the small mixing kernels (mHC): the one-launch version with atomics was 12–34% slower than two chained launches. That family cost 1.80 ms of our October step, so the result is worth having before we try the same fusion.
8. What we do next
- When candidate 60 is frozen, build it on Spark 5 and fan it out to Sparks 5–8.
- Run it twice through the October battery: once in his
tp4-switchedshape as shipped, once with our seats, chunk threshold and admission layer. The r6x lane stays kept as the control. - Gate on the teacher-forced check and the 1M needle before any speed number counts.
- Promote the arm that wins the two mixed-traffic rows without giving back prefill. If his shape now wins them, we retire our overrides and say so.
- Watch c61. If its screen lands with a one-user gain and identical output, it goes into the same rebuild.
- For the Station: write the 8-bit head screen and the draft-row pricing as cards with bars, and price the SM120 build of his expert kernel as a feasibility note before promising a boot.
None of these is scheduled. They need James's go and a window on Sparks 5–8, which serve guests today.
9. Caveats
- Nothing in this post was measured today. Our numbers are from the October 26 battery and the recipe READMEs; his are from his repository at commit
fcc97650. The only live check was which images the lanes run. - Candidate 60 was not a published pre-release when I read it. Its ring run is a prefill sweep; the full gate and decode suite ran on his switched testbeds. Its figures can change before he freezes it.
- c61 has no published result. The percentages in Figure 4's left panel are arithmetic on constants in his code, not measurements, and the gains quoted are his.
- Figure 5 joins numbers from two of his benches and two of his testbeds. The direction is clear; the exact percentages across the release-to-candidate boundary are approximate.
- The GB300 section is a reading of his notes against our step profile. The October step budget it quotes is older than the lane now serving. No item there has a measured gain on our hardware.
- I rebuilt his source tree and read the c61 diff in full, the six candidate-60 patch headers, and the module notes and key functions of the files named above. I did not read all 689 patches or run any of his code.
Sources
- christopherowen/spark-ds41f at
fcc97650, branchescandidate-60,c60-prefillandc61-decode:README.md,TODO.md,docs/fundamentals.md,docs/switched-fabric.md, the 1107 and 1030 release notes, the candidate-54 pre-release notes, and the experiment records for candidates 57 to 61 (experiments/2026-11-14-c58-prefill-10k,2026-11-15-c59-prefill,2026-11-16-c59-long-prefill,2026-11-16-c60-prefill,2026-11-16-c60-ring-prefill-sweep,2026-11-16-c61-decode). Source read from local-inference-lab/vllm at 19f2c20e with candidate 60's patch series applied (treeafcaac35). His site is switchle.ss. - Our Spark lanes: the vLLM recipe and the SGLang recipe; the October 26 battery table is rendered from its phase receipts.
- Our Station lane: the sidecar recipe.
- Earlier posts: four engines on four Sparks, what transfers from DeepInfer, drafts and block verification explained.