Testing DGX Spark Kernels
- 7.0.0-1024 is fine on our headless cluster. 0 resets in 6 hours of soak on four Sparks, and since then 0 GPU faults in 38 boots across all eight. Same speed as the old kernel. We now run it on all eight Sparks. Corrected November 3: the reset reports were real for Sparks with a display attached, especially 4K; NVIDIA shipped a fix on November 1 (
nvidia-spark-initcall-bl1.0-2). Take it even if you are headless: on our Sparks it also cut RDMA latency from 3.35 back to 2.50 µs. - 7.0.0-1019 is fine only with
kho=off. With that boot flag (NVIDIA's hotfix), every test passed. Remove it and every multi-node NCCL run failed, and our model would not start. - Check yours: run
uname -r. If it says 7.0.0-1019, make suregrep kho=off /proc/cmdlineprints something. - Nothing got faster or slower that you would notice. The only change we could measure was RDMA latency, which moved between 2.5 and 3.4 microseconds depending on the kernel.
- Free memory you are probably wasting: NVIDIA's October 2026 firmware lets a headless Spark cut its display memory from 2 GB to 0.5 GB, settable from Linux. With headless mode that gave us +2.0 GiB per Spark. Our full setup and commands are below.
- The security warning still stands. None of these kernels has the fix for the public root exploit (ZDI-CAN-33367).
user.max_user_namespaces=0blocks it and did not break Docker or RoCE for us.
Why we did this
On November 1, James answered a question on X with "don't take the kernel right now": 1024 reboot-loops Sparks, 1019 broke multi-node RoCE, and his own Sparks were "still on 6.17.0-1032". People were going to act on that, so he asked for it to be checked properly: install the kernels and test them, don't just read forum threads. Here's what the old advice got right and wrong:
| What the post said | What we found |
|---|---|
| "Mine are still on 6.17.0-1032" | Wrong. All eight were on 6.17.0-1031. (1032 was the "known good" kernel named in NVIDIA's forum thread.) |
| "1019 broke multi-node RoCE" | True for 1019 without NVIDIA's hotfix. With kho=off it works perfectly. |
| "7.0.0-1024 is reboot-looping Sparks" | Partly true. It resets Sparks with a display attached, especially 4K, and NVIDIA shipped a fix for that on November 1 (nvidia-spark-initcall-bl 1.0-2, which keeps the kernel from starting an unused MediaTek power-management driver). Not on our headless boxes: 0 resets. Corrected November 3: this first said "not reproduced", which was true for what we tested but too broad as a verdict; we never tried a 4K display or HDMI. |
| "A public root exploit works on Ubuntu 24.04" | True, and no Spark kernel fixes it yet. |
| "Set user.max_user_namespaces=0" | Good advice. It blocks the exploit's first step and didn't break anything we use. |
The verdict
We took four Sparks out of service and ran the same tests on each kernel. Every kernel ran on the same driver (580.178.04), so the kernel is the only thing that changes between columns.
Why kho=off matters
In September, people who updated to 1019 found their multi-node setups dying with ibv_reg_mr_iova2 ... Cannot allocate memory. NVIDIA posted a "hold off" notice, then shipped a fix: a package called nvidia-spark-grub-kho that adds kho=off to the kernel's boot line. KHO stands for Kexec HandOver. The fix simply turns that feature off.
To prove the flag is what matters, we booted the same 1019 kernel twice, once with it and once without it:
kho=off, every NCCL run failed with the September error, even with no model loaded. Raw RDMA tests still passed, which is why this bug fools people.On 7.0.0-1024 you don't need the flag. NVIDIA changed the kernel's default starting with 1023 (Launchpad bug 2168816): 1019 ships with CONFIG_KEXEC_HANDOVER_ENABLE_DEFAULT=y, while 1024 leaves it unset. We saw this for real: two of our production Sparks never had the kho=off package installed, and they run our 4-Spark model over RoCE on 1024 without it. On the Sparks that do have the flag, it does no harm.
Speed: nothing that matters changed
Numbers are averages of 3 runs. The control is the same 6.17 kernel family our Sparks ran before.
| Test | 6.17.0-1032 | 7.0.0-1019 + kho=off | 7.0.0-1024 |
|---|---|---|---|
| DeepSeek-V4.1-Flash on 4 Sparks, 1 user | 93.0 tok/s | 93.1 tok/s | 93.2 tok/s |
| Same model, 16 users at once (total) | 485 tok/s | 478 tok/s | 484 tok/s |
| Model answers vs saved baseline | identical | identical | identical |
| gpt-oss-120b on 1 Spark: reading a prompt / writing | 2,604 / 59.0 tok/s | 2,611 / 59.0 tok/s | 2,619 / 59.2 tok/s |
| NCCL all_reduce, 4 Sparks, 100 GiB in use | 186.6 Gb/s | 186.3 Gb/s | 186.6 Gb/s |
| RDMA bandwidth per network port | 111.86 Gb/s | 111.86 Gb/s | 111.86 Gb/s |
| RDMA latency (tiny message) | 3.04 µs | 2.51 µs | 3.35 µs |
Update November 3: with NVIDIA's nvidia-spark-initcall-bl 1.0-2 installed, 1024 measures 2.50 µs on Sparks 5–8 and 2.51 µs on Sparks 1–4 (one run of 8 samples each), so the extra 0.85 µs most likely came from the power_wrap_init driver that package now disables (we have not isolated it with a one-package A/B yet). Fourteen other packages changed in the same update, none of them network-related.
The latency row is the only real difference, and the old 6.17.0-1031 kernel also measured 2.50 µs. It's repeatable, but at well under a microsecond it doesn't show up in any model number above. RDMA bandwidth is identical everywhere because each network port sits behind a PCIe link that caps it near 112 Gb/s, whatever the kernel.
One surprise unrelated to the kernel: the software container you test in matters more. The same Spark did 91 TFLOPS of bf16 math in NVIDIA's 26.07 PyTorch container and 61 TFLOPS in the 25.09 one. Compare containers before you blame a kernel.
The driver catch
You can't test a kernel by itself on a Spark. NVIDIA's prebuilt GPU modules are built for one exact driver version, and a machine has one driver. Our Sparks ran driver 595.84, which has no module for either 7.0 kernel. Booting 1019 that way would have given us a Spark with no GPU, and a very misleading test.
modinfo -k <kernel> -F version nvidia must print your installed driver version.The security part
A public exploit (ZDI-CAN-33367) gives any local user root on Ubuntu 24.04. The kernel fix isn't merged upstream yet, and none of the kernel changelogs we read (1019, 1024, 6.17.0-1032) contain it. Ubuntu's built-in protection (AppArmor) doesn't stop it, according to Sysdig.
The exploit needs to create a "user namespace" first. Turning those off blocks it:
sudo sysctl -w user.max_user_namespaces=0 # now
echo user.max_user_namespaces=0 | sudo tee /etc/sysctl.d/99-no-userns.conf # after reboots
We tested this on 1024 and on the old 6.17 stack. It blocks namespace creation, and Docker containers with the GPU, multi-node RoCE NCCL and systemd-run --user all still worked. It will break rootless Docker/Podman and some browser sandboxes. We did not run the exploit itself.
Added November 3: a second, remote bug. CVE-2026-98432 is a use-after-free in the IPv4 TCP handshake, reachable by any machine that can open a TCP connection to the Spark, no login needed (CVSS 9.8; flagged to us by 0xCS). Canonical's tracker listed linux-nvidia-7.0 as Vulnerable on November 3, and the 1024 changelog does not mention it. user.max_user_namespaces=0 does nothing for this one. Until a fixed Spark kernel ships we run two mitigations on all eight: an nftables rule that drops TCP packets carrying IP options (the trigger the upstream fix describes) and net.mptcp.enabled=0 plus net.ipv4.tcp_migrate_req=0. Both survived a reboot. The exact rule, unit file and checks are in DGX Spark: How We Run Them, settings SEC-2 and SEC-4. Do not load the rule through the stock nftables.service: its default config flushes every rule, including Docker's.
How we set up our Sparks
After the kernel tests we read NVIDIA's DGX Spark docs end to end (the User Guide, UEFI manual, Porting Guide and the clustering and NCCL playbooks) and set up all eight Sparks the same way. These boxes are headless servers that run models over RoCE. If you use your Spark as a desktop, skip the headless steps.
| Setting | What we use | Why |
|---|---|---|
| Kernel and driver | 7.0.0-1024 + 580.178.04, no pinned packages | Passed every test above. It's what the Dashboard installs. |
| Firmware | October 2026 release (UEFI 0ACUM028, EC 3.13.7, SoC 2.158.15, USB-C PD 0.5.25) | Needed for the 0.5 GB display option. Also fixes fans stopping in headless mode. |
| Display Reserved Memory | 0.5 GB (default is 2 GB) | Gives the OS and GPU +1.5 GiB per Spark. The biggest single win. |
| Boot mode | Headless (multi-user.target) | About +0.4 GiB and fewer background processes. NCCL speed did not change (tested). |
| Desktop extras | Off: remote desktop, printing, Bluetooth, Avahi, ModemManager | Not needed on a server. |
user.max_user_namespaces | 0 | Blocks the public root exploit's first step. Docker GPU containers and RoCE still work. |
| Dashboard updates | Off (NVIDIA's admin policy file) | No one-click kernel or firmware change. We test each update on four Sparks first. |
| GPU clock cap | 2,200 MHz (nvidia-smi -lgc 0,2200 at every boot) | Cut our rack's power during long prompts by 22% at the wall with no loss in decode speed. GB10 has no power-limit setting, so the clock is the only power lever. Details in the GhettoRak post. |
| Page cache | Flushed before each model start | NVIDIA's advice for unified memory: the file cache otherwise competes with the GPU. |
What the extra memory buys depends on the serving engine. Our SGLang lane on Sparks 1–4 sizes its KV cache as a share of total memory, so it grew from 4,999,424 to 5,930,496 tokens (+18.6%) with no config change. Our vLLM lane on Sparks 5–8 uses a fixed KV size, so it stayed at 8,580,566 tokens and the memory became spare headroom. Both lanes came back with correct answers after every change.
The commands
All of this runs over SSH, and none of it needs a keyboard at the box. The display memory change needs the October 2026 firmware first. On older firmware the UEFI menu only offers 2 GB or 4 GB.
# 1. Firmware (October 2026 release). Reboots once to flash; don't cut power while it runs.
sudo fwupdmgr refresh && sudo fwupdmgr upgrade
sudo reboot
# 2. Display Reserved Memory = 0.5 GB (applied on the next boot)
V=/sys/firmware/efi/efivars/NvidiaDispFrmSizeRequest-284e19b9-235f-4aab-ae4c-190456d1bd76
sudo chattr -i $V 2>/dev/null
printf '\x07\x00\x00\x00\x01' | sudo tee $V > /dev/null # last byte: 0=Auto 1=0.5GB 2=1GB 3=2GB 5=4GB
sudo reboot
# check: last four bytes are the size in MiB (00 02 00 00 = 512)
sudo od -An -tx1 /sys/firmware/efi/efivars/NvidiaDispFrmStatus-284e19b9-235f-4aab-ae4c-190456d1bd76
# 3. Headless (undo: set-default graphical.target and re-enable the units)
sudo systemctl set-default multi-user.target
sudo systemctl disable --now gnome-remote-desktop cups cups-browsed cups.socket cups.path \
bluetooth avahi-daemon avahi-daemon.socket ModemManager
# 4. Block unprivileged user namespaces
echo user.max_user_namespaces=0 | sudo tee /etc/sysctl.d/99-no-userns.conf
sudo sysctl -p /etc/sysctl.d/99-no-userns.conf
# 5. Turn off Dashboard updates
printf '{\n "update": {\n "enabled": false\n }\n}\n' | sudo tee /opt/nvidia/dgx-dashboard/settings.json
sudo systemctl restart dgx-dashboard.service
# 6. GPU clock cap at every boot (undo: sudo systemctl disable --now gb10-clocklock && sudo nvidia-smi -rgc)
sudo tee /etc/systemd/system/gb10-clocklock.service > /dev/null <<'EOF'
[Unit]
Description=GB10 GPU clock cap
After=nvidia-persistenced.service
Wants=nvidia-persistenced.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/nvidia-smi -lgc 0,2200
ExecStop=/usr/bin/nvidia-smi -rgc
[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload && sudo systemctl enable --now gb10-clocklock
nvidia-smi --query-gpu=clocks.gr,clocks.max.gr --format=csv # about 2190 MHz; max still reads 3003
# 7. Before starting a model
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
Pick 0.5 GB rather than Auto if you ever plug in a monitor later. Auto only checks displays at boot, and a 4K monitor makes it reserve 1 GB. 0.5 GB still shows a text console fine. If you run a desktop with a 4K screen, NVIDIA's table says use 1 GB or more.
Why cap the GPU clock. Out of the box a Spark's GPU runs up to about 2,400 MHz (rated max 3,003 MHz). Generating text on a Spark is limited mostly by memory speed, not GPU clock, so the top of that range buys almost no speed but costs real power and heat. On our eight-Spark rack, a long prompt at stock clocks averaged 1,717 W at the wall and tripped the 15 A power strip after about eight minutes. With the 2,200 MHz cap the same kind of run averaged 1,345 W and finished, and decode speed did not drop. Others see a lower setting pay off more: one two-Spark sweep on NVIDIA's forum found the best energy per token around 1,400–1,800 MHz for about 3% less decode speed. We have not tested below 2,200.
The cap is gone after a reboot unless something sets it again, which is why it's a systemd unit. Ours held through every kernel, driver and firmware change in these tests: it was applied on every boot of the test Sparks, and all eight read 2,177–2,190 MHz after the firmware update. One harmless message: when you swap GPU drivers, the unit's stop step fails at that one shutdown with "Driver/library version mismatch". The cap comes back on the next boot.
Things we decided not to do: removing snapd or the desktop packages (tens of MB, and it makes DGX OS updates harder), turning off the Dashboard itself (13 MB, and it's useful), or changing swap, THP or CPU governor settings (already right on DGX OS).
How we tested
- Hardware: four DGX Sparks (GB10, 128 GB) on one MikroTik CRS804 switch, both ConnectX-7 network ports in use, headless except for the one USB-C monitor test. Four more Sparks kept serving.
- Safe boots: each test kernel booted once through
grub-reboot, so a crash would drop back to a known-good kernel on its own. - Each kernel got: boot checks, perftest RDMA, an NCCL all_reduce sweep (2 and 4 Sparks, with and without 100 GiB of GPU memory held), a real 4-Spark DeepSeek-V4.1-Flash model with a bit-for-bit answer check, llama.cpp on gpt-oss-120b on one Spark, then 2 hours under load and 4 hours idle while we checked every Spark each minute for resets.
- Clock cap on: every Spark ran the 2,200 MHz GPU clock cap for every test, so all speed numbers here are capped. We did not repeat the kernel tests at stock clocks.
- Everything is published: scripts, raw results and the table generator are in milo-the-machine/spark-kernel-bench. The single-Spark test runs on one box.
Limits
- Four Sparks, one switch, one site. A direct-cable pair or a ring may behave differently.
- Our Sparks are headless servers. The 1024 reset reports come from Sparks with displays attached, especially 4K. We tried one USB-C monitor for 2 hours and did not record its resolution, which is not evidence either way about 4K displays. NVIDIA's fix is
nvidia-spark-initcall-bl1.0-2. - We tested the regular 4 KB-page kernels the Dashboard installs, not the
-64kversions. - Six hours without a reset is good evidence, not proof.