October 22, 2026 · Milo (session model: custom/dsv41-flash-uva, DeepSeek-V4.1-Flash; reasoning setting not recorded) · one DGX Station GB300 (MSI XpertStation WS300) · companion to the sidecar results and the Station config

Adding an RTX PRO 6000 Blackwell Max-Q to the DGX Station GB300

Created · Updated

INSTALLED The 96 GB, 300 W Max-Q is in the Station's one double-wide PCIe 5.0 x16 slot and booted on the first attempt at 16:33 CDT on October 21, 2026. The card is the easy part. The surprise around it was power: firmware had already cut the GB300 module to 1,250 W before any NVIDIA sloshing software existed on the box. The one loose end — the 6000 has no mini-DisplayPort, so the JetKVM's old cable didn't fit — is closed: an active DP → HDMI adapter went in and the JetKVM console is working again.
Card96 GB / 300 W97,887 MiB visible; limits 250–325 W
Module cap1,250 Wenforced; 1,600 W requested
Static split1,550 Wof a 1,600 W supply
First boot16:33 CDTOct 21, one power cycle
Card resident66,213 MiB89 cold experts/layer at install; 95,682 MiB with today's 129
GPU-to-GPU0 linkstopo SYS; P2P not supported
Contents
  1. Where the model lives
  2. What went in
  3. The swap
  4. The power budget moved before the card did
  5. 120 V is not the limit
  6. The console cable had to change
  7. Two GPUs, no P2P
  8. What it bought
  9. Still open
  10. Receipts

Where the model lives

The card's job here is memory placement, not spare FLOPS. DeepSeek-V4.1-Flash-1017 is 40 MoE layers of 384 experts with 6 active per token: 268.9 GiB of routed experts and 189.2 GiB of Engram in a 475 GiB checkpoint, against 250.7 GiB of HBM and 494.5 GiB of Grace in this Station. The experts cannot all sit in HBM beside a 1M-token KV cache, so roughly 62 GiB of them have to live somewhere else — and every decode step that routes a token to one of those pays for wherever "somewhere else" is.

Where DeepSeek-V4.1-Flash lives in one DGX Station GB300, before and after the RTX PRO 6000 sidecar, and in the current Pg61 recipe Three memory pools on one Station with sizes and bandwidths: GB300 HBM3e 250.7 GiB at 7.1 TB/s, Grace LPDDR5X 494.5 GiB at 396 GB/s, RTX PRO 6000 Max-Q 95.6 GiB at 1.79 TB/s. Panel A, the GB300-only v23 lane: 208.9 GiB of routed experts plus 10.2 GiB of dense weights in HBM, 60 GiB of experts offloaded into Grace beside 189.2 GiB of Engram, and a 24 GB RTX PRO 4000 holding no model bytes. Panel B, the v25 sidecar lane: 206.6 GiB of hot experts in HBM and 62.3 GiB of cold experts resident on the RTX PRO 6000, with Grace holding only Engram. Panel C, the Pg61 recipe since November 10: 255 hot experts per layer (178.6 GiB, derived) plus a 26.5 GiB KV cache in HBM, 129 cold experts per layer (90.3 GiB, derived) on the RTX PRO 6000, which has 93.4 GiB resident, and Grace holding Engram plus a 48 GiB host KV tier. A bandwidth ladder compares HBM 7100, GDDR7 1792, NVLink-C2C 900 spec, LPDDR5X 396, measured C2C gather 355 and PCIe 5.0 x16 one-way 64 gigabytes per second. Where DeepSeek-V4.1-Flash lives in one Station 40 MoE layers × 384 experts, 6 active per token · routed experts 268.9 GiB · Engram 189.2 GiB · bar length ∝ GiB dense hot experts off-HBM experts Engram KV cache not model / spare A · Before: GB300 only (v23 lane) — 60 GiB of experts read out of Grace GB300 · HBM3e 250.7 GiB visible · 7.1 TB/s routed experts in HBM 208.9 experts 208.9 + dense 10.2 derived remainder 31.6 = KV cache + graphs + free RTX PRO 4000 22.4 GiB · display only 24 GB card — display only, no model bytes Grace · LPDDR5X 494.5 GiB visible · 396 GB/s Engram 189.2 60.0 GiB v23 kept 60 GiB of routed experts in Grace — whatever the positional offloader walked to first, not a ranking. Every token that routes to one reads those weights over C2C: the slowest read on the box, and it sits on the critical path. B · After: GB300 + RTX PRO 6000 (v25 lane) — the same role, computed on the card GB300 · HBM3e 250.7 GiB visible · 7.1 TB/s routed experts in HBM 206.6 experts 206.6 + dense 10.2 derived remainder 33.9 = KV cache + graphs + free RTX PRO 6000 Max-Q 95.6 GiB visible · 1.79 TB/s 62.3 GiB 62.3 experts + 2.4 scratch = 64.7 GiB resident of 95.6 Grace · LPDDR5X 494.5 GiB visible · 396 GB/s Engram 189.2 The 89 ranked cold experts (62.3 GiB) now live on the 6000 and are computed there: weights never move, only activations cross PCIe. The GB300 waits 0.015 ms per handoff (5.7M measured) while its hot experts run in parallel. Grace keeps only Engram. C · Today: Pg61 recipe (since Nov 10) — 129 cold experts per layer, card nearly full GB300 · HBM3e 250.7 GiB visible · 7.1 TB/s hot experts 178.6 experts 178.6 + dense 10.2 + KV 26.5 derived remainder 35.4 = graphs + free RTX PRO 6000 Max-Q 95.6 GiB visible · 1.79 TB/s cold 90.3 GiB 90.3 experts + 3.1 scratch = 93.4 GiB resident (95,682 MiB) Grace · LPDDR5X 494.5 GiB visible · 396 GB/s Engram 189.2 KV 48 Fewer hot experts free HBM for a 26.5 GiB KV cache (11.7M tokens, 11× a 1M request); Grace adds a 48 GiB host KV tier. Expert GiB scaled from the v25 split by expert count (derived); KV and card residency measured in Card HC1 on 2026-11-10. The hierarchy those bytes are read from — GB/s, linear scale Solid: datasheet peak. Dashed: measured on this box — a 256-byte gather over C2C, which is what the model actually asks for. HBM3e (GB300) 7,100 GB/s GDDR7 (RTX 6000) 1,792 GB/s NVLink-C2C (spec) 900 GB/s LPDDR5X (Grace) 396 GB/s C2C, measured gather 355 GB/s PCIe 5.0 x16, one way 64 GB/s Same model, same HBM. The card does not add capacity the model was short of — it takes the slowest tier off the critical path. Sizes: nvidia-smi, free, boot logs (A, B: Oct 22; C: Nov 10). Bandwidths: NVIDIA datasheets, except the measured C2C row.

Figure 1. Where the model's bytes sit in three lanes — GB300 only (A), the first sidecar lane on October 22 (B), and today's Pg61 recipe (C) — and the bandwidth of each tier. Bar length is proportional to capacity. A and B sizes were read on 2026-10-22 (66,213 MiB on the card). C's KV cache (26.54 GiB) and card residency (95,682 MiB) are measured in Card HC1; its expert GiB are scaled from B's split by expert count, and the 48 GiB Grace KV tier is the recipe's --kv-offloading-size 48. Bandwidths are NVIDIA datasheets except the measured C2C row, our own 256-byte gather microbenchmark.

Before the card that tier was Grace: 60 GiB of experts read over NVLink-C2C one gather at a time, on the critical path. With the card it is 62.3 GiB of ranked cold experts resident on the 6000 and computed there. Weights stop moving; only activations cross the PCIe slot — a few KB per token row, 0.015 ms per layer, measured over 5.7M handoffs. Today's Pg61 recipe pushes the same idea further: 129 cold experts per layer on the card, which leaves the GB300 room for a 26.5 GiB KV cache.

What went in

Read from the box, not from the invoice: nvidia-smi, lspci and uname on 2026-10-22.

 Before (2026-09-01 → 2026-10-21)After (2026-10-21 →)
CardNVIDIA RTX PRO 4000 Blackwell SFFNVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, PN 900-5G153-2200-000
Memory24 GB96 GB GDDR7 ECC — 97,887 MiB reported, 1,792 GB/s
Power70 W, no auxiliary plug300 W default, range 250–325 W, 1× 16-pin (12V-2x6)
Displays4× Mini DisplayPort4× full-size DisplayPort 2.1b
PCI address0004:01:00.00004:01:00.0 — same slot, same address
PCI IDRTX PRO 400010de:2bb4, arch 12.0 (SM120)
RoleDisplay + JetKVMDisplay + JetKVM + MoE expert sidecar

The host side is unchanged, and every number below is scoped to it: GB300 at 0009:06:00.0 (10de:31c3, SM103), 256,703 MiB HBM, GPU limit 1,300 W, Grace 72 cores with 494 GiB and no swap, Ubuntu 24.04.5 on kernel 7.0.0-1019-nvidia-64k, driver 595.91.07, CUDA 13.2.

Max-Q, not the 600 W Workstation Edition. The two card classes land in different power rows: an RTX rated 76–300 W leaves the module 1,250 W, and one rated above 300 W drops it to 950 W. Same 96 GB of memory either way, 300 W of heat instead of 600 W, in a chassis that has to share 1,600 W. The 600 W card was never an option for this box.

The swap

Powered down at 14:05 CDT, card collected, box back up at 16:33. One power cycle, no BIOS fight, no reseat.

RTX PRO 4000 SFF in the Station's double-wide slot, next to the GB300's copper cold plates
Second angle on the RTX PRO 4000 SFF, showing its single-slot cooler and the liquid-cooling loop behind it

Before: the RTX PRO 4000 SFF in the double-wide slot. 24 GB, 70 W, no auxiliary power, mini-DisplayPort only. Photo by James, unaltered.

RTX PRO 6000 Blackwell Max-Q seated in the same slot, with its gold fin stack, the 16-pin cable, and the STEREO/SYNC connectors visible

After: the Max-Q in the same slot. The RTX PRO 6000 silkscreen sits at the far end, next to the STEREO and SYNC connectors that give the professional cards away. Photo by James, unaltered.

The front half of the Station's interior: Grace cold plate with braided coolant lines, three M.2 modules, and DIMM slots

The front half, none of it touched: Grace cold plate and coolant lines, three M.2 modules, DIMMs. Photo by James, unaltered.

If you do this yourself: leave the cold plates, coolant tubes and the two 360 mm radiators alone. Opening the side panel can set the BMC's chassis-intrusion flag — a log entry, not a fault. A loose 16-pin is not a crash here either: MSI's chassis pulls its own power limit and keeps running, so a half-latched connector shows up as a slow box, not a dead one. Latch it until it clicks.

The power budget moved before the card did

The Station's supply is 1,600 W and it is shared between the GB300 module (GB300 plus Grace) and an optional RTX card. NVIDIA's calcGb110Cap table sets the module's share at boot from the rating of the installed RTX, not from whatever you set later:

Installed RTX ratingGB300 module cap
no RTX1,600 W
≤ 75 W (the 4000 SFF)1,600 W
76–300 W (this Max-Q)1,250 W
> 300 W (the 600 W Workstation Edition)950 W

What the box reports, live:

GPU 00000009:06:00.0   (GB300)
  Module Power Readings
    Current Power Limit        : 1250.00 W   <-- enforced
    Requested Power Limit      : 1600.00 W
    Default / Max Power Limit  : 1600.00 W

GPU 00000004:01:00.0   (RTX PRO 6000 Max-Q)
    Current Power Limit        : 300.00 W
    Min / Max Power Limit      : 250.00 / 325.00 W

There is no vsloshd on this box. Not installed, not running — that daemon moves watts between module and card at runtime on NVIDIA's DGX OS 7.5 images. So the 1,250 W is not sloshing: it is a static clamp set below host software, with 1,600 W still showing as requested. Nothing on the box can move that line, and nothing needs to.

DGX Station GB300 · 1,600 W supply, split as enforced Bigger card class, smaller module share. The card did not take the watts; the firmware took them before the card arrived. GB300 module (GB300 + Grace) RTX PRO 6000 1,250 W enforced 300 W 50 W unallocated 1,250 + 300 = 1,550 W of 1,600 W. No vsloshd installed, so nothing reallocates the leftovers at runtime. The driver still reports the module as requested 1,600 W and default 1,600 W. The enforced value is the one that binds. With the 70 W RTX PRO 4000 in place, the same table gave the module 1,600 W. This is what 230 W of extra card costs.

Figure 2. Static split at boot, as reported by nvidia-smi -q -d POWER on 2026-10-22.

The pre-install plan was wrong in the safe direction. It expected 300 W of card plus a 1,600 W module against a 1,600 W supply, and said to stop and confirm before powering on. The real sum is 1,550 W: the clamp we did not know about is what made it safe.

What the clamp costs when the lane is busy

Power-by-phase sampling on the sidecar lane — five phases, nvidia-smi module draw and clock-event reasons:

PhaseGB300 module p99SW power cap activeGB300-only control
idle, model loaded385 W0.00% of samples361 W · 0.00%
C1 decode861 W0.00%872 W · 0.00%
C32 decode1,263 W2.42%972 W · 0.00%
64K-token prefill1,353 W89.47%1,279 W · 65.00%
32× 8K prefill1,342 W85.71%1,300 W · 51.72%

The GB300 was already hitting its power cap in prefill before the card arrived — 65% of 64K-prefill samples on the GB300-only arm — so this is not a new cliff. But the sidecar lane spends more of prefill capped (89%), and instantaneous module draw runs above the 1,250 W line while the platform clamps after the fact. Prefill on this box is power-bound, and 230 W of extra card is part of why.

DO NOT Do not install vsloshd to "recover" the 50 W. The documented best case is 50 W of module headroom, and only when the module sits within 50 W of its cap while the card has more than 100 W spare — not the C1 case, where the card idles around 90–145 W. The downside is worse: the card's policy maximum is 325 W, and if the algorithm reads that as its rating, the module drops into the >300 W row at 950 W. Leave it alone.

Electrical. Worst case is 1,600 W DC through a Platinum supply, ≈1,800 W at the wall, ≈15 A at 120 V. That fits a dedicated 20 A circuit (16 A continuous) with a little margin and does not fit a 15 A circuit. The Station has its own dedicated 20 A circuit, nothing else on it.

120 V is not the limit

This supply delivers its full 1,600 W from 115 to 240 V. A US 120 V wall sits inside the full-power band; the only de-rated band is 100–114 V, which is Japan. The full treatment — clock loss per phase, peaks versus averages, when 240 V is worth it — is GB300 with RTX 6000, and 120V Power.

AC inputPSU capability
100–114 V1,300 W
115–240 V1,600 W — our wall is 120 V

The measured peak agrees with the nameplate. Our worst prefill instant was a 1,353 W module with a 194 W card: 1,547 W DC, ≈14.5 A. Fans, pumps, drives and the NIC add roughly 60–120 W, which puts a full-system peak at 1,607–1,667 W DC, ≈15.0–15.6 A against 16 A of continuous capacity — 0.4–1.0 A of margin. Thin enough that the word "dedicated" is doing real work. The circuit is not what limits this machine; it also has no room for a space heater.

What 240 V would actually change

The signature of a wall throttle would be clocks falling while the power limits still read full. We have never seen that here; what we see is sw_power_cap, which is the module cap working as the firmware intends. A ±3% spread between runs is the neutral band on this box, not a symptom.

The console cable had to change

The 4000 had four mini-DisplayPorts, with a mini-DP dongle for the JetKVM. The 6000 has four full-size DisplayPort 2.1b outputs, so the dongle did not fit. Fixed: an active full-size DP → HDMI adapter is in, and the JetKVM console works again.

Two GPUs, no P2P

$ nvidia-smi topo -m        # GB300 <-> RTX
        CPU    NIC
GPU0    SYS    SYS
GPU1    SYS    SYS

$ nvidia-smi topo -p2p r    # P2P read
GPU0    NS
GPU1    NS

No NVLink and no peer-to-peer over PCIe, so anything split across the two GPUs moves through host memory. That is the only available path, not a tuning choice. On driver R595 a CUDA process also sees one GPU modality (ATS for the GB300, non-ATS for the RTX), and /etc/cuda-gpu-select/env pins CUDA_VISIBLE_DEVICES=1 so containers keep getting the GB300.

What it bought

NOW · NOV 13 Nothing is serving on the Station right now — on purpose. James gave the box to a 48-hour research campaign that runs until November 14 at 8:11 PM CST, looking for a better all-round recipe. Both GPUs have been idle between experiment steps since the evening of November 12. The recipe it will come back to is Pg61, which is kept intact. It does not restart automatically; it comes back when James says so.

The card is not display and not "more FLOPS on the side": it holds the cold experts and computes them locally, so activations cross a PCIe slot instead of tens of GiB of weights crossing NVLink-C2C. It bought this in two steps.

Day one: v25, October 22

89 cold experts per layer on the card, against the GB300-only lane. The full same-window measurement is in DeepSeek-V4.1-Flash on a GB300 + RTX PRO 6000 Sidecar:

one user, real text187 → 279tok/s, +49%
first token0.66 → 0.19 sat C1
16 users3.0×945 → 1,414 tok/s
1M prompt81 → 33 sprefill 12.3K → 30.1K tok/s

Twelve quality gates passed in the same window — teacher-forced flips, GSM8K-1319 flat at 96.97%, BFCL dev and held-out within 1 point, needle 500/500, per-class decode 20/20 — so the card is not buying speed with accuracy.

Today's recipe: Pg61

Since October 29 the sidecar lane has run Pg61: Garner McCloud's DS4.1 engine on stock vllm/vllm-openai:v0.31.0, under our patches and the same fail-closed watchdog on the second card. On November 10 its map moved to 255 hot / 129 cold, so the card now holds 95,682 MiB of its 97,887. The numbers come from two different tests, so they are listed separately and not multiplied together:

MeasureResultSource
One user, per class, vs h2-REF (the previous sidecar lane)1.31× mean (1.26–1.36× by class), e.g. code 495 vs 369 tok/sCard G5, Oct 29–31: 40 prompts per class, 3 seeds
One user, mixed prompts412.6 tok/s, 7.64 ms per stepCard HC1, Nov 10
32 users, aggregate1,582–1,818 tok/s across 6 repsCard HC1, two control boots
First token, 61K-token prompt~1.5 sCard HC1
KV cache at boot11.7M tokens (11× a 1M request)Card HC1

Receipts and the rollback command are in the J-M-Recipes sidecar recipe. The 255-vs-275 map test is in Card HC1: 275 hot made long prompts 11–12% faster but missed the single-user bar and halved the KV pool, so 255 stayed. Pg61 is still labeled experimental.

Still open

  1. vsloshd. Decision made: not installing it. Reopen only if module power sits at ≥1,200 W with power-cap events during prefill and an OEM-supplied package appears.
  2. Firmware. No BIOS, BMC or VBIOS release is documented to change the module cap. Do not chase one for power; if firmware is wanted for other reasons it comes from the OEM as a Galaxy-GB300-WS bundle over Redfish, not LVFS.
  3. Soak. Pg61's 48-hour clock kept restarting through early November; the last start was November 10 at 5:38 AM CST. The lane was still up with no watchdog trip on the afternoon of November 12, past the 48-hour mark. That uptime alone does not close the soak gate, and the lane has been dark since for the research campaign. The soak is still owed — it is a serving question, not an install one.

Receipts

# what is in the box
nvidia-smi -L
nvidia-smi --query-gpu=index,name,memory.total,power.limit,power.min_limit,power.max_limit,pci.bus_id --format=csv
nvidia-smi -q -d POWER            # GPU and Module Power Readings, both GPUs
lspci -nn | grep -iE "vga|3d|display"
systemctl is-active vsloshd; dpkg -l | grep -Ei "vslosh|smbpbi"

# topology
nvidia-smi topo -m
nvidia-smi topo -p2p r
cat /etc/cuda-gpu-select/env

Boot history, from the box: journalctl --list-boots shows the previous boot ending 2026-10-21 14:05:44 CDT and the current one starting 16:33:31 CDT. Lane state, power-by-phase tables and the sidecar gates live in ~/dsv41/results/ledger.md and ~/dsv41/m3/ on the Station.

Sources