J&M Labs Blog by Milo

Building the future, locally

← Back to Home

GhettoRak: an 8-node DGX Spark cluster

In plain English. GhettoRak is eight NVIDIA DGX Spark mini computers on a wire shelf. Two network switches tie them together so they can split one big AI model between them. Each Spark has one fast cable to a switch, and one extra-fast cable joins the two switches. It works: all eight talk to each other at full speed. With both switches in and the GPU clocks capped, the whole rack runs on one ordinary 15 A power strip, but only just: our heaviest load, GLM-5.3 on all eight reading a 754K-token prompt, averaged 1,412 W and peaked at 1,457 W (November 13), a little over the 1,440 W such a strip should carry for long. It held. Lighter models stay well under. New October 29: the big GB300 Station is on the same network now, with two 200G links. From the GPU's memory it now uses both at full speed, 392 Gb/s; from ordinary memory both share one narrower PCIe link and stop at about 231.

8
DGX Sparks, one per shelf
2
MikroTik CRS804 switches
200G
one cable per Spark
24 GB/s
all 8, NCCL all_reduce at 256 MiB
392 Gb/s
Station, both ports, GPU memory (new)
1.46 kW
peak at the wall, GLM-5.3 long prompt, both switches
2,200 MHz
GPU clock cap, all eight

Wiring

How GhettoRak is wired, October 29, 2026 Sparks 1 to 4 plug into switch 2 and Sparks 5 to 8 into switch 1, one 200G cable each. The two MikroTik CRS804 switches are joined by one 400G cable. The GB300 Station joins switch 2 with two 200G links, live since October 29. Management runs separately: house LAN to switch 1 to switch 2. Measured October 29: 128 of 128 jumbo pings pass including the Station, and all eight Sparks reach 24 GB/s in an all_reduce at 256 MiB. How it is wired Each Spark has one 200G cable. Four Sparks per switch. One 400G cable joins the switches. Spark 1 DGX Spark Spark 2 DGX Spark Spark 3 DGX Spark Spark 4 DGX Spark Spark 5 DGX Spark Spark 6 DGX Spark Spark 7 DGX Spark Spark 8 DGX Spark Switch 2 · CRS804 Sparks 1–4 · 2 m cables · 14″ shelf Switch 1 · CRS804 Sparks 5–8 · 1 m cables · 10″ shelf 400G 2 × 200G GB300 Station · new both ports live Oct 29 · reaches all 8 Sparks Management is separate house LAN → switch 1 → switch 2 (copper) Measured Oct 29: 128/128 jumbo pings incl. the Station · 8-Spark all_reduce 24 GB/s · 0 switch errors

As built, October 29, 2026, including the new GB300 Station link.

SwitchCage 1Cage 2Cage 3Cage 4
Switch 2Sparks 4, 3Sparks 1, 2400G linkStation p0, p1
Switch 1Sparks 8, 7Sparks 6, 5400G linkempty

Each Spark cage takes one breakout cable that splits into two 200G ends (2 m on switch 2, 1 m on switch 1). Which Spark sits on which end was read from the switches, not traced by hand.

Shelves

GhettoRak side view: what sits on each shelf A 54-inch chrome wire rack on 5-inch casters. Top shelf at 50 inches is bare. Sparks 1 to 8 sit one per shelf from 46 down to 18 inches, 4 inches apart. Switch 2 at 14 inches, switch 1 at 10, the UniFi management switch at 6, and power bricks with two smart strips at 1 inch. What sits on each shelf 12 × 18 × 54″ wire rack, built sideways so 18″ is the depth. Fronts face you; every cable leaves the back. 50″ Top shelf · bare 46″ Spark 1 42″ Spark 2 38″ Spark 3 34″ Spark 4 30″ Spark 5 26″ Spark 6 22″ Spark 7 18″ Spark 8 14″ Switch 2 · CRS804 10″ Switch 1 · CRS804 6″ UniFi Pro XG 8 · management bricks 2 smart strips 1″ 8 power bricks + 2 Tapo strips front ← 18″ deep → back House rules • One Spark per shelf • Shelves 4″ apart • Fronts face you • All cables out the back • Bungees hold the Sparks • Zip ties hold the switches • No extra fans • Keep the gold intake clear of walls

Side view, to scale. Shelf heights are measured from the floor deck.

The frame is a Shelving Inc SC121854-5 (12″ × 18″ × 54″) plus eight extra SI-S1218C decks, on 5″ threaded casters. It is built sideways so the 18″ side is the depth: a CRS804 is 387 mm deep with its handles, and 391 mm fits between the posts. Two Sparks do not fit side by side, so it is one per shelf. Build bottom up: bricks and strips, management switch, switch 1, switch 2, then Sparks 8 to 1. Zip-tie each switch to its deck and hold each Spark down with 6″ mini bungee cords as you go; cables last. CAD source: inference-box-ghettorak.scad.

Nothing printed holds anything down. The only prints are the green labels (inference-box-label-clip.scad + inference-box-label-measured.scad, PLA, on end, 5 mm brim) and these wire-shelf hooks in PETG for cable slack.

Photos

GhettoRak from the front: a tall chrome wire rack on casters, one gold DGX Spark per shelf with a green Spark 1 label, and switches, the management switch and power gear on the lower shelves.
Front, October 29. One gold Spark per shelf; switches and power below.
GhettoRak from the back: black braided fabric cables and white copper cords run down the posts, held with straps, with green tags marking each Spark's cables.
Back, October 29. Every cable leaves this side; green tags mark each Spark's cables.

Photos by James, resized for the web, otherwise unaltered.

New: the Station link

Live since October 29. The DGX Station GB300 now sits on the Spark network. One breakout cable in switch 2's fourth cage gives each of its two network ports a 200G link. From the Station, every Spark is one or two switches away.

GB300 Station to Spark RDMA speed and latency, measured October 29, 2026 RDMA write bandwidth in gigabits per second. Station to one Spark port 98, to one Spark over both of its functions 176, one Station port to two Sparks 196 which is line rate, the other Station port alone 196, both Station ports at once 231 out and 211 in from host memory, because both share one PCIe Gen5 x8 link, and 392 both ways from GPU memory through Data Direct. Latency 2.46 microseconds to a Spark on the same switch and 3.50 across the 400G link. The Station link: how fast it is RDMA write speed (Gb/s) between the GB300 Station and the Sparks. ib_write_bw, 8 s per test, Oct 29. Station → one Spark port 98 Spark limit per port Station → one Spark, both ports 176.2 One Station port → two Sparks 196 line rate Other Station port → two Sparks 196 line rate Both ports, host memory 230.8 PCIe Gen5 x8 limit Both ports, host memory, inbound 211.3 same link Both ports, GPU memory (Data Direct) 392.1 · full speed Both ports, GPU memory, inbound 392.1 · full speed 400 = two ports × 200G 0 100 200 300 400 Latency one tiny write, typical Spark on the same switch 2.46 µs Spark across the 400G link 3.50 µs +1.0 µs per extra switch Host memory shares a PCIe Gen5 x8 link (~231 Gb/s). GPU memory via Data Direct runs both ports at 392.

Measured October 29–30, 2026, with the Station serving its production model. Host memory unless marked GPU.

Why the Station's two network ports share about 231 Gb/s: the PCIe path The Grace CPU connects to the ConnectX-8 card's built-in PCIe switch through one PCIe Gen5 x8 link, about 252 Gb/s raw and 231 measured. Behind that switch sit both 200G network ports, both 8 TB drives of the models RAID, the 10G house network port and USB. A second Gen6 x16 path from the card straight to the GB300 GPU, called Data Direct, runs both ports at 392 Gb/s since a boot-setting fix on October 30. Why both ports share ~231 Gb/s Inside the Station: everything the network card carries goes through one narrow link to the CPU. Grace CPU + memory 72 cores GB300 GPU HBM PCIe Gen5 x8 ~252 Gb/s raw · 231 measured ConnectX-8 card · built-in PCIe switch fast inside (Gen6 x16), but everything below leaves through the x8 link above Fabric port 0 200G → switch 2 Fabric port 1 200G → switch 2 /models drives 2 × 8 TB NVMe House LAN (eno3) 10G USB Data Direct · Gen6 x16 straight to the GPU · 392 measured Host memory: ~231 Gb/s through the x8 link. GPU memory: 392 Gb/s through Data Direct. Read from the Station's PCIe tree (lspci) and measured with ib_write_bw, October 30, 2026.

The Station's network path, read from its PCIe tree on October 30, 2026.

Method: ib_write_bw -F -x 3 -q 4 -D 8 --report_gbits (64 KiB messages, one run with 1 MiB) and ib_write_lat (2-byte writes, 20,000 iterations, typical value), RoCE v2. Station: ConnectX-8 firmware 40.47.1088. GPU-memory runs: perftest 26.07.8 built with CUDA and dma-buf (--use_cuda_dmabuf, plus --use_data_direct for Data Direct).

Speed

NCCL all_reduce bus bandwidth by message size, October 29, 2026 Bus bandwidth in GB/s against message size from 32 KiB to 256 MiB. Four Sparks on one switch and four Sparks forced across the 400G link draw the same curve. All eight Sparks are slower below about 4 MiB because the ring has twice as many hops, and reach about 24 GB/s at 256 MiB, the same as four. How fast the Sparks talk to each other NCCL all_reduce bus bandwidth (GB/s) by message size. Higher is better. Average of two runs each. 0 5 10 15 20 25 32K 128K 512K 2M 8M 32M 128M 256M message size (bytes, log scale) 4 Sparks, one switch 4 Sparks, every hop across the 400G link All 8 Sparks, both switches The 400G link costs nothing measurable. 8 Sparks are slower on small messages (twice the hops), full speed on big ones.

Measured October 29, 2026, with the two-switch layout above. Two runs per line, averaged.

Bus bandwidth, GB/s1 MiB4 MiB256 MiB
4 Sparks, one switch (switch 1 / switch 2)10.7 / 11.520.8 / 20.923.9 / 23.7
4 Sparks, every hop across the link (2 runs)10.9 / 9.720.1 / 20.423.7 / 23.8
All 8 Sparks (2 runs)7.4 / 7.218.6 / 18.624.0 / 24.0

Method: torch.distributed all_reduce, one rank per Spark, fp32, 20 warm-up + 100 timed iterations, slowest rank counts, bus bandwidth = 2(n−1)/n × bytes ÷ time. NCCL 2.30.7 in lmsysorg/sglang:v0.5.22-cu130 with NCCL_PROTO=^LL128 and NCCL_MAX_NCHANNELS=8. This is a network test, not model speed. Why those two NCCL settings matter: NCCL tuning in plain English.

Power

Rack power at the wall with both switches, measured November 12 and 13, 2026 Whole-rack wall power with both switches installed and GPU clocks capped at 2,200 MHz. GLM-5.3 on all eight Sparks reading a 754K-token prompt: 1,412 watts mean and 1,457 watts peak, 17 watts over the 1,440 watt continuous limit of a 15 amp strip; the strip held. Two DeepSeek-V4.1-Flash lanes on long prompts: 1,245 watts mean and 1,324 watts peak. Idle with no model loaded: 504 watts. Power: both switches, one 15 A strip Whole rack at the wall (watts), GPU clocks capped at 2,200 MHz. GLM-5.3 on all eight · 754K-token prompt · November 13 Two DeepSeek-V4.1-Flash lanes · ~212K-token prompts · November 12 Idle · November 13 Long prompt, mean 1,412 W Long prompt, peak 1,457 W · 17 W over the line Long prompts, mean 1,245 W Long prompts, peak 1,324 W · 116 W under Idle, no model loaded 504 W 1,440 W = 80% of a 15 A strip 0 500 1,000 GLM-5.3 sat over the line for about 100 s of a 12-minute prompt. The strip held. For long GLM-5.3 runs, move one Spark to a second outlet.

Whole rack at the wall, summed from the outer strip's outlet meters (10-second samples), both switches in. DeepSeek measured November 12, 2026; GLM-5.3 and idle just after midnight, November 13.

Method: GLM-5.3 NVFP4 on vLLM, tensor-parallel over all eight, with the October 16 settings (4-bit KV cache, 1M context, MTP depth 3) and the same two needle-in-a-haystack prompts (513,814 and 754,233 tokens, both passed). DeepSeek: both live DeepSeek-V4.1-Flash TP4 lanes (Sparks 1–4 on SGLang, Sparks 5–8 on vLLM), fed cold prompts of random words (~212K tokens, 16-token answers) so no cache helps. Load driven from another machine. Tapo P316M outlet meters read through Home Assistant into Prometheus, 10-second samples, first 20 seconds of each phase dropped.

Copy it

Step-by-step guide with checks: Networking DGX Station and DGX Sparks with MikroTik CRS804.

Each switch (RouterOS 7.25). Keep the factory bridge for management only and turn its hardware offload off, so the fabric bridge gets the switch chip:

/interface/bridge/port/set [find bridge=bridge] hw=no
/interface/bridge/add name=fabric protocol-mode=rstp igmp-snooping=no
:foreach p in={"qsfp56-dd-1-1";"qsfp56-dd-1-5";"qsfp56-dd-2-1";"qsfp56-dd-2-5"} do={
  /interface/ethernet/set [find name=$p] auto-negotiation=no \
    speed=200G-baseCR4 fec-mode=fec91 l2mtu=9216 mtu=9000
  /interface/bridge/port/add bridge=fabric interface=$p hw=yes
}
# cage 3: the 400G link to the other switch
/interface/ethernet/set [find name=qsfp56-dd-3-1] auto-negotiation=no \
    speed=400G-baseCR8 fec-mode=fec91 l2mtu=9216 mtu=9000
/interface/bridge/port/add bridge=fabric interface=qsfp56-dd-3-1 hw=yes
# ECN marking on the default traffic class (cuts RoCE loss into a busy receiver)
/interface/ethernet/switch/qos/tx-manager/queue/set [find traffic-class=1] ecn=yes

Each Spark (N = Spark number). Delete the two auto-created "Wired connection" profiles first, then one netplan file, no gateway:

network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp1s0f0np0:
      addresses: [10.0.0.N/24]
      mtu: 9000
      dhcp4: no
      dhcp6: no
      accept-ra: no
      link-local: []
    enP2p1s0f0np0:
      addresses: [10.0.1.N/24]
      mtu: 9000
      dhcp4: no
      dhcp6: no
      accept-ra: no
      link-local: []

GB300 Station. Its ConnectX-8 will not link with the switch's forced 200G while it is auto-negotiating, so force it. Same addresses and ARP rules as the Sparks, with .9 as the host number:

sudo ethtool -s enP1p3s0f0np0 autoneg off speed 200000 duplex full lanes 4
sudo ethtool --set-fec enP1p3s0f0np0 encoding rs
sudo ip link set enP1p3s0f0np0 mtu 9000
sudo ip addr add 10.0.0.9/24 dev enP1p3s0f0np0
# same for enP1p3s0f1np1 with 10.0.1.9/24
sudo sysctl -w net.ipv4.conf.enP1p3s0f0np0.arp_ignore=1 net.ipv4.conf.enP1p3s0f0np0.arp_announce=2

On the switch, the two Station legs are set up exactly like the Spark legs (cage 4, lanes 1 and 5). Save them with netplan (addresses, MTU) and a small boot service (the ethtool and sysctl lines). For GPU-memory transfers at full speed, also run sudo rdma_topo check; if it reports wrong ACS values, run sudo rdma_topo write-grub-acs, sudo update-grub and restart.

NCCL:

NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
NCCL_SOCKET_IFNAME=enp1s0f0np0
NCCL_IB_ROCE_VERSION_NUM=2
NCCL_IB_GID_INDEX=3
NCCL_PROTO=^LL128
NCCL_MAX_NCHANNELS=8

Pin NCCL_SOCKET_IFNAME, or NCCL may start up over Docker or Tailscale. Check it worked: both links up at 200G with MTU 9000 on every Spark, then ping -M do -s 8972 between every pair on both subnets (112 of 112 for eight Sparks).

Software

ItemNow (Sparks checked November 3; switches and Station October 29)
OSUbuntu 24.04 (DGX OS), kernel 7.0.0-1024-nvidia, headless
GPU driver580.178.04, NVIDIA's stock driver; nothing held. Dashboard updates are off, and we test each update on four Sparks first.
Spark firmwareOctober 2026 release; display memory set to 0.5 GB
ConnectX-7 firmware28.45.4028
GPU clockscapped at 2,200 MHz at boot
SwitchesRouterOS 7.25 on both (RouterBOARD firmware 7.24.2), ECN on
GB300 Station NICConnectX-8, firmware 40.47.1088; card ports Gen6 x16, link to the CPU Gen5 x8

Kernel 7.0 is fine now. On November 2 and 3 we tested 7.0.0-1019 and 7.0.0-1024 on four of these Sparks, then moved all eight to 7.0.0-1024 on the stock 580.178.04 driver. 7.0.0-1019 does break multi-node RDMA unless you boot with kho=off; 7.0.0-1024 does not need it. Results and our full setup: Testing DGX Spark Kernels.

Buy

QtyWhatPart
112″ × 18″ × 54″ chrome wire rack, 5 shelves (build it sideways)SC121854-5
8Extra 12 × 18 shelves with clips (13 decks total)SI-S1218C
15″ threaded casters, 4-pack (threaded, not stem)SI-APC5WB-4
2MikroTik CRS804 switchCRS804-4DDQ-hRM
4400G → 2 × 200G breakout cable, MikroTik-coded, one per pair of Sparks: two 2 m, two 1 mNADDOD Q2Q56-400G-CU2 / -CU1
1400G cable, 1 m, switch to switchNADDOD QDD-400G-CU1
1400G → 2 × 200G breakout for the GB300 Station, 1 m (ours is Mellanox-coded)NADDOD Q2Q56-400G-CU1
215 A smart strip with per-outlet power metersTapo P316M
110G management switch, fanlessUSW-Pro-XG-8-PoE
9Cat6a patch cords (8 Sparks + switch 1 → switch 2)—
121 ft power cords for the Spark bricksC5 → 5-15P
16″ mini bungee cords with plastic-coated hooks, 15-pack (hold the Sparks)Joneaz B09CM1JN5S
1Long blue nylon zip ties (hold the switches)—

Skip: taller posts (54″ is enough), 0.5 m breakouts (could not make one work here), InfiniBand OSFP splitters and QSFP112 cables (wrong socket for the CRS804), 400G optics for the Station until 2 × 200G proves too slow, extra fans, and a UPS inside the frame.

Next

  1. Use Data Direct from real software (NCCL 2.27.3+, UCX 1.20+) for Station-to-Spark model and cache transfers.
  2. Give one Spark its own wall outlet before long GLM-5.3 runs: on one strip it peaks at 1,457 W.