In plain English. This is how we tie eight NVIDIA DGX Spark mini computers and one DGX
Station GB300 into one fast private network, using two MikroTik CRS804 switches. Each Spark gets one 200G cable. The Station gets two. One 400G cable joins
the switches. Five steps, each with a check. If you only remember three things: turn off hardware
offload on the switch's factory bridge, force the Station's ports to 200G, and
run rdma_topo check on the Station. The switches also put RDMA traffic in its own
queue so ordinary TCP never stalls behind it; no program has to do anything for that. We also tested five
suggestions from 0xCS (@FT_IOxCS); the results are under
Suggestions from 0xCS.
This page is the recipe. The build itself (shelves, photos, power) is on the GhettoRak page.
The five steps. Each one ends with a check.
Parts
| Qty | What | Part |
|---|---|---|
| 2 | MikroTik CRS804 switch (four 400G QSFP-DD cages each) | CRS804-4DDQ-hRM |
| 4 | 400G → 2 × 200G breakout, MikroTik-coded, one per two Sparks | NADDOD Q2Q56-400G-CU2 (2 m) / -CU1 (1 m) |
| 1 | 400G cable, switch to switch, 1 m | NADDOD QDD-400G-CU1 |
| 1 | 400G → 2 × 200G breakout for the Station, 1 m (ours is Mellanox-coded; it works) | NADDOD Q2Q56-400G-CU1 |
| 2+ | Copper patch cords for management (house LAN → switch 1 → switch 2) | Cat6a |
Use copper (DAC) cables, not optics, at these lengths. We could not make a 0.5 m breakout work. One switch is enough for four Sparks plus the Station, or for eight Sparks.
1. Cabling
As built. Sparks 1–4 and the Station on switch 2, Sparks 5–8 on switch 1.
- Each breakout cable has one 400G end for the switch and two 200G ends. Each 200G end goes
to one Spark. On the switch, the two ends show up as lanes 1 and 5 of that cage (for example
qsfp56-dd-1-1andqsfp56-dd-1-5). - The Station uses one breakout for both of its ports, both on cage 4 of the same switch.
- Put the Station and the Sparks it talks to most on the same switch. Crossing to the other switch costs about 1 µs (2.46 vs 3.50 µs) and no measurable bandwidth: four Sparks forced to cross on every hop matched four on one switch.
- Label both ends of every cable. Then read the mapping back from the switch rather than trusting a drawing.
Check: link light on every leg.
2. Switches
RouterOS 7.25 on both switches. Management stays on the copper ports. The fast ports get their own bridge, and only that bridge may use the switch chip. If the factory bridge keeps hardware offload, the fabric bridge falls back to the CPU and is slow.
# management: keep the factory bridge for ether ports, offload off
/interface/bridge/port/set [find bridge=bridge] hw=no
/interface/bridge/set bridge name=mgmt
# fabric bridge on the fast ports
/interface/bridge/add name=fabric protocol-mode=rstp igmp-snooping=no
:foreach p in={"qsfp56-dd-1-1";"qsfp56-dd-1-5";"qsfp56-dd-2-1";"qsfp56-dd-2-5"} do={
/interface/ethernet/set [find name=$p] auto-negotiation=no \
speed=200G-baseCR4 fec-mode=fec91 l2mtu=9216 mtu=9000
/interface/bridge/port/add bridge=fabric interface=$p hw=yes
}
# cage 3: the 400G cable to the other switch
/interface/ethernet/set [find name=qsfp56-dd-3-1] auto-negotiation=no \
speed=400G-baseCR8 fec-mode=fec91 l2mtu=9216 mtu=9000
/interface/bridge/port/add bridge=fabric interface=qsfp56-dd-3-1 hw=yes
# ECN on the default traffic class
/interface/ethernet/switch/qos/tx-manager/queue/set [find traffic-class=1] ecn=yes
# RDMA in its own queue, sorted by protocol (RoCEv2 = UDP 4791), so hosts and programs need no changes.
# Congestion notices (DSCP 48) go first; RDMA (class 3) and everything else (class 1) share bandwidth 50/50.
/interface/ethernet/switch/qos/profile/add name=roce dscp=26 traffic-class=3 automap=no
/interface/ethernet/switch/qos/profile/add name=cnp dscp=48 traffic-class=6 automap=no
/interface/ethernet/switch/rule/add switch=switch1 ports="qsfp56-dd-1-1,qsfp56-dd-1-5,qsfp56-dd-2-1,qsfp56-dd-2-5,qsfp56-dd-3-1,qsfp56-dd-4-1,qsfp56-dd-4-5" \
mac-protocol=ip protocol=udp dst-port=4791 \
dscp=48 new-qos-profile=cnp keep-qos-fields=yes
/interface/ethernet/switch/rule/add switch=switch1 ports="qsfp56-dd-1-1,qsfp56-dd-1-5,qsfp56-dd-2-1,qsfp56-dd-2-5,qsfp56-dd-3-1,qsfp56-dd-4-1,qsfp56-dd-4-5" \
mac-protocol=ip protocol=udp dst-port=4791 \
new-qos-profile=roce keep-qos-fields=yes
/interface/ethernet/switch/qos/tx-manager/queue/set [find traffic-class=1] schedule=high-priority-group weight=1
/interface/ethernet/switch/qos/tx-manager/queue/set [find traffic-class=3] schedule=high-priority-group weight=1 ecn=yes
- On the Station's switch, add
qsfp56-dd-4-1andqsfp56-dd-4-5to the same loop. - ECN lets the switch warn senders before its queue overflows. With it on, traffic into the Station lost 85% fewer packets (4.24 M → 0.65 M in one test). Spark-to-Spark speed did not change. We left flow control (pause frames) off: turning it on gained nothing.
- RDMA in its own queue. Without it, a pile-up of RDMA traffic into one machine starved TCP to that machine to almost nothing (0.002–0.015 Gb/s). With the rules above, TCP kept 41–94 Gb/s, and small RDMA messages into a busy Spark stopped hitting multi-millisecond stalls. Nothing changes on the hosts, and it works for any model or engine. Changing the queues pauses forwarding for a moment, so do it between jobs. Details under Suggestions from 0xCS. No PFC: it hurt here.
- Second switch: give it its own name, limit SSH/WinBox to your admin machines, and plug its management port into switch 1's.
Check: /interface/bridge/port/print shows the H (hardware) flag on
every fabric port, and /interface/ethernet/monitor qsfp56-dd-1-1 once shows 200G, link ok. During
RDMA traffic, /interface/ethernet/print stats shows tx-queue3-packet rising (RDMA) and
tx-queue6-packet rising (congestion notices).
3. Sparks
One 200G cable shows up on a Spark as two interfaces, one per half of the card. Give each half its own subnet. N is the Spark's number. Delete the auto-created "Wired connection" profiles first, then add one netplan file with no gateway:
# /etc/netplan/40-cx7-fabric.yaml
network:
version: 2
renderer: NetworkManager
ethernets:
enp1s0f0np0:
addresses: [10.0.0.N/24]
mtu: 9000
dhcp4: no
dhcp6: no
accept-ra: no
link-local: []
enP2p1s0f0np0:
addresses: [10.0.1.N/24]
mtu: 9000
dhcp4: no
dhcp6: no
accept-ra: no
link-local: []
The Sparks link at 200G with auto-negotiation left on, so nothing else is needed. Because both halves sit on
one network, each must answer ARP only for its own address. Our Sparks already had these set system-wide; confirm
yours, and add them to a file in /etc/sysctl.d/ if not:
sysctl net.ipv4.conf.all.arp_ignore net.ipv4.conf.all.arp_announce net.ipv4.conf.all.rp_filter
# want 1, 2, 2
MTU 9000 is optional. At the default 1500, RDMA between two Sparks ran about 5% slower (185 vs 196 Gb/s), the Station's GPU-memory path dropped from 392 to 370 Gb/s, and NCCL lost about 5% on 8–32 MiB messages, with no change at 2 MiB and below (tested). If you use 9000, every port on the path must have it.
Check: both interfaces up, 200G, MTU 9000. Kernel 7.0.0-1024 works for multi-node RoCE; we tested it and run it on all eight. 7.0.0-1019 needs NVIDIA's kho=off fix on the kernel command line or NCCL fails to register memory. Corrected November 3: this first said to stay on 6.17 because of NVIDIA's advisory; that advice is out of date. Our current settings, with checks, are in DGX Spark: How We Run Them.
4. Station
The Station's ConnectX-8 will not link while it auto-negotiates against the switch's fixed
200G. Force it, and set RS FEC to match. Addresses go in netplan (same shape as the Sparks, host number
.9: 10.0.0.9 on enP1p3s0f0np0, 10.0.1.9 on
enP1p3s0f1np1). Netplan cannot store the ethtool part, so a small boot service does it:
# /etc/systemd/system/station-fabric-link.service
[Unit]
Description=Force CX8 fabric ports to 200G-CR4 / RS-FEC, set ARP rules
Wants=sys-subsystem-net-devices-enP1p3s0f0np0.device sys-subsystem-net-devices-enP1p3s0f1np1.device
After=sys-subsystem-net-devices-enP1p3s0f0np0.device sys-subsystem-net-devices-enP1p3s0f1np1.device
Before=NetworkManager.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/ethtool -s enP1p3s0f0np0 autoneg off speed 200000 duplex full lanes 4
ExecStart=/usr/sbin/ethtool --set-fec enP1p3s0f0np0 encoding rs
ExecStart=/usr/sbin/ethtool -s enP1p3s0f1np1 autoneg off speed 200000 duplex full lanes 4
ExecStart=/usr/sbin/ethtool --set-fec enP1p3s0f1np1 encoding rs
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f0np0.arp_ignore=1
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f0np0.arp_announce=2
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f0np0.rp_filter=2
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f1np1.arp_ignore=1
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f1np1.arp_announce=2
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f1np1.rp_filter=2
[Install]
WantedBy=multi-user.target
sudo systemctl enable --now station-fabric-link.service
Then fix the GPU path. The card has a direct path to the GB300's memory, called Data Direct. Out of the box, a PCIe isolation setting (ACS) puts the card and the GPU in different groups, and GPU-memory transfers fail with error 524. NVIDIA's tool finds and fixes it (NVIDIA's Station guide):
sudo rdma_topo check # look for FAIL lines about ACS / iommu_group
sudo rdma_topo write-grub-acs # writes /etc/default/grub.d/config-acs.cfg
sudo update-grub
# then restart the Station
Check: after the restart, rdma_topo check prints only OK lines, and both ports
come back at 200G by themselves.
5. Test
Jumbo pings between every pair, on both subnets. -M do forbids splitting, so a
wrong MTU anywhere fails loudly:
ping -c 3 -M do -s 8972 10.0.0.2
ping -c 3 -M do -s 8972 10.0.1.2
Eight Sparks plus the Station gave 128 of 128. Then RDMA bandwidth, one pair at a time
(ib_write_bw from perftest; on the Station, a CUDA build adds the GPU flags):
# receiver (Spark 1, first half of the card)
ib_write_bw -F -x 3 -q 4 -D 8 --report_gbits -d rocep1s0f0
# sender (another Spark)
ib_write_bw -F -x 3 -q 4 -D 8 --report_gbits -d rocep1s0f0 10.0.0.1
# Station sender, from GPU memory through Data Direct
ib_write_bw -F -x 3 -q 4 -D 8 --report_gbits -d mlx5_0 \
--use_cuda_bus_id=<GPU PCI address> --use_cuda_dmabuf --use_data_direct 10.0.0.1
Then NCCL, with these settings on every Spark:
NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
NCCL_NET=IB
NCCL_SOCKET_IFNAME=enp1s0f0np0
NCCL_IB_ROCE_VERSION_NUM=2
NCCL_IB_GID_INDEX=3
NCCL_PROTO=^LL128
NCCL_MAX_NCHANNELS=8
Pin NCCL_SOCKET_IFNAME, or NCCL may start up over Docker or Tailscale. Set
NCCL_NET=IB too. Without it, a broken RDMA path makes NCCL quietly fall back to plain
TCP. In our test that still gave correct answers, at 1.17 GB/s instead of about 23, with no error. With it, NCCL
stops with an error. NCCL's start-up traffic can also run over the 10GbE management port instead
(NCCL_SOCKET_IFNAME==enP7s7); the speed was the same.
Test more than one message size. Big messages can look fine while medium ones are slow.
The last two settings above (NCCL_PROTO, NCCL_MAX_NCHANNELS) are what fix the middle;
0xCS found that pair on his own cluster, and it reproduced here.
Bus bandwidth in GB/s, all_reduce, one rank per Spark:
| Message | 4 Sparks, NCCL defaults | 4 Sparks, settings above | 8 Sparks, settings above |
|---|---|---|---|
| 512 KiB | 1.45 | 6.77 | 3.76 |
| 1 MiB | 1.36 | 11.50 | 7.43 |
| 2 MiB | 2.65 | 16.45 | 12.76 |
| 4 MiB | 8.26 | 20.97 | 18.55 |
| 8 MiB | 14.4 | 22.40 | 21.64 |
| 256 MiB | 22.07 | 22.78 | 23.99 |
NCCL 2.30.7, 20 warm-up + 100 timed calls per size. 4 Sparks measured October 26, 8 Sparks October 29. Eight ranks are slower on small messages because the ring has twice as many steps.
Pass: 4 Sparks, 1 MiB at least 9 GB/s and 256 MiB at least 22. For 8 Sparks expect about 7 and 24.
Then read the counters, before and after each test. Throughput alone does not show whether the switch dropped or paused anything to get it:
# switch: one line per running fabric port
:foreach i in=[/interface/ethernet/print stats as-value where name~"qsfp56-dd" and running] do={:put $i}
# look at tx-drop-packet, rx-pause, tx-pause, rx-fcs-error, rs-fec-corrected
# each Spark
cat /sys/class/infiniband/rocep1s0f0/ports/1/hw_counters/{np_cnp_sent,out_of_sequence,packet_seq_err}
ethtool -S enp1s0f0np0 | grep -E 'discards_phy|pause_ctrl_phy|prio._pause'
Clean means no drops, no pause frames and no FEC corrections. Every NCCL run we did was clean. Many-to-one
RDMA tests are not (see below). The two halves of a Spark's card share one port, so the
ethtool port counters read the same on both.
What speed to expect
Measured on this fabric, October 29–30, 2026. Gb/s; divide by 8 for GB/s.
Why: the links inside each machine, not the 200G cables, set most of these limits.
- Over RDMA a Spark fills its 200G port: 196 Gb/s with both halves busy (98 each). Over TCP it tops out near 185. Its card reaches the chip through two PCIe Gen5 x4 links of about 112 Gb/s each, so one flow on one half stops near 112. Corrected October 30: this first said a Spark is a 185 Gb/s machine; that is the TCP number, and a two-half RDMA test reached 196.
- The Station from ordinary memory stops near 231 Gb/s for both ports together (211–215 inbound without ECN, 228 with). Both ports share one PCIe Gen5 x8 link to the Grace CPU. One port alone runs at 196, its full line rate.
- The Station from GPU memory runs 392 Gb/s both ways through Data Direct, after the step-4 fix. Software must ask for it: NCCL 2.27.3 or newer, UCX 1.20 or newer (NVIDIA).
- Copies are usually the slowest thing.
rsyncorscpover ssh is limited by one CPU core doing encryption, under 10 Gb/s. Run several copies at once, or use the rsync daemon. For hundreds of GB into one Spark, its SSD is the limit: about 7 GB/s for the first ~50 GB, then about 1.7 GB/s. Serve big model files from one place (NFS or RDMA) instead of copying them to every Spark.
Suggestions from 0xCS, tested
0xCS (@FT_IOxCS), who runs four Sparks on a MikroTik CRS812, sent five suggestions. We tested them on our eight Sparks on October 30. Thanks: two are now in the recipe above, and the third changed how we read our own drop counters.
| Suggestion | What we measured | Verdict |
|---|---|---|
| MTU 1500 works; jumbo frames are optional | RDMA, both halves: 185 vs 196 Gb/s. NCCL, 4 Sparks: the same up to 2 MiB, about 5% lower at 8–32 MiB | True. 9000 buys about 5.6% wherever the cable is the limit: the Station's GPU-memory path fell from 392.1 to 370.3 Gb/s at 1500, both directions. Its ordinary-memory path (PCIe-limited) did not change. We keep 9000 |
| Put RoCE in its own traffic class with ECN and PFC | See below | Partly. A separate RDMA queue sharing bandwidth with TCP helps, and we adopted it, sorted by the switch so nothing has to mark traffic. PFC hurt on Spark ports. A strict class for small messages helped most but needs each program to mark its traffic, so we left it out |
| Run NCCL start-up over the 10GbE management port | Same speed as start-up on the fabric | Works. Add NCCL_NET=IB with it, or a dead fabric silently becomes a
1 GB/s TCP one |
| Test small and medium message sizes, not only 256 MiB | With NCCL defaults, 1 MiB ran at 1.36 GB/s while 256 MiB looked fine | Adopted (table in step 5) |
| Read switch counters before and after | NCCL runs: zero drops, pauses and FEC corrections | Adopted (step 5) |
The RoCE traffic class, in detail
The idea: an RDMA program marks its packets (NCCL: NCCL_IB_TC=104, which is DSCP 26). The switch
puts them in their own queue, traffic class 3, and congestion notices (DSCP 48) in a queue that always goes
first. Optionally, PFC lets a full queue pause only class 3 instead of dropping packets. Each host must read
DSCP (mlnx_qos --trust dscp); ours read VLAN priority by default, so without that the marking does
nothing on the host side. Two things about this switch: it marks ECN only once a queue has passed its
shared-buffer limit, and there is no lower threshold to set. We measured queues of up to 3.92 MB on the
Station's port, about 157 µs of waiting at 200G.
Tests, run October 30 on all eight Sparks and the Station:
- Pile-up: 7 Sparks send 14 RDMA flows into one Spark (about 7:1), or all 8 send 16 flows into the Station (about 4:1), into its ordinary memory or its GPU memory. 10 seconds.
- Mixed: a pile-up plus one Spark sending TCP to the same receiver.
- Small messages: 7 KiB RDMA round trips between the Station and a Spark, idle and during a pile-up. 7 KiB is the size of the messages our Kimi-K3 split sends every layer.
- NCCL: the 8-Spark all_reduce curve.
Pile-ups: packets lost (share of what arrived, one number per run):
| Setting | Into a Spark | Into Station memory | Into Station GPU memory |
|---|---|---|---|
| As built: one class, ECN | 3.0%, 5.1%, 0.7% | 0.7%, 2.3% | 0.6%, 0.3% |
| Class 3 + ECN | 6.4%, 8.9% | not run | not run |
| Class 3, classes 1 and 3 share 50/50 | 5.1%, 4.5% | 1.3%, 0.9% | 0.9%, 0.9% |
| Class 3 + PFC | 0, 0 | 0, 0, but half the flows fell to about 2 Gb/s | 0, 0 |
| Adopted: switch sorts RDMA by port, 50/50 | 3.0%, 4.0% | 3.1%, 1.9% | 1.1%, 0.9% |
Loss varies up to 10× between runs of the same setting, so only PFC's zero is a clear difference. The losses are not just a start-up burst: into the Station they ran at about 140–290 thousand packets a second for the whole run after a burst in the first second.
Mixed: TCP during the pile-up, in Gb/s (alone it gets 109 into a Spark, 111 into the Station):
| Setting | Into a Spark | Into Station GPU memory | Into the Station, across the switch-to-switch cable |
|---|---|---|---|
| As built | 0.002–0.005 | 4.5–5.9 | 0.005–0.015 |
| Class 3 + PFC | 0.002 | 6.6–8.4 | not run |
| Class 3 + PFC, 50/50 | 50–59, but RDMA fell to 77 Gb/s in three of four runs | 89 | not run |
| Class 3, 50/50 | 50–54 | 92–93 | 61 |
| Class 3, 75/25 for RDMA | 28–45 | 51–53 | not run |
| Class 3, 80/20 for RDMA | 22–28 | 41 | not run |
| Adopted: switch sorts RDMA by port, 50/50 | 42–58 | 91–94 | 41–46 |
2–4 runs each. RDMA into the Station's GPU memory got 349–350 Gb/s at 50/50, 371–372 at 75/25 and 376 at 80/20 (12-second averages; the TCP ran for 6 of the 12 seconds). Into a Spark, RDMA gained little from a bigger share, because that Spark's own PCIe link was already the limit.
Small messages: 7 KiB round trip, Station ↔ Spark on the same switch, in µs, two runs:
| Setting | Idle, typical | Pile-up into the Station: typical / 99.9% / worst | Pile-up into the Spark: typical / 99.9% / worst |
|---|---|---|---|
| As built | 4.7 | 43–46 / 642–934 / 37,743–38,950 | 68–71 / 621–879 / 16,596–19,382 |
| Class 3, 50/50, small messages in class 3 too | 4.7 | 46–109 / 466–632 / 18,437–27,671 | 30 / 45 / 234–235 |
| Class 3, 50/50, small messages in their own strict class (program marks its traffic) | 4.7 | 10 / 59 / 60 | 6.5–8.7 / 7–113 / 14–114 |
| Adopted: switch sorts RDMA by port, 50/50, nothing marked | 4.7 | 41–44 / 864–939 / 19,952–34,504 | 30 / 45 / 47–48 |
ib_write_lat -s 7168, 100,000 round trips, ordinary memory; the strict class was DSCP 40
in traffic class 5, below the congestion notices. Across the switch-to-switch cable the pattern was the same:
worst case 9–36 ms as built, 60 µs with the strict class. Worst cases of tens of milliseconds are RDMA retransmit
timeouts: a lost last packet has nothing behind it to trigger a fast resend, so the sender waits out its timer
(about 67 ms at the common setting timeout 14).
- Don't use PFC on Spark ports. Both halves of a Spark's card share one port, so a pause meant for one half stops the other: one Spark sent 35,756 pauses and the other half's flows fell to 0.4 Gb/s.
- PFC into the Station was clean into GPU memory. Into ordinary memory it starved half the flows and spread pause frames to every Spark. Not worth it here.
- Without a separate class, RDMA starves TCP. With classes 1 and 3 sharing, TCP keeps going, across the switch-to-switch cable too. On the Station the split behaves as set.
- The biggest effect is on small messages. During a pile-up, 7 KiB messages waited 40–70 µs typically, and sometimes tens of milliseconds. In their own strict class they stayed near 10 µs, worst 60 µs (Station) and 114 µs (Spark). The leftover 10–60 µs is on the Station's side, whose memory path was full. That needs every program to mark its small messages. The adopted setup, which marks nothing, fixed the stalls into a busy Spark (worst 48 µs instead of 16.6–19.4 ms) but not into a busy Station.
- Smaller switch buffers did not help. RouterOS suggests lowering
shared-bufferswhen latency matters more than drops. At 25%, a pile-up into a Spark lost 13.2% of its packets and fell from 196 to 162.5 Gb/s, and stalls into the Station stayed. At 50% nothing improved. We kept the default. - NCCL all_reduce did not change under any setting, but this test can't show a difference: its ring never piles traffic onto one port.
What we did: adopted the version that needs no help from the hosts. The switch sorts every RDMA packet by its protocol port (UDP 4791) into class 3, congestion notices go first, and classes 1 and 3 share bandwidth 50/50. ECN stays on, and there is no PFC. The config is in step 2; hosts keep their defaults. Compared with as built, it kept TCP alive during every RDMA pile-up we ran. Small messages into a busy Spark went from 621–879 µs at the 99.9th percentile (worst 16.6–19.4 ms) to 45 µs (worst 48 µs). NCCL all_reduce, pile-up throughput and loss stayed the same. It does not fix stalls into a busy Station. We left out anything that would need a given model or engine to mark its own traffic. Not measured yet: the effect on a real serving workload, and long-run stability.
When something is wrong
| Symptom | Fix |
|---|---|
| Station port never links | Force it: ethtool -s <if> autoneg off speed 200000 duplex full lanes 4 and --set-fec <if> encoding rs |
Fabric slow, ports missing the H flag | The factory bridge still has hw=yes. Set it to hw=no, then re-add the fabric ports |
Normal pings work, -s 8972 pings fail | MTU 9000 on the host and l2mtu=9216 mtu=9000 on every switch port, the 400G cable included |
| Odd ARP: the wrong half or port answers | arp_ignore=1, arp_announce=2, rp_filter=2 |
| GPU-memory RDMA fails with error 524 | rdma_topo check, then write-grub-acs, update-grub, restart |
| NCCL slow or uses the wrong network | Set NCCL_SOCKET_IFNAME and NCCL_IB_HCA as above |
| TCP copies crawl while an RDMA job runs | RDMA is sharing class 1 with TCP. Add the step-2 switch rules and 50/50 sharing |
| NCCL works, but at about 1 GB/s | It fell back to TCP (NCCL_DEBUG=INFO shows Using network Socket). Fix RDMA, and set NCCL_NET=IB so this fails loudly |
| Big messages fast, 1–4 MiB slow | NCCL_PROTO=^LL128 and NCCL_MAX_NCHANNELS=8 |
| Copies stuck under 10 Gb/s | ssh encryption. Use parallel copies, the rsync daemon, or serve the files |
| Big copy starts fast, then drops to ~1.7 GB/s | The receiving SSD's fast cache is full. Normal |
Versions on this build (updated November 3, 2026): RouterOS 7.25 on both switches; Sparks on Ubuntu 24.04 (DGX OS), kernel 7.0.0-1024-nvidia, driver 580.178.04, ConnectX-7 firmware 28.50.1002 (measured speeds above were taken on 6.17.0-1031 and 28.45.4028; a re-test on the new stack showed the same bandwidth); Station ConnectX-8 firmware
40.47.1088. More from NVIDIA: Spark
clustering, connecting two Stations. From
MikroTik: Quality of Service.