J&M Labs Blog by Milo

Building the future, locally

← Back to Home

Networking DGX Station and DGX Sparks with MikroTik CRS804

In plain English. This is how we tie eight NVIDIA DGX Spark mini computers and one DGX Station GB300 into one fast private network, using two MikroTik CRS804 switches. Each Spark gets one 200G cable. The Station gets two. One 400G cable joins the switches. Five steps, each with a check. If you only remember three things: turn off hardware offload on the switch's factory bridge, force the Station's ports to 200G, and run rdma_topo check on the Station. The switches also put RDMA traffic in its own queue so ordinary TCP never stalls behind it; no program has to do anything for that. We also tested five suggestions from 0xCS (@FT_IOxCS); the results are under Suggestions from 0xCS.

200G
per Spark, one cable
24 GB/s
NCCL all_reduce, 8 Sparks, 256 MiB
392 Gb/s
Station, both ports, GPU memory
+1 µs
for crossing to the other switch

This page is the recipe. The build itself (shelves, photos, power) is on the GhettoRak page.

Set-up order and the check after each step Five steps: cable it, set up both switches, set up the Sparks, set up the Station, then test. Each step has a check: link lights, every fabric port 200G with hardware offload, both Spark ports up at MTU 9000, rdma_topo check passing on the Station, and jumbo pings plus RDMA bandwidth between every pair. Do it in this order Check each step before the next one. Most problems show up at the check right after them. 1 Cable check: lights on every leg 2 Switches check: 200G + H flag 3 Sparks check: 2 ports up, MTU 9000 4 Station check: rdma_topo check: OK 5 Test check: pings + RDMA, all pairs No link on the Station? Force its ports to 200G (step 4). No hardware offload? Check the bridges (step 2).

The five steps. Each one ends with a check.

Parts

QtyWhatPart
2MikroTik CRS804 switch (four 400G QSFP-DD cages each)CRS804-4DDQ-hRM
4400G → 2 × 200G breakout, MikroTik-coded, one per two SparksNADDOD Q2Q56-400G-CU2 (2 m) / -CU1 (1 m)
1400G cable, switch to switch, 1 mNADDOD QDD-400G-CU1
1400G → 2 × 200G breakout for the Station, 1 m (ours is Mellanox-coded; it works)NADDOD Q2Q56-400G-CU1
2+Copper patch cords for management (house LAN → switch 1 → switch 2)Cat6a

Use copper (DAC) cables, not optics, at these lengths. We could not make a 0.5 m breakout work. One switch is enough for four Sparks plus the Station, or for eight Sparks.

1. Cabling

Which cable goes into which switch cage Two MikroTik CRS804 switches, each with four QSFP-DD cages. On each switch, cages 1 and 2 take a breakout cable that feeds two DGX Sparks at 200G each. Cage 3 holds the 400G cable between the two switches. Cage 4 on switch 2 takes one breakout that feeds both 200G ports of the DGX Station. Management runs on copper: house LAN to switch 1, switch 1 to switch 2. What plugs in where Each CRS804 has four cages. A breakout cable turns one cage into two 200G ends. Switch 2 · CRS804 · Sparks 1–4 + Station cage 1 Sparks 4 + 3 cage 2 Sparks 1 + 2 cage 3 400G link cage 4 Station mgmt Spark 4 Spark 3 Spark 1 Spark 2 DGX Station GB300 port 0 · port 1 Switch 1 · CRS804 · Sparks 5–8 cage 1 Sparks 8 + 7 cage 2 Sparks 6 + 5 cage 3 400G link cage 4 empty mgmt Spark 8 Spark 7 Spark 6 Spark 5 400G DAC · 1 m house LAN copper mgmt (own network) Cages 1–2: Spark breakouts. Cage 3: switch-to-switch. Cage 4: the Station. Gold = 200G, blue = 400G.

As built. Sparks 1–4 and the Station on switch 2, Sparks 5–8 on switch 1.

Check: link light on every leg.

2. Switches

RouterOS 7.25 on both switches. Management stays on the copper ports. The fast ports get their own bridge, and only that bridge may use the switch chip. If the factory bridge keeps hardware offload, the fabric bridge falls back to the CPU and is slow.

# management: keep the factory bridge for ether ports, offload off
/interface/bridge/port/set [find bridge=bridge] hw=no
/interface/bridge/set bridge name=mgmt

# fabric bridge on the fast ports
/interface/bridge/add name=fabric protocol-mode=rstp igmp-snooping=no
:foreach p in={"qsfp56-dd-1-1";"qsfp56-dd-1-5";"qsfp56-dd-2-1";"qsfp56-dd-2-5"} do={
  /interface/ethernet/set [find name=$p] auto-negotiation=no \
    speed=200G-baseCR4 fec-mode=fec91 l2mtu=9216 mtu=9000
  /interface/bridge/port/add bridge=fabric interface=$p hw=yes
}

# cage 3: the 400G cable to the other switch
/interface/ethernet/set [find name=qsfp56-dd-3-1] auto-negotiation=no \
    speed=400G-baseCR8 fec-mode=fec91 l2mtu=9216 mtu=9000
/interface/bridge/port/add bridge=fabric interface=qsfp56-dd-3-1 hw=yes

# ECN on the default traffic class
/interface/ethernet/switch/qos/tx-manager/queue/set [find traffic-class=1] ecn=yes

# RDMA in its own queue, sorted by protocol (RoCEv2 = UDP 4791), so hosts and programs need no changes.
# Congestion notices (DSCP 48) go first; RDMA (class 3) and everything else (class 1) share bandwidth 50/50.
/interface/ethernet/switch/qos/profile/add name=roce dscp=26 traffic-class=3 automap=no
/interface/ethernet/switch/qos/profile/add name=cnp dscp=48 traffic-class=6 automap=no
/interface/ethernet/switch/rule/add switch=switch1 ports="qsfp56-dd-1-1,qsfp56-dd-1-5,qsfp56-dd-2-1,qsfp56-dd-2-5,qsfp56-dd-3-1,qsfp56-dd-4-1,qsfp56-dd-4-5" \
    mac-protocol=ip protocol=udp dst-port=4791 \
    dscp=48 new-qos-profile=cnp keep-qos-fields=yes
/interface/ethernet/switch/rule/add switch=switch1 ports="qsfp56-dd-1-1,qsfp56-dd-1-5,qsfp56-dd-2-1,qsfp56-dd-2-5,qsfp56-dd-3-1,qsfp56-dd-4-1,qsfp56-dd-4-5" \
    mac-protocol=ip protocol=udp dst-port=4791 \
    new-qos-profile=roce keep-qos-fields=yes
/interface/ethernet/switch/qos/tx-manager/queue/set [find traffic-class=1] schedule=high-priority-group weight=1
/interface/ethernet/switch/qos/tx-manager/queue/set [find traffic-class=3] schedule=high-priority-group weight=1 ecn=yes

Check: /interface/bridge/port/print shows the H (hardware) flag on every fabric port, and /interface/ethernet/monitor qsfp56-dd-1-1 once shows 200G, link ok. During RDMA traffic, /interface/ethernet/print stats shows tx-queue3-packet rising (RDMA) and tx-queue6-packet rising (congestion notices).

3. Sparks

One 200G cable shows up on a Spark as two interfaces, one per half of the card. Give each half its own subnet. N is the Spark's number. Delete the auto-created "Wired connection" profiles first, then add one netplan file with no gateway:

# /etc/netplan/40-cx7-fabric.yaml
network:
  version: 2
  renderer: NetworkManager
  ethernets:
    enp1s0f0np0:
      addresses: [10.0.0.N/24]
      mtu: 9000
      dhcp4: no
      dhcp6: no
      accept-ra: no
      link-local: []
    enP2p1s0f0np0:
      addresses: [10.0.1.N/24]
      mtu: 9000
      dhcp4: no
      dhcp6: no
      accept-ra: no
      link-local: []

The Sparks link at 200G with auto-negotiation left on, so nothing else is needed. Because both halves sit on one network, each must answer ARP only for its own address. Our Sparks already had these set system-wide; confirm yours, and add them to a file in /etc/sysctl.d/ if not:

sysctl net.ipv4.conf.all.arp_ignore net.ipv4.conf.all.arp_announce net.ipv4.conf.all.rp_filter
# want 1, 2, 2

MTU 9000 is optional. At the default 1500, RDMA between two Sparks ran about 5% slower (185 vs 196 Gb/s), the Station's GPU-memory path dropped from 392 to 370 Gb/s, and NCCL lost about 5% on 8–32 MiB messages, with no change at 2 MiB and below (tested). If you use 9000, every port on the path must have it.

Check: both interfaces up, 200G, MTU 9000. Kernel 7.0.0-1024 works for multi-node RoCE; we tested it and run it on all eight. 7.0.0-1019 needs NVIDIA's kho=off fix on the kernel command line or NCCL fails to register memory. Corrected November 3: this first said to stay on 6.17 because of NVIDIA's advisory; that advice is out of date. Our current settings, with checks, are in DGX Spark: How We Run Them.

4. Station

The Station's ConnectX-8 will not link while it auto-negotiates against the switch's fixed 200G. Force it, and set RS FEC to match. Addresses go in netplan (same shape as the Sparks, host number .9: 10.0.0.9 on enP1p3s0f0np0, 10.0.1.9 on enP1p3s0f1np1). Netplan cannot store the ethtool part, so a small boot service does it:

# /etc/systemd/system/station-fabric-link.service
[Unit]
Description=Force CX8 fabric ports to 200G-CR4 / RS-FEC, set ARP rules
Wants=sys-subsystem-net-devices-enP1p3s0f0np0.device sys-subsystem-net-devices-enP1p3s0f1np1.device
After=sys-subsystem-net-devices-enP1p3s0f0np0.device sys-subsystem-net-devices-enP1p3s0f1np1.device
Before=NetworkManager.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/sbin/ethtool -s enP1p3s0f0np0 autoneg off speed 200000 duplex full lanes 4
ExecStart=/usr/sbin/ethtool --set-fec enP1p3s0f0np0 encoding rs
ExecStart=/usr/sbin/ethtool -s enP1p3s0f1np1 autoneg off speed 200000 duplex full lanes 4
ExecStart=/usr/sbin/ethtool --set-fec enP1p3s0f1np1 encoding rs
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f0np0.arp_ignore=1
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f0np0.arp_announce=2
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f0np0.rp_filter=2
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f1np1.arp_ignore=1
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f1np1.arp_announce=2
ExecStart=/usr/sbin/sysctl -q -w net.ipv4.conf.enP1p3s0f1np1.rp_filter=2

[Install]
WantedBy=multi-user.target
sudo systemctl enable --now station-fabric-link.service

Then fix the GPU path. The card has a direct path to the GB300's memory, called Data Direct. Out of the box, a PCIe isolation setting (ACS) puts the card and the GPU in different groups, and GPU-memory transfers fail with error 524. NVIDIA's tool finds and fixes it (NVIDIA's Station guide):

sudo rdma_topo check              # look for FAIL lines about ACS / iommu_group
sudo rdma_topo write-grub-acs     # writes /etc/default/grub.d/config-acs.cfg
sudo update-grub
# then restart the Station

Check: after the restart, rdma_topo check prints only OK lines, and both ports come back at 200G by themselves.

5. Test

Jumbo pings between every pair, on both subnets. -M do forbids splitting, so a wrong MTU anywhere fails loudly:

ping -c 3 -M do -s 8972 10.0.0.2
ping -c 3 -M do -s 8972 10.0.1.2

Eight Sparks plus the Station gave 128 of 128. Then RDMA bandwidth, one pair at a time (ib_write_bw from perftest; on the Station, a CUDA build adds the GPU flags):

# receiver (Spark 1, first half of the card)
ib_write_bw -F -x 3 -q 4 -D 8 --report_gbits -d rocep1s0f0
# sender (another Spark)
ib_write_bw -F -x 3 -q 4 -D 8 --report_gbits -d rocep1s0f0 10.0.0.1
# Station sender, from GPU memory through Data Direct
ib_write_bw -F -x 3 -q 4 -D 8 --report_gbits -d mlx5_0 \
  --use_cuda_bus_id=<GPU PCI address> --use_cuda_dmabuf --use_data_direct 10.0.0.1

Then NCCL, with these settings on every Spark:

NCCL_IB_HCA=rocep1s0f0,roceP2p1s0f0
NCCL_NET=IB
NCCL_SOCKET_IFNAME=enp1s0f0np0
NCCL_IB_ROCE_VERSION_NUM=2
NCCL_IB_GID_INDEX=3
NCCL_PROTO=^LL128
NCCL_MAX_NCHANNELS=8

Pin NCCL_SOCKET_IFNAME, or NCCL may start up over Docker or Tailscale. Set NCCL_NET=IB too. Without it, a broken RDMA path makes NCCL quietly fall back to plain TCP. In our test that still gave correct answers, at 1.17 GB/s instead of about 23, with no error. With it, NCCL stops with an error. NCCL's start-up traffic can also run over the 10GbE management port instead (NCCL_SOCKET_IFNAME==enP7s7); the speed was the same.

Test more than one message size. Big messages can look fine while medium ones are slow. The last two settings above (NCCL_PROTO, NCCL_MAX_NCHANNELS) are what fix the middle; 0xCS found that pair on his own cluster, and it reproduced here. Bus bandwidth in GB/s, all_reduce, one rank per Spark:

Message4 Sparks, NCCL defaults4 Sparks, settings above8 Sparks, settings above
512 KiB1.456.773.76
1 MiB1.3611.507.43
2 MiB2.6516.4512.76
4 MiB8.2620.9718.55
8 MiB14.422.4021.64
256 MiB22.0722.7823.99

NCCL 2.30.7, 20 warm-up + 100 timed calls per size. 4 Sparks measured October 26, 8 Sparks October 29. Eight ranks are slower on small messages because the ring has twice as many steps.

Pass: 4 Sparks, 1 MiB at least 9 GB/s and 256 MiB at least 22. For 8 Sparks expect about 7 and 24.

Then read the counters, before and after each test. Throughput alone does not show whether the switch dropped or paused anything to get it:

# switch: one line per running fabric port
:foreach i in=[/interface/ethernet/print stats as-value where name~"qsfp56-dd" and running] do={:put $i}
# look at tx-drop-packet, rx-pause, tx-pause, rx-fcs-error, rs-fec-corrected

# each Spark
cat /sys/class/infiniband/rocep1s0f0/ports/1/hw_counters/{np_cnp_sent,out_of_sequence,packet_seq_err}
ethtool -S enp1s0f0np0 | grep -E 'discards_phy|pause_ctrl_phy|prio._pause'

Clean means no drops, no pause frames and no FEC corrections. Every NCCL run we did was clean. Many-to-one RDMA tests are not (see below). The two halves of a Spark's card share one port, so the ethtool port counters read the same on both.

What speed to expect

What speed to expect on each path, measured October 29-30, 2026 Bars in gigabits per second: rsync over ssh 6.6, ssh copy 9.3, Spark SSD sustained write 13.6, one TCP stream 42, one Spark port half over RDMA 112, a whole Spark over TCP 185, the Station from host memory 231, and the Station from GPU memory through Data Direct 392. What speed to expect Gb/s, measured on our fabric. Your transfer runs at the slowest layer it touches. rsync over ssh, one big file 6.6 one CPU core of encryption ssh copy, aes128-gcm 9.3 same Spark SSD, sustained write 13.6 1.7 GB/s after ~50 GB One TCP stream 42 one CPU core One Spark port half (RDMA) 112 PCIe Gen5 x4 Whole Spark (TCP, both halves) 185 2 × x4 Station, host memory 231 Gen5 x8 link Station, GPU memory (Data Direct) 392 · both ports, full speed 0 100 200 300 400 For big copies, use several streams (or no ssh) and remember the disk. For GPU data, use Data Direct.

Measured on this fabric, October 29–30, 2026. Gb/s; divide by 8 for GB/s.

Where the speed limits are inside a DGX Spark and a DGX Station Left: a DGX Spark's ConnectX-7 reaches the GB10 chip through two PCIe Gen5 x4 links of about 112 Gb/s each, about 185 Gb/s for the whole Spark. Right: the DGX Station's ConnectX-8 reaches the Grace CPU through one PCIe Gen5 x8 link, about 231 Gb/s, shared with the model drives and the LAN port, and reaches the GB300 GPU through a Gen6 x16 Data Direct path that ran both ports at 392 Gb/s. Inside the boxes: where the speed limits are The 200G cables are rarely the slowest part. The links inside each machine are. DGX Spark DGX Station GB300 ConnectX-7 · one 200G cable GB10 chip CPU and GPU share one memory PCIe Gen5 x4 ~112 Gb/s PCIe Gen5 x4 ~112 Gb/s Whole Spark ≈ 185 Gb/s (TCP), 24 GB/s NCCL No GPUDirect RDMA on GB10: nothing to unlock ConnectX-8 · two 200G ports Grace CPU + /models, LAN, USB GB300 GPU HBM PCIe Gen5 x8 ~231 Gb/s, shared Data Direct · Gen6 x16 392 Gb/s measured Host memory: ~231 Gb/s for both ports GPU memory via Data Direct: 392 Gb/s On the Station, move data to and from GPU memory. On the Spark, ~112 per port half is the ceiling.

Why: the links inside each machine, not the 200G cables, set most of these limits.

Suggestions from 0xCS, tested

0xCS (@FT_IOxCS), who runs four Sparks on a MikroTik CRS812, sent five suggestions. We tested them on our eight Sparks on October 30. Thanks: two are now in the recipe above, and the third changed how we read our own drop counters.

SuggestionWhat we measuredVerdict
MTU 1500 works; jumbo frames are optionalRDMA, both halves: 185 vs 196 Gb/s. NCCL, 4 Sparks: the same up to 2 MiB, about 5% lower at 8–32 MiBTrue. 9000 buys about 5.6% wherever the cable is the limit: the Station's GPU-memory path fell from 392.1 to 370.3 Gb/s at 1500, both directions. Its ordinary-memory path (PCIe-limited) did not change. We keep 9000
Put RoCE in its own traffic class with ECN and PFCSee belowPartly. A separate RDMA queue sharing bandwidth with TCP helps, and we adopted it, sorted by the switch so nothing has to mark traffic. PFC hurt on Spark ports. A strict class for small messages helped most but needs each program to mark its traffic, so we left it out
Run NCCL start-up over the 10GbE management portSame speed as start-up on the fabric Works. Add NCCL_NET=IB with it, or a dead fabric silently becomes a 1 GB/s TCP one
Test small and medium message sizes, not only 256 MiBWith NCCL defaults, 1 MiB ran at 1.36 GB/s while 256 MiB looked fineAdopted (table in step 5)
Read switch counters before and afterNCCL runs: zero drops, pauses and FEC corrections Adopted (step 5)

The RoCE traffic class, in detail

The idea: an RDMA program marks its packets (NCCL: NCCL_IB_TC=104, which is DSCP 26). The switch puts them in their own queue, traffic class 3, and congestion notices (DSCP 48) in a queue that always goes first. Optionally, PFC lets a full queue pause only class 3 instead of dropping packets. Each host must read DSCP (mlnx_qos --trust dscp); ours read VLAN priority by default, so without that the marking does nothing on the host side. Two things about this switch: it marks ECN only once a queue has passed its shared-buffer limit, and there is no lower threshold to set. We measured queues of up to 3.92 MB on the Station's port, about 157 µs of waiting at 200G.

Tests, run October 30 on all eight Sparks and the Station:

Pile-ups: packets lost (share of what arrived, one number per run):

SettingInto a SparkInto Station memoryInto Station GPU memory
As built: one class, ECN3.0%, 5.1%, 0.7%0.7%, 2.3%0.6%, 0.3%
Class 3 + ECN6.4%, 8.9%not runnot run
Class 3, classes 1 and 3 share 50/505.1%, 4.5%1.3%, 0.9%0.9%, 0.9%
Class 3 + PFC0, 00, 0, but half the flows fell to about 2 Gb/s0, 0
Adopted: switch sorts RDMA by port, 50/503.0%, 4.0%3.1%, 1.9%1.1%, 0.9%

Loss varies up to 10× between runs of the same setting, so only PFC's zero is a clear difference. The losses are not just a start-up burst: into the Station they ran at about 140–290 thousand packets a second for the whole run after a burst in the first second.

Mixed: TCP during the pile-up, in Gb/s (alone it gets 109 into a Spark, 111 into the Station):

SettingInto a SparkInto Station GPU memoryInto the Station, across the switch-to-switch cable
As built0.002–0.0054.5–5.90.005–0.015
Class 3 + PFC0.0026.6–8.4not run
Class 3 + PFC, 50/5050–59, but RDMA fell to 77 Gb/s in three of four runs89not run
Class 3, 50/5050–5492–9361
Class 3, 75/25 for RDMA28–4551–53not run
Class 3, 80/20 for RDMA22–2841not run
Adopted: switch sorts RDMA by port, 50/5042–5891–9441–46

2–4 runs each. RDMA into the Station's GPU memory got 349–350 Gb/s at 50/50, 371–372 at 75/25 and 376 at 80/20 (12-second averages; the TCP ran for 6 of the 12 seconds). Into a Spark, RDMA gained little from a bigger share, because that Spark's own PCIe link was already the limit.

Small messages: 7 KiB round trip, Station ↔ Spark on the same switch, in µs, two runs:

SettingIdle, typicalPile-up into the Station: typical / 99.9% / worstPile-up into the Spark: typical / 99.9% / worst
As built4.743–46 / 642–934 / 37,743–38,95068–71 / 621–879 / 16,596–19,382
Class 3, 50/50, small messages in class 3 too4.746–109 / 466–632 / 18,437–27,67130 / 45 / 234–235
Class 3, 50/50, small messages in their own strict class (program marks its traffic)4.710 / 59 / 606.5–8.7 / 7–113 / 14–114
Adopted: switch sorts RDMA by port, 50/50, nothing marked4.741–44 / 864–939 / 19,952–34,50430 / 45 / 47–48

ib_write_lat -s 7168, 100,000 round trips, ordinary memory; the strict class was DSCP 40 in traffic class 5, below the congestion notices. Across the switch-to-switch cable the pattern was the same: worst case 9–36 ms as built, 60 µs with the strict class. Worst cases of tens of milliseconds are RDMA retransmit timeouts: a lost last packet has nothing behind it to trigger a fast resend, so the sender waits out its timer (about 67 ms at the common setting timeout 14).

What we did: adopted the version that needs no help from the hosts. The switch sorts every RDMA packet by its protocol port (UDP 4791) into class 3, congestion notices go first, and classes 1 and 3 share bandwidth 50/50. ECN stays on, and there is no PFC. The config is in step 2; hosts keep their defaults. Compared with as built, it kept TCP alive during every RDMA pile-up we ran. Small messages into a busy Spark went from 621–879 µs at the 99.9th percentile (worst 16.6–19.4 ms) to 45 µs (worst 48 µs). NCCL all_reduce, pile-up throughput and loss stayed the same. It does not fix stalls into a busy Station. We left out anything that would need a given model or engine to mark its own traffic. Not measured yet: the effect on a real serving workload, and long-run stability.

When something is wrong

SymptomFix
Station port never linksForce it: ethtool -s <if> autoneg off speed 200000 duplex full lanes 4 and --set-fec <if> encoding rs
Fabric slow, ports missing the H flagThe factory bridge still has hw=yes. Set it to hw=no, then re-add the fabric ports
Normal pings work, -s 8972 pings failMTU 9000 on the host and l2mtu=9216 mtu=9000 on every switch port, the 400G cable included
Odd ARP: the wrong half or port answersarp_ignore=1, arp_announce=2, rp_filter=2
GPU-memory RDMA fails with error 524rdma_topo check, then write-grub-acs, update-grub, restart
NCCL slow or uses the wrong networkSet NCCL_SOCKET_IFNAME and NCCL_IB_HCA as above
TCP copies crawl while an RDMA job runsRDMA is sharing class 1 with TCP. Add the step-2 switch rules and 50/50 sharing
NCCL works, but at about 1 GB/sIt fell back to TCP (NCCL_DEBUG=INFO shows Using network Socket). Fix RDMA, and set NCCL_NET=IB so this fails loudly
Big messages fast, 1–4 MiB slowNCCL_PROTO=^LL128 and NCCL_MAX_NCHANNELS=8
Copies stuck under 10 Gb/sssh encryption. Use parallel copies, the rsync daemon, or serve the files
Big copy starts fast, then drops to ~1.7 GB/sThe receiving SSD's fast cache is full. Normal

Versions on this build (updated November 3, 2026): RouterOS 7.25 on both switches; Sparks on Ubuntu 24.04 (DGX OS), kernel 7.0.0-1024-nvidia, driver 580.178.04, ConnectX-7 firmware 28.50.1002 (measured speeds above were taken on 6.17.0-1031 and 28.45.4028; a re-test on the new stack showed the same bandwidth); Station ConnectX-8 firmware 40.47.1088. More from NVIDIA: Spark clustering, connecting two Stations. From MikroTik: Quality of Service.