Noting actually says those ports have to be for management.

Most people probably would use them for something else.

Power usage is also very reasonable! 400GE switches and routers pull a ridiculous amount of power.

On their video Patrick points out that this is now lower than $1 per Gbps, cheaper than any of the cheapest AliExpress switches.

Please be careful with units. There are high-end consumer SSDs that will deliver data at 14+GB/s, which is much higher than 25Gb/s.

The second sentence of the article:

> We have a number of these now and have been running our RDMA network backend for the 8x NVIDIA GB10 cluster for several months now, using one.

I have a friend who uses something similar for his 4 NVIDIA GDX Spark cluster.

I purchased my SSD based on price and I have a AM4 APU so I'm limited to pci-e 3, but a quick check with crystaldiskmark says I'm hitting about 3.6 GB/s on sequential reads in seq1m q8t1 ... That would saturate 25gbps (assuming you have the right access patterns). seq1m q1t1 does 2.3 GB/s on my system, close enough to 25 gbps.

By my math, pci-e 5.0 should do 4x the throughput, so if you had 4 pci-e ssds you wanted to serve over the network and you had the pci-e lanes, you could saturate 400 gbps.

I'm pretty happy with my little baby network with a handful of 10G and 2.5G ports, and I'm cheap so if I made a big upgrade it'd probably be something used that has more 10g ports plus a 40G I could DAC to the adjacent home server ... but my home servers are barely powerful enough to push 10g anyway. (Something like a DCS-7050TX-48 goes for ~ $100 and would be a big upgrade in capacity I don't use over the S3300-28X I currently run)

DGX Sparks have 200Gbps interlinks for clustering. Apparently switches are the only practical way to scale such clusters beyond 2 nodes even though each Spark has 2 such ports.

Yes, AI... this video does a good job walking through: https://www.youtube.com/watch?v=QJqKqxQR36Y

(tl;dw - guy connects first four DGX and then another 4 "DGX-alikes". Uses 400gbps breakout cables (you can split interfaces, TIL) so he has 2 ports on the first switch, and then gets this one, I believe. You can buy cables that are 1x400 to 4x100gig, etc.

I mean at lot of what a homelab is experimenting. For example I have an absurd amount of communication 56GbE Mellanox switches but never really saturate it.