I have an X99 Taichi and X570 Taichi that I would like to connect together at high speed. I have noticed that the system integrator versions of the Connectx-4 are quite cheap. For example, I’m looking at some HPE 640sfp28 ones at the moment. I’m just concerned that I could run into vendor locking and future transceiver / cable issues rather than being able to grab something commonly available such as a Mikrotik product. Are there any vendors that should specifically be avoided? Perhaps all of them and stick to first party Nvidia cards? Is this vendor locking not an issue, or perhaps even worse and the network cards require vendor specific code on the motherboard?
Update: I bought the cards and the cable. I may buy some weird cheap 10gig DAC to test it’s compatibility, but at the cost of 25gig DACs I didn’t feel like being a guinea pig.
Unfortunately I have found that the Mellanox Connectx-4 HPE 640sfp28 will not function as the second card in direct attached PCIE lanes bifurcation. It will function in the top slot with my GTX 1050 in the bottom 8 CPU lanes, but it will not function with my RTX 3080 in the top 8 lane slot and the network card in the bottom 8 lane slot. (RTX 3080 only fits in the top slot with my case as a bracket is in the way of it fitting in the other slot, thus why the GTX 1050 for display output) The card would also function in the 4 lane chipset slot, but that defeats the point of having gone out of my way and paying significantly extra to get a dual 8 lane slot board.
Due to an issue of one of the cards being damaged, I have yet to be able to establish a point to point connection between the NAS and the desktop. The card showed code 10 failed to start but I am unsure if this is normal without a transceiver or dac inserted. win 10 22h2, Mallanox MLNX_WinOF2 3.10.52010
Since I am still unsure how I’m going to handle the desktop situation and I only have one good card, I put it in the NAS and it shows up just fine in TrueNAS. (without a connection of course)
1 Like
I think my failed to start might have been from installing the Mellanox drivers instead of just letting windows do it’s thing with updates because I have both the 25 gig cards installed and a link up and running. I set up my NAS as 192.168.0.1/30 and the PC as 192.168.0.2 255.255.255.252. I really wish I could see the temperature of the cards in TrueNAS and HWiNFO to know that I am supplying enough cooling.
I was also stuck putting the network card on the chipset lanes because I forgot the GPU is larger than 2 slots and would prevent me from installing my M.2 riser. I guess that means I can get a dual M.2 riser instead. haha.
2 Likes
So an update on the speeds. I found out that my current drives are very much not worth having 25gig. They wouldn’t even saturate 5gig.
That being said - I do have 256GB of memory in the NAS so there’s plenty of room for ZFS Cache.
Optane doesn’t brrr as fast the network (P1600X 58GB). So testing with the WD SN750 1TB boot drive.
Network will saturate write buffer of my Patriot Viper VPN100 SSD (at 65% filled) if the file is in the ZFS Cache.
Testing with a Softperfect RAM disk reveals that I must be running into the SMB CPU bottleneck that Wendel was talking about when he did his video on 25gig. The speeds were actually no faster than copying to an SSD.
Outside of setting up with a very small subnet, I have not done any optimizations.
2 Likes
Not sure what you mean about 25G DAC costs. They’re not much more than 10G. I picked up this one to test my cards would actually link at 25G since my network is currently only 10G.
The cpu bottleneck is why I use NFS and not SMB. Also, if you’re doing a direct connection between the PC and NAS then look at increasing your MTU. That will lower your cpu load.
1 Like
You probably aren’t if you don’t have ducting. Zip tie a fan to the card or use one of those PCIe PCB blanks with a fan in it as all enterprise grade networking requires a TON of CFM.
OEM cards will interface with the BMC to allow monitored temps and controlled cooling via fan ramping.
Don’t try to implement this after the fact unless you’re a college student bored out of your mind, just install another fan set to a slightly intolerable noise level and move on.
Unfortunately Microsoft has chosen to lock SMB Direct (the RDMA functionality) behind “for Workstation” editions of client windows. So if you want to get faster SMB performance you need to upgrade to that version of windows and make sure all the RDMA stuff is enabled. At that point SMB becomes capable of something like 80-90gbps I believe as long as it isnt bound by something else in the system…
I don’t recall specifically mentioning the DAC costs aside from being concerned about being locked into using a specific vendor and being subject to whatever those cost at the time. As far as when I mentioned the cost of 25gig over 10gig, it was about 10 dollars more per network card and about $30 more for the 3 meter DAC. Ironically, when I got my replacement 25gig card, it came with two 10gig transceivers but I had already purchased the 10gig DAC for the 10gig card I put in a third system.
This is the cooling setup. I know the spec is 300 linear feet per minute but no clue what that would actually equate to since all fans are cubic feet rated not linear.
1 Like
No side panel?
Should be fine
scoot it closer if you can and zip tie it through the power supply holes (not through the power supply, or don’t this is Merica’) and you’ll be fine
glass - it’s a Fractal Pop Air case. Got it hoping that I’d be able to put the GPU in the middle slot and the NIC in the top slot so I didn’t need to have it in chipset lanes like a pleb, but GPU too fat. The old case would never have had a cooling issue because it had a giant 120mm fan on the side panel (DIYPC Ranger R4-R).
GPU too fat or it would be closer. Only way to get closer is to buy smaller than 80mm
Gonna need a smaller fan closer to the cards, up against the glass it’ll struggle to pull enough air