I’m using a Mellanox ConnectX-3 CX354A NIC to connect my Arch Linux PC to a TrueNAS host at 40GbE. Using nfs, I get far more performance over RDMA instead of TCP (which makes sense), but I get full system crashes when stressing it too much. I never had issues with it before, but when I tried doing a 1.4TB transfer (which went at around ~1.2GiB/s) my system would completely freeze, requiring a force shutdown (holding down the power button). Checking journalctl logs, I get a ton of mlx4_core 0000:0a:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0015 address=0xbd6a20c0 flags=0x0020] type errors (like, thousands) which end in mlx4_core 0000:0a:00.0: Internal error detected:, after which it crashes (the logs are cut off at the colon, it doesn’t actually report what the “internal error” is. Here is a pastebin of the last couple dozen lines, everything before is repetition of similar page fault errors with just different memory addresses.
I’ve tried recompiling my kernel and the included drivers, but I couldn’t find any external package drivers for Arch. I did some googling which led to the suggestion that there were too many things on the same iommu group, but the NIC is on group 20 and there is nothing else in that group. Any ideas?
Relevant specs:
Threadripper 1920x
Asus ROG Strix X399-E Gaming
ConnectX-3 CX354A
64GB Corsair Vengeance LPX 3200mhz memory (4x16)
The transfer is writing to an mdadm raid 0 array of 4x Solidigm P44 Pros on a Gigabyte Quad M.2 Card (yes, I know these are gen4 drives on a gen3 platform, I’m upgrading soon I’m just waiting on my motherboard to come in)
Reading from a TrueNAS Scale Dragonfish server on a Dell R730XD with 12x 8TB HDDs in raidz2 with another CX354A NIC, running through a HP FlexFabric 5900AF switch.
Edit: Intended to add lspci output for the NIC but forgot to, here it is
Subsystem: Mellanox Technologies Device [15b3:0050]
Flags: bus master, fast devsel, latency 0, IRQ 37, IOMMU group 20
Memory at ba800000 (64-bit, non-prefetchable) [size=1M]
Memory at 483f800000 (64-bit, prefetchable) [size=8M]
Expansion ROM at ba700000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Capabilities: [48] Vital Product Data
Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
Capabilities: [60] Express Endpoint, IntMsgNum 0
Capabilities: [c0] Vendor Specific Information: Len=18 <?>
Capabilities: [100] Alternative Routing-ID Interpretation (ARI)
Capabilities: [148] Device Serial Number 00-10-e0-00-01-5a-60-70
Capabilities: [154] Advanced Error Reporting
Capabilities: [18c] Secondary PCI Express
Kernel driver in use: mlx4_core
Kernel modules: mlx4_core