VFIO/Passthrough Issues - Threadripper Pro & Asus WRX80E-Sage

Software & Operating Systems VFIO
21

ugggg @DotBowder you are seeing SATA errors too? I recently upgraded to the latest BIOS and I have tried the SATA in both SATA RAID mode as well as AHCI. After about 1TB of data written, I get errors in dmesg and ubuntu appears to kick out the drive (/dev/sda goes away). the most common error I see is “write beyond end of disk.” Have tried with a Samsung QVO 8TB but also recently saw same errors with an Exos 18TB.

Is this along the lines of what you’re seeing with ZFS? dumb question but do you have good cooling on your SATA drives? My case does not have fans there and I saw SMART reporting temps of 55C or so, which seemed a bit high.

I’d love to hear if you have any more info on your experience with the built-in SATA because I’m almost ready to RMA the hard drive.

22

@oarphoarph

Indeed, I see a similar behavior to you. After about 1TB of data written... The drives appear totally fine for like 5 seconds, then speeds drop down to the 10MBps, then 5MBps, then 1MBps, then plateau at nothing. ZFS logs indicated that the drives were not responding or were slow to respond. Then my ZFS pool kicks offline (after much waiting for the drives to respond).

It does seem to be agitated and initiated by intensive writing, but it doesn’t matter what drive I put into it. It happens with SanDisk SSD Plus 1TB, ADATA SU635 256GB, and Crutial BX500 1TB, Seagate Exos 14TB drives, and Seagate 8TB drives.

My best guess is that the SATA controller has a firmware bug, or the linux drivers associated with them are bugged. I also tried passing through drives to a windows machine via the qemu scsi adapter, and they do continue to work for long periods of time, but, they are no where near as performant as they should be and I have no real idea if they’re kicking out bad data or not (haven’t tested).

The way that I have worked around this issue is to use the U.2/MINISAS style connectors in SATA mode (must change in bios). When plugging in drives to these plugs, things work without any problems. The sata ports are still borked when I use the U.2 ports – What is REALLLY weird about this is:

  1. I disable the SATA controller in the bios
  2. I set the U.2 to SATA mode in the bios.
  3. I plugin drives to both the U.2 and the SATA ports
  4. All of the drives are visible. - Even though I disabled the SATA controllers in the bios.

See below an example of 8 drives on U.2 and 4 drives in SATA ports.

/dev/sda - U.2/MINISAS
/dev/sdb - U.2/MINISAS
/dev/sdc - U.2/MINISAS
/dev/sdd - U.2/MINISAS
/dev/sdi - U.2/MINISAS
/dev/sdj - U.2/MINISAS
/dev/sdk - U.2/MINISAS
/dev/sdl - U.2/MINISAS

/dev/sde - SATA
/dev/sdf - SATA
/dev/sdg - SATA
/dev/sdh - SATA

Even if I turn off the SATA controllers in the bios, a new sata controller sprouts up next to the U.2 one to show the SATA devices. The SATA devices still don’t perform properly though, only the U.2 ones.

root@pve-01:~# ls -hal /dev/disk/by-id | grep -v “part” | grep -e “sd[a-z]”
lrwxrwxrwx 1 root root 9 Feb 5 12:23 ata-CT1000BX500SSD1_2125E5AD9163 → …/…/sde
lrwxrwxrwx 1 root root 9 Feb 5 12:23 ata-CT1000BX500SSD1_2125E5AD929B → …/…/sdf
lrwxrwxrwx 1 root root 9 Feb 5 12:23 ata-SanDisk_SSD_PLUS_1000GB_202774801540 → …/…/sdg
lrwxrwxrwx 1 root root 9 Feb 5 12:23 ata-SanDisk_SSD_PLUS_1000GB_203805A0017F → …/…/sdh
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZL29YDP2 → …/…/sdi
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZTM070J3 → …/…/sdj
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZTM0EHTE → …/…/sdc
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZTM0ER63 → …/…/sdb
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZTM0EYW2 → …/…/sda
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST14000NM001G-2KJ103_ZTM0FDJV → …/…/sdd
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST8000VN0022-2EL112_ZA1J4CAV → …/…/sdl
lrwxrwxrwx 1 root root 9 Feb 5 11:00 ata-ST8000VN004-2M2101_WKD18FVF → …/…/sdk

root@pve-01:~# ls -hal /dev/disk/by-path/ | grep -v -e “part” -e “.0 ->” | grep -e “sd[a-z]”
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:2b:00.0-ata-1 → …/…/sda # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:2b:00.0-ata-2 → …/…/sdb # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:2b:00.0-ata-3 → …/…/sdc # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:2b:00.0-ata-4 → …/…/sdd # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 12:23 pci-0000:2c:00.0-ata-1 → …/…/sde # SATA Port
lrwxrwxrwx 1 root root 9 Feb 5 12:23 pci-0000:2c:00.0-ata-2 → …/…/sdf # SATA Port
lrwxrwxrwx 1 root root 9 Feb 5 12:23 pci-0000:2c:00.0-ata-5 → …/…/sdg # SATA Port
lrwxrwxrwx 1 root root 9 Feb 5 12:23 pci-0000:2c:00.0-ata-6 → …/…/sdh # SATA Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:31:00.0-ata-5 → …/…/sdi # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:31:00.0-ata-6 → …/…/sdj # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:31:00.0-ata-7 → …/…/sdk # U.2 Port
lrwxrwxrwx 1 root root 9 Feb 5 11:00 pci-0000:31:00.0-ata-8 → …/…/sdl # U.2 Port

root@pve-01:~# lspci | grep SATA
26:00.0 IDE interface: ASMedia Technology Inc. ASM1061 SATA IDE Controller (rev 02)
27:00.0 IDE interface: ASMedia Technology Inc. ASM1061 SATA IDE Controller (rev 02)
2b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
2c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
31:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

In this example, I have the SSDs plugged into the SATA ports, but I can’t trust them to work properly, so I don’t have them in use.

Anyway, that’s my headache of an experience that I’ve had with SATA devices on this motherboard.

1 Like
23

@DotBowder wow! thank you so so much for the details here!

edit: @DotBowder I contacted ASUS support and they’re gonna let me send in a request for their lab to reproduce. thank you again! if you don’t mind I’ll note your linux version in affected softwares.

One unrelated question: i have wanted to try GPU passthru into Windows on this machine. I’ve only tried virtualbox and vmware (free version). is there a free solution that works, or have you used something more fancy?

Some more details on the SATA woes:
So what I’ve seen in dmesg is Ubuntu turning the SATA speed link from 6g to 3g then 1.5g then finally punting the drive. That might help explain the degradation you see with ZFS. That said, if I initially connect a drive and don’t do many writes at first, I’m able to get full 250MB/s or more for about 1TB before “write beyond end of drive.” If I physically unplug and re-plug the device then (hot-swap), I usually get an immediate 6g->3g->1.5g degradation, as if ubuntu wants to just kick out the drive. After reboot, ubuntu will accept the drive again but I still get errors.

Yes I saw your link about the adapters, thank you! In my case I need those M.2 slots though so I’m trying to figure out a similar mitigation.

Your note about the Windows passthru is really interesting, thank you! I myself have seen issues only with Ubuntu, but both vanilla ext4 on the disk as well as mdadm raid across two devices. If the passthru works as you say then perhaps it is indeed a kernel issue. My theory about the 1TB limit that I’m seeing is that there’s some sort of AHCI feature for large drives that’s getting borked.

I am going to try contacting ASUS. Thank you again for all these details! They also screwed up nvme RAID in the new BIOS-- new BIOS does not remember PCIe bifurcation settings (for Asus Hyper M.2 card) after a power disconnect. So reboots it’ll remember, but if you have a full unplug, you have to manually set up again in BIOS.

24

Here’s an output from inxi.

root@pve-01:~# inxi -F
System:    Host: pve-01 Kernel: 5.13.19-3-pve x86_64 bits: 64 Console: tty 6 Distro: Debian GNU/Linux 11 (bullseye) 
Machine:   Type: Desktop System: ASUS product: N/A v: N/A serial: N/A 
           Mobo: ASUSTeK model: Pro WS WRX80E-SAGE SE WIFI v: Rev 1.xx serial: 210483465700035 
           UEFI: American Megatrends v: 0701 date: 09/15/2021 
CPU:       Info: 32-Core (4-Die) model: AMD Ryzen Threadripper PRO 3975WX s bits: 64 type: MT MCP MCM 
           L2 cache: 16 MiB 
           Speed: 4267 MHz min/max: 2200/3500 MHz Core speeds (MHz): 1: 4267 2: 4250 3: 3413 4: 4186 5: 4231 6: 4116 
           7: 3996 8: 4245 9: 3593 10: 3632 11: 3840 12: 3598 13: 4218 14: 3592 15: 3594 16: 3580 17: 3590 18: 4266 
           19: 4111 20: 4117 21: 4126 22: 4253 23: 4251 24: 4251 25: 4214 26: 4248 27: 4244 28: 4256 29: 4247 
           30: 4247 31: 4243 32: 4216 33: 4223 34: 4177 35: 4239 36: 3384 37: 3400 38: 3975 39: 4230 40: 4170 
           41: 4235 42: 3964 43: 4220 44: 3588 45: 3592 46: 4243 47: 3592 48: 3592 49: 3595 50: 3592 51: 3585 
           52: 3588 53: 4216 54: 4217 55: 4218 56: 4223 57: 4215 58: 4228 59: 4352 60: 4221 61: 4216 62: 4217 
           63: 4108 64: 3593 
Graphics:  Device-1: NVIDIA TU116 [GeForce GTX 1650 SUPER] driver: vfio-pci v: 0.2 
           Device-2: ASPEED Graphics Family driver: ast v: kernel 
           Device-3: NVIDIA TU102 [GeForce RTX 2080 Ti Rev. A] driver: vfio-pci v: 0.2 
           Device-4: NVIDIA GP102 [GeForce GTX 1080 Ti] driver: vfio-pci v: 0.2 
           Device-5: NVIDIA GP102 [GeForce GTX 1080 Ti] driver: vfio-pci v: 0.2 
           Device-6: NVIDIA GP102 [GeForce GTX 1080 Ti] driver: vfio-pci v: 0.2 
           Display: server: No display server data found. Headless machine? tty: 120x40 
           Message: Advanced graphics data unavailable in console for root. 
Audio:     Device-1: NVIDIA TU116 High Definition Audio driver: vfio-pci 
           Device-2: NVIDIA TU102 High Definition Audio driver: vfio-pci 
           Device-3: Advanced Micro Devices [AMD] Starship/Matisse HD Audio driver: N/A 
           Device-4: NVIDIA GP102 HDMI Audio driver: vfio-pci 
           Device-5: NVIDIA GP102 HDMI Audio driver: vfio-pci 
           Device-6: NVIDIA GP102 HDMI Audio driver: vfio-pci 
           Sound Server: ALSA v: k5.13.19-3-pve 
Network:   Device-1: Intel Ethernet 10G X550T driver: ixgbe 
           IF: enp36s0f0 state: up speed: 10000 Mbps duplex: full mac: fc:34:97:a5:cc:7c 
           Device-2: Intel Ethernet 10G X550T driver: ixgbe 
           IF: enp36s0f1 state: down mac: fc:34:97:a5:cc:7d 
           IF-ID-1: fwbr100i0 state: up speed: 10000 Mbps duplex: unknown mac: 96:36:96:d9:2d:dc 
           IF-ID-2: fwbr100i1 state: up speed: 10000 Mbps duplex: unknown mac: da:a2:a9:ea:dc:37 
           IF-ID-3: fwbr100i2 state: up speed: 10000 Mbps duplex: unknown mac: 1a:3f:58:43:61:68 
           IF-ID-4: fwbr100i3 state: up speed: 10000 Mbps duplex: unknown mac: 9a:44:54:94:31:1d 
           IF-ID-5: fwbr100i4 state: up speed: 10000 Mbps duplex: unknown mac: 2e:0b:16:ac:9f:43 
           IF-ID-6: fwbr100i5 state: up speed: 10000 Mbps duplex: unknown mac: ea:1d:e3:50:cd:30 
           IF-ID-7: fwbr101i0 state: up speed: 10000 Mbps duplex: unknown mac: f6:72:7c:60:ce:93 
           IF-ID-8: fwbr101i1 state: up speed: 10000 Mbps duplex: unknown mac: e2:6e:09:31:a2:04 
           IF-ID-9: fwbr101i2 state: up speed: 10000 Mbps duplex: unknown mac: aa:1f:97:77:49:e6 
           IF-ID-10: fwbr101i3 state: up speed: 10000 Mbps duplex: unknown mac: 0e:35:0d:a1:4b:75 
           IF-ID-11: fwbr101i4 state: up speed: 10000 Mbps duplex: unknown mac: 86:5b:3c:ec:65:4c 
           IF-ID-12: fwbr101i5 state: up speed: 10000 Mbps duplex: unknown mac: f2:cc:01:b0:41:33 
           IF-ID-13: fwbr104i0 state: up speed: 10000 Mbps duplex: unknown mac: 62:20:9c:5e:59:10 
           IF-ID-14: fwbr301i0 state: up speed: 10000 Mbps duplex: unknown mac: 66:e0:b5:a4:d9:f3 
           IF-ID-15: fwbr301i1 state: up speed: 10000 Mbps duplex: unknown mac: 66:9a:08:9a:db:8a 
           IF-ID-16: fwbr301i2 state: up speed: 10000 Mbps duplex: unknown mac: 02:08:33:ca:4b:13 
           IF-ID-17: fwbr301i3 state: up speed: 10000 Mbps duplex: unknown mac: 9a:2b:86:ec:80:c0 
           IF-ID-18: fwbr301i4 state: up speed: 10000 Mbps duplex: unknown mac: 7a:1d:0c:f8:00:8f 
           IF-ID-19: fwbr301i5 state: up speed: 10000 Mbps duplex: unknown mac: 7e:92:c8:57:af:d9 
           IF-ID-20: fwbr304i0 state: up speed: 10000 Mbps duplex: unknown mac: 3e:72:0b:92:75:6c 
           IF-ID-21: fwbr304i1 state: up speed: 10000 Mbps duplex: unknown mac: fa:a6:74:7e:ab:b7 
           IF-ID-22: fwbr304i2 state: up speed: 10000 Mbps duplex: unknown mac: ea:8d:37:6e:5c:4f 
           IF-ID-23: fwbr304i3 state: up speed: 10000 Mbps duplex: unknown mac: 82:2d:45:ec:10:23 
           IF-ID-24: fwbr304i4 state: up speed: 10000 Mbps duplex: unknown mac: ee:51:7c:ac:b7:6a 
           IF-ID-25: fwbr304i5 state: up speed: 10000 Mbps duplex: unknown mac: 1e:88:68:41:ca:80 
           IF-ID-26: fwbr501i0 state: up speed: 10000 Mbps duplex: unknown mac: 76:43:01:87:54:6c 
           IF-ID-27: fwln100i0 state: up speed: 10000 Mbps duplex: full mac: 66:9e:a0:7c:56:6d 
           IF-ID-28: fwln100i1 state: up speed: 10000 Mbps duplex: full mac: ee:04:32:c8:1a:ba 
           IF-ID-29: fwln100i2 state: up speed: 10000 Mbps duplex: full mac: 02:89:06:98:93:a3 
           IF-ID-30: fwln100i3 state: up speed: 10000 Mbps duplex: full mac: da:36:ae:bf:78:73 
           IF-ID-31: fwln100i4 state: up speed: 10000 Mbps duplex: full mac: 4a:cc:95:8c:de:c4 
           IF-ID-32: fwln100i5 state: up speed: 10000 Mbps duplex: full mac: ea:be:17:a6:f2:7d 
           IF-ID-33: fwln101i0 state: up speed: 10000 Mbps duplex: full mac: b2:ec:dd:da:83:37 
           IF-ID-34: fwln101i1 state: up speed: 10000 Mbps duplex: full mac: a2:80:a6:5f:98:d1 
           IF-ID-35: fwln101i2 state: up speed: 10000 Mbps duplex: full mac: 62:c0:0b:b8:88:9a 
           IF-ID-36: fwln101i3 state: up speed: 10000 Mbps duplex: full mac: 96:8f:25:d1:0b:2f 
           IF-ID-37: fwln101i4 state: up speed: 10000 Mbps duplex: full mac: 66:23:01:e4:ca:d7 
           IF-ID-38: fwln101i5 state: up speed: 10000 Mbps duplex: full mac: 8e:3a:a6:e0:9e:8b 
           IF-ID-39: fwln104i0 state: up speed: 10000 Mbps duplex: full mac: 3a:20:a2:ec:05:3a 
           IF-ID-40: fwln301i0 state: up speed: 10000 Mbps duplex: full mac: 1a:d5:82:f7:32:f7 
           IF-ID-41: fwln301i1 state: up speed: 10000 Mbps duplex: full mac: 7a:ca:c4:65:c4:2f 
           IF-ID-42: fwln301i2 state: up speed: 10000 Mbps duplex: full mac: be:69:96:11:be:19 
           IF-ID-43: fwln301i3 state: up speed: 10000 Mbps duplex: full mac: 9a:b6:1d:7c:d1:b7 
           IF-ID-44: fwln301i4 state: up speed: 10000 Mbps duplex: full mac: 2a:dc:66:59:79:0f 
           IF-ID-45: fwln301i5 state: up speed: 10000 Mbps duplex: full mac: da:8c:35:83:d8:be 
           IF-ID-46: fwln304i0 state: up speed: 10000 Mbps duplex: full mac: f2:7d:5b:44:86:f9 
           IF-ID-47: fwln304i1 state: up speed: 10000 Mbps duplex: full mac: b6:a0:75:f3:78:91 
           IF-ID-48: fwln304i2 state: up speed: 10000 Mbps duplex: full mac: 32:ce:74:a5:a1:b2 
           IF-ID-49: fwln304i3 state: up speed: 10000 Mbps duplex: full mac: aa:b4:12:e4:8b:7a 
           IF-ID-50: fwln304i4 state: up speed: 10000 Mbps duplex: full mac: ce:21:88:e3:eb:b6 
           IF-ID-51: fwln304i5 state: up speed: 10000 Mbps duplex: full mac: f6:83:c3:ef:53:cc 
           IF-ID-52: fwln501i0 state: up speed: 10000 Mbps duplex: full mac: ba:4b:34:39:37:43 
           IF-ID-53: fwpr100p0 state: up speed: 10000 Mbps duplex: full mac: c2:cd:cd:6a:2a:cf 
           IF-ID-54: fwpr100p1 state: up speed: 10000 Mbps duplex: full mac: 7e:28:66:2d:d5:c5 
           IF-ID-55: fwpr100p2 state: up speed: 10000 Mbps duplex: full mac: 36:43:13:0a:37:d6 
           IF-ID-56: fwpr100p3 state: up speed: 10000 Mbps duplex: full mac: ca:87:7e:c0:0f:62 
           IF-ID-57: fwpr100p4 state: up speed: 10000 Mbps duplex: full mac: 22:29:74:c5:b4:15 
           IF-ID-58: fwpr100p5 state: up speed: 10000 Mbps duplex: full mac: 22:6d:c8:cb:84:ff 
           IF-ID-59: fwpr101p0 state: up speed: 10000 Mbps duplex: full mac: ce:61:a2:e0:c8:ef 
           IF-ID-60: fwpr101p1 state: up speed: 10000 Mbps duplex: full mac: 92:0d:08:d6:12:9a 
           IF-ID-61: fwpr101p2 state: up speed: 10000 Mbps duplex: full mac: d6:c3:62:ae:95:bc 
           IF-ID-62: fwpr101p3 state: up speed: 10000 Mbps duplex: full mac: 2a:a3:e4:07:71:3e 
           IF-ID-63: fwpr101p4 state: up speed: 10000 Mbps duplex: full mac: 2a:42:68:e5:dd:29 
           IF-ID-64: fwpr101p5 state: up speed: 10000 Mbps duplex: full mac: 5a:b5:a8:e7:50:e1 
           IF-ID-65: fwpr104p0 state: up speed: 10000 Mbps duplex: full mac: ba:23:6d:e5:d8:c7 
           IF-ID-66: fwpr301p0 state: up speed: 10000 Mbps duplex: full mac: 9a:72:b0:0f:83:22 
           IF-ID-67: fwpr301p1 state: up speed: 10000 Mbps duplex: full mac: 82:f3:b2:bc:82:f5 
           IF-ID-68: fwpr301p2 state: up speed: 10000 Mbps duplex: full mac: b2:ec:a4:69:ac:26 
           IF-ID-69: fwpr301p3 state: up speed: 10000 Mbps duplex: full mac: 0a:23:5a:c7:40:76 
           IF-ID-70: fwpr301p4 state: up speed: 10000 Mbps duplex: full mac: 12:5b:fa:14:41:52 
           IF-ID-71: fwpr301p5 state: up speed: 10000 Mbps duplex: full mac: 7a:61:94:8d:8a:ed 
           IF-ID-72: fwpr304p0 state: up speed: 10000 Mbps duplex: full mac: ce:e8:d3:38:f5:01 
           IF-ID-73: fwpr304p1 state: up speed: 10000 Mbps duplex: full mac: da:f2:cc:b9:ea:83 
           IF-ID-74: fwpr304p2 state: up speed: 10000 Mbps duplex: full mac: 6e:86:4f:ac:4f:70 
           IF-ID-75: fwpr304p3 state: up speed: 10000 Mbps duplex: full mac: ea:9f:32:17:45:c6 
           IF-ID-76: fwpr304p4 state: up speed: 10000 Mbps duplex: full mac: c2:e8:48:de:e6:5a 
           IF-ID-77: fwpr304p5 state: up speed: 10000 Mbps duplex: full mac: ce:c9:bd:08:03:9d 
           IF-ID-78: fwpr501p0 state: up speed: 10000 Mbps duplex: full mac: 7a:a0:72:42:b3:98 
           IF-ID-79: tap100i0 state: unknown speed: 10 Mbps duplex: full mac: 9a:90:3b:38:fd:5e 
           IF-ID-80: tap100i1 state: unknown speed: 10 Mbps duplex: full mac: 56:01:db:c3:39:2c 
           IF-ID-81: tap100i2 state: unknown speed: 10 Mbps duplex: full mac: 72:14:e2:dd:be:74 
           IF-ID-82: tap100i3 state: unknown speed: 10 Mbps duplex: full mac: e6:41:57:71:ad:d9 
           IF-ID-83: tap100i4 state: unknown speed: 10 Mbps duplex: full mac: f2:cb:c3:bb:a9:c0 
           IF-ID-84: tap100i5 state: unknown speed: 10 Mbps duplex: full mac: be:e8:22:99:25:06 
           IF-ID-85: tap101i0 state: unknown speed: 10 Mbps duplex: full mac: 06:ab:c1:7a:9b:d8 
           IF-ID-86: tap101i1 state: unknown speed: 10 Mbps duplex: full mac: 92:80:d0:6a:e0:90 
           IF-ID-87: tap101i2 state: unknown speed: 10 Mbps duplex: full mac: 62:c2:7b:b1:ae:cf 
           IF-ID-88: tap101i3 state: unknown speed: 10 Mbps duplex: full mac: 56:84:e1:37:a2:08 
           IF-ID-89: tap101i4 state: unknown speed: 10 Mbps duplex: full mac: 52:7e:9a:76:78:d1 
           IF-ID-90: tap101i5 state: unknown speed: 10 Mbps duplex: full mac: 5e:b9:95:ce:7f:5e 
           IF-ID-91: tap104i0 state: unknown speed: 10 Mbps duplex: full mac: 96:5d:8c:cd:d3:f5 
           IF-ID-92: tap301i0 state: unknown speed: 10 Mbps duplex: full mac: 56:ab:1c:98:72:48 
           IF-ID-93: tap301i1 state: unknown speed: 10 Mbps duplex: full mac: fa:dc:3e:2e:8f:9d 
           IF-ID-94: tap301i2 state: unknown speed: 10 Mbps duplex: full mac: aa:5e:dc:a5:b1:ed 
           IF-ID-95: tap301i3 state: unknown speed: 10 Mbps duplex: full mac: 72:a0:26:c4:1f:ca 
           IF-ID-96: tap301i4 state: unknown speed: 10 Mbps duplex: full mac: de:99:3f:7c:4d:64 
           IF-ID-97: tap301i5 state: unknown speed: 10 Mbps duplex: full mac: 1e:cc:0c:a4:41:6a 
           IF-ID-98: tap304i0 state: unknown speed: 10 Mbps duplex: full mac: ea:40:99:47:91:b9 
           IF-ID-99: tap304i1 state: unknown speed: 10 Mbps duplex: full mac: 72:4c:dd:8d:ae:a2 
           IF-ID-100: tap304i2 state: unknown speed: 10 Mbps duplex: full mac: 46:99:de:4d:77:57 
           IF-ID-101: tap304i3 state: unknown speed: 10 Mbps duplex: full mac: 16:b4:12:f1:41:62 
           IF-ID-102: tap304i4 state: unknown speed: 10 Mbps duplex: full mac: a2:62:3e:78:13:13 
           IF-ID-103: tap304i5 state: unknown speed: 10 Mbps duplex: full mac: be:66:ef:aa:ba:72 
           IF-ID-104: tap501i0 state: unknown speed: 10 Mbps duplex: full mac: 9e:d3:de:9b:98:23 
           IF-ID-105: vmbr0 state: down mac: 66:ca:24:33:b1:da 
           IF-ID-106: vmbr1 state: up speed: 10000 Mbps duplex: unknown mac: 56:44:56:2d:4c:f0 
           IF-ID-107: vmbr2 state: down mac: 26:bc:27:cb:02:33 
           IF-ID-108: ztyxau3ooa state: unknown speed: 10 Mbps duplex: full mac: 06:a1:6a:f0:0c:c0 
Bluetooth: Device-1: Intel AX200 Bluetooth type: USB driver: usbfs 
           Message: Required tool hciconfig not installed. Check --recommends 
RAID:      Device-1: backup type: zfs status: ONLINE level: mirror size: 7.27 TiB free: 1.8 TiB 
           Components: Online: N/A 
           Device-2: cache type: zfs status: - level: no-raid size: N/A free: N/A 
           Components: Online: N/A 
           Device-3: rpool type: zfs status: ONLINE level: no-raid size: 222 GiB free: 156 GiB 
           Components: Online: 1: nvme3n1p3 
           Device-4: tank type: zfs status: ONLINE size: 38.2 TiB free: 25.6 TiB 
           Array-1: mirror status: ONLINE size: 12.7 TiB free: 8.69 TiB 
           Components: Online: N/A 
           Array-2: mirror status: ONLINE size: 12.7 TiB free: 8.76 TiB 
           Components: Online: N/A 
           Array-3: mirror status: ONLINE size: 12.7 TiB free: 8.15 TiB 
           Components: Online: N/A 
Drives:    Local Storage: total: raw: 100.23 TiB usable: 145.6 TiB used: 3.2 TiB (2.2%) 
           ID-1: /dev/nvme0n1 vendor: Corsair model: Force MP600 size: 465.76 GiB 
           ID-2: /dev/nvme1n1 vendor: Corsair model: Force MP600 size: 465.76 GiB 
           ID-3: /dev/nvme2n1 vendor: Corsair model: Force MP510 size: 894.25 GiB 
           ID-4: /dev/nvme3n1 vendor: Corsair model: Force MP510 size: 223.57 GiB 
           ID-5: /dev/nvme4n1 model: T-FORCE TM8FP7001T size: 931.51 GiB 
           ID-6: /dev/nvme5n1 model: T-FORCE TM8FP7001T size: 931.51 GiB 
           ID-7: /dev/nvme6n1 model: T-FORCE TM8FP7001T size: 931.51 GiB 
           ID-8: /dev/nvme7n1 model: T-FORCE TM8FP7001T size: 931.51 GiB 
           ID-9: /dev/sda vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB 
           ID-10: /dev/sdb vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB 
           ID-11: /dev/sdc vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB 
           ID-12: /dev/sdd vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB 
           ID-13: /dev/sde vendor: Crucial model: CT1000BX500SSD1 size: 931.51 GiB 
           ID-14: /dev/sdf vendor: Crucial model: CT1000BX500SSD1 size: 931.51 GiB 
           ID-15: /dev/sdg vendor: SanDisk model: SSD PLUS 1000GB size: 931.52 GiB 
           ID-16: /dev/sdh vendor: SanDisk model: SSD PLUS 1000GB size: 931.52 GiB 
           ID-17: /dev/sdi vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB 
           ID-18: /dev/sdj vendor: Seagate model: ST14000NM001G-2KJ103 size: 12.73 TiB 
           ID-19: /dev/sdk vendor: Seagate model: ST8000VN004-2M2101 size: 7.28 TiB 
           ID-20: /dev/sdl vendor: Seagate model: ST8000VN0022-2EL112 size: 7.28 TiB 
Partition: ID-1: / size: 209.35 GiB used: 60.07 GiB (28.7%) fs: zfs raid: rpool/ROOT/pve-1 
Swap:      Alert: No Swap data was found. 
Sensors:   System Temperatures: cpu: 51.6 C mobo: 35.0 C 
           Fan Speeds (RPM): fan-1: 0 fan-2: 0 fan-3: 0 fan-4: 0 fan-5: 0 fan-6: 0 fan-7: 0 
Info:      Processes: 1321 Uptime: 4d 3h 05m Memory: 251.51 GiB used: 164.77 GiB (65.5%) Init: systemd runlevel: 5 
           Shell: Bash inxi: 3.3.01

I am using PVE Proxmox Virtual Environment as a hypervisor which uses QEMU and KVM for virtualization. The hypervisor itself is not designed to provide a desktop experience, so, I pass through video cards and USB devices/controllers to Linux and Windows virtual machines.

Proxmox is designed as a dedicated hypervisor – so it’s build specifically to host virtual machines. However, Ubuntu is not designed from the ground up as a hypervisor. If you are running Ubuntu server, you may be better in switching to Proxmox for managing virtual machines. If you use this machine as a desktop and are using Ubuntu Desktop, you would likely need to install and use libvirt & virt-manager to manage virtual machines in the QEMU/KVM stack. (GUI friendly)

This guide here looks like it has some of the major relevant steps to managing virtual machines with QEMU/KVM on a debian based desktop distro Pop!_OS and performing gpu passthrough. https://github.com/bryansteiner/gpu-passthrough-tutorial This guide would likely have most of the important steps you would have to follow to get it working in Ubuntu Desktop.

If you can purcahse another Asus Hyper M.2 Card and have another 16x slot available, you can move the 2x M.2 cards that share the PCIE lanes with the U.2 connectors. (Also, I haven’t tested, but it may be possible that in SATA mode the U.2 connectors do not use the PCIE lanes)

25

Thank you again @DotBowder !! I’ve filed my info and your details to Asus and I hope they get out a fix. This should be a really compelling case to them because between us there are I think 8 distinct SATA drives tested as well as two different Linux installs. IDK why Windows might work but they can’t just say this is a system-specific issue.

And thanks for the info on PVE. I’ve never heard of it before now! I’ve used QEMU and mostly Docker, but never something like this. I mainly want to try Photoshop and some other Windows stuff and it’s great to know this stuff will work on hardware comparable to what I have.

Finally for the disks, I already have 2 Hyper M.2 cards and a total of 10 NVME devices, and the mobo manual claims you can only do 12 nvme devices (tho I’m not sure if that’s a limit of 12 for “hardware RAID” thru the BIOS settings or 12 no matter what). Another problem is I’m really tight on physical space due to GPUs and other stuff in the case.

One last tidbit: I’ve found that these QNAP dual 2.5 inch adapters appear to work with our Asus mobo (modulo Asus SATA errors): https://www.amazon.com/QNAP-Dual-SATA-Adapter-Converter/dp/B07RLM8W68 The adapter requires the mobo support SATA bifurcation… something my Mac won’t do but the Asus board does. They’re really handy cuz you can double your 2.5 inch drive density (e.g Samsung QVO SSDs) and gain some IOPS even tho the SATA 6g is shared.

26

@DotBowder Well I’ve gone through a couple loops with Asus support and they want me to RMA the board :frowning: As far as I can tell, their lab did not actually do anything like, say, just try to use linux with a SATA disk :stuck_out_tongue: and they are trying to just cover their butts and say it’s a one-off hardware problem. I can’t RMA the board because I use it for work… I’ll report here if Asus can find a way to be more productive. In the meantime, I guess your SATA adapter is the most viable fix.

27

Oh my, how weird things have gotten today. @oarphoarph

So, I was fiddling around with my server and managed to make my way to the proxmox forum. They advertised the 5.15 kernel is available for those who wish to test it.

Thinking, “I can always revert back to an older kernel” I went ahead and installed the 5.15 kernel. As you can see from my prior posts, my kernel version was 5.13.19-3.

Well, after updating to kernel 5.15, bam, SATA errors appeared on my U.2 connected drives! For kicks and giggles, I swapped all of the drives back to the SATA ports instead of the U.2 ports. Lo and behold, there are no SATA errors.

I did not touch the hardware when I upgraded from kernel 5.13 to 5.15. So, I have high confidence that there is some kind of interaction with the controller, but I haven’t found anything in the kernel patch notes that points to a fix. So :person_shrugging: I have no idea what the difference is.

I would suggest you see what kernel version your Ubuntu installation is on, and see if you can switch to a different kernel and see if there are any differences. As of right now, I’m on 5.15 (not yet supported by proxmox) and my SATA ports work. (And seemingly my U.2 ports as SATA ports do not work.)

28

@DotBowder Thanks for the info! Well I’m glad your SATA drives work!

I also have interesting news… I had to re-wire my machine a bit and I noticed that the Asus board actually has two different SATA controllers. Previously I had all my SATA drives plugged into the ports farthest from the CPU (the ones with E in the names in the pic). Do you know if these are the ports you were using? These apparently belong to a secondary ASmedia controller according to the manual. This time, I plugged the cables into the ports closest to the CPU, which the manual says are part of the AMD controller. Thus far I at least am able to copy about 2TB of data, which I have not been able to do before. Will keep testing tho.

Screen Shot 2022-02-20 at 11.34.26 AM

Also, Asus sent me an unclear message that suggested to me there might be some conflict between Hyper M.2 cards / NMVE and the SATA stuff. Maybe the secondary ASMedia SATA controller is junked if you use lots of NVME. I think we are both using Hyper M.2 / lots of NVME ?

2 Likes
29

I know I’m necroing a super old thread - but did you get your sata issues resolved?

I had a proxmox machine die, I was about to move my virtualized truenas server over, but saw some of the warnings you guys mentioned.