Ryzen3000 performs very great for kvm gaming

SirTeddy · July 13, 2019, 12:56pm

Cpu 3700x no graphic
Gpu1 1080 (highest)
Gpu2 1063 (medium & high)
Asus x570 prime pro
650watt
Two gaming virtual machines
With no additional graphic

AnotherDev · July 13, 2019, 3:11pm

Hey man, nice pictures. Do you mind going into a bit more detail?

Is this a headless KVM install, with the two GPUs strictly being used for Windows virtual machines in KVM? Or, are you using one for VM and one for local?

Pixo · July 13, 2019, 5:04pm

How are the IOMMU groups?

This is the only board that would fit 2 GPU, Vive wireless card, sound card, 10 GB NIC, 2x NVME and even has COM port ( frees one slot that can be used for wifi if needed ).
Only feature missing is BIOS flashback.

SirTeddy · July 14, 2019, 12:32am

Two vms . Linux will use gpu1 during system boot. After “virsh start vm1”,gpu1 will be used by vm1. Then “virsh start vm2”,vm2 has the gpu2 (no strict order for two “virsh”）.then we can go to play.

SirTeddy · July 14, 2019, 12:49am

Here is “lspci”


00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1480
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 1481
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1440
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1441
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1442
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1443
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1444
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1445
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1446
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1447
01:00.0 Non-Volatile memory controller: Silicon Motion, Inc. Device 2263 (rev 03)
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57ad
03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
05:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1485
05:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Device 149c
05:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 149c
06:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
07:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
08:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
08:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
09:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1)
09:00.1 Audio device: NVIDIA Corporation Device 10f1 (rev a1)
0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 148a
0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1485
0b:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1486
0b:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 149c
0b:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1487
0c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
0d:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

Then iommu groups


/sys/kernel/iommu_groups/17/devices/0000:03:09.0
/sys/kernel/iommu_groups/17/devices/0000:06:00.0
/sys/kernel/iommu_groups/7/devices/0000:00:04.0
/sys/kernel/iommu_groups/15/devices/0000:03:05.0
/sys/kernel/iommu_groups/5/devices/0000:00:03.1
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/21/devices/0000:09:00.0
/sys/kernel/iommu_groups/21/devices/0000:09:00.1
/sys/kernel/iommu_groups/11/devices/0000:00:14.3
/sys/kernel/iommu_groups/11/devices/0000:00:14.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/18/devices/0000:07:00.0
/sys/kernel/iommu_groups/18/devices/0000:03:0a.0
/sys/kernel/iommu_groups/8/devices/0000:00:05.0
/sys/kernel/iommu_groups/16/devices/0000:05:00.3
/sys/kernel/iommu_groups/16/devices/0000:05:00.1
/sys/kernel/iommu_groups/16/devices/0000:03:08.0
/sys/kernel/iommu_groups/16/devices/0000:05:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:03.2
/sys/kernel/iommu_groups/14/devices/0000:02:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:03.0
/sys/kernel/iommu_groups/12/devices/0000:00:18.3
/sys/kernel/iommu_groups/12/devices/0000:00:18.1
/sys/kernel/iommu_groups/12/devices/0000:00:18.6
/sys/kernel/iommu_groups/12/devices/0000:00:18.4
/sys/kernel/iommu_groups/12/devices/0000:00:18.2
/sys/kernel/iommu_groups/12/devices/0000:00:18.0
/sys/kernel/iommu_groups/12/devices/0000:00:18.7
/sys/kernel/iommu_groups/12/devices/0000:00:18.5
/sys/kernel/iommu_groups/2/devices/0000:00:01.2
/sys/kernel/iommu_groups/20/devices/0000:08:00.0
/sys/kernel/iommu_groups/20/devices/0000:08:00.1
/sys/kernel/iommu_groups/10/devices/0000:0c:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:08.0
/sys/kernel/iommu_groups/10/devices/0000:0b:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:08.3
/sys/kernel/iommu_groups/10/devices/0000:00:08.1
/sys/kernel/iommu_groups/10/devices/0000:0b:00.3
/sys/kernel/iommu_groups/10/devices/0000:0b:00.1
/sys/kernel/iommu_groups/10/devices/0000:0d:00.0
/sys/kernel/iommu_groups/10/devices/0000:00:08.2
/sys/kernel/iommu_groups/10/devices/0000:0b:00.4
/sys/kernel/iommu_groups/0/devices/0000:00:01.0
/sys/kernel/iommu_groups/19/devices/0000:04:00.0
/sys/kernel/iommu_groups/9/devices/0000:0a:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:07.0
/sys/kernel/iommu_groups/9/devices/0000:00:07.1

I choose this motherboard because it has sli support.(only this board has intro of sli during pre-order)Ryzen 3000 is so strong ,maybe vm2 can just use the x4 slot without sli (intel can do this because of high speed DMI bridge)

Goalkeeper · July 14, 2019, 1:01am

Woah. Is that the Mac OS dock I see? Did you get hackintosch going on the ryzen 3000?

Also if use the ` key three times ( ``` ) it will clean up the lspci and iommu text. For example:

00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1444
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1445
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1446
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1447
01:00.0 Non-Volatile memory

SirTeddy · July 14, 2019, 1:06am

Sadly,no. I just use macbook pro to ssh linux system. I was able to boot macos vm 2 years ago. I’ll try it if I have enough time

Pixo · July 14, 2019, 8:46am

Thank you for the information.
The groups have regressed on the CPU side as audio, USB and Encryption controller are on the same group. This was the same at first for 1st gen Ryzen so there is a possibility it will be split in the future.
On the chipset side all is that is connected is the integrated NIC so its hard to say if all will be split. But the NIC is in its own group and that looks promising.

Sorted the info for whoever want to have easier overview of the groups:

1  0000:00:01.1  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
2  0000:00:01.2  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
3  0000:00:02.0  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
4  0000:00:03.0  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
5  0000:00:03.1  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
6  0000:00:03.2  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1483
7  0000:00:04.0  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
8  0000:00:05.0  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
9  0000:00:07.0  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
9  0000:00:07.1  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
9  0000:0a:00.0  Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 148a
10 0000:00:08.0  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1482
10 0000:00:08.1  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
10 0000:00:08.2  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
10 0000:00:08.3  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1484
10 0000:0b:00.0  Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1485
10 0000:0b:00.1  Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1486
10 0000:0b:00.3  USB controller: Advanced Micro Devices, Inc. [AMD] Device 149c
10 0000:0b:00.4  Audio device: Advanced Micro Devices, Inc. [AMD] Device 1487
10 0000:0c:00.0  SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
10 0000:0d:00.0  SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
11 0000:00:14.0  SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
11 0000:00:14.3  ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
12 0000:00:18.0  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1440
12 0000:00:18.1  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1441
12 0000:00:18.2  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1442
12 0000:00:18.3  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1443
12 0000:00:18.4  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1444
12 0000:00:18.5  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1445
12 0000:00:18.6  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1446
12 0000:00:18.7  Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1447
13 0000:01:00.0  Non-Volatile memory controller: Silicon Motion, Inc. Device 2263 (rev 03)
14 0000:02:00.0  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57ad
15 0000:03:05.0  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a3
16 0000:03:08.0  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
16 0000:05:00.0  Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1485
16 0000:05:00.1  USB controller: Advanced Micro Devices, Inc. [AMD] Device 149c
16 0000:05:00.3  USB controller: Advanced Micro Devices, Inc. [AMD] Device 149c
17 0000:03:09.0  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
17 0000:06:00.0  SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
18 0000:03:0a.0  PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 57a4
18 0000:07:00.0  SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
19 0000:04:00.0  Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
20 0000:08:00.0  VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1080] (rev a1)
20 0000:08:00.1  Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
21 0000:09:00.0  VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1)
21 0000:09:00.1  Audio device: NVIDIA Corporation Device 10f1 (rev a1)```

SirTeddy · July 15, 2019, 6:58am

Did you try pcie_acs_downstream=override

I didnt try it,either because it was good enough for me at that time

Pixo · July 15, 2019, 12:58pm

The ACS oweride would work but my hope is that it would not be needed on x570.
Its a hack so not that great, and as such it will never be upstreamend.

eric.frederich · July 15, 2019, 1:12pm

This is the exact motherboard I was looking at. Having been out of the game for a while I remember Asus makes quality stuff. I would like to get a 3900X and this board to run Unraid.
What can you tell me about these IOMMU groups? Are they good, bad? What am I looking for?
Would I be able to pass an entire NVMe to a VM?

Currently I run Unraid and have just a single Linux VM which I use as a desktop. I’d like to set it up to game on a Windows VM. What else should I be looking at with this IOMMU stuff?

SirTeddy · July 15, 2019, 2:40pm

Checkout my & Pixo’s replies above.

NVMe controller is in a single group. √
I put 2 GPUs, and they are all in single group. √
Audio card and usb controller are in the same group. ×
kernel 4.19.58

Maybe try acs_override.Or they may deprated in the future.

SirTeddy · July 15, 2019, 2:41pm

You mean acs_override patch has disadvantages?
I 'm intrested in it ,pls tell me more.

SirTeddy · July 15, 2019, 2:41pm

Checkout my & Pixo’s replies above.

NVMe controller is in a single group. √
I put 2 GPUs, and they are all in single group. √
Audio card and usb controller are in the same group. ×
kernel 4.19.58

Maybe try acs_override.Or they may deprated in the future.

hondaman · July 15, 2019, 3:26pm

Thats not very good groups to be honest. I was hoping for better with x570

Pixo · July 15, 2019, 5:10pm

Zen1 had this grouping and later it was split. Seems that its back on Zen2. And i read somewhere its the same even on Zen1 with the latest AGESA. So there is a hope it will be split again with later AGESA update.
EDIT: And the PCH should be the same silicon as on the CPU and there the USB and SATA are split.

@SirTeddy the ACS patch disadvantages is that it needs a patched kernel and it needs to be maintained out-of-tree.
And it creates a hole in VM for possible escape. It lies about HW isolation between devices so they can communicate across VMs and host. Not a big problem for home users. But due to this it will newer be mainlined.

SirTeddy · July 16, 2019, 4:51am

Wow,so acs_override may cause hw fault?

I think audio card split is important,but not usb.I didnt passthrough usb controller ,but the usb devices could be hotpluged and be splited into two vms(even with hub).

Pixo · July 16, 2019, 6:41am

It will not cause hw faults.
It just means that the devices can communicate device-to-device with each other when they should not be able to do that. Its not that big deal for home users running gaming VM, but for enterprise and cloud providers its a big no.

As for the USB. Qemu has a mechanism of passing only the USB device. As USB is digital interface, its similar how you can forward network packet through virtual NIC to VM.

coel13 · July 16, 2019, 3:01pm

@SirTeddy Are you assigning 4c4t for each gaming VM? I am planning to build Unraid server for:

2 gaming VM (3c3t each)
NAS (Unraid)
Docker (Unraid)
VM (not heavy job, testing only, e.g. Kali, etc.)

I am not sure if 3700x is enough for this or I have to get 3900x?

SirTeddy · July 17, 2019, 8:05am

4c8t for each vm. If you assign 3c6t for each gaming vm,it remains 2c4t,an i3 7100.No problem for nas.If all your job is not so heavy,3700x is enough.pls put 2 channel dimms,splited disk assignment and consider for 2gpus’ thermal problem.