1366 5520 PCI-Passthrough awful Performance

Larbalappen · January 3, 2018, 6:43pm

Hello,

i tried many times to get PCI-Passthrough via KVM in arch to work and succeeded in doing so after ditching X58-Asus-Boards because they have some VT-d issues. I use a
Dell T5500
X5675
6x4GB Registered DDR3@ 1333MHz
RX 480 8GB
The Linux card is a FX 1800 until i buy a better one when the Virtual Machine runs.

I followed https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF and used pci-stub because vfio-pci wouldn’t claim one device of the Card.
IOMMU and DMAR are working just fine.

The machine works but with wierd colors as seen above. When Idle, the CPU is around 25% as Windows Task manager reports, it is slow as hell and when i install graphics drivers i cant see anything. Wait a bit, reboot the machine, no picture after UEFI boot logo. I tried a 1070 from a friend once. CPU load was the same, but with a working Picture after driver install.

Why? What have i done wrong? Is it because i dont have a UEFI? If so: SeaBIOS clients are an option i never got to work but admittedly i have not invested too much time into it. Would Sandy-Bridge be a better option? 2011 Boards which are ECC compatible and working VT-d are very expensive. I want to reuse my RAM if possible because RAM is sooo damn expensive. Some good recommendations are appreciated as well.

Thank you for helping. Have a great day

SgtAwesomesauce · January 3, 2018, 6:58pm

That’s a problem. Did you stub and then bind vfio-pci to the card once you’ve booted?

Larbalappen · January 3, 2018, 7:25pm

No. I scrapped vfio-pci completely and only used pci-stub.

SgtAwesomesauce · January 3, 2018, 7:35pm

pci-stub is deprecated. Don’t use it.

That’s your main problem.

Larbalappen · January 3, 2018, 7:37pm

Well ok. That might explain the weird colors, but not the performance, am i right? Why is the Performance soo bad, even when using CPU-Pinning?

SgtAwesomesauce · January 3, 2018, 7:42pm

Let’s fix the GPU issues before we go after the performance issues.

First, how old is that FX card? You might want to check if it supports UEFI. If not, switch to seabios.

On the topic of performance, are you using static hugepages?

Which distro are you using?

Honestly, if you want to change hardware, I’d just go Ryzen.

Have you set up IO Threads? If not, do that.

Larbalappen · January 3, 2018, 7:52pm

[quote=“sgtawesomesauce, post:6, topic:123078”]
First, how old is that FX card? You might want to check if it supports UEFI. If not, switch to seabios.
[/quote] Socket 1366 does not support UEFI. There is no UEFI board. Not X58, not 5520. The FX is only for Linux, not for passing through.

[quote=“sgtawesomesauce, post:6, topic:123078”]
On the topic of performance, are you using static hugepages?
[/quote] I did. I dont have a Linux running on that machine right now, but i can install it later.

[quote=“sgtawesomesauce, post:6, topic:123078”]
Which distro are you using?
[/quote] Archlinux

[quote=“sgtawesomesauce, post:6, topic:123078”]
Honestly, if you want to change hardware, I’d just go Ryzen.
[/quote] Ram is expensive. I’d need a X370 board, a ryzen and ram. That would cost a lot because RAM is expensive.

Which one do you recommend? I think i used Noop.

SgtAwesomesauce · January 3, 2018, 8:01pm

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win10-gpu</name>
  <uuid>5c25a5d5-5be1-4519-859e-f5b7df7a38f5</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <iothreads>4</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='7'/>
    <iothreadpin iothread='1' cpuset='0-1'/>
    <iothreadpin iothread='2' cpuset='2-3'/>
    <iothreadpin iothread='3' cpuset='4-5'/>
    <iothreadpin iothread='4' cpuset='6-7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.10'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/ovmf/gitmaster/ovmf_code_x64.bin</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10-gpu_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='novideo'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>

That’s the CPU configuration I’ve got for my Passthrough VM on a Ryzen 1700 system.

Take note of how I set up the <iothreadpin section.

Larbalappen · January 3, 2018, 8:02pm

I will try this later and report. Thank you.

SgtAwesomesauce · January 3, 2018, 8:18pm

Sorry, I was thinking the 1800 is guest card for some reason. RE: the 480, which exact card is it?

Have you added the modules to your initcpio? If not, make sure you do that.

That’s true. Any board change would cost a lot. My point is that if you’re going to upgrade, Ryzen isn’t a bad option. I think you’ll probably be okay on this system for a while though.

Larbalappen · January 3, 2018, 8:30pm

Powercolor RX 480 Red Dragon 8gb

Yes i did

If my 1366 would not work i thought about 2011 to have :

UEFI
DDR3 ECC
Good Price
Good Performance

The Price is difficult because of the very expensive Boards, which have VT-d working and work with ECC-Memory.
2011 Price: 100€ CPU, 200-350 MB, IDK Cooler.
AM4 Price: 200€ CPU, 150 MB, Cooler 0€?, RAM 300.

SgtAwesomesauce · January 3, 2018, 8:33pm

Some 480’s have the reset bug. I don’t know about this specific one, but I’ll do some digging and get back to you.

I’m fairly certain it’s not your system that’s causing the problem. (aside from the GPU, potentially)

FurryJackman · January 3, 2018, 9:03pm

Well… Don’t use it for GPUs. That’s the big thing. Some USB controllers don’t hook onto vfio-pci. So that has to use pci-stub.

Larbalappen · January 3, 2018, 9:13pm

Thank you very much. I will start my arch installation when you give the ok:D

Larbalappen · January 4, 2018, 2:41pm

I am actually running the 480 in a VM now. I got no problem with reset, only with reinitialization. Which is fine for now. Although i have seen a post somewhere around here that fixes that. I still got not the best performance. IOThreads are now on cores, that are not taken by the vm and it works kinda fine.

SgtAwesomesauce · January 4, 2018, 6:39pm

Good, Sorry I never got back to you, had a bit of a shitshow dealing with the fallout from the Intelpocalypse at work.

So, reinitialization and reset are essentially the same thing. The problem on some AMD cards is the reinitialization, which is done when a bus reset is attempted.

Larbalappen · January 4, 2018, 7:22pm

This is why we should slowly but surely switch to RISC. But hey, thats just my opinion.

Do you think following Linux Host, Windows Guest, GPU passthrough reinitialization fix would be a good idea?
Performance seems to be ok after some time. I still got some Problems with sound. AC97 drivers are bluescreening my vm, ich9 is very distorted. What are you doing with your audio?
Do you have experience with stripping down Windows, so it wont take so much CPU?
Because of Hyperthreading Processor/Core ID are garbled up and i really don’t get what to do with my cpupins. May i invite you to my teamspeak?

SgtAwesomesauce · January 4, 2018, 8:27pm

The vulnerability not about RISC vs CISC. It’s about speculative execution and branch prediction. But yes, I do agree that RISC would be nicer to use.

Worth a try.

Switch to Linux. Seriously though, I don’t. How long have you kept the system up? You might start to see improvements if you let windows just do it’s thing for a bit.

I don’t typically use teamspeak. Let me see if I can get an RPM package for it.