I have always wondered if it matters on what I pass through in a VM, wether it be 30/40 nvidia or 6000/7000 AMD GPU, intel not included for, well… Obvious reasons (not being out, yet… kinda)
And I only heard from one friend of my that I “Require” an nvidia for passthrough, if so, might I ask on why exactly? Thanks in forward.
AMD cards might still be susceptible to a reset bug. What happens is you start up a virtual machine and do the work you need to get done. Then, shut down the virtual machine; the next time you need to use it, it wouldn’t boot. To be able to use the virtual machine again, the host system must be rebooted. Hence the term reset bug. Nividia cards are just more accessible to pass through. There are a lot of other considerations when deciding which graphic card makes an excellent candidate to be passed through to a VM.
“Reset bug” as it’s mentioned would probably not be a big issue to me, since I won’t be really using the VM multiple times in 1 day. If I do, eh, one reset isn’t a big issue.
Reset bug also applies if you need to restart a VM, for updates for example. Plus it can also cause instability for the host or other adverse effects like high i/o latency if the HW is left in a „uncertain“ state.
That said, I have a GeForce RTX 3080ti and a Radeon RX 6600 in the same system and the AMD card behaves if I the VMs are shutdown properly e.g. going through the standard shutdown sequence of the guest OS.
As for Nvidia vs AMD. I have yet to break vfio for Nvidia cards. I can halt VMs, I can switch between VMs as much as I want, the Nvidia card just works ™. With Nvidia all cards released since the GTX 7xx series with UEFI support should work out of the box, doesn’t matter if it’s the cheap 30$ GT 710 or high end 3090. With AMD (except the latest RX 66xx - RX 69xx) you have to research - what AMD GPU Plattform it is based on, what bugs regarding vfio does it have etc.
The RX 6600 card is great for Linux Wayland and for MacOS VMs and that’s mainly the reason I use one.
Good to hear the wonderful news of RX6000 from you. I’m looking to buy a new AMD GPU (thought not urgently). Radeon RX6600/XT is one of my considerations!
Is my below understanding correct?
Btw, may I confirm or kindly ask you to do a test? Put MacOS VM into sleep/hibernate. Once done. Wake it up from QEMU monitor. Does the MacOS VM kernel panic and then restarts automatically? The KP may indicate “GPU not found” during wake-up. Thanks
Can do, but I use Proxmox as headless host. It’s either a spice based output on the webfrontend or dedicated GPU output. But I can try suspending or putting on sleep mode and see if it breaks. I’ll also check the logs.
Edit: Regarding your questions. I can run and switch to various VMs including Mac OS.
I don’t bind my GPUs to a vfio-pci dummy driver, but I blacklisted all GPU drivers.
I don’t use any vendor reset or reset mitigation outside of what Proxmox provides - dunno if they have anything baked in. (Proxmox is mostly stock Debian, their PVE modules and ZFS support)
The only two times I broke vfio was with a misconfigured core boot- boot loader and with a misconfigured Linux VM which couldn’t shut down because I passed two USB ports from a PCIe USB card which was already passed to the same guest. I switched between 3 different Windows VM in oder to update the GPU driver plus back and forth to the Mac OS VM without problems, including various VM restarts.
I put the Mac OS VM to sleep. I could resume it via the webgui but it was not responding (no error, but no display either) and didn’t react to keypresses. Unfortunately I forgot to ping or connect to it remotly.
I checked /var/log/messages and systemd journal, could not find any AMD-V related error messages, though.
I could halt the VM and restart it. vfio still worked.
Thanks a lot. My take away is that resume from sleep/hibernate in MacOS VM is still a challenge. Other than that RX6000 seems very compatible with the MacOS driver. Great news!
The RX 6600 needs the lilu.kext and whatevergreen.kext and the entry agdpmod=pikera in the config.plist of coreboot. Both kext should come by default with the modified version of coreboot for VMs. (Dunno about a bare metal Hackintosh)
I try to keep ‘hacks’ to minimum as much as possible. My MacOS VM with RX560 passed through seems working the same with or without whatevergreen. Hence, I don’t use. 
Separately, below I include some sample AMD-V error logs when my MacOS VM reboots:
Aug 25 05:17:29 zen kernel: AMD-Vi: Completion-Wait loop timed out
Aug 25 05:17:29 zen kernel: iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=26:00.0 address=0x1001dc5c0]
Aug 25 05:17:30 zen kernel: AMD-Vi: Completion-Wait loop timed out
Aug 25 05:17:30 zen kernel: AMD-Vi: Completion-Wait loop timed out
Aug 25 05:17:30 zen kernel: iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=26:00.0 address=0x1001dc600]
Aug 25 05:17:30 zen kernel: iommu ivhd0: AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=26:00.0 address=0x1001dc620]
The AMD RX 5xx series works out of the box for Hackintosh without the need for tweaks. Unfortunately you can get these GPUs only used nowadays and most
have been abused for mining. I got burned once and decided for the RX 6600 as next price to performance option.
Lilu.kext is basically only a framework and whatevergreen.kext includes workarounds which have to be enabled via config.plist - in the case of of this AMD card it corrects the HW IDs so it is recognized by Mac OS. In this configuration it doesn’t inject driver functionality or other executable code which might break after a update.
I greped for AMD in the journal and the logs but could not find anything similar to the errors you listed.
Edit: I think it’s not much different than running core boot in the first place the kexts are enabled by default anyways.
According to one of VFIO authors at Redhat, the errors indicate the passed through GPU doesn’t play well with the guest OS (mainly its GPU driver). So it’s one of the reasons I want a newer AMD GPU.
WhateverGreen collects a bunch of patches for various GPUs. I believe WG may cover issues of some edge cases. For my RX560, I didn’t notice a difference in three years of use. That’s why I don’t use.
RX6600/XT is natively supported in MacOS too. Have you tried to run without Whatevergreen? It’ll be another bonus if it does without WG.
I tried running it without, only got a black screen / no signal after the Apple splash / loading screen.
As far as I know the Mac OS or vfio issues might also be vendor specific. Saphire cards seem to have the best compatibility with Hackintosh systems.
Black screen most likely is an issue of connector config. WhateverGreen will automatically fix it. It’s possible to do it manually in the config plist.Sadly I don’t (want to) understand the thing a bit. My RX560 is lucky to match the default of MacOS driver I believe.
Thanks for trying. RX6600/XT sounds very good to me already.