Return to Level1Techs.com

3960X / Radeon VII / QEMU / macOS Catalina / Passthrough issues

Hi all,

I’m trying to get Catalina going properly with the following setup:
Gigabyte TRX40 Designare
Threadripper 3960X 24-core
4x32GB = 128GB DDR4
Radeon VII 16GB
Latest Manjaro on kernel 5.5 (no MCE bug)

I’m currently at a stage that I can use 16 cores and 64GB RAM with decent albeit not native performance, and the GPU is recognized in macOS but as far as I can tell it’s functioning in VESA-mode (no actual hardware acceleration). Also, when booting Manjaro I need to physically disconnect the Radeon VII DisplayPort cable, otherwise Manjaro boots to a black screen which isn’t fixed by unplugging afterwards (I just reboot). I need to plug the DP-cable when in Manjaro for it to boot normally.

I have mostly followed Github kholia/OSX-KVM, but after changing distros from Ubuntu to Fedora to Manjaro I had to start looking around to try to fix problems for GPU and virbr0/tun0 passthrough and such.

I would like to know how the following things:

  • How to increase the maximum CPU core count in QEMU to 24 out of 24 (if possible, otherwise highest possible);
  • I find it hard to figure out whether SMT is being accounted for, altough it does seem the case even with below config (threads=1);
  • Can the maximum system memory be allocated? macOS now seems to think it has 4x16GB installed;
  • How to passthrough the Radeon VII fully, as it is functioning in VESA mode now (with or without Whatevergreen.kext in Clover EFI)
  • How to fix the need to physically disconnect the DP-cable of the passthrough GPU;
  • How to fix virbr0 attached to tun0 permanently, for any ethernet passthrough (currently succesfully using the Titan Ridge AIC -> TB3<>TB2 -> TB2 <> ETH adapters;

I hope these troubleshoot files are enough, but I’m more than willing to supply any other information needed to succesfully complete this project!

lspci -nn -k -D

0000:04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 20 [Radeon VII] [1002:66af] (rev c1)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Vega 20 [Radeon VII] [1002:081e]
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu
0000:04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 20 HDMI Audio [Radeon VII] [1002:ab20]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Vega 20 HDMI Audio [Radeon VII] [1002:ab20]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel
0000:21:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP108 [GeForce GT 1030] [10de:1d01] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] GP108 [GeForce GT 1030] [1462:8c98]
	Kernel driver in use: nouveau
	Kernel modules: nouveau

Note: GT1030 is placed in PCIE16_1, Radeon VII is placed in PCIE8_1. GT1030 is primary UEFI boot device in BIOS.

/etc/modprobe.d/vfio.conf:

options vfio-pci ids=1002:66af,1002:ab20,1022:149c

Note: GPU, GPU audio, USB

/etc/mkinitcpio/conf

MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd kvm=amd) BINARIES=() FILES="" HOOKS="base udev autodetect modconf block keyboard keymap lvm2 filesystems"

/etc/default/grub

GRUB_CMDLINE_LINUX_DEFAULT="quiet apparmor=1 security=apparmor udev.log_priority=3 amd_iommu=on iommu=1 rd.driver.pre=vfio-pci"

Note: I feel the VESA-problem may be related to Grub boot arguments, but I’m not sure.

My boot.sh file:

#!/bin/bash

# qemu-img create -f qcow2 mac_hdd_ng.img 128G
#
# echo 1 > /sys/module/kvm/parameters/ignore_msrs (this is required)

############################################################################
# NOTE: Tweak the "MY_OPTIONS" line in case you are having booting problems!
############################################################################

# This works for High Sierra as well as Mojave. Tested with macOS 10.13.6 and macOS 10.14.4.

MY_OPTIONS="+pcid,+ssse3,+sse4.2,+popcnt,+avx,+aes,+xsave,+xsaveopt,check"

# OVMF=./firmware
OVMF="./"

qemu-system-x86_64 -enable-kvm -m 65536 -cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,$MY_OPTIONS\
	  -machine q35 \
	  -smp cpus=16,cores=16,threads=1,sockets=1 \
	  -vga none \
	  -device vfio-pci,host=04:00.0,bus=pcie.0,multifunction=on \
	  -device vfio-pci,host=04:00.1,bus=pcie.0 \
	  -device vfio-pci,host=47:00.1,bus=pcie.0 \
	  -device vfio-pci,host=47:00.3,bus=pcie.0 \
	  -usb -device usb-kbd -device usb-tablet \
	  -device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" \
	  -drive if=pflash,format=raw,readonly,file=$OVMF/OVMF_CODE.fd \
	  -drive if=pflash,format=raw,file=$OVMF/OVMF_VARS-1024x768.fd \
	  -smbios type=2 \
	  -device ich9-intel-hda -device hda-duplex \
	  -device ich9-ahci,id=sata \
	  -drive id=Clover,if=none,snapshot=on,format=qcow2,file=./'Mojave/CloverNG.qcow2' \
	  -device ide-hd,bus=sata.2,drive=Clover \
	  -drive id=MacHDD,if=none,file=./mac_hdd_ng.img,format=qcow2 \
	  -device ide-hd,bus=sata.4,drive=MacHDD \
	  -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -device vmxnet3,netdev=net0,id=net0,mac=52:54:00:c9:18:27 \
	  -monitor stdio \

Clover configuration is iMacPro1,1 supplied (but updated) Clover version and drivers. Kexts:
(optional) Whatevergreen.kext
AppleALC.kext (not yet configured any audio)
AppleMCEReporterDisabler.kext (known to prevent kernel panics for AMD-based Hackintoshes)
Lilu.kext (master of Whatevergreen, AppleALC)

Like I said, if anything else is needed, please do let me know. I understand I’m asking many questions at once, so my primary concern would be GPU/CPU passthrough, networking and minor issues like cabling coming last.

Have you set the Boot VGA GPU to the GT 1030? Gigabyte BIOSes allow that to be the option.

Also, don’t use qcow for images, use raw for your storage files.

Lilu and WhateverGreen are always a good idea to have on a Clover based Hackintosh with AMD cards. Without Lilu and WhateverGreen, my RX 580 won’t even boot.