Threadripper 2990wx + Vega 64 + KVM GPU passthrough freeze on Apple Logo

Operating Systems & Open Source Linux
1

Hello, I’ve been struggling with this setup for over a week now.

I’ve seen people having threadripper&vega 64 passthrough successfully, and there’s not much info for me to debug the cause of this freeze, so I’m stuck.

GPU seems to be able to passthrough, usb passthrough works fine, but as soon as I boot from clover’s menu it will start loading, showing Apple Logo and a bar, the bar moves for couple of seconds, then freeze.
Without GPU passthrough, no freeze, everything works fine.

My setup:
CPU: Threadripper 2990wx
Motherboard: Asus ROG ZENITH EXTREME
passthrough GPU: ASUS ROG STRIX Radeon RX Vega64 8GB
host GPU: Radeon Pro WX4100
host OS: Ubuntu 18.04.1 & Fedora 28
macOS version: High Sierra 10.13.6
clover version: 4674

I followed https://github.com/kholia/OSX-KVM and binds vfio-pci to vega 64. lspci -nnk shows that kernel driver in use for both video & audio of vega are vfio-pci.

My monitor attached to vega 64 can receive signal & display ovmf & clover boot screen properly.
But when I boot macOS from Clover, it will freeze on Apple Logo.
If I add -v to boot args of Clover, it will show the following,
and freeze at the last line “initialize normal”, google shows me nothing

IMG_20181029_195323
IMG_20181029_195323.jpg3931x2399 2.42 MB

I first tried this in ubuntu, then fedora, using same disk image installed macOS HS, freeze in the exact same way.
I checked ovmf & clover’s resolution are the same.
I downloaded & added the romfile for my Vega 64, nothing changed.
Then I tried to change smbios to iMac 14,2, nothing changed.

My clover config:

Screenshot%20from%202018-11-02%2006-26-53
Screenshot from 2018-11-02 06-26-53.png639x630 121 KB

config.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
	<key>ACPI</key>
	<dict>
		<key>DSDT</key>
		<dict>
			<key>Debug</key>
			<false/>
			<key>DropOEM_DSM</key>
			<dict>
				<key>ATI</key>
				<false/>
				<key>Firewire</key>
				<false/>
				<key>HDA</key>
				<false/>
				<key>HDMI</key>
				<false/>
				<key>IDE</key>
				<false/>
				<key>IntelGFX</key>
				<false/>
				<key>LAN</key>
				<false/>
				<key>LPC</key>
				<false/>
				<key>NVidia</key>
				<false/>
				<key>SATA</key>
				<false/>
				<key>SmBUS</key>
				<false/>
				<key>USB</key>
				<true/>
				<key>WIFI</key>
				<false/>
			</dict>
			<key>Fixes</key>
			<dict>
				<key>AddPNLF_1000000</key>
				<true/>
				<key>DeleteUnused_400000</key>
				<true/>
				<key>FIX_ACST_4000000</key>
				<true/>
				<key>FIX_ADP1_800000</key>
				<true/>
				<key>FIX_RTC_20000</key>
				<true/>
				<key>FIX_S3D_2000000</key>
				<true/>
				<key>FixRegions_10000000</key>
				<true/>
			</dict>
			<key>Name</key>
			<string>q35-acpi-dsdt.aml</string>
			<key>ReuseFFFF</key>
			<false/>
		</dict>
		<key>SSDT</key>
		<dict>
			<key>DropOem</key>
			<false/>
			<key>Generate</key>
			<false/>
		</dict>
	</dict>
	<key>Boot</key>
	<dict>
		<key>Debug</key>
		<false/>
		<key>DefaultLoader</key>
		<string>boot.efi</string>
		<key>DefaultVolume</key>
		<string>LastBootedVolume</string>
		<key>Legacy</key>
		<string>PBR</string>
		<key>Secure</key>
		<false/>
		<key>Timeout</key>
		<integer>5</integer>
		<key>XMPDetection</key>
		<string>Yes</string>
	</dict>
	<key>CPU</key>
	<dict>
		<key>QEMU</key>
		<true/>
		<key>UseARTFrequency</key>
		<false/>
	</dict>
	<key>Devices</key>
	<dict>
		<key>USB</key>
		<dict>
			<key>FixOwnership</key>
			<false/>
			<key>Inject</key>
			<true/>
		</dict>
	</dict>
	<key>GUI</key>
	<dict>
		<key>Mouse</key>
		<dict>
			<key>DoubleClick</key>
			<integer>500</integer>
			<key>Enabled</key>
			<false/>
			<key>Mirror</key>
			<false/>
			<key>Speed</key>
			<integer>8</integer>
		</dict>
		<key>Scan</key>
		<true/>
		<key>ScreenResolution</key>
		<string>1600x1200</string>
		<key>Theme</key>
		<string>embedded</string>
	</dict>
	<key>Graphics</key>
	<dict>
		<key>Inject</key>
		<false/>
		<key>NvidiaSingle</key>
		<false/>
	</dict>
	<key>KernelAndKextPatches</key>
	<dict>
		<key>AppleRTC</key>
		<false/>
		<key>AsusAICPUPM</key>
		<false/>
		<key>Debug</key>
		<false/>
		<key>KernelCpu</key>
		<false/>
		<key>KernelHaswellE</key>
		<false/>
		<key>KernelLapic</key>
		<false/>
		<key>KernelPm</key>
		<false/>
	</dict>
	<key>RtVariables</key>
	<dict>
		<key>BooterConfig</key>
		<string>0x28</string>
		<key>CsrActiveConfig</key>
		<string>0x67</string>
		<key>ROM</key>
		<string>UseMacAddr0</string>
	</dict>
	<key>SMBIOS</key>
	<dict>
		<key>BiosReleaseDate</key>
		<string>09/17/2018</string>
		<key>BiosVendor</key>
		<string>Apple Inc.</string>
		<key>BiosVersion</key>
		<string>IM142.88Z.0131.B00.1809171347</string>
		<key>Board-ID</key>
		<string></string>
		<key>BoardManufacturer</key>
		<string>Apple Inc.</string>
		<key>BoardSerialNumber</key>
		<string></string>
		<key>BoardType</key>
		<integer>10</integer>
		<key>BoardVersion</key>
		<string>1.0</string>
		<key>ChassisAssetTag</key>
		<string>iMac-Aluminum</string>
		<key>ChassisManufacturer</key>
		<string>Apple Inc.</string>
		<key>ChassisType</key>
		<string>0x0D</string>
		<key>EfiVersion</key>
		<string>133.0.0.0.0</string>
		<key>Family</key>
		<string>iMac</string>
		<key>FirmwareFeatures</key>
		<string>0xE00FE137</string>
		<key>FirmwareFeaturesMask</key>
		<string>0xFF1FFF3F</string>
		<key>LocationInChassis</key>
		<string>Part Component</string>
		<key>Manufacturer</key>
		<string>Apple Inc.</string>
		<key>Mobile</key>
		<false/>
		<key>PlatformFeature</key>
		<string>0x01</string>
		<key>ProductName</key>
		<string>iMac14,2</string>
		<key>SerialNumber</key>
		<string></string>
		<key>SmUUID</key>
		<string></string>
		<key>Version</key>
		<string>1.0</string>
	</dict>
	<key>SystemParameters</key>
	<dict>
		<key>InjectKexts</key>
		<string>Detect</string>
		<key>InjectSystemID</key>
		<true/>
	</dict>
</dict>
</plist>

boot-passthrough.sh

#!/bin/bash

MY_OPTIONS="+aes,+xsave,+avx,+xsaveopt,+xsavec,+xgetbv1,+avx2,+bmi2,+smep,+bmi1,+fma,+movbe"

qemu-system-x86_64 -enable-kvm -m 12G -cpu Penryn,kvm=on,vendor=GenuineIntel,+invtsc,vmware-cpuid-freq=on,$MY_OPTIONS\
	-mem-prealloc \
	-machine pc-q35-2.9 \
	-smp 4,cores=2 \
	-usb -device usb-kbd -device usb-tablet \
	-nographic \
	-vga none \
	-serial none \
	-parallel none \
	-device vfio-pci,host=0b:00.0,multifunction=on,romfile=$HOME/Asus.RXVega64.8192.170815.rom \
	-device vfio-pci,host=0b:00.1 \
	-device vfio-pci,host=0c:00.3,bus=pcie.0 \
	-device isa-applesmc,osk="ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" \
	-drive if=pflash,format=raw,readonly,file=OVMF_CODE.fd \
	-drive if=pflash,format=raw,file=OVMF_VARS.fd \
	-smbios type=2 \
	-device ich9-intel-hda -device hda-duplex \
	-device ide-drive,bus=ide.2,drive=MacHDD \
	-drive id=MacHDD,if=none,file=./mac_hdd.img \
	-device ide-drive,bus=ide.1,drive=Clover \
	-drive id=Clover,if=none,snapshot=on,format=qcow2,file=./'Clover.qcow2' \
	-device ide-drive,bus=ide.0,drive=MacDVD \
	-drive id=MacDVD,if=none,snapshot=on,media=cdrom,file=./'HighSierra-10.13.6.iso' \
	-netdev tap,id=net0,ifname=tap0,script=no,downscript=no -device e1000-82545em,netdev=net0,id=net0,mac=52:54:00:c9:18:27 \
	-monitor stdio

So what might be causing my macOS to freeze?
Thanks!

1 Like
2

That’s one beast of a machine! What are you doing with all these cores? Are you using looking glass? Have you tried passing through an Nvidia GPU?

3

Not sure there are all that many mac users on the forum maybe @FaunCB might have some input, but pretty sure he messes with older mac stuff.

4

disclosure: I’m a mac (plus Linux, FreeBSD, Windows) user, but i’ve not messed with virtualising macOS. I’ve built a hackintosh on hardware known to work, but that was ages ago. All of my current mac use is with real macs.

The line that sticks out to me is the one about applekeystore failing.

Have you tried with an earlier version of macOS? I think high sierra made changes to the UEFI boot security, depending on the age of the image you are installing from maybe there are new hoops to jump through.

edit:
reading some more… doubt that said hoops are GPU specific. Sorry can’t be more help…

Interested to know if you do get it working though, because i’d like a macOS vm :smiley:

5

I mess with everything lol. I’ll look later.

6

this is definitively mac’s drm kicking in;
you’ll need custom/‘hacked’/unlocker kvm/vm.

I had same issue long time ago trying to run mac what i believe version 10 or 11 leopard in vmware on intel. I had to use this to have it working:

7

I don’t know about Linux, but on windows unlocker patches one of vmware’s executables to allow macOS guest. Unless he switches to vmware how can he possibly make use of unlocker with kvm?

8

Its your mobo. I’m unfamiliar with the core count on the 2990wx, but the count plus mobo, plus if your dsdt/ssdt is correct are the large factors to cherk here.

Good luck.

9

Something else i thought of today… I have an Intel board and to run macOS in a VM it’s necessary to set “Execute Disable bit” to disabled in BIOS. On AMD boards the same setting is called NX Mode. Try disabling it.

2 Likes
10

Could be onto something there. I believe when i built my PC hackintosh i had to tweak some of the BIOS options for intel VT-D or VT-whatever, too.

11

I’ve already answered you in reddit, but I think it’s a good idea to post it here too:

The issues is that Vega GPU’s have a PCI Express bridge inside them (you can see this by using lspci -t -v). However, when using vfio, we are passing the GPU (and HDMI audio) directly. This works in Linux and Windows, but the Mac drivers expect the bridge probably, so they don’t work properly. I’m not sure if this explanation is correct, but I’ve managed to get it working by changing:

-device vfio-pci,host=0b:00.0,multifunction=on,romfile=$HOME/Asus.RXVega64.8192.170815.rom \
-device vfio-pci,host=0b:00.1 \

to

-device ioh3420,id=root_port1,chassis=1,slot=2,bus=pcie.0 \
-device x3130-upstream,id=upstream_port1,bus=root_port1     \
-device xio3130-downstream,id=downstream_port1,chassis=11,slot=21,bus=upstream_port1 \
-device vfio-pci,host=0b:00.0,bus=downstream_port1,multifunction=on \
-device vfio-pci,host=0b:00.1,bus=downstream_port1 \
3 Likes
12

Can you post your working command please? I’m having similar issues with a 5700 card