It was an issue introduced ca 2021 and most certainly fixed later. Manjaros default precompiled kernel had an issue where KVM VMs with more than 8GB RAM had 2 minutes and more boot time added where the qemu process simply stalled while memory was assigned. I found I/o latency was quite bad. After each kernel update I had to add the kernel fix again and had to make sure the Nvidia driver works. This was just one of the various things I had to work around. Most of the issues were gone when I switched to Proxmox.
I can’t complain about Proxmox performance. I currently play lots of games which require high performance of the GPU (Raytracing, latest game releases) I also use it for VR.
I mean energy efficiency surely not comparable to Apple Silicon. But yours is an EPYC. Running MacOS or not you’re burning constant energy mostly anyway even at idle.
Just curious to hear your experience about “poor efficiency” of running a MacOS VM in addition to everything else running on your EPYC box.
With 12 dedicated cores, the Vm would ‘idle’ in linux at 20% cpu over all cores, GPU performance of the sapphire pro card was meh (as in sometimes stuttering during heavy refreshes (I use multiple virtual desktops), and the GPU would draw constantly 25-30W without any way of powering down to a lesser state. There was no way to sleep or hybernate the VM
The M1 mac I replaced it with draws less than 10w at idle, and sleeps and hibernates flawlessly.
As for the epyc processor still drawing power, that is true, and I repurposed it to run my tree truenas NAS and my forbidden router, plus all the windows/linux Vms I used for work, so I am still using it with vfio, but only for one VM/GPU combo. This allowed me to power down my physical truenas box (55w at idle) that now hosts a proxmox backup server that I power on only four hours a day while my photovoltaic production is at max, and to which I back up all my proxmox VMs, plus some ZFS datasets from truenas, and the root of the proxmox server itself …
“25-30W” is a bogus value because the driver has this bug which can’t properly report power consumption for at least the Baffin family of GPUs. Your Sapphire Pro seems a Baffin based equivalent of Radeon RX560.
Depends on your graphics workloads, hard to judge on the stuttering. Virtual desktop doesn’t hurt as long as you only hooks up one physical display.
That’s true, at least a common symptom for all Baffin based GPUs. Seems related to the driver again. I wonder if newer GPUs have better luck. I once asked a RX6600XT owner here but I forgot what he told me back then…LOL
On the bright side, “Re-open all application upon startup” works pretty well in MacOS. So one can shutdown the VM instead. The experience is pretty close to resume from sleep. Though I’ll admit if you’re in the middle of a set of long tasks, I would really appreciate the exact state that’s possible with “resume from sleep.”
As idle as a VM can be with 20+ apps open and not used, three of which are Chrome, Firefox and Safari, two of which are Teams and Outlook, one of which is VSCode?
but still, the load on the VM does not match the underlying kvm load by at least an order of magnitude …
‘as measured at the wall’ with and without the GPU, and even when the GPU was not yet passed to the OSX VM …
No virtual desktops in use, straight displayport out or, in my case, displayport to hdmi to hdbaset to hdmi input, but still the stutter is in the window drawing, not in the signal coming to the monitor (4k@60Hz4:4:4)
Sleep with a hackintosh is not an option whatsoever, even less with an AMD GPU
Call me a dinosaur, I never ever want to shut down and re-open my desktop, run times of my client machines is almost always equal to my server machines … e.g months at a time, usuallly down only because of updates or me going on vacation. That is one of th ereasons why Windows has never worked for me. Linux may work as a daily driver but to this day I still need to find a Linux distro for which 4k output doean’t suck, hard, and that can handle remote desktop sessions that are at least bearable (and no, xvnc is not bearable for my use case) and that doesn’t suck at font rendering …
If it was ‘top’ from the Linux host, the arithmetics should be 20% ( out of 1200% in your case). So it has to go through a normalisation process. After that you got 1.7% CPU utilisation for your MacOS VM. Not bad at all…
What a coincidence! My “rx560” with 4GB idle at between 25-30W as reported by MacOS Baffin driver. I had to patch the open-source monitor software to have the correct offset. Once done, wattage reporting goes very well and accurate.
I dug out my previous conversation with the dude [0]. Seems rx6600/xt still no luck. There could be quite a few of potential issues in wake up properly. In my case, I’m pretty sure GPU is the last hurdle I need to cross. But probably not worth the time to investigate further as we all know MacOS on AMD64 will be over around year 2028.
I’m running my “rx560” at 4K@60Hz RGB444 with “look a like 1440p” scaled resolution. Perfectly fine. A physical display really makes a VM pop and real…
Windows 10 VM hibernates and wakes up well for me with the same “rx560” gpu. Hibernation is a must-have for Windows. Because “reopen applications on startup” doesn’t exist for Windows at all…
I’m not sure what will be my Daily Desktop after 2028. Going Linux is one possibility. Buying a Mac mini with 7 angstrom is another possibility. Good that I don’t have to decide for now.
Managed to get it working on an Asrock x370 Taichi, 1600x and R9 390 (host) and an GTX 1070 (guest) it wasn’t the smoothest experience but it worked until it didn’t and in my glee I neglected to take good notes between the start of it working and when it stopped.
I’ll need to dig out the book on that one, was really just a “Can I do it” project that I didn’t get to use a ton since the 390 has artifacts that the warranty service just sent back to me the same way twice. Looking forward to getting back into it more now that time has passed and I have some better ideas on how to take advantage of it.
I have been pursuing PCI passthrough since I first saw your video on it where I believe you used a R9 390X in the guest to run GTA5. I First attempted on the FX platform which I did get to work but with poor performance. It wasn’t until I built a Ryzen system with a 1700X where I had great success and continued to use this for the last few years.
As of last year I stopped using it as proton performance had gotten good enough to the point that I realized I was not spinning up the VM anymore. I did attempt passthrough with a 5600xt on my proxmox server which was surprisingly more headache than I thought. I sort of lost interest in it and has been just sitting idle on my server wasting current the last few months due to lack of interest. As I saw this thread I am considering to stick the 5600xt into my main rig and reconfigure the VM. Besides gaming, I would like to have the VM available for any other use case that comes up.
The biggest headache I can recall was audio. I had given up on it and just bought a sound card and passed that through to the VM. I had built a passive crossover and had both my host and guest connected to my 5.1 system at the same time. To be honest, I have not been keeping up with the latest in VFIO breakthrough in the last few years so I am not aware of alternative solutions that may have cropped up since so I am eager to see where this goes.
On the QEMU front it’s still not great, I did write a Jack audio device for QEMU which resolved most of the problems but it’s still a pain to setup.
Otherwise if you are using Looking Glass, audio is a breese as we have fully implemented audio via the SPICE transport, along with a PID loop to maintain audio clock sync with the guest. No guest software needs to be installed either, works with the emulated intel hardware flawlessly.
Would love to win this system, Keep being Epic L1
Currently have struggled getting AMD Link working on my VM with RX460. Took a lot of tinkering to get it passed to the VM but now I want to setup the remote gaming streaming so the kids can access it with their phones. One step at a time its coming along.
Was there any issues with the Radeon cards and their USB ports? In my experience those tend to be ASMedia stuff which is super stuttery when it comes to simultanious mouse input and audio or has issues with reset when passed through.
i am running a 7800x3d on the X670E Arous Master with a 4080.
i tried a Sapphire 7900XTX first which was the card of my dreams but sadly had reset issues so i had to downgrade to team green (card is louder and imo in general worse in every aspect).
but i can reattach and detach it without any problems.
i use the iGPU for the host and the dGPU for VFIO or native gaming depending on my use case.
Sadly nobara has a acs patch baked in (with no opt out).
The setup is running for nearly a year now without any problems on the lookingglass / vfio side.
(7800X3D iGPU still has lots of system freezes and hangs on linux tho).
As soon as the 8900XTX (or what ever is next) will fix the reset /reattach issues ill happily switch back to my beloved team red <3
if i can help in any way i am happy to offer what i can
PS: My VFIO also is used for Dolby Atmos gaming in my living room since linux apparently still has problems with good audio 3d audio hdr and such stuff ^^
PPS: my first VFIO /looikingglas setup was a X370 Hero VI with a WX3200 for the Host and a 6800XT for the VM. But my 6800XT also had a reset bug
PPPS: My only VFIO problem is that the games i set it up for(because no proton support) started to also block VFIO and only support windows native -.- causing me not to be able to play them)
So does this mean I need to drag all three of my C612 rigs out? I have been using VFIO for a while now and it works really well. I am using older stuff but I think the basics are still the same regardless. I am using one for a living room gaming VM and I was running Parsec clients for a while until my cluster bellied up. I have some hardware I could deff share my journey with.
Nvidia 1030 (msi passive heatsink) - passthrough for Manjaro VM
My story begins roughly around November 2018 when I started to build my PC for my Homelab Proxmox Cluster with the goal of having my gaming machine in a rack hidden away anywhere. It took me around 3-4 days until I had a working running setup with 2 VMs (Windows, Manajro) and both of the GPUs as PCI-Passthrough. During that time it had some hiccups booting up but was working ~85% of time without any problems.
After a kernel update, do not know which one exactly, around 6-7 months ago I was surprised that the system got even more stable with only one bug left when booting up. I use an 4PC - USB3 hub for switching input devices between the two VMs and my Work Laptop (Home Office) and everytime when the hub is connected to the Manjaro VM the kernel is freaking out with problems of the GPU (1030) I use as passthrough on the VM. There is also the Problem with the 1030 and initial setup on Manjaro but here I am not sure if the problem is more the 1030 or the broken X-Server configuration, because when I try to boot after installing the Manjaro OS I only get a black screen but when I delete the X-Configuration which btw. worked during the installation path, everything is going great again.
My biggest problems with my current setup are Monitors and Audio.
Currently I run HDMI/DP Cables from both of the GPUs to my monitors and switch manual between them (input select). It is not ideal but because I am using split monitors (monitor 1 Work laptop, monitor 2 Manjaro - watching Youtube videos psst) it is the only way possible.
For Audio I am using 3 Soundblaster XG1 to a “Little Bear MC5 Mini 4 channel mixer” going into a Soundbar/Steelseries Arctis nova pro wireless.
This needs to be replaced with proper Speakers and Headphones but it is working for now.
What I wish for: (And where you awesome community may help me with)
A “companion” app or something like that which runs on all 3 machines (windows, manjaro, laptop) which can switch the inputs of the Monitors. Would be great if it had some sort of API where I can use an ESP with some buttons to switch
A “companion” app which creates a virtual audio interface and sends the audio to a third device where I can mix up to 4 input channels.
For both of my wishes I also thought of a DIY device with knobs (volume), buttons (monitor switch) and a DAC. Don’t know if something like that does already exist.
Also I do discourage anyone to use the 1030 as passthrough if you already have a nvidia card in your system. I still think that if you want to use one card (in my case nvidia) as full passthrough device then the second one should be AMD and vice versa.
PS: Forgot to mention but even Steam Index USB passthrough is working on both Windows and Linux without any problems.
I ran the full looking glass VFIO stack for a couple years, but I just kept running into updates breaking things; by the end, I even had my own kernel with reverted commits.
My setup considered of two Nvidia cards, a 2060 super (guest) and a 1050ti (host). The 1050ti isn’t supported on the newer open source drivers and that didn’t make things any easier. I ran Wayland on my host and that came with another sleuth of issues. Blacklisting Nvidia drivers wasn’t an option because I needed it for my host GPU, so I bound VFIO-pci to the guest gpu through kernel parameters. I ran windows as my guest is and had the best performance using a dedicated nvme ssd as it’s boot drive. It also allowed me to boot directly to the guest OS without any vms (even though it wouldn’t actually be a guest), which I did on a few occasions. I pinned CPU cores, and had a qemu hook to disable host process scheduling on those cores when the vm booted. Tuning gave me significant performance gains, and I found that I could run WSL with paravirtualized gpu acceleration in my windows VM with little overhead to mess around with AI models.
Things fell apart when there were new kernel version that were incompatible with Nvidia drivers (arch btw) and then after a kernel update (around 6.1?) the VFIO-pci binding would mess with the Nvidia framebuffer, and I had to revert some commits on mainline, but then further changes made it difficult to rebase and update. Oh and then my host ssd died.
Come to think of it… another nice use for VFIO I would like to do. Just it had not ocurred to me before now… is when people come over occasionally. And we want to do the LAN PC gaming together. But it’s actually quite nice not to then need a 2nd dedicated PC required in order to actually do that. But instead rather use that same (pre-existing) VFIO passthru guest VM. And then be streaming the guest machine through to the TV instead. To act as the 2nd physical machine. Over local lan via moonlight / sunshine / similar local network streaming. Would be quite a lot more convenient for those occasions. Heck, maybe even could also take my old 5700xt and then put into the home server. For a 3rd machine… but maybe VFIO isn’t so well supported on the earlier navi1 generation i suppose? Not sure.
I did this with dummy plugs in my proxmox server. I used Parsec for folks to remote in and play. This actually works well. I have a pair of 1070’s I did this with. I’ll post everything I have going on when I get home with pictures included. Mind you I am also using two 2699v3’s and two 2690v3’s in two servers. I have a third I need to setup. More on that later.
(card is louder and imo in general worse in every aspect).
