GPU Passthrough: Advice on my boot failure issue / Mobo+GPU combo?

My goal is to install two graphics cards, and eventually do GPU passthrough with one of them.

My cards are:

  • NVidia GeForce GTX 1660 Ti
  • Radeon RX 580

My system:

  • Mobo: ASRock X570 Phantom Gaming 4
  • CPU: AMD Ryzen 7 2700X
  • 32G ECC memory
  • Currently running Debian Linux

However, whenever there is any graphics card in the bottom x16 PCIe slot (PCIE3), the machine does not boot.

Note: it does get to the BIOS setup screen. However, after that the screen goes blank, and makes no progress. I do not see the GRUB boot loader screen, for instance.

In this hung state, there is a video signal (my monitor would show a “no signal” message otherwise), but it is blank. Also, the motherboard light is off, so it has passed all of those health checks. Lastly, in this hung state, the network light for the LAN is off (in case that is helpful).

Here are things I’ve tried:

  1. Nvidia card in the PCIE1 slot, alone - boots OK
  2. Radeon card in the PCIE1 slot, alone - boots OK
  3. Nvidia card in the PCIE3 slot, alone - no boot
  4. Radeon card in the PCIE3 slot, alone - no boot
  5. Nvidia card in the PCIE1 slot, Radeon in the PCIE3 slot - no boot (this is what I eventually want to work).

I have updated to the newest available BIOS version (P1.30). No change. I have reset CMOS. No change.

The manual indicates that PCIE3 should support a graphics card in x4 mode. But even with a single card in this slot, I cannot make it work.

Anyone have advice on this?

I contacted ASRock support, and gave them the above information, and I got a 1-line reply back: “Motherboard support no more than 2 video cards”.

So, either they misunderstood me, or they meant to write “no more than 1 video card”.

But from the documentation I find, and even the printed manual, the second x16 PCIe slot should support a graphics card. And even with a single card in the system, in that second slot, it does not boot.

So, the tech support does not seem… very helpful (:

Anyone here have ideas?

Is it possible to set the second PCIe 16x to PCIe 3.0? I know neither card is 4.0 and it should be automatically downgrading to what the card supports, but… Wendell did suggest there was some instability with 3.0 and 4.0 mix. Or is it just implied that the chipset will downgrade if the CPU only supports 3.0?

Well you’re using Ryzen 2000 on a X570 like I plan to do, hopefully you can get it sorted.

Thanks for the reply. I believe it’s all operating in 3.0 mode, since my CPU doesn’t support more. In the BIOS all that stuff is set to “auto”.

However, I did resolve this issue!
I had to disable CSM, to force pure-UEFI booting. On my BIOS, that setting is named Boot\CSM Compatibility (set to Disabled).

Of course, once I did that, my OS failed to boot, since it was installed under CSM, and so was not visible to the UEFI boot code.

Reinstalling my OS with UEFI (creating a EFI System Partition for the boot code) fixed that problem. I probably could have salvaged the old install by doing some partition surgery, but it wasn’t worth it for me, so I just did a new install (saved my /home beforehand).

So now I have both vid cards installed, and they are both visible to the OS.

I haven’t gotten the VMs set up yet, but making progress…

1 Like

Good deal. Honestly had to look CSM up as I don’t believe I have that option. I want to say that is similar or the same as UEFI + Legacy mode? Well at least you got it to boot. I’ll read up about it.

same as UEFI + Legacy mode

Yes, I believe this is a synonym. Different BIOSes use different terminology.

CSM/Legacy is the thing that makes UEFI look like old-style plain BIOS booting.
In truth, calling the thing whose settings I tweaked the “BIOS” is not accurate (even though that’s what the screen says), since UEFI and BIOS are really entirely separate technologies. UEFI can “fake” looking like a BIOS though, with Legacy mode/CSM.

The terminology is a bit of a mess (:

Fun with names…

So I guess you can say that option rom loading is borked or not useable by design on a X570 chipset bus? At least with the old bios way of doing things.

Looks like Intel wants to kill CSM by next year, so I guess that’s a thing that might cause a slight stir. At this point I think most/all distros have support for efi so it’s just a matter of people making the change and breaking old habits.

I didn’t think I’d be using this bit of wisdom so soon, but I was forced to retire my X470 board and there happened to be a X570 laying around. Totally forgot about this post until the system wouldn’t boot with 2 x GPU… then it finally clicked. Anyways, goodbye CSM.

Your suffering wasn’t in vain :stuck_out_tongue:

Edit So, it might be related, it might not, but after getting booted up with both GPUs installed I fixed my VFIO configuration, rebooted, and launched my VM I already had working on my X470. Got the following error in my libvirt log-

qemu-system-x86_64: vfio: Unable to power on device, stuck in D3

I rebooted and try to go into the bios and realized I was back to square one without video. I can only figure CSM was somehow triggered because power cycling didn’t help and clearing the cmos would just leave me with CSM on again. So I removed the 2nd GPU again. Booted into the bios, and applied the latest BIOS for my board as I had already noticed my version was a little sketchy (it listed every used SATA port twice when autodetecting drives for one). And now… my VM still doesn’t work… :frowning: Same error.

So I’m tinkering but I’m guessing I’ll either need a newer kernel or wait for another bios update. I know one thing… I really wish UEFI was the default for the bios now. Anytime it gets reset, like everytime you update the bios, you’ll have to remove the 2nd GPU…

What QEMU and OS/kernel versions are you using?

I have it on my list to try newer ones, but it’s a stab in the dark.

Well, at long last, I got this working!
On an X570, no less.

It required 2 kernel patches, but seems pretty solid so far.

I put some info here, if you’re interested: https://old.reddit.com/r/VFIO/comments/cvvxvj/success_on_x570_ryzen_2000/