Black screen after making changes to mkinitcpio.conf

Software & Operating Systems VFIO
1

I bought a new system

CPU: AMD Ryzen 9 3900X

Mobo: MSI MEG X570 UNIFY

Desktop VGA (slot 3): Sapphire Pulse AMD RX560 4GB

Gaming VGA (passthroug, slot 1): EVGA Nvidia1080ti

Previously I had an Intel system, so I didn’t have the RX560, I was using the IGP.

So, after a fresh install of Majaro 20, Kernel 5.7.9, i started the process of setting a new KVM.

The steps I followed:

boot kernel parameters: amd_iommu=on iommu=pt vfio-pci.ids=10de:1b06,10de:10ef,14e4:1677 systemd.unified_cgroup_hierarchy=1

reboot

check with: sudo dmesg | grep -i -e DMAR -e IOMMU

sudo nano /etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1b06,10de:10ef,14e4:1677

sudo nano /etc/mkinitcpio.conf

MODULES=(… vfio_pci vfio vfio_iommu_type1 vfio_virqfd …)

HOOKS=(… modconf …)

sudo mkinitcpio -P / sudo mkinitcpio -p linux54

reboot

After the reboot, I am having 3 black screens (1 on DP of 1080, 2 on DVI & HDMI of RX560)

(I had to undo changes to .conf files and run mkinitcpio -P from Live CD to boot)

What am I doing wrong?

System:

Host: amd-manjaro Kernel: 5.7.9-1-MANJARO x86_64 bits: 64
compiler: gcc v: 10.1.0 Console: tty 1 Distro: Manjaro Linux
Machine:
Type: Desktop System: Micro-Star product: MS-7C35 v: 2.0 serial: N/A
Mobo: Micro-Star model: MEG X570 UNIFY (MS-7C35) v: 2.0
serial: Kxxxxxx UEFI: American Megatrends v: A.40 date: 07/11/2020
Battery:
Device-1: hidpp_battery_0 model: Logitech G603 Wireless Gaming Mouse
charge: 100% (should be ignored) status: Discharging
CPU:
Topology: 12-Core model: AMD Ryzen 9 3900X bits: 64 type: MT MCP
arch: Zen L2 cache: 6144 KiB
flags: avx avx2 lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
bogomips: 182455
Speed: 3599 MHz min/max: 2200/3800 MHz Core speeds (MHz): 1: 3599
2: 3598 3: 2102 4: 1983 5: 3596 6: 3589 7: 2057 8: 2058 9: 3777
10: 2091 11: 2199 12: 2203 13: 2056 14: 2200 15: 2804 16: 1974 17: 2132
18: 2114 19: 3715 20: 2119 21: 2200 22: 2199 23: 2199 24: 2814
Graphics:
Device-1: NVIDIA GP102 [GeForce GTX 1080 Ti] vendor: eVga.com.
driver: nouveau v: kernel bus ID: 2d:00.0
Device-2: AMD Baffin [Radeon RX 550 640SP / RX 560/560X]
vendor: Sapphire Limited driver: amdgpu v: kernel bus ID: 2e:00.0
Display: server: X.Org 1.20.8 driver: amdgpu,nouveau FAILED: ati
unloaded: modesetting resolution: 2560x1440~60Hz, 1920x1200~60Hz
OpenGL: renderer: NV132 v: 4.3 Mesa 20.1.3 direct render: Yes
Audio:
Device-1: NVIDIA GP102 HDMI Audio vendor: eVga.com.
driver: snd_hda_intel v: kernel bus ID: 2d:00.1
Device-2: AMD Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]
vendor: Sapphire Limited driver: snd_hda_intel v: kernel
bus ID: 2e:00.1
Device-3: AMD Starship/Matisse HD Audio vendor: Micro-Star MSI
driver: snd_hda_intel v: kernel bus ID: 30:00.4
Device-4: Logitech HD Pro Webcam C920 type: USB
driver: snd-usb-audio,uvcvideo bus ID: 1-2.1:4
Device-5: Creative Sound BlasterX G6 type: USB
driver: hid-generic,snd-usb-audio,usbhid bus ID: 5-3:3
Sound Server: ALSA v: k5.7.9-1-MANJARO
Network:
Device-1: Broadcom and subsidiaries NetXtreme BCM5751 Gigabit Ethernet
PCI Express
driver: tg3 v: kernel port: N/A bus ID: 25:00.0
IF: enp37s0 state: up speed: 1000 Mbps duplex: full
mac: 00:10:18:1b:59:6c
Device-2: Realtek RTL8125 2.5GbE vendor: Micro-Star MSI driver: r8169
v: kernel port: f000 bus ID: 27:00.0
IF: enp39s0 state: down mac: 2c:f0:5d:3b:8b:df
Drives:
Local Storage: total: 11.14 TiB used: 1.17 TiB (10.5%)
ID-1: /dev/nvme0n1 vendor: Samsung model: SSD 970 EVO 500GB
size: 465.76 GiB
ID-2: /dev/nvme1n1 vendor: Samsung model: SSD 970 EVO 250GB
size: 232.89 GiB
ID-3: /dev/sda vendor: Samsung model: SSD 840 EVO 250GB
size: 232.89 GiB
ID-4: /dev/sdb vendor: Samsung model: SSD 850 EVO 500GB
size: 465.76 GiB
ID-5: /dev/sdc vendor: Toshiba model: HDWE140 size: 3.64 TiB
ID-6: /dev/sdd vendor: Toshiba model: HDWE140 size: 3.64 TiB
ID-7: /dev/sde type: USB vendor: Western Digital
model: WD30EZRX-00DC0B0 size: 2.73 TiB
RAID:
Hardware-1: Advanced Micro Devices [AMD] Device driver: ahci v: 3.0
bus ID: 2b:00.0
Hardware-2: Advanced Micro Devices [AMD] Device driver: ahci v: 3.0
bus ID: 2c:00.0
Partition:
ID-1: / size: 123.50 GiB used: 11.00 GiB (8.9%) fs: ext4
dev: /dev/nvme1n1p3
ID-2: /home size: 72.24 GiB used: 1.07 GiB (1.5%) fs: ext4
dev: /dev/nvme1n1p4
ID-3: swap-1 size: 32.00 GiB used: 121.5 MiB (0.4%) fs: swap
dev: /dev/nvme1n1p2
Sensors:
System Temperatures: cpu: 40.2 C mobo: N/A
Fan Speeds (RPM): N/A
GPU: device: amdgpu temp: 45 C fan: 1005 device: nouveau temp: 27 C
fan: 0
Info:
Processes: 490 Uptime: 2h 49m Memory: 7.76 GiB used: 5.58 GiB (71.9%)
Init: systemd Compilers: gcc: 10.1.0 Shell: bash v: 5.0.18 inxi: 3.0.37

2

Hello and congratulations on your new system!

To what device does ID “14e4:1677” belong to?

Can you post the output of “lspci -k”?

From archwiki: PCI_passthrough_via_OVMF#Isolating_the_GPU

Two methods exist for providing the device IDs. Specifying them via kernel parameters
Alternatively, the IDs may be added to a modprobe conf file.

You are on the right path - You only need to use one of the methods. I’d suggest going the kernel parameter route. Try adding

vfio_pci vfio vfio_iommu_type1 vfio_virqfd vfio-pci.ids=10de:1b06,10de:10ef,14e4:1677

to your boot parameters. Regenerate your boot loader (I’m not sure what manjaro uses, GRUB’s been a stable for some time, but I’ve seen systemd boots around too) and reboot.
Run “dmesg | grep -i vfio” to check if vfio-pci was bound to your device. Also “lspci -k” should show the drivers in use too.

If the GPU fails to be bound - I’d suggest to change the way the cards are situated in your pci-e slots. (Nvidia to slot 3, AMD to slot 1) Iirc binding the boot gpu (pci-e slot 1) is a big PITA.

Good luck!

3

Thank you for your reply.

“14e4:1667” is a PCI Gigabit Ethernet controller, I am using. Some games don’t behave well with NAT, like CoD.

I do have the kernel parameters, but I thought this is not enough…
Are you saying that steps

sudo nano /etc/modprobe.d/vfio.conf

sudo nano /etc/mkinitcpio.conf

are not required?

Also, my system is water cooled and moving the 1080ti to another slot, requires too much work…

4

“14e4:1667” is a PCI Gigabit Ethernet controller, I am using.

Okay, I was just curious. I’ve never had a problem with the internet inside a vm. (granted I’ve only played rust and pubg)

You should be able to do the passthrough via kernel parameters. Just remember to regenerate the boot loader.
Archwiki Kernel_parameters#GRUB
On ubuntu, I’ve always edited /etc/default/grub and ran update-grub.

I’m not 100% sure about not using the kernel modules, I can’t recall exactly how I got mine working and cannot check atm. But I think I only messed around with kernel parameters… Then again this post that I have bookmarked guides to use kernel modules

play-games-in-windows-on-linux-pci-passthrough-quick-guide

Also, my system is water cooled and moving the 1080ti to another slot, requires too much work…

Big ouch - Hopefully you get it working w/o messing with the watercooling. I hope somebody with more experience can guide how to bind the pci-e slot 1 card to vfio-pci. (iirc you need to mess around in bios to somehow switch the boot gpu into something else… as my understanding is that pci-e slot 1 is initialized on boot as default and bound to the underlying OS)

Found this with a quick google search:
redhat-vfio-users

5

So, since my motherboard was defective, I sent if for RMA, and picked a GB X570 Aorus Master, that has an option in the BIOS to assign initial/primary VGA.

However, it does not work…

If I enter BIOS, the display output goes to the monitor connected to Radeon (set as primary vga, slot PCIe2). But as soon as it goes for the GRUB, the screen goes blank and I get nothing there. Instead, I have to monitor when disk stops working, and I login blindly, and after that, my monitor shows the desktop.

I read plenty of threads in here and other forums, but most of the replies were “change the setting to BIOS” and worked. I did, but doesn’t seem to work.

Any suggestions?

6

I’m a bit out of my depth here but from my readings adding “video=efifb:off” to the kernel parameters should be the way.

archwiki recommends switching the the gpu’s around sadly.

level1techs-howto-headless-vfio-host also suggest using “video=efifb:off”

Can you check is the nvidia card bound to vfio-pci after boot? with “lspci -k”

7

I checked with “lspci -k” and it was bount.

As for the “video=efifb:off”, I tired it before. I am going to try it again, with the new mobo.

8

If the GPU is bound - You should be able to fire a VM up and give it the GPU… But the blind logging in sucks - Hopefully the kernel parameter fixes that.

Remember when setting up the passthrough - The GPU needs to be hooked up to a monitor. (It can be the same monitor GPU2 is hooked up - just different input)

9

I added in the kernel parameters:
video=vesafb:off,efifb:off
It didn’t work and I got an error while initializing desktop display.

However, when I changed a BIOS setting:
CSM > Enabled
then worked!
Not perfect, but it did!
Now I am able to view the login screen and the VGA on PCIe1 slot is ignored.

Thank you for your input!

1 Like