VFIO setup on MSI GE66 Raider laptop

Hello all, first time posting noob here. Long time viewer of the Leve1Tech YT channels.

So recently I’ve been watching more videos and reading up on VFIO stuff. A friend of mine did a couples computer set up with Unraid and got me wanting to give it a try on my laptop. Though not with Unraid. Just running Windows within Arch. I just recently got my first high end computer in the last five years. The full model of the laptop is MSI GE66 Raider 10SFS-048US. The specs are

  • Intel Core i7 10750H

  • 32GB DDR4 RAM

  • NVIDIA RTX 2070 Super

  • WD NVME 500GB SSD (Arch Linux)

  • WD NVME 1TB SSD (Windows 10 x64 Home)

I’ve seen some posts online saying how it is possible to do VFIO on laptops and have the dGPUs work fine along with the iGPU. I found a post here on the forum that was awesome, though it was made for Fedora 29 and 30 I think. It hadn’t been updated for a while. It was some scripts that automated the setup of VIFO on a laptop. I think it was set up for a laptop. Haven’t been able to find anything for Arch. Did some searching and can’t find anything specific for my laptop. I’m not quite sure where to start. I would love to have Arch as my main OS and run Windows within Arch when I want to run games and other windows applications. I’m not sure what other info I should include in the post. I have been using Linux for a while (mostly Arch), though I am still a major major newbie.

First thing you need to do is to enable IOMMU in bios if there’s an option, and check iommu grouping.
This script will tell you if there’s chance:

You can attach output here, for assessment :slight_smile:

Edit: IOMMU is called VT-d on intel :wink:

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 02)
00:02.0 VGA compatible controller [0300]: Intel Corporation UHD Graphics [8086:9bc4] (rev 05)
00:04.0 Signal processing controller [1180]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem [8086:1903] (rev 02)
00:12.0 Signal processing controller [1180]: Intel Corporation Comet Lake PCH Thermal Controller [8086:06f9]
00:14.0 USB controller [0c03]: Intel Corporation Comet Lake USB 3.1 xHCI Host Controller [8086:06ed]
00:14.2 RAM memory [0500]: Intel Corporation Comet Lake PCH Shared SRAM [8086:06ef]
00:14.3 Network controller [0280]: Intel Corporation Wireless-AC 9462/9560 [8086:06f0]
00:15.0 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH Serial IO I2C Controller #0 [8086:06e8]
00:15.2 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH Serial IO I2C Controller #2 [8086:06ea]
00:16.0 Communication controller [0780]: Intel Corporation Comet Lake HECI Controller [8086:06e0]
00:1b.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #17 [8086:06c0] (rev f0)
00:1b.4 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #21 [8086:06ac] (rev f0)
00:1d.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #9 [8086:06b0] (rev f0)
00:1d.6 PCI bridge [0604]: Intel Corporation Device [8086:06b6] (rev f0)
00:1d.7 PCI bridge [0604]: Intel Corporation Device [8086:06b7] (rev f0)
00:1f.0 ISA bridge [0601]: Intel Corporation Comet Lake LPC Controller [8086:068d]
00:1f.3 Audio device [0403]: Intel Corporation Comet Lake PCH cAVS [8086:06c8]
00:1f.4 SMBus [0c05]: Intel Corporation Comet Lake PCH SMBus Controller [8086:06a3]
00:1f.5 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH SPI Controller [8086:06a4]
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104M [GeForce RTX 2070 SUPER Mobile / Max-Q] [10de:1e91] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)
01:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)
02:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller [1b21:3242]
03:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:5006]
04:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:5006]
05:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:3100] (rev 01)
06:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS525A PCI Express Card Reader [10ec:525a] (rev 01)

This is the output that the script gave me. I do have VT-d and Virtualization Technology options enabled in the bios.

Iommu set to enable? Iommu=on at boot time as a kernel boot param?

Yeah I forgot that, but its gonna be rather:

intel_iommu=on iommu=pt

Anyway, WoofahRayet you should go through guide at:

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

There’s not really much point to repeating here what is already well written there.

Edit: Right, just realized that you may be confused as new Linux user. What Wendell and I are trying to say with short-hands obvious to us is, that your IOMMU groups don’t work properly :slight_smile:

Gotcha. So since the IOMMU groups aren’t working right, I won’t be able to get it working at all?

Not yet. It means you have do extra steps we mentioned (and I forgot initially to tell you :/. But it will be best if you just follow guide I linked. There are proper IOMMU groups shown in it, that you should get form script.

Once you get them then post it here :slight_smile:

Screenshot_20201202_184717_virt_manager

When following the instructions to set up the VM in virt-manager, I get this error when I click begin installation.

Here is what it says in the details part -

Unable to complete install: 'Requested operation is not valid: network 'default' is not active'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/createvm.py", line 2001, in _do_async_install
    installer.start_install(guest, meter=meter)
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 691, in start_install
    domain = self._create_guest(
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 639, in _create_guest
    domain = self.conn.createXML(install_xml or final_xml, 0)
  File "/usr/lib/python3.8/site-packages/libvirt.py", line 4035, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirt.libvirtError: Requested operation is not valid: network 'default' is not active

I went to the link to activate the default libvirt network.

Running virsh net-define /usr/share/libvirt/networks/default.xml
yields this result -
error: Failed to open file '/usr/share/libvirt/networks/default.xml': No such file or directory

Not sure what I can do at the moment :frowning_face:

Edit - Forgot to mention I added intel_iommu=on iommu=pt to grub.

What about:

virsh net-create default
virsh net-start

I had this problem around year ago, so cant remember exactly what I did, but something in those lines :wink:

What about your IOMMU groups? How do they look now?

I ran dmesg | grep -i -e DMAR -e IOMMU after restarting and got -

[    0.010773] ACPI: DMAR 0x0000000097801000 0000A8 (v01 INTEL  EDK2     00000002      01000013)
[    0.065699] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=6c9f4ed1-945e-479f-a705-ce7db00758df rw intel_iommu=on iommu=pt loglevel=3
[    0.065740] DMAR: IOMMU enabled
[    0.154891] DMAR: Host address width 39
[    0.154892] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.154897] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.154898] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.154901] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.154902] DMAR: RMRR base: 0x0000009853e000 end: 0x00000098787fff
[    0.154903] DMAR: RMRR base: 0x0000009b000000 end: 0x0000009f7fffff
[    0.154905] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.154906] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.154906] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.158037] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.418175] iommu: Default domain type: Passthrough (set via kernel command line)
[    0.810253] DMAR: No ATSR found
[    0.810317] DMAR: dmar0: Using Queued invalidation
[    0.810320] DMAR: dmar1: Using Queued invalidation
[    0.810387] pci 0000:00:00.0: Adding to iommu group 0
[    0.810402] pci 0000:00:01.0: Adding to iommu group 1
[    0.810412] pci 0000:00:02.0: Adding to iommu group 2
[    0.810421] pci 0000:00:04.0: Adding to iommu group 3
[    0.810435] pci 0000:00:12.0: Adding to iommu group 4
[    0.810452] pci 0000:00:14.0: Adding to iommu group 5
[    0.810462] pci 0000:00:14.2: Adding to iommu group 5
[    0.810471] pci 0000:00:14.3: Adding to iommu group 6
[    0.810489] pci 0000:00:15.0: Adding to iommu group 7
[    0.810498] pci 0000:00:15.2: Adding to iommu group 7
[    0.810511] pci 0000:00:16.0: Adding to iommu group 8
[    0.810529] pci 0000:00:1b.0: Adding to iommu group 9
[    0.810548] pci 0000:00:1b.4: Adding to iommu group 10
[    0.810559] pci 0000:00:1d.0: Adding to iommu group 11
[    0.810580] pci 0000:00:1d.6: Adding to iommu group 12
[    0.810597] pci 0000:00:1d.7: Adding to iommu group 13
[    0.810624] pci 0000:00:1f.0: Adding to iommu group 14
[    0.810633] pci 0000:00:1f.3: Adding to iommu group 14
[    0.810644] pci 0000:00:1f.4: Adding to iommu group 14
[    0.810653] pci 0000:00:1f.5: Adding to iommu group 14
[    0.810658] pci 0000:01:00.0: Adding to iommu group 1
[    0.810663] pci 0000:01:00.1: Adding to iommu group 1
[    0.810666] pci 0000:01:00.2: Adding to iommu group 1
[    0.810670] pci 0000:01:00.3: Adding to iommu group 1
[    0.810686] pci 0000:02:00.0: Adding to iommu group 15
[    0.810704] pci 0000:03:00.0: Adding to iommu group 16
[    0.810717] pci 0000:04:00.0: Adding to iommu group 17
[    0.810732] pci 0000:05:00.0: Adding to iommu group 18
[    0.810744] pci 0000:06:00.0: Adding to iommu group 19
[    0.810783] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    0.842656] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <[email protected]>
[    0.842657] AMD-Vi: AMD IOMMUv2 functionality not available on this system
[    0.917089]     intel_iommu=on

Running this script

shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

yielded -

IOMMU Group 1:
        00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 02)
        01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU104M [GeForce RTX 2070 SUPER Mobile / Max-Q] [10de:1e91] (rev a1)
        01:00.1 Audio device [0403]: NVIDIA Corporation TU104 HD Audio Controller [10de:10f8] (rev a1)
        01:00.2 USB controller [0c03]: NVIDIA Corporation TU104 USB 3.1 Host Controller [10de:1ad8] (rev a1)
        01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller [10de:1ad9] (rev a1)
IOMMU Group 10:
        00:1b.4 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #21 [8086:06ac] (rev f0)
IOMMU Group 11:
        00:1d.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #9 [8086:06b0] (rev f0)
IOMMU Group 12:
        00:1d.6 PCI bridge [0604]: Intel Corporation Device [8086:06b6] (rev f0)
IOMMU Group 13:
        00:1d.7 PCI bridge [0604]: Intel Corporation Device [8086:06b7] (rev f0)
IOMMU Group 14:
        00:1f.0 ISA bridge [0601]: Intel Corporation Comet Lake LPC Controller [8086:068d]
        00:1f.3 Audio device [0403]: Intel Corporation Comet Lake PCH cAVS [8086:06c8]
        00:1f.4 SMBus [0c05]: Intel Corporation Comet Lake PCH SMBus Controller [8086:06a3]
        00:1f.5 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH SPI Controller [8086:06a4]
IOMMU Group 15:
        02:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM3242 USB 3.2 Host Controller [1b21:3242]
IOMMU Group 16:
        03:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:5006]
IOMMU Group 17:
        04:00.0 Non-Volatile memory controller [0108]: Sandisk Corp Device [15b7:5006]
IOMMU Group 18:
        05:00.0 Ethernet controller [0200]: Intel Corporation Device [8086:3100] (rev 01)
IOMMU Group 19:
        06:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS525A PCI Express Card Reader [10ec:525a] (rev 01)
IOMMU Group 2:
        00:02.0 VGA compatible controller [0300]: Intel Corporation UHD Graphics [8086:9bc4] (rev 05)
IOMMU Group 3:
        00:04.0 Signal processing controller [1180]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem [8086:1903] (rev 02)
IOMMU Group 4:
        00:12.0 Signal processing controller [1180]: Intel Corporation Comet Lake PCH Thermal Controller [8086:06f9]
IOMMU Group 5:
        00:14.0 USB controller [0c03]: Intel Corporation Comet Lake USB 3.1 xHCI Host Controller [8086:06ed]
        00:14.2 RAM memory [0500]: Intel Corporation Comet Lake PCH Shared SRAM [8086:06ef]
IOMMU Group 6:
        00:14.3 Network controller [0280]: Intel Corporation Wireless-AC 9462/9560 [8086:06f0]
IOMMU Group 7:
        00:15.0 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH Serial IO I2C Controller #0 [8086:06e8]
        00:15.2 Serial bus controller [0c80]: Intel Corporation Comet Lake PCH Serial IO I2C Controller #2 [8086:06ea]
IOMMU Group 8:
        00:16.0 Communication controller [0780]: Intel Corporation Comet Lake HECI Controller [8086:06e0]
IOMMU Group 9:
        00:1b.0 PCI bridge [0604]: Intel Corporation Comet Lake PCI Express Root Port #17 [8086:06c0] (rev f0)
virsh net-create default
virsh net-start

gave these errors -

[email protected]: ~ $ virsh net-create default
error: Failed to open file 'default': No such file or directory

[email protected]: ~ $ virsh net-start
error: command 'net-start' requires <network> option

Can’t seem to find what to do to fix the network thing in virt-manager.

Maybe you missing some packages, try:

sudo pacman -S ebtables iptables bridge-utils

But looking at your IOMMU groups, theres problem. Your GTX is bundled with other stuff. That’s not good. but maybe someone will suggest workaround for this.

Oh, and it supposed to be virsh net-start default. I ate one word :wink:

Doing virsh net-start default gave the errors -

error: failed to get network default
error: Network not found: no network with matching name default

Yeah bummer it’s grouped weird. Hopefully something can be figured out. Gonna do more googling to see if something might help.

You have to always start at last failed command. So if you were missing packages I mentioned, then you have to create default net first.
I’m off to bed, because its morning :wink: But here you have details about it:
https://wiki.libvirt.org/page/Networking

You should be also able to create default network in virt-manager GUI somewhere.

Ah gotcha okey. Thank you got your input! Will continue to do some research and read over the wiki.