Hello all,
I am playing around with setting up PCIE passthrough with my old system which was the z390 master mobo. I am using truenas scale 22.12.1 currently as the host os.
Only really using that as a hypervisor as I only need one vm to run for database server on that machine and having direct access to the storage vs going over a virtual adapter for network is faster. And passing through a GPU and a USB card I can use that VM directly like you could with unraid passthrough.
When I check lspci though, the GPU is put in group 1 if I have the card in either the first or second PCIE x16 slot.
I have tried an override I found for the bootloader with the below which didn’t seem to make any differences.
midclt call system.advanced.update ‘{“kernel_extra_options”: “intel_iommu=on pcie_acs_override=downstream”}’
I have also tried directly adding “intel_iommu=on pcie_acs_override=downstream” to grub and updating it with little success as well.
I changed a number of things in the BIOs which none of which seemed to have any impact on where it was placed.
Forgot to add this:
# dmesg |grep -i iommu
[ 0.000000] Command line: BOOT_IMAGE=/ROOT/22.12.1@/boot/vmlinuz-5.15.79+truenas root=ZFS=boot-pool/ROOT/22.12.1 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N intel_iommu=on pcie_acs_override=downstream
[ 0.014379] Kernel command line: BOOT_IMAGE=/ROOT/22.12.1@/boot/vmlinuz-5.15.79+truenas root=ZFS=boot-pool/ROOT/22.12.1 ro libata.allow_tpm=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1 intel_iommu=on zfsforce=1 nvme_core.multipath=N intel_iommu=on pcie_acs_override=downstream
[ 0.014437] DMAR: IOMMU enabled
[ 0.014459] DMAR: IOMMU enabled
[ 0.054014] DMAR-IR: IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 2.477770] iommu: Default domain type: Passthrough (set via kernel command line)
[ 2.576724] DMAR: IOMMU feature fl1gp_support inconsistent
[ 2.576725] DMAR: IOMMU feature pgsel_inv inconsistent
[ 2.576726] DMAR: IOMMU feature nwfs inconsistent
[ 2.576727] DMAR: IOMMU feature pasid inconsistent
[ 2.576729] DMAR: IOMMU feature eafs inconsistent
[ 2.576730] DMAR: IOMMU feature prs inconsistent
[ 2.576731] DMAR: IOMMU feature nest inconsistent
[ 2.576732] DMAR: IOMMU feature mts inconsistent
[ 2.576733] DMAR: IOMMU feature sc_support inconsistent
[ 2.576736] DMAR: IOMMU feature dev_iotlb_support inconsistent
[ 2.576996] pci 0000:00:00.0: Adding to iommu group 0
[ 2.577008] pci 0000:00:01.0: Adding to iommu group 1
[ 2.577014] pci 0000:00:01.1: Adding to iommu group 1
[ 2.577020] pci 0000:00:02.0: Adding to iommu group 2
[ 2.577029] pci 0000:00:12.0: Adding to iommu group 3
[ 2.577039] pci 0000:00:14.0: Adding to iommu group 4
[ 2.577045] pci 0000:00:14.2: Adding to iommu group 4
[ 2.577051] pci 0000:00:14.3: Adding to iommu group 5
[ 2.577060] pci 0000:00:16.0: Adding to iommu group 6
[ 2.577066] pci 0000:00:17.0: Adding to iommu group 7
[ 2.577074] pci 0000:00:1b.0: Adding to iommu group 8
[ 2.577091] pci 0000:00:1c.0: Adding to iommu group 9
[ 2.577100] pci 0000:00:1c.5: Adding to iommu group 10
[ 2.577109] pci 0000:00:1c.6: Adding to iommu group 11
[ 2.577117] pci 0000:00:1d.0: Adding to iommu group 12
[ 2.577133] pci 0000:00:1f.0: Adding to iommu group 13
[ 2.577139] pci 0000:00:1f.3: Adding to iommu group 13
[ 2.577146] pci 0000:00:1f.4: Adding to iommu group 13
[ 2.577152] pci 0000:00:1f.5: Adding to iommu group 13
[ 2.577158] pci 0000:00:1f.6: Adding to iommu group 13
[ 2.577161] pci 0000:01:00.0: Adding to iommu group 1
[ 2.577165] pci 0000:02:00.0: Adding to iommu group 1
[ 2.577168] pci 0000:02:08.0: Adding to iommu group 1
[ 2.577171] pci 0000:02:09.0: Adding to iommu group 1
[ 2.577175] pci 0000:03:00.0: Adding to iommu group 1
[ 2.577180] pci 0000:05:00.0: Adding to iommu group 1
[ 2.577183] pci 0000:06:00.0: Adding to iommu group 1
[ 2.577186] pci 0000:06:00.1: Adding to iommu group 1
[ 2.577199] pci 0000:09:00.0: Adding to iommu group 14
[ 2.577207] pci 0000:0a:00.0: Adding to iommu group 15
[ 2.588314] AMD-Vi: AMD IOMMUv2 functionality not available on this system - This is not a bug.