[SOLVED] RTX 4090 Passthrough : AMD, Proxmox

Operating Systems & Open Source Virtualization
, ,
1

Hi everyone, I’m having a hard time to setup a gpu passthrough even though the VGA Controller named NVIDIA with the right number is there on my Ubuntu VM, let’s recap all the info

lspci -vv (BEFORE SETUP)

lspci -vv and info on the device I want to passthrough

0f:00.0 VGA compatible controller: NVIDIA Corporation AD102 [GeForce RTX 4090] (rev a1) (prog-if 00 [VGA controller])
Capabilities: [e00 v1] Data Link Feature <?>
Kernel modules: nvidiafb, nouveau

0f:00.1 Audio device: NVIDIA Corporation AD102 High Definition Audio Controller (rev a1)
Subsystem: NVIDIA Corporation AD102 High Definition Audio Controller
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel

lspci -nnv | grep VGA

lspci -nnv | grep VGA

03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1) (prog-if 00 [VGA controller])
0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD102 [GeForce RTX 4090] [10de:2684] (rev a1) (prog-if 00 [VGA controller])

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

dmesg | grep -e DMAR -e IOMMU -e AMD-Vi

[ 0.276619] AMD-Vi: Using global IVHD EFR:0x0, EFR2:0x0
[ 0.714320] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[ 0.718366] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[ 0.718368] AMD-Vi: Extended features (0x58f77ef22294a5a, 0x0): PPR NX GT IA PC GA_vAPIC
[ 0.718371] AMD-Vi: Interrupt remapping enabled
[ 0.718591] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).

dmesg | grep 'remapping'

dmesg | grep ‘remapping’

[ 0.442503] x2apic: IRQ remapping doesn’t support X2APIC mode
[ 0.718371] AMD-Vi: Interrupt remapping enabled

cat /etc/modules

cat /etc/modules

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

modprobe files

echo “blacklist nouveau” >> /etc/modprobe.d/blacklist.conf
echo “blacklist nvidia*” >> /etc/modprobe.d/blacklist.conf
echo options vfio-pci ids=10de:2684,10de:22ba disable_vga=1 > /etc/modprobe.d/vfio.conf

lspci -vv (AFTER SETUP/REBOOT)

0f:00.0 VGA compatible controller: NVIDIA Corporation AD102 [GeForce RTX 4090] (rev a1) (prog-if 00 [VGA controller])
Capabilities: [e00 v1] Data Link Feature <?>
Kernel driver in use: vfio-pci
Kernel modules: nvidiafb, nouveau

0f:00.1 Audio device: NVIDIA Corporation AD102 High Definition Audio Controller (rev a1)
Subsystem: NVIDIA Corporation AD102 High Definition Audio Controller
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

lspci inside the ubuntu VM once configured and passthrough

01:00.0 VGA compatible controller: NVIDIA Corporation Device 2684 (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation Device 165b
Physical Slot: 0
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

I passed the pcie as per the doc and installed Nvidia drivers from Nvidia, no nvidia-smi available but the driver seems there, also the gpu name doesn’t appear on lspci but only the code :fearful:

If you guys have ANY clues that would be awesome, thanks in advance!

UPDATE : FIXED just disable secure boot for the passthrough vm … :slight_smile:

1 Like
2

I looked at your hardware list and it only listed one descret graphic card. As far as I know, you need either two graphic cards, one for the VM and one for the host.

4

I am glad you fixed your first problem with passthough. I have the same motherbroad as you. Been saving for a Nividia 4090. As soon as I save enough for a Nividis 4090, I will build a Proxmox server. So I will be wstching your progress very carefuly.

5

Thank you for the update, i struggeled with this on very similar hardware. Ill try that out when i get it back up and running :call_me_hand: