Falcon Northwest MIG - 8 gamers 1 PC with RTX Pro 6000

wendell · August 17, 2025, 1:17am

Background

TODO – check out our awesome FalconNorthwest Threadripper Pro 96 core build – with Dual RTX Pro 6000!

Paths to GPU Virtualization

It isn’t super relevant to this guide per se but confusion on this can lead to misunderstanding. There are 4-ish ways to get real GPU compute inside a full virtual machine*

GPU pass-through: This is a pcie device that shows up as a pcie device and one passes through the entire device to a virtual machine. This is how gamers and enthusiasts have done it, and a lot of our 5+ year old content on this topic is based on this approach. There are gotchas, especially around how the GPU devices themselves handle PCIe reset events.
SR-IOV: Single Root I/O virtualization. This functions like #1 except the device itself can be configured to show up on the PCIe bus multiple times. Intel iGPUs also permit this functionality (but are so anemic as to be borderline useless, even though they support it.) There is new home with Battlemage/BattleMatrix and a small set of SR-IOV unlockable Arc-based A770s from the initial production run.
MIG: Multi-instance GPU. This is a new-ish thing from Nvidia, and what we’re discussing here. It’s similar to SR-IOV and, contrary to some reports, does work with QEMU/KVM. It’s a true hardware partition. ← This one is us, and what we’re going to do in this how-to.
vGPU: Another nvidia technology, but this one also requires a license or subscription from nvidia to use. Unlike the other technologies. This is effectively co-operative multi-tasking on a gpu, and mostly doesn’t support vRam oversubscription but can support oversubscription of other GPU resources.

*Note that containerization services have more options and tend to be more flexible. Docker and Kubernetes can co-operatively share GPU resources in some additional ways than what is discussed here. A licenseless vGPU type

Getting Started

It helps if you are starting with Linux, have the nvidia drivers installed, ideally version 575 or newer, and that nvidia-smi returns meaningful information.

The first step is to put your GPU in compute mode. This disables the displayport outputs.

Our setup here has an RTX A4000 for host/console output, and two RTX PRO 6000 Blackwell edition cards.

nvidia-smi

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.181                Driver Version: 570.181        CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A4000               Off |   00000000:21:00.0 Off |                  Off |
| 38%   57C    P0             41W /  140W |       0MiB /  16376MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:F1:00.0 Off |                   On |
| 38%   37C    P1            162W /  600W |     256MiB /  97887MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:F6:00.0 Off |                   On |
| 33%   34C    P1            160W /  600W |     256MiB /  97887MiB |     N/A      Default |
|                                         |                        |              Enabled |
+-----------------------------------------+------------------------+----------------------+

Enabling multi-instance GPU persists across reboots. The documentation suggests that

sudo nvidia-smi -i 1 -mig 1

would work to enable MIG / compute mode on the first RTX Pro 6000. However it errors out. displaymodeselector from July 25, 2025 or newer does correctly set the MIG mode, however. You will have to run that utility to set mig, not nvidia-smi, and then reboot.

From here we can verify that mig profiles are available. The available profiles vary from card to card:

nvidia-smi mig -lgip
+-------------------------------------------------------------------------------+
| GPU instance profiles:                                                        |
| GPU   Name               ID    Instances   Memory     P2P    SM    DEC   ENC  |
|                                Free/Total   GiB              CE    JPEG  OFA  |
|===============================================================================|
|   1  MIG 1g.24gb         14     0/4        23.62      No     46     1     1   |
|                                                               1     1     0   |
+-------------------------------------------------------------------------------+
|   1  MIG 1g.24gb+me      21     0/1        23.62      No     46     1     1   |
|                                                               1     1     1   |
+-------------------------------------------------------------------------------+
|   1  MIG 1g.24gb+gfx     47     0/4        23.62      No     46     1     1   |
|                                                               1     1     0   |
+-------------------------------------------------------------------------------+
|   1  MIG 1g.24gb+me.all  65     0/1        23.62      No     46     4     4   |
|                                                               1     4     1   |
+-------------------------------------------------------------------------------+
|   1  MIG 1g.24gb-me      67     0/4        23.62      No     46     0     0   |
|                                                               1     0     0   |
+-------------------------------------------------------------------------------+
|   1  MIG 2g.48gb          5     0/2        47.38      No     94     2     2   |
|                                                               2     2     0   |
+-------------------------------------------------------------------------------+
|   1  MIG 2g.48gb+gfx     35     0/2        47.38      No     94     2     2   |
|                                                               2     2     0   |
+-------------------------------------------------------------------------------+
|   1  MIG 2g.48gb+me.all  64     0/1        47.38      No     94     4     4   |
|                                                               2     4     1   |
+-------------------------------------------------------------------------------+
|   1  MIG 2g.48gb-me      66     0/2        47.38      No     94     0     0   |
|                                                               2     0     0   |
+-------------------------------------------------------------------------------+
|   1  MIG 4g.96gb          0     0/1        95.00      No     188    4     4   |
|                                                               4     4     1   |
+-------------------------------------------------------------------------------+
|   1  MIG 4g.96gb+gfx     32     0/1        95.00      No     188    4     4   |
|                                                               4     4     1   |
+-------------------------------------------------------------------------------+

I selected profile 47, the third entry, above. This is 24b+gfx and the card supports up to 4 instance. Each instance has 24g vram and performs really well for gaming and other workloads.

It’s also possible to split the card in half for two 48g vram instances and some other configurations.

To create the instances:

sudo nvidia-smi mig -cgi 47,47,47,47 -C

This does not survive reboot. Nvidia does provide a daemon stub that is usable to automate this on boot:

Overall nvidia’s documentation is reasonably good and should also be used as a reference.

QEMU Configuration

One thing missing from Nvidia’s docs is how easy it is to go from here to a working QEMU swam. With two RTX Pro 6000s we now have 8x 24g virtualizable gpu instances.

 nvidia-smi -L
GPU 0: NVIDIA RTX A4000 (UUID: GPU-d92f7379-5e22-4dd6-70cc-7d41bf5ad9fb)
GPU 1: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (UUID: GPU-5*****)
  MIG 1g.24gb     Device  0: (UUID: MIG-f46****)
  MIG 1g.24gb     Device  1: (UUID: MIG-d17****)
  MIG 1g.24gb     Device  2: (UUID: MIG-d16****)
  MIG 1g.24gb     Device  3: (UUID: MIG-d1c****)
GPU 2: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (UUID: GPU-4****)
  MIG 1g.24gb     Device  0: (UUID: MIG-7be****)
  MIG 1g.24gb     Device  1: (UUID: MIG-a4d****)
  MIG 1g.24gb     Device  2: (UUID: MIG-e65****)
  MIG 1g.24gb     Device  3: (UUID: MIG-9cf****)

The UUIDs from the output of nvidia-smi are necessary – the MIG 1.24gb type Device entry’s UUID is what will be assigned in QEMU’s configuration
XML.

Make sure not to use the MIG UUID prefix. Also the display=off or on depends on the card you’re using. Off in RTX Pro 6000’s case.

Be sure to update the device’s bus and slot for what makes sense in your passthrough secnario.

<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
 <source>
 <address uuid='f46****'/>
 </source>
 <address type='pci' domain='0x0000' bus='0xxx' slot='0xyy function='0x0'/>
</hostdev>

What about windows as a host?

It should be okay since it is okay on the A100 with hyper-v, but it isn’t.

Not supported!? Lies. It is supported elsewhere, but not with this platform. This is yet another instance where there just isn’t feature parity from Nvidia on Windows vs Linux.

And it’s extra baffling given older enterprise GPUs also work here. MIG isn’t that new. Given hyper-v does actually work okay with sr-iov, I thought it might be possible to put the card into the right mode then soft-reset the machine back to windows from linux. However as I said in the video, It doesn’t work for some reason.

Can you create a windows VM and nest virtualization with the virtual devices created by the utilities on linux? While not recommended that does seem to work, too.

vGPU as a licensed solution is, of course, is still possible directly with bare metal Linux. But who wants that if they don’t have to?

This was also driver 580 – newer than on linux FWIW.

jxdking · August 17, 2025, 3:47pm

Does it have pcie reset bug?

H-i-v-e · August 18, 2025, 6:42pm

Though the question nobody seems to be able to answer, is there a way to make it work with Looking Glass? (Without the video output and a dummy plug)

wendell · August 18, 2025, 6:45pm

Yes, you will need to install the redhat display driver but that works even on gaming GPUs now.

H-i-v-e · August 18, 2025, 6:51pm

That’s the one included in the virtio ISO with all the other virtualization drivers, right?

wendell · August 18, 2025, 7:08pm

yep.

H-i-v-e · August 19, 2025, 5:39pm

@wendell what is the correct sequence to do this. I obviously need nvidia-smi installed, but when I am logged into Gnome on Fedora 42 with the drivers loaded and try to spawn MIG instances, it does not work. However when I switch into the multi-user.target and unload the drivers it works. Is there a correct way to do this from within a graphical session? Or do I not need the Nvidia drivers at all?

I obviously have a host GPU, an Intel card with a different set of drivers to be precise.

wendell · August 19, 2025, 5:43pm

displaymodeselector tool makes the card outputs go dead by default “compute mode” then nvidia has a systemd service to assign mig profils at boot

H-i-v-e · August 19, 2025, 5:46pm

I already used the display mode selector. Though Fedora refused to create instances because the device was apparently already in use. I’ll reboot and try again to maybe provide a precise description of the problem.

H-i-v-e · August 19, 2025, 5:48pm

Also interesting note, the current displaymode selector, downloaded from the official site today, was not able to enable compute mode in Linux with Secure Boot already disabled. I booted into a Windows 11 installation from a different disc and it worked immediately.

H-i-v-e · August 19, 2025, 5:53pm

sudo nvidia-smi mig -cgi 35,35 -C
Unable to create a GPU instance on GPU 1 using profile 35: In use by another client
Failed to create GPU instances: In use by another client

That is basically the output I get when I try to do it from withing Gnome.

I have the Nvidia driver installer, the CUDA toolkit and the Container toolkit. Something is interfering here. I already used the displaymodeselector to enable compute mode and disable display ports.

Ah nvidia-smi shows that there is a process /usr/bin/gnome-shell running on both Nvidia GPUs … that might be a problem.

H-i-v-e · August 19, 2025, 6:50pm

Okay so the default driver includes display output and if you have this one installed the gnome-shell process is spawned on Nvidia devices even though compute mode is enabled. It’s a pain to disable this. Simplest solution to circumvent this issue is to install the compute-only driver from Nvidia as described in their documentation.

After this spawning MIG instances works as expected since no process is hogging the GPU.

wendell · August 19, 2025, 7:28pm

nvtop can also show what processes are running on the GPU if that’s useful

H-i-v-e · August 20, 2025, 7:45pm

lspci

22:00.0 3D controller: NVIDIA Corporation GB202GL [RTX PRO 6000 Blackwell Workstation Edition] (rev a1)

nvidia-smi -L

GPU 0: NVIDIA RTX PRO 6000 Blackwell Workstation Edition (UUID: GPU-aaaaaaaaaaaa)
  MIG 2g.48gb     Device  0: (UUID: MIG-07xxxxxx)
  MIG 2g.48gb     Device  1: (UUID: MIG-67yyyyyy)

virt-manager

<hostdev mode="subsystem" type="mdev" managed="no" model="vfio-pci" display="off">
  <source>
    <address uuid="07xxxxxx"/>
  </source>
  <address type="pci" domain="0x0000" bus="0x22" slot="0x00" function="0x0"/>
</hostdev>

Results in

libvirt.libvirtError: device not found: mediated device '07xxxxxx' not found

MDEV menu in virt-manager shows that no devices are available.

@wendell Do I need to bind vfio-pci somehow?

/sys/bus/mdev does not exist either …

lsmod | grep vfio does not show nvidia_vgpu_vfio

brandon.c · August 22, 2025, 8:35am

Today, I noticed this video NVIDIA posted to YouTube a few weeks ago about MIG with RTX PRO 6000 Blackwell: https://www.youtube.com/watch?v=Ahi9Ta-gv9I

According to this video, using MIG with a hypervisor (passing a partition to a VM) requires using vGPU and is only compatible with the Server card and not the Workstation cards. It acknowledges you can pass an entire GPU to a guest without using vGPU, and you could use MIG within that guest (although this isn’t very useful).

The video also says that graphic-mode instances aren’t currently supported and will come in Q3 2025.

Now, I am not sure if this is a hard limitation of the device and drivers or if vGPU is just automating some of the process of creating MIG instances and configuring the guest OS to use them. However, according to that video, passing a MIG instance to a guest OS is not possible on a workstation card and requires a vGPU license on the server card.

H-i-v-e · August 22, 2025, 10:09am

I know this video but it is a little confusing given that Wendell apparently had it working on Archlinux before making this thread.

wendell · August 22, 2025, 1:21pm

passing a mig instance to a guest is the easy part. it’s grid drivers after that


<hostdev mode='subsystem' type='mdev' managed='no' model='vfio-pci' display='off'>
  <source>
    <address uuid='593207-0bca-4900-b0b1-2844832cd2f'/>
  </source>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</hostdev>

no secure boot, UEFI = on

the UUID you get from the mig instance list. it’s the UUID of the instance.

p.s. it looks to me like there’s a path to not grid drivers given nouveau seemed to load and be accelerated somewhat when remotely accessing the instance. there’s also the sr iov path but that didn’t seem to work for me. that’s what I thought should work with hyper v but it didn’t

H-i-v-e · August 23, 2025, 9:07am

If you try it please report back if Wendell’s instructions worked for you please.

H-i-v-e · September 2, 2025, 7:07pm

Anyone except Wendell managed to make it work? My computer does not see any mdevs of any kind and the /sys/bus/pci/devices/0000:ID:00.0 directory there is also no mdev_supported_types entry.

H-i-v-e · September 4, 2025, 8:25pm

I am not able to replicate this, neither under Fedora nor under Arch.

@wendell I am still confused, which driver you used under Arch. The repositories have only the full-driver apparently and on the Nvidia homepage in the download section for the RTX Pro 6000 Workstation Edition the installer script also installs the full display driver. Where did you get a compute only driver for Arch? Would you maybe be able to give a more in-depth desciption of the setup process and which settings you used

@eousphoros @MarcoBarroca @Veratu @grimulkan @brandon.c Sorry for mentioning all of you but I wanted to ask if anyone of you with an RTX 6000 is able to replicate the passthrough of a MIG instance that Wendell describes here and can give some insight into how you did it?