Return to Level1Techs.com

2 Gamers 1 GPU with Hyper V GPU-P (GPU Partitioning finally made possible with HyperV)

Hi everyone,
First time posting on here. Tried my best to type this out to help out anyone who wants to give this a try.

# What does this GPU-P function do?

A few months back, Microsoft apparently implemented a HyperV function called GPU-P (More a less an early release) into Windows 10 Pro/Enterprise (1903 - 20H1) and possibly Windows Server 2016, 2019, and 2022 (People reported errors on server OS versions. Check back on these when more major server platform updates hit). Windows 11 looks to be a Go for this as well for the Pro/Enterprise editions as well, I believe same setup process will work, Let me know if changes or additions need to be made for this edition of Windows.

What does this GPU-P function do?

This is a replacement to the Legacy GPU RemoteFX which allowed the ability to pass through GPU acceleration to HyperV VMS. With GPU-P, this will allow you to partition different functions of a physical GPU (can be an enterprise or consumer card). I do not believe that this is considered SR-IOV, it can at least give us a somewhat realistic expectation of what to expect if we partition a GPU from the GTX Series to the RTX series of GPUS. AMD Cards work as well.

From What I can tell (feel free to correct me cause my sources are limited) we are sharing a GPU driver from the Host computer to give the VM a driver to use to bypass the code 43 error for any GPU. Apparently, it works with Nvidia drivers that came out before the VM passthrough enablement. When setting up the drivers, you can enable different kinds of functions (Cuda, NVENC, etc) Currently, This should enable NVENC encoding and Cuda support for applications with the DLLs listed below. If the DLLs are missing for what you need, you will be able to search your System32 folder to locate the right DLLs.



Disclaimer

I cannot guarantee that everything will be stable.



Video Guide

I made a Video Guide for people that would like to follow along visually to set this up. I will make another for Parsec later on when I got time.

For those who want to use no monitors

You will need to have headless monitor adapters (any kind will work to my knowledge). Use two or more if using multiple VMs as it will randomly use one. Also, it seems to resolve black screen issues when trying to stream with Rainway.

If you are a Parsec User, you will need to set up a USB Monitor Driver which would make a fake VGA/USB display. Instructions Below.

Instructions for GPU-P setup

NOTE: These are my instructions on how to set up an Nvidia card. AMD cards are similar. For steps 2 and 3, you will need to search in the Device manager to locate different driver files. If I have the time and if my Radeon 7770HD cards have encoders, I will try and update this to help AMD users.

  1. When setting up HyperV. Create VMs with Gen 2. You will also need to disable enhanced sessions and Checkpoints for VMs. By Default, each HyperV VM is set to one CPU core and RAM will be set to Dynamic (if on Windows 10 and not server.). This can be changed on the VM’s Settings





  2. On your host machine, go to C:\Windows\System32\DriverStore\FileRepository\
    and copy the nv_dispi.inf_amd64 folder to C:\Windows\System32\HostDriverStore\FileRepository\ on your VM (This folder will not exist, so make sure to create it)



  3. Next you will need to copy items from C:\Windows\System32\ from your host to C:\Windows\System32\

    In your hosts System32 folder, do a search for NV*.* and copy all.


    !!! PLEASE NOTE IF YOU HAVE UPDATED YOUR HOST’S VIDEO DRIVERS, YOU WILL HAVE TO RECOPY ALL DRIVERS TO YOUR VM’S OR CODE 43 WILL REAPPEAR !!!


  4. You will notice that you do not have any sort of Audio Driver installed on the VM.

    You will need to have a virtual audio cable installed to pass audio. One Virtual Audio Cable software I can recommend is VB Cable. You can get the free version or support their product. One thing to note is when you mute the audio in the VM, it will not mute or change the volume level.

  5. Shut VM down and configure your Powershell script for the VM. You can change out the Values as you see fit or change them to be the same as mine from the Video.



  6. Run PowerShell on the host system as admin to start partitioning the GPU with Powershell with the script below.

    PowerShell Script

    This will be trial and error to get settings right. You can use the PowerShell command “Get-VMPartitionableGpu” to locate the total amount of resources you have available and divide it for each VM you want to assign resources to. Also if you have multiple GPUs, The name field will match the hardware ID of the card.

For users using Windows 11, Please see this comment in the thread, It looks like people can directly assign GPUS to have a partition from and not let Windows auto select one for you 2 Gamers 1 GPU with Hyper V GPU-P (GPU Partitioning finally made possible with HyperV) - #137 by JayRyl

    #The output of all the values of Get-VMPartitionableGpu will look like this

    Name                    : \\?\PCI#VEN_10DE&DEV_1F02&SUBSYS_251619DA&REV_A1#4&2283f625&0&0019#{064092b3-625e-43bf-9eb5-d
                              c845897dd59}\GPUPARAV
    ValidPartitionCounts    : {32}
    PartitionCount          : 32
    TotalVRAM               : 1000000000
    AvailableVRAM           : 1000000000
    MinPartitionVRAM        : 0
    MaxPartitionVRAM        : 1000000000
    OptimalPartitionVRAM    : 1000000000
    TotalEncode             : 18446744073709551615
    AvailableEncode         : 18446744073709551615
    MinPartitionEncode      : 0
    MaxPartitionEncode      : 18446744073709551615
    OptimalPartitionEncode  : 18446744073709551615
    TotalDecode             : 1000000000
    AvailableDecode         : 1000000000
    MinPartitionDecode      : 0
    MaxPartitionDecode      : 1000000000
    OptimalPartitionDecode  : 1000000000
    TotalCompute            : 1000000000
    AvailableCompute        : 1000000000
    MinPartitionCompute     : 0
    MaxPartitionCompute     : 1000000000
    OptimalPartitionCompute : 1000000000
    CimSession              : CimSession: .
    ComputerName            : MY-COMPUTER
    IsDeleted               : False

When Setting up the partitions, try to leave headroom on the GPU or it might throw an error stating that there aren’t any resources available. Try to leave 10% - 15% available If you seem to be getting errors when starting up a second VM (or VM#x) that would be also using the same card as another VM(s).

PowerShell script

$vm = "ENTER YOUR VM NAME HERE"
Remove-VMGpuPartitionAdapter -VMName $vm
Add-VMGpuPartitionAdapter -VMName $vm
Set-VMGpuPartitionAdapter -VMName $vm -MinPartitionVRAM 1
Set-VMGpuPartitionAdapter -VMName $vm -MaxPartitionVRAM 11
Set-VMGpuPartitionAdapter -VMName $vm -OptimalPartitionVRAM 10
Set-VMGpuPartitionAdapter -VMName $vm -MinPartitionEncode 1
Set-VMGpuPartitionAdapter -VMName $vm -MaxPartitionEncode 11
Set-VMGpuPartitionAdapter -VMName $vm -OptimalPartitionEncode 10
Set-VMGpuPartitionAdapter -VMName $vm -MinPartitionDecode 1
Set-VMGpuPartitionAdapter -VMName $vm -MaxPartitionDecode 11
Set-VMGpuPartitionAdapter -VMName $vm -OptimalPartitionDecode 10
Set-VMGpuPartitionAdapter -VMName $vm -MinPartitionCompute 1
Set-VMGpuPartitionAdapter -VMName $vm -MaxPartitionCompute 11
Set-VMGpuPartitionAdapter -VMName $vm -OptimalPartitionCompute 10
Set-VM -GuestControlledCacheTypes $true -VMName $vm
Set-VM -LowMemoryMappedIoSpace 1Gb -VMName $vm
Set-VM -HighMemoryMappedIoSpace 32GB -VMName $vm
Start-VM -Name $vm

  1. Installing a Streaming Application (Rainway and Parsec)

Installing Rainway

  • If you are at this point and have a real or headless monitor hooked to the GPU, You can install Rainway and get going. You may be able to still use the fake VGA/USB Monitor in the Parsec Guide Below.

    Link to Rainway: Game Streaming - Play Anywhere | Rainway

Installing Parsec

  • Parsec will work with GPU-P, The headless Monitor trick may or may not work for this. One way that me and a friend were able to get this to work (with or without Parsec Warp subscription) was to install a fake VGA/USB Monitor Driver.

    To download the driver, this is the AMYUNI Technologies Forum page where the driver is found.
    Link to Forum page: Activating a Secondary Display on Windows 10 when no Monitor is Connected -Amyuni Technologies

    A1. Extract the downloaded folder “usbmmidd.zip” and you can move this to a more general location like C:\ProgramFiles or root of the C: Drive.

    A2. After moving this folder, you will want to run the “usbmmidd.bat” file to install the driver.

    image

    !!!NOTE!!! In HyperV, once the driver is installed, your mouse may cease to function cause a second monitor will be present in the VM that cannot be seen. To resolve this, you will have to navigate to the settings menu set to only show on Display 1 only using the keyboard.

    image

    A3. !!YOU MAY NOT NEED TO DO THIS STEP!!, the driver may have to be initialized again. You can follow the steps here incase it needs to be done every restart. It may cause more failed monitor drivers in the device manager.

We will have to make a Task scheduler that starts every time the VM starts.

In Windows, search for Task Scheduler in search

Select “Task Scheduler Library” and Create New Task.

In the Creation of the Task, Set these options in the General Tab.

In the Triggers tab, create new trigger and set At startup.

In the Actions tab, make a new action. Have it Start a program and select the usbmmidd.bat file.

The Conditions tab can be left alone.

In the Settings tab, make the task run as soon as possible, stop the task if runs longer than 1 hour, and force it to stop if task does not end.

A4. After hitting ok, you will be all set and can install Parsec.

Link to Parsec: https://parsec.app/



From here on out, everything should be working. Install stuff from steam or get prereqs for other software.


Notes about multiple GPUs

So you are aware, multiple GPUs will work but no guarantee you will get assigned the card you want as it may randomly select a GPU every time the host computer reboots. If it’s two or more of the same GPU type then don’t worry about getting assigned a slower card.



Notes about software compatibilities:

  • Nvidia Geforce Experience: Doesn’t look like it will work as it will not detect an Nvidia GPU is installed. Even after putting in GPU drivers.

  • AMD’s Radeon Software: Looks to be in the same boat as Geforce experience from my testing.

  • Parsec: I know Parsec works, even Parsec Warp users can use this with the virtual display driver to get better screen resolutions. If you are a Warp user and want to have two Displays (HyperV monitor with additional USB/VGA virtual monitor), Warp users can use this. But be warned, the viewing window on HyperV may not work when setting the monitor settings back to extend, and have to navigate by a keyboard. Recommendation to use a parsec installed app rather than a web app to use.

  • Rainway: Far as I can tell, fully works in an unofficial way and will run whatever you need in any amount of VMs.

  • Open stream: From the Reddit post comments I’ve seen, this works. I haven’t fully investigated this. If you have any results, lets us know in the reply section!

  • MSI Afterburner: Run this on the Host computer rather than the VM. Seems to give it a bit more performance when increasing some clock speeds of ram and GPU core in the VMs. Might not have a consistent experience across the VMs.

  • OBS: From the looks of it, Hardware encoding with NVENC is working when using the DLLs listed above. If NVENC Works with this, then there may be other applications that may follow as well.

  • Cuda enable apps: From testing out the program Meshroom to try out Cuda requirements, it recognized Cuda and used it. So Might be working or hit or miss with other apps.



My rigs I have used to test GPU-P

Here is the hardware I have used to do my testing:

Computer 1: My Workstation/gaming computer - Also used for trying Rainway and Cuda with Meshroom.
CPU: Ryzen 9 5900X
MOBO: AORUS Master B550
RAM: HyperX Fury 32 GB (16GB x 2) @3600MHz
GPU: Zotac Mini RTX 2070 8GB (Not a super or TI Varient)
Storage: too many SSDs

Computer 2: Big Bertha (It’s a Computer, not a Howitzer, and not a mining rig) - Also used for doing Multi GPU testing and Rainway.
CPUS: Dual Xeon x5650 @2.60 GHz (usually run at 3.0 GHz at turbo)
MOBO: X8DAH-F SuperMicro Dual Socket LGA 1366 (PCIE Gen 2 speed)
RAM: 28 GB of ECC @ 1333MHz (4GB x 7 Kit. Last 4 GB for 32GB died or slot died on MOBO)
GPUs:
GPU1 - PNY GTX 1070 TI 8GB
GPU2 - ASUS White GTX 1060 3GB



Final thoughts:

I hope to hear if people decide to go up and beyond with this. I plan to try and max out the 32 Partitions on my 2080 to see if I can get something like the ARMA 3 benchmarks to run 32 times. Not a usual game to benchmark but I think it is possible.

Even though Microsoft hasn’t put out much documentation for official use, this is still worth a try in my opinion.

Let me know there are questions and I will try to do my best and answer them. Though this software is still quite Vague, I’ll do my best.77

29 Likes

can you post some stuff with your setup and/or screenshots?

I did the remotefx stuff and I’ve been tinkering with this but it seems like somethings up with the 3090 because it had been working okay-ish.

3 Likes

Fantastic stuff but as @wendell said some screen shots would be great to see :slight_smile:

1 Like

@wendell

I did two videos on this on YouTube but can’t for some reason post a link.

The specs of the computer I did my cloud streaming on

Gigabyte AORUS MASTER B550
Ryzen 9 5900X
32 gb ddr4 3600Mhz
Zotac mini RTX 2070 (not super or Ti)

My first system I did this on was an old dual hex Xeon system running on a super micro board with pci 2.0 speeds. I’ll get the information later.

28gb ecc ram
Gpu 1: pny 1070ti
Gpu2: asus white 1060 3GB

If I can I’ll upload photos but I cannot seem post YouTube video links. Writing this on my phone so I’ll fix the information in the comment when I get home

1 Like

Awesome thank you so much!

1 Like

image

These are the two videos that I did on this.
Though this is a photo

1 Like

Thanks wendel

I did some clean up/organization and added notes to this post if something was confusing.

Can’t add any more photos for instructions cause of limitations of my new profile status, whenever that passes I’ll update it.

If anyone tries this, I’d like to hear the results!

Thanks,

1 Like

Maybe after about 30 Revisions of this post to add in the proper information and photos. I think it is completed.

Let me know if more revisions need to be made LOL.

1 Like

Hi! This sounds amazing. I’ve been running Parsec (1 VM per GPU) on my Proxmox servers via GPU Passthrough as remote workstations, but this method caps out at 3 users per server. Due to the constraints of my 4U rackmount cases, I can only fit 2x double slot GPUs and 1x single slot GPU.

If we could run multiple VM’s with Parsec per GPU eventually, it would free up a lot of PCIE lanes, power draw and heat.

I am curious about 2 things:

  1. Is there any particular known error or limitation with Parsec that is preventing using multi VMs?

  2. Is there any noticeable performance overhead or caveats from using GPU-P so far?

Thanks!

EDIT: Found some info on 1. Reddit Post

2 Likes

@WarwicK7

My experience with parsec (with Warp subscription as well) seems to have a weird limitation with me and might not for others.

I partitioned two VMs to a 1070TI and I could only get one of the VMs to actually stream, the other would keep giving me a display issue error but would hardware encode stream on the other. Tested using Headless Monitor Dongles and also the Virtual Display Driver that comes with Warp subscription.

The only thing was when I did this, I was using a PCIE Gen2 system and I haven’t tried yet with my 2070 in my system with a PCIE Gen4 slot.

To answer performance caveats. Since it is a single GPU core, I cannot seem to find a way to fully regulate core performance per VM (VRAM for sure can be regulated but not the physical processor from my testing). The PowerShell script above shows different settings where there might be some trial and error on getting different results.

So if you were playing Crysis in one VM, and tried to run it in the other using one GPU. One VM may run better or they both may just equalize out the framerates. It all depends on what kind of workload is going on per vm.

Now if it was a totally unoptimized game Cough arma 3 cough that wouldn’t even use over 20% of the GPU’s processer, it might not take much of a hit on the other VM and might have same performance.

Running arma 3 benchmarks on two vms with a rtx 2070 but had the vms limited to 4 cpu cores and 4GBs of ram.

Making a video to see if I can get 10 vms running but will have to sacrifice some ram.

Hope this answers your question!

This is really awesome. Thanks for the clear tutorial.

I’ve got this working on my AMD 6800XT too, so AMD cards shouldn’t be a problem I think.
I exported the Driver Store files using a program called “DriverStoreExplorer”, I did not know which folder I needed to copy at first. It appeared to be the one called “u0366490.inf_amd64_c0dea8a43cb81731”, though I think this probably differs per install/driver version.

For the nvidia files in system32 you named I just exported everything with “AMD” in the name from the C:\Windows\System32\ folder. This seems to work, though I’m not sure if you need all of them.

Finally I used this as my powershell script:

$vm = “Win10-GPU-1”
Remove-VMGpuPartitionAdapter -VMName $vm
Add-VMGpuPartitionAdapter -VMName $vm
Set-VMGpuPartitionAdapter -VMName $vm -MinPartitionVRAM 1
Set-VMGpuPartitionAdapter -VMName $vm -MaxPartitionVRAM 4000000000
Set-VMGpuPartitionAdapter -VMName $vm -OptimalPartitionVRAM 3999999999
Set-VMGpuPartitionAdapter -VMName $vm -MinPartitionEncode 1
Set-VMGpuPartitionAdapter -VMName $vm -MaxPartitionEncode 4611686018427387903
Set-VMGpuPartitionAdapter -VMName $vm -OptimalPartitionEncode 4611686018427387902
Set-VMGpuPartitionAdapter -VMName $vm -MinPartitionDecode 1
Set-VMGpuPartitionAdapter -VMName $vm -MaxPartitionDecode 4000000000
Set-VMGpuPartitionAdapter -VMName $vm -OptimalPartitionDecode 3999999999
Set-VMGpuPartitionAdapter -VMName $vm -MinPartitionCompute 1
Set-VMGpuPartitionAdapter -VMName $vm -MaxPartitionCompute 4000000000
Set-VMGpuPartitionAdapter -VMName $vm -OptimalPartitionCompute 3999999999
Set-VM -GuestControlledCacheTypes $true -VMName $vm
Set-VM -LowMemoryMappedIoSpace 1Gb -VMName $vm
Set-VM -HighMemoryMappedIoSpace 32GB -VMName $vm
Start-VM -Name $vm

In the VM it looks something like this:

Performance seems really good too. When just one of the benchmarks is running I get around 300fps in the VM, compared to around 350 on the host with the same settings.
Host:


VM:

Two VM’s at the same time, getting around 230fps:

5 Likes

@NetworkNoob NICE MAN!!!

AMD Drivers are tricky. I do not have a good enough AMD GPU to use for a Tutorial nor do I have the money to get one with proper encoders.

Were you able to get any sort of card computation like the equivalent to NVENC encoding or CUDA tasks? I know those aren’t the terms for AMD but I’m not versed on the AMD Realm

Glad to see that the newest gen cards can blow VM Tasks like this out of the water like it’s nothing. :slight_smile:

2 Likes

@cheif22 I haven’t had the time to test any of that yet unfortunately. I’ll see if I can test some more the next few days :smiley:
Anyways I’m actually really happy I’ve got this working. I was looking for something like this on a consumer card for a while now.

2 Likes

@NetworkNoob

Good to hear. I wish you good luck!

Let us know your findings

Thanks!

2 Likes

Working on something

running low on Ram LOL but still impressive that arma can run on 4 VMs but not peg the GPU core.


This the top VM row photo above, bottom VM row photo below

4 Likes

Only real annoyance here is the slightly laggy RDP. I wonder how hard it would be to get looking glass running on a setup like this? Seems Microsoft is working on allowing Linux to run as the root partition on Hyper-V, which would make this pretty much perfect.

4 Likes

True that. the current setup is limited to the latency of the streaming application. Unless you were to just use the hyperV viewer which does sorta proves useless since it doesn’t keep the mouse within the window and screen resolution limited to 1080p(Probably a Hyper-V trick somewhere to fix the resolution limitations).

I haven’t tried Looking Glass at all or on Hyper-V so I cannot confirm anything.

1 Like

Hey all parsec users, I believe me and a friend may have found a workaround with some issues.

I’m running through it again to test it and will update the guide if I’m able to

1 Like

For any Parsec Users, Guide has been updated once again LOL.

2 Likes