How to run a process on a specific GPU?

I have an NVidia 1660 card with my main screen attached and a 5090 with my second screen attached.
What I would like to do is have everything run on the 1660 by default, then specific processes I want to run on the 5090.

I have not had any issue specifying the card for CUDA workloads, but have not been able to get this working for graphical workloads.

If I include both my GPUs in my xorg.conf then xrandr --listproviders will just report whatever GPU the main screen is attached to (the 1660)

My 1660 is index 0 and 5090 index 1 when checking nvidia-smi.

__NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-1 _NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia glxinfo | grep "OpenGL renderer" 

Will report both the cards if both are configured in xorg.conf or just the one card there if only one is configured.

__NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-1 _NV_PRIME_RENDER_OFFLOAD=1 __GLX_VENDOR_LIBRARY_NAME=nvidia firefox

Will launch firefox, but when I run nvidia-smi, firefox does not show up. So it seems the environment variables are doing something, but not making the process run on GPU 1.

If I try variations on the __NV_PRIME_RENDER_OFFLOAD_PROVIDER value (different sources online seem to give different examples), again trying to launch firefox, all don’t show firefox running on either GPU in nvidia-smi. I tried:

  • NVIDIA-1
  • GPU-3b389bf5-f9ed-85ed-5cab-f1dc6ea2a4b2 (the guid given by nvidia-smi -L)
  • GPU-1
  • NVIDIA-G1

I have tried the above with the 5090 in the xorg.conf and without it, both with the same results.

Running firefox normally you can see it on the GPU in nvidia-smi.

I also tried with DRI_PRIME=1 and this makes no difference.

Does anyone have any advice on how to make a process run on a specific GPU?

2 Likes

made an account just to comment on this, since it has been one of my biggest pains with linux (and more specifically Wayland / KWin).

I am in a similar boat with a 3070 and a 5070 Ti. I don’t know what DE / compositor you run on, but if it is Wayland / KWin, you’re out of luck. Save yourself hours of headaches and in Linux run one GPU, both monitors.

Windows’ WDDM and CORS / CASO (?) (Cross Adapter Resource something something) play nicely with two gpus, weirdly enough.

I tried the same setup as yours, everything on the 3070, one card per monitor. Or both monitors on the 5070 Ti, just with some processes running on the 3070. No luck either way. With KWIN_DRM_DEVICES=/dev/dri/card1:/dev/dri/card0 (or something like that). Wayland / KWin will not really let you do that. If the compositor chose one GPU as “master”, and you want to render something on a secondary GPU, many apps (or wayland itself?) do not play nice being offloaded to another GPU.

Closest I got, after hours and hours and more agonizing hours of trying, but don’t count on me to remember what I did, was to get NVDEC on the 3070 to almost decode a video when the display (and KDE) was rendered by the 5070 Ti. I was met with a crash and a static green frame on videos and so I gave up. That is as close as I got dual-gpu, outside of CUDA, + NVDEC hw acceleration (in browsers) to play along.

Other apps, realistically any other app i tried, aside from, say, ComfyUI or llama.cpp, that can make use of CUDA_VISIBLE_DEVICES, was a no-go.

Games, however, do make use of offloading.

For games VKD3D_VULKAN_DEVICE=0/1 should work, as well as DXVK_FILTER_DEVICE_NAME=“NVIDIA GeForce RTX 5070 Ti"

Iirc I played around with these variables

__NV_PRIME_RENDER_OFFLOAD=1

__GLX_VENDOR_LIBRARY_NAME=nvidia

__NV_PRIME_RENDER_OFFLOAD_PROVIDER=NVIDIA-G0 browser-of-choice

I think I tried every “use this so process runs on another GPU” variables, like

EGL_VISIBLE_DEVICES=pci-bus-id (in my case 04:00.0)

AQ_DRM_DEVICES=/dev/dri/card0

CUDA_VISIBLE_DEVICES=x

__NV_PRIME_RENDER_OFFLOAD_DEVICE=GPU-guid in your post

On x11, iirc, it’s easier to run apps per-gpu (and in your case per-monitor), however, they are treated as separate “workspaces” (for lack of a better word), you cannot move one window from Monitor 2 to Monitor 1. They would be functionally disconnected and independent from each other.

If anyone here has got any other ideas, I’m all ears.

On Windows actually it doesn’t work perfectly either. On my main PC I have a RTX 4000 PRO SFF and a RTX 5090. Sometimes I want to render something in the RTX 5090 i.e. (which is headless) but it still uses the PRO. Or viceversa. I set the GPU on Settings → Graphics for an specific GPU but it just ignores it lol.

1 Like

Thanks, I am using X11/Xorg, so no Wayland related compatibility issues. The nearest I have got is running a program on a specific display (which then runs it on the GPU that display is attached to).

But that’s not what I want to do, I want to render and display on the 1660 by default and then for certain programs render on the 5090 and display on the 1660.

I tried adding my main monitor to my xorg.conf twice, once with the 1660 set as the device and again with the 5090 set as the device:


Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Samsung 1660" 0 0
    Screen      1  "Dell 5090" 2560 180
    Screen      2  "Samsung 5090" 0 0
    InputDevice    "keyboard" "CoreKeyboard"
    InputDevice    "mouse" "CorePointer"
    Option         "Xinerama" "0"
    Option       "AllowNVIDIAGPUScreens" "true"
EndSection

Section "ServerFlags"
    Option         "AutoAddDevices" "false"
EndSection

Section "InputDevice"
    Identifier     "mouse"
    Driver         "evdev"
    Option         "Device" "/dev/input/event2"
    Option         "AutoServerLayout" "true"
EndSection

Section "InputDevice"
    Identifier     "keyboard"
    Driver         "evdev"
    Option         "Device" "/dev/input/event3"
    Option         "AutoServerLayout" "true"
EndSection

Section "Monitor"
    Identifier     "Dell 22"
    VendorName     "Dell"
    ModelName      "P2214H"
    HorizSync       30.0 - 83.0
    VertRefresh     56.0 - 76.0
    Option         "DPMS"
EndSection

Section "Monitor"
    Identifier     "Samsung 32"
    VendorName     "Samsung"
    ModelName      "U32H85x"
    HorizSync       30.0 - 135.0
    VertRefresh     30.0 - 75.0
EndSection

Section "Device"
    Identifier     "5090"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "NVIDIA GeForce RTX 5090"
    BusID          "PCI:13:0:0"
EndSection

Section "Device"
    Identifier     "1660"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "NVIDIA GeForce GTX 1660 SUPER"
    BusID          "PCI:12:0:0"
EndSection

Section "Screen"
    Identifier     "Dell 5090"
    Device         "5090"
    Monitor        "Dell 22"
    DefaultDepth    24
    Option         "Stereo" "0"
    Option         "nvidiaXineramaInfoOrder" "HDMI-0"
    Option         "metamodes" "nvidia-auto-select +0+0"
    Option         "SLI" "Off"
    Option         "MultiGPU" "off"
    Option         "BaseMosaic" "off"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

Section "Screen"
    Identifier     "Samsung 1660"
    Device         "1660"
    Monitor        "Samsung 32"
    DefaultDepth    24
    Option         "Stereo" "0"
    Option         "nvidiaXineramaInfoOrder" "DP-2"
    Option         "metamodes" "nvidia-auto-select +0+0"
    Option         "SLI" "Off"
    Option         "MultiGPU" "off"
    Option         "BaseMosaic" "off"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

Section "Screen"
    Identifier     "Samsung 5090"
    Device         "5090"
    Monitor        "Samsung 32"
    DefaultDepth    24
    Option         "Stereo" "0"
    Option         "nvidiaXineramaInfoOrder" "DP-2"
    Option         "metamodes" "nvidia-auto-select +0+0"
    Option         "SLI" "Off"
    Option         "MultiGPU" "off"
    Option         "BaseMosaic" "off"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection

But this doesn’t work, if I try DISPLAY=:0.2 firefox I get an error that display can’t be opened. (:0.0 will open on my 1660 screen and :0.1 on the 5090 screen).

Most of the info I can find online relates to offloading rendering to an NVidia GPU when your display out is attached to an Intel iGPU.

I don’t have a lot of knowledge about X11, but yes, all those _NV_PRIME_OFFLOAD_ are, as you said, mostly for hybrid laptops (one intel, one NVIDIA for example). I don’t think multi-dgpu offload is a priority for anyone, sadly.

CUDA apps work via CUDA VISIBLE DEVICES
Vulkan apps should have its own offload thingy that should work ?

Other apps might not run (easily) on another GPU.

One thing I could think of is running a second X Server at :1 instead of multiple X Screens in one X Server?

I am not smart enough to know how to setup xorg.conf, or something in xorg.conf.d with that, but probably you could figure it out lmao.

Then run with DISPLAY=:1 firefox ?

L.E:

Could also try Cosmic DE if you got some time to spare? Found something on google - there’s a variable called COSMIC_RENDER_DEVICE=<> (as seen in this github issue)

Alternatively, could keep an eye out on this ? Runtime GPU switching (#308) · Issues · Plasma / KWin · GitLab

There are probably much better solutions for this, but you could probably set up a separate user with distrobox or something, and then try setting up different device permissions for them so that your main user just can’t run apps on the 5090, and start the CUDA apps through distrobox. You’ll want to make damn sure you have the same version of Mesa on the host though lmao

Or maybe get a dummy adapter for the 1660 and set the monitor connected to the 5090 to mirror that? Not sure if it’ll work, but those are pretty cheap.

I don’t know much about xorg config either, unfortunately.

I don’t think running multiple xorg servers would help as I would still have the problem of how to render/compute on one GPU but output on the other.

I gave cosmic a try, but was unable to get it working. On Debian it didn’t seem to install any services, and manually trying to start it I get an error about Xwayland I’ve been unable to find a solution for:
ERROR cosmic_comp::xwayland: Failed to start Xwayland. err=Os { code: 2, kind: NotFound, message: "No such file or directory" }

I also tried on Arch and couldn’t get it running there, possibly an issue relating to mismatched Nvidia drivers:
Jun 07 10:14:57 lxc-arch greetd[12579]: PAM unable to dlopen(/usr/lib/security/pam_limits.so): libsystemd.so.0: failed to map segment from shared object Jun 07 10:14:57 lxc-arch greetd[12579]: PAM adding faulty module: /usr/lib/security/pam_limits.so Jun 07 10:14:57 lxc-arch greetd[12579]: PAM unable to dlopen(/usr/lib/security/pam_lastlog2.so): libsqlite3.so.0: failed to map segment from shared object Jun 07 10:14:57 lxc-arch greetd[12579]: PAM adding faulty module: /usr/lib/security/pam_lastlog2.so Jun 07 10:14:57 lxc-arch greetd[12579]: error: terminal: unable to set kernel display mode: ENOTTY: Not a typewriter Jun 07 10:14:57 lxc-arch greetd[12578]: unable to start greeter: terminal: unable to set kernel display mode: ENOTTY: Not a typewriter Jun 07 10:14:57 lxc-arch greetd[12578]: error: unable to reset VT: terminal: unable to set kernel display mode: ENOTTY: Not a typewriter

Regarding using a dummy plug and mirroring, I’m not sure how to mirror a display? None of the xorg config suggestions I found worked. And the xrandr suggestions I found all seem to be when using mirroring a screen connected to the same card. My attempts using xrandr didn’t work either, e.g.
DISPLAY=:0.0 xrandr --output NVIDIA-0.DP-2 --mode 1920x1080 --output NVIDIA-1.HDMI-0 --mode 1920x1080 --same-as "Samsung 1660" warning: output NVIDIA-0.DP-2 not found; ignoring warning: output NVIDIA-1.HDMI-0 not found; ignoring

Using additional containers or VMs is not the ideal solution but may be the only way to get it working. I can think of a couple of possible solutions:

  1. Have the 5090 assigned to a different container, then use moonshine/sunshine to access software running on the 5090 from the container running the 1660. The 1660 would have the screen(s) attached and the 5090 no physical screens. Issue with this setup is we need two containers that are basically the same apart from the attached GPU, so it is a bit of duplication (though I’d imagine quite a bit of the storage could be shared between the two).
  2. Similar to option 1, have 1 container / vm config with the 1660 only, then another config with the 5090 (and could have the 1660 attached as well). Have the same storage attached to both, and only ever run one of the two options. Then just use the 5090 one when I need access to it for GPU intensive GUI stuff like video editing. Otherwise use the 1660 one, which leaves the 5090 free for non-GUI stuff like running LLMs. The issue with this option is that if I’m doing GUI stuff with the 5090 then want to use it for non-GUI stuff, I’d need to completely shutdown the 5090 container/vm, start the 1660 container/vm and then re-open everything that I was working on.

I will give the above options a go and see if I can get either working. Though any other suggestions would be welcome!

1 Like