I’m trying to set up a multi gpu workstation (currently with gtx 980 and gtx 1060) for GPU rendering and I’m having some issues. Has anyone managed to get it running on Linux and can give me some pointers?
First problem is probably more hardware related. I don’t see an option to set a gpu priority in BIOS (i have a Asrock EP2C602 moba). This means a wrong GPU is set as primary. Even worse if I connect monitors to gpu I want to be primary (gtx 980) I stop seeing any feedback from the system as soon as grub initializes - I can’t see even the grub options screen (judging by drivers working and click on the sound card the os starts though). Are there any settings / grub tweaks I can try to use gtx 980 as a primary?
While I’ve been looking into BIOS the only gpu related setting I’ve noticed is uefi/legacy setting for video (there are also separate settings for other pcie devices). But sadly when I switch it to “uefi first” I can’t boot at all - I hear a single beep then after a (long) while I’m back to BIOS.
I can get the system running if I follow up what system forces me to and use gtx 1060 as a primary card and connect monitors to it. I use proprietary drivers. Nvidia-settings panel shows both cards. But I can’t display anything on the other card - settings let me know that Xorg reboot is required if I want to see screens connected to 2nd gpu. This does not work though (note - I’ve saved the conf file ).
Moreover no app I’ve tried (Blender, Octane benchmark…) sees both cards. I can use 1060 only (gpu monitors are connected to).
So while the 2nd device is seen (nvidia-settings, lspci etc) no app , or Xorg itself seems to be able to use it.
Drives me nuts…
Welcome to the world of Xorg being uncooperative with multiple GPUs. If you had AMD cards, you could switch to Wayland no problem, but because Nvidia isn’t being cooperative with the Wayland nor Nouveau people, you’re stuck on broken Xorg. Xorg will never support multiple monitors over multiple GPUs, period.
Xorg has supported that for over a decade now. My current setup is using a R9 380 for the main display and a RX 560 for second monitor. Never tried dual NVidia graphics setup though, only AMD/AMD and Radeon/NVidia.
Yes, but you can’t do a spanned desktop, only 2 individual displays with their individual stuff with Xorg. That’s something I call broken beyond repair UX wise. You’re not going to set environment variables for displays for which EACH of your applications has to load nowadays, it’s wildly inefficient. If you wanted to drag a window from one X screen to another, YOU CAN’T. Wayland will solve this eventually and allow a proper spanned desktop across multiple GPUs once the compositors get up to speed. Xinerama does not allow 3D acceleration across spanned displays and a copy of the desktop has to exist (extremely inefficently) on each GPU as opposed to running specific applications on specific GPUs. I for one prefer a spanned desktop that intelligently determines what GPU is doing what. Wayland is a step closer to that.
The thing is that I don’t even want to display on multiple monitors connected over multiple gpus. Only one card is intended as a display device.
Second gpu is intended only as additional compute device for 3d rendering.
AMD is not an option - major gpu 3d rendering engines are CUDA only (Redshift, Octane… - even Cycles has much better support on CUDA).
At this point I can’t even get intended card (gtx 980) to work as a main display device. And I’m not even talking about Xorg - as soon as grub takes over it switches to gtx 1060 so the console displays only over through it. Haven’t had a clue it could be so painful…
Yesterday was a tricky day. At one point I decided to update bios… only to loose efi entries… Had to learn howto use efi shell to even boot my pc. So at least it was a learning experience .
So any tips are welcome so hopefully I could avoid more misadventures .
Two things. I cannot help with grub but the above post mentioned trying different PCI-e slots. If the 1060 is only for GPGPU then PCI-e 8x or 4x is more than enough.
1.) Try with only the 980 installed, Setup XRandR and then add the additional card. I don’t think grub is your issues here. I think that you are having an issue with libdrm. If you are using the proprietary NVIDIA driver, then ensure that you blacklist nouveau. The NVIDIA driver supports SLI and should be able to handle multiple GPUs. It is the same driver that is used in MS Windows and MacOS X.
2.) Gnome 3 does support the NVIDIA eglstreams solution and Wayland itself does support multiple GPUs at this point. The support is not complete but there are people currently using it. I cannot vouch for NVIDIA because they generally go their own way but if option 1 does not work, option 2 may.
both issues are fixed.
Quite annoyingly I don’t know what really made the difference in issue 1 (system selecting wrong gpu as primary). After the mentioned removal of the battery and thus clearing all the uefi and bios settings I simply plugged the cards into the same slots, connected monitors to gtx 980 and all went as intended (and again note - it was not Xorg that was the problem, I experienced incorrect card selection just as soon as boot loader taken over from bios, way before Xorg starts). Don’t know whether I’m happy or annoyed. Happy that it works, annoyed because I don’t know why (and thus I’m a bit nervous tweaking settings as I don’t know what may break things).
After that I simply reinstalled the drivers + cuda and the packages I use to test this setup immediately detected both gpus. And boy, Octane render with multi gpu is a joy. That being said I hope Redshift gets wider adoption in the coming months as it looks the best for my needs (but sadly does not have any inegration with the apps I use).
Just in case anyone here has a similar, but perhaps slightly different issue, I had problems with dual GTX 1070s that was only really solved by using the Nvidia driver in Mosaic mode. Otherwise my desktop environment would crash if I could get both cards outputting to monitors at all. I still have some issues with RTCWake not bringing things back appropriately, but I’m in far better shape than I was prior (and yes, I black listed nouveau).
Not pointing fingers here or something, more of a PSA.
One of the main reasons for linux is the open source nature of it and the possibilities that come with it. And then you get hardware that will never get proper open source drivers? … That just doesn’t make sense.
Apart from my general disgust for nvidia as a company, this is a practical real world limitation of a product any linux user should be aware of.
Please don’t buy nvidia cards for linux desktop use.
Fringe “gamers that are slowly transitioning to Linux” won’t know that off the bat, and likely built systems with Nvidia cards… They’re getting screwed big time cause of the Wayland mess and the Nouveau mess. I’m wagering to bet Nvidia will never release reclocking code for Maxwell and Pascal.
I personally might get a Ryzen Mobile 2700U laptop and load it with Linux, or if the ASUS ROG Ryzen 7 laptop has Thunderbolt 3 anytime in the future, I may do passthrough with that laptop if it ever gets a variant with a Thunderbolt port.
I’m not buying computer stuff for ideological reasons - they’re 2nd thought when I have the option to choose. In this instance I don’t have this luxury. All worthwhile gpu renderers (for example Octane, Redshift) are CUDA only.
But even leaving this aside (which I can’t) the state of opensource drivers for AMD is not so awesome too. While Opengl performance is pretty great, OpenCL (and Vulkan) support is rather bad. And it is a must if you’re doing fe compositing (or other task demanding on graphics compute side) . So again at this point in time for me AMD is a no go. I’m not saying a definitive no though - if the tools I need work with AMD cards and the drivers catch up so I could do my work I’ll gladly switch.
It’s not that I’m an nvidia fan (especially considering how pissed I am because of the whole Nvidia vs Wayland issue) - I just buy tools that do the trick for me. And that’s Nvidia for now.
I don’t think that anyone would fault you for your choice. How ever, they would trust that you educate yourself and do the research to know what your are getting yourself into.
Some people feel that they have to be the defenders of the F/LOSS world. It is a phase that we all go through. You seem more educated on the happens and why NV has such a bad reputation in the F/LOSS community. Maybe one day you can transition to hardware supported in the open and still do the awesome stuff that your doing now.
.