I ran into an issue with the new system I was building that is using an eVGA X299 DARK motherboard, where to my surprise it doesn’t allow me to select the PCIe slot for “Initial Display Device” like other MSI/Gigabyte boards. Supposedly by default the motherboard picks the topmost slot in use as primary display (Radeon VII in this case), however if I mix in an older card like a Quadro K4200 or a 7970, the motherboard instead picks that card as primary display regardless of slot position!?!
Ideally the configuration I want to use is to set PCIe slot1(x16) and PCIe slot4 (x16) as the GPU pass-through devices and had hoped to use a cheap nVidia GT710 in the lowest PCIe slot6 (x8 gen2) for the Linux host. However regardless of what I try, as soon as I designate the primary card as vfio-pci I lose the display for the GT710. Even though I thought it was working as I was using a display attached to the GT710, it appears that is it only working as part of the nouveau driver allowing the primary display device to share it to the GT710. Meaning that if I try to install the nvidia specific driver or set vfio-pci on the primary card, I lose the ability to display anything to the GT710 from that point forward. Is there any way to configure it such that Linux always tries to use GT710 as the primary display device regardless of the other cards, so I can maintain the ideal airflow/PCIe x16 lanes for the pass-through GPUs?
I have tried both MINT 19.2 and the latest Manjaro to the same effect, aside from buying a different motherboard where I can configure that in the bios, is there any thing I can configure to work around this motherboard’s missing feature?
Complain on the EVGA forums, they will listen to this complaint and re-engineer the BIOS. And I agree this is a really big oversight for VFIO. If they can build Prime95-like CPU stress tools in the UEFI, they can add this feature. If you’re getting no traction on the forums, contact Jacob Freeman of EVGA directly.
Tried that first and was told it wasn’t supported on any board without iGPU support and told to request the feature to EVGA support as a request for enhancement. Within an hour I received their response:
Blockquote
Hello,
Thank you for your feedback on our motherboards. At this time, none of our motherboards support a feature where you can specify which PCIe slots give the primary display. By default it would only be the top card which would provide a display. We do appreciate your feedback and we will be sure to pass it along to our product management team to look into with future products and/or possibly future updates.
Regards,
EVGA
… so I don’t think that feature is coming any time soon.
That is why I was hoping for a workaround in Linux. I saw a bunch of old questions on this back in the day of Ubuntu 10.x-13.x related to two nVidia GPU and generating /etc/X11/xorg.conf in the nvidia-setting and then swapping the BUS IDs. Not sure if this will work where the primary card is an ATI, but its worth a try once I get a chance to tinker with the system again.
i have the same problem BUT, i’ve never seen any options in the vmware quest bios. i pretty have to have way to force linux to 1 of the GPUs for display.
Mounted on a motherboard X570: ROG STRIX X570-E GAMING
ATI Vega 64 Gigabyte RX Vega64 8G GDDR5 GAMING OC
Nvidia GTX 1060 MSI GeForce GTX 1060 Aero ITX OC
An extra 4 Ethernet intel I350
OS: Manjaro
This motherboard automatically switch the 2 PCI-E slot to 8X. (when it saw the 2 cards…)
There is no way to choose the main GFX in the bios,
So I try to swap them, but it did not change the main GFX.
Some time the Nvidia is the primary sometime the ATI is the primary.
that make me feel crazy.
At the end I was only able to pass-through the GTX 1060 to my VM, But I want to pass tass-througth the VEGA64
Maybe I should complain to Asus for the motherboard.
normally your bios runs through your pci-e slot from top to bottom(so top slot and downwards if you look at it) to find a GPU, and if none is found it looks on the CPU, which i guess just shows up as a pci-e card, through some kind of voodoo magic.
the Bios then hands what ever it finds over as a “main gpu” to your OS.
so if your what ever card you dont wanna use is placed in the physically highest? slot, it would be detected first and handed over to your OS as primary GPU, and what ever you put in the lower slot as secondary e.g. not what were using to render desktops etc etc.
disabling the GPU the BIOS handed to your OS as “primary GPU”, causes all sorts of funny things, and i wouldn’t recommend it.
just place what ever card you wanna passthrough lower on your MOBO, and it should work.
i tried this once with a APU where i could passthrough the APU GPU, but not the nvidia 750TI(or something), because that was what the BIOS handed to my OS, since it was positioned in the first pci-1 slot.
I dunno if the tech evolved since then it’s been a few years.
After the initial response from EVGATech_JoseC on the support forum stating that:
Hello! Unfortunately there is no option to change the primary display output on the X299 Dark. If this is an option you would like to see, you can always leave your suggestions and comments here https://www.evga.com/support/suggestions.asp
and a similar response from the support suggestion submission, I figured it isn’t likely. This is an OC focused board and I don’t think it is worth their time to add this feature to the BIOS and I doubt Jacob Freeman would think this is a high priority. The board technically supports setting whatever slot you want to use a primary by setting dip switches to disable all other slots, but that defeats what I am trying to do.
Again that is why I was hoping for a workaround in Linux, but I may just have to pick up a different board from MSI/Gigabyte with the newer X299X revision for the whopping +4 additional lanes on CascadeLake ;).
I finally gave up on Manjaro and went back to my default distro Mint 19.2. In case anyone else is trying to figure this out here is what I did:
After a clean install on Mint 19.2:
Installed the nvidia drivers using Administration-> Driver Manager (or install manually from an xterm)
add the appropriate blacklists to /etc/default/grub: GRUB_CMDLINE_LINUX_DEFAULT="quiet modprobe.blacklist=noveau,radeon,ati intel_iommu=on splash"
sudo update-grub
reboot then use sudo nvidia-xconfig which will create /etc/X11/xorg.conf that will need to be update with the desired nVidia device’s BusID
Use lspci | grep "VGA" to find your nVidia card’s BusID and convert the first 2 digits from hex to dec.
(e.g. b3:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1)) means the BusID is PCI:179:0:0 in my example
edit sudo nano /etc/X11/xorg.conf and add: BusID “PCI:179:0:0” to the Section “Device” under VendorName
Finally reboot and your primary display should be from the nVidia card, which may not be the primary display device according to the BIOS, but will be used as such in Linux so you can setup your other display adapters as vfio-pci move to the interesting stuff.
Same should apply with Debian. Vanilla Debian should be a better base for VFIO anyways, because Mint is using a lot of alternate repositories and that could be a headache to debug.
Should be a good learning experience for Debian once you get the hang of it though.
Coming back to this thread cause I just learned that Gigabyte are literally the only ones that allow changing of the Boot GPU to a different slot, even if it’s through the chipset.
If you’re looking for a VFIO board, go Gigabyte.
On X299, the favorites are the X299 Gaming 7 Pro and the X299 Designare EX.
On X399, The X399 Designare is extremely solid, and is the base for the System76 Thelios systems.
On TRX40, the TRX40 Aorus Master looks good.
For all these selections, if you look in the manual, look for “Initial Display Output” in the BIOS options.
What your system is doing is putting the best hardware to vfio first I think. Ifxyou put in a K4200 and it outright ignorse the rad 7, then it recognizes that you’re doing something and is setting up for it. The DARK series has a lot of features like this.
No, this is nothing that the DarkX is doing right to optimize for VFIO as the only way it behaves exactly as described in the manual is when I disable slots with the dip switches. For example I was playing around with just an old 7970 and the Radeon VII, regardless of which card is in the upper most PCIe slot the 7970 is always chosen regardless of its position in the remaining slots. This works fine on my old Gigabyte X99 setup, but I really was hoping to use the Radeon VII on my newer build.
I have tried about every major distro (Manjaro, Fedora, CentOS, Ubuntu, MINT, MX Linux) and was able to get them all working with the display config I want, but it is still a major pain that could have been resolved by the BIOS option as anything that requires selection during boot or changes in the BIOS mean you need to use a KVM to swap displays.
I lucked out an got a 3950X with the Gigabyte Aorus Master X570 combo that I am still waiting on, but at least from the manual PDF the board also supports the display option except I can’t confirm yet.