Return to Level1Techs.com

[Arch] Gnome + Wayland with Intel, Nvidia, and AMD in One System? Or X11 GPU Offloading?


#1

Hi all,

I’ve been doing some digging around and I’m pretty sure at this time that it can’t be done because of Mutter but just in case I missed something obvious in my configurations I’m asking for some help. I’m trying to get Wayland running again with Gnome after I added an AMD GPU to my system and I wanted to do some comparative testing with my new high refresh rate monitor since I noticed prior to adding the card and running Wayland on the Intel chip that even though it says 100Hz both in Display options and on the monitor OSD, it’s definitely not 100Hz. Adding the AMD card has forced X11, which is displaying 100Hz correctly but sometimes I can see a “scan line” when watching videos or playing games with or without using Looking Glass, the tearing caused by that is noticable so I leave it at 95 Hz and it seems to not do that but I still have some dropped frames with Chrome/Chromium, especially on 60 FPS videos though it seems Firefox is fine. A while back I had similar issues with dropped frames on X11 and switching to Wayland fixed it, hence why I wanted to try it again, but I didn’t have a high refresh monitor at that time.

I mainly want to see if it will fix the tearing at 100Hz but another big reason I added another GPU was because the Intel iGPU cannot keep up with the demand from Looking Glass to display that many frames at 1440p, intel_gpu_top showed it was basically pegged even once overclocked. I have a few AMD cards lying around from when I used to mine Litecoin so I figure why not try to use one? Yes I realize that I am not benefiting from Gsync by doing this but I am trying to achieve the best of both worlds, I’m waiting for the L1T 2 monitor KVM switch to ship because my current KVM switch cannot do 100Hz so for the games that I can get 60+ FPS in anyway or just generally using Windows I’m very happy using Looking Glass and will 100% be using the Gsync capabilities of the monitor when the situation calls for it. I’m using the KVM switch for USB only right now.

Mobo: Asus Z270E
CPU: i7 7700K w/ HD 630
Top slot: GTX 1080Ti - Used for vfio passthrough only, no Nvidia drivers installed, noueveu blacklisted
Middle slot: R9 280X
Monitor: Asus PG348Q (3440x1440, 60-100Hz)
Linux 4.14.12-3 (linux-ck, I added ACS override patch but not i915 VGA arbiter from linux-vfio)

GPU placement is restricted because I run a hardline watercooled build so unless I drain the loop and make new tubes I can’t move the hardware around (AMD card still air cooled), but if I know I can get it working I have no problem spending some time on that. I can’t move the AMD GPU to the bottom slot right now because there’s a tube in the way but I don’t think that’s a factor here.

I cannot disable the Intel iGPU from the BIOS, it must be set as the primary otherwise when I boot the Nvidia card cannot be properly grabbed by vfio, and I can’t pick a secondary card so it will use the top slot, the options are Auto, iGFX, and PEG. Unless I aggressively blacklist i915 it will still be loaded when GDM starts (which starts with Wayland when i915 is loaded). Preventing i915 from loading disables the option of logging in with Wayland and I had to manually configure X11 to use the AMD GPU due to multiple IDs otherwise GDM would not start.

00:02.0 VGA compatible controller: Intel Corporation HD Graphics 630 (rev 04)
01:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1)
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X]

/etc/modprobe.d/blacklist.conf

#Ignore nouveau
blacklist nouveau

#Ignore radeon so we use amdgpu instead for sure
blacklist radeon

#Ignore the intel driver
blacklist i915
install i915 /bin/false

/usr/share/X11/xorg.conf.d/20-amdgpu.conf

Section "OutputClass"
	Identifier "AMDgpu"
	MatchDriver "amdgpu"
	Driver "amdgpu"
EndSection

Section "Device"
	Identifier "Screen0"
	Driver "amdgpu"
	BusID "PCI:2:0:0"
	Option "DRI" "3"
	Option "TearFree" "on"
EndSection

/etc/mkinitcpio.conf

MODULES="dm_mod vfio vfio_iommu_type1 vfio_pci vfio_virqfd amdgpu nct6775"

Jan 08 21:35:36 archlinux kernel: Command line: initrd=\intel-ucode.img initrd=\initramfs-linux-ck.img root=/dev/mapper/lessaj--desktop_lvm-root rw acpi_enforce_resources=lax intel_iommu=on intel_iommu=igfx_off pcie_acs_override=downstream default_hugepagesz=1G hugepagesz=1GB hugepages=12 quiet pci=noaer radeon.si_support=0 radeon.cik_support=0 amdgpu.si_support=1 amdgpu.cik_support=1

I have added MUTTER_ALLOW_HYBRID_GPUS=1 to /etc/environment which will allow me to log in to an X session on the AMD card from a Wayland GDM, but of course I can’t see GDM unless I move one of my monitors from the AMD card to the iGPU because it is defaulting there and a Wayland session will just load on the iGPU.

From what I can tell from researching, the main reason Wayland is not available is this:

1441:Jan 08 21:35:41 lessaj-desktop gnome-shell[566]: Can't initialize KMS backend: could not find drm kms device

However I can also see the below occuring prior to the above, which makes me think I’m just missing something:

825:Jan 08 21:35:36 archlinux kernel: [drm] amdgpu kernel modesetting enabled.
828:Jan 08 21:35:36 archlinux kernel: [drm] initializing kernel modesetting (TAHITI 0x1002:0x6798 0x1458:0x3001 0x00).

.
.
.
Having said all that, if it is not possible at this time to use Wayland with the AMD card given the rest of the hardware in the system, is it possible to configure X to offload work to the AMD card and display it on the Intel iGPU? This would allow me to actually be able to see the boot process, not have to redo my loop, achieve better framerates with OpenGL, and possibly allow 100Hz correctly as well but I’m still unsure about that.

Any and all help/advice would be greatly appreciately, if any additional info is required I can certainly provide that.


#2

Do you have another monitor attached? If so, Gnome/mutter locks the actual refresh rate to that of the lowest refresh rate display on the system.


Side note, this is quite long winded, which is probably why you aren’t getting much response.


#3

Yes I do I have 2 monitors and the other is only 60, I didn’t know that! Okay well at least now I know how to fix that, but the Intel iGPU still isn’t up to the task for what I need it for so still stuck trying to figure out how to get Wayland to work using the AMD card. I think I might need to move it to the top slot and set PEG as the primary graphics and then it would work but that means draining my loop even just to try it. :frowning:

I know, it’s hard to be concise and detailed while keeping it short… Working in an IT support role there’s nothing that irks me more than lack of detail when reporting issues so I’d rather include more detail, as long as it’s relevant. :slight_smile: I couldn’t find many posts about the KMS error and resolutions, and the ones I did find weren’t that similar to my situation, mostly laptops with Optimus/Prime. I kept seeing mention of something similar to nvidia-drm.modeset=1 but I don’t know if there is an equivalent for amdgpu, couldn’t find anything…


#4

I get that. Sorry. I hadn’t had my coffee yet and was walking into a meeting as I responded, so I didn’t have time to read the whole post.

So, as far as multi-gpu wayland, on Fedora, it just works out of the box. You might want to start there.

As far as which GPU does the heavy lifting, I do believe it’s either the primary GPU (the way you mentioned) or the GPU that the display is connected to. I’m not entirely sure.

I’m not aware of an equivalent either.

I used to have a 280 and it was a real piece of work when it comes to Linux. I’m not sure what to say about those KMS errors.

Saw this thread and thought you might be interested in it if you haven’t seen it already:

Spoiler: try booting a vanilla kernel.


#5

I have some new SSDs that are going into my server, I can use one temporarily as a test with Fedora to see if it works out of the box, just going to end up wiping it and cloning the existing drive anyway. I think it does pick up the primary GPU from the mobo since I saw that when it tried with the Nvidia card as primary vfio would not bind, the error was something like Unable to read PCI header. I forget exactly, only saw it once and not in front of my machine to check previous journals. If I don’t blacklist i915 even without a monitor connected it loads GDM there.

I did come across that thread but not quite the same issues as me. I will try the vanilla kernel that ships with Arch tonight, I don’t really expect any different results due to the above noted except that I won’t be able to start my VM, cards are in the same group and need ACS patch. Worth trying anyway. :slight_smile:


#6

No dice with vanilla kernel and no dice with Fedora. I actually can’t seem to get amdgpu to work at all with Fedora, I can see the module is loaded but the monitors don’t come to life at all. If I load radeon instead they do show the Fedora splash but GDM doesn’t come up until I plug in to the Intel iGPU. I also can’t seem to blacklist i915 even with install i915 /bin/false in blacklist it still loads. Also these options don’t seem to work:

Jan 10 23:13:32 fedora kernel: PCI: Unknown option `radeon.si_support=0'
Jan 10 23:13:32 fedora kernel: PCI: Unknown option `radeon.cik_support=0'
Jan 10 23:13:32 fedora kernel: PCI: Unknown option `amdgpu.si_support=1'
Jan 10 23:13:32 fedora kernel: PCI: Unknown option `amdgpu.cik_support=1'

At one point I at least had it where I could see something on the other monitor but it looked like this… Cursor was fine though haha.


#7

Damn, that’s unique.

Are you running 4.14 kernel?

Did you rebuild your initrd? (dracut -f --kver $(uname -r))


TBH, I didn’t really expect much from a vanilla kernel, but I did want to verify the problems weren’t caused by kernel, since that can sometimes be the case.


#8

Yes first thing I did was update the kernel, it shipped with 4.13.9 I think, updated to 4.14.11. Unless I unplugged cables from my other cards even trying to install it was flickering because it couldn’t figure out what to send output to I think.

Yes I did, I also did dracut --add-driver "amdgpu" to make sure it was there and verified with lsinitrd /boot/initrd-4.14* | grep amdgpu (going off memory here). I still have it installed on the spare disk but I put my system back to normal after I gave up.

I think I’m going to have to bite the bullet and drain my loop to move the AMD card to the top slot and set PEG as the primary to see if that works. I can work on that this weekend.


#9

Yeah, that might be what’s needed. I’m not really sure where to go from here. Sorry!


#10

No worries, appreciate the help anyway! Will report my results.


#11

Unfortunately spent the entire weekend converting my server to RAID1 with larger SSDs instead of just one day like I expected so I didn’t get any time to work on this this weekend. I just came across this article this morning from Phoronix, maybe there is light at the end of the tunnel anyway… Maybe during the week I’ll work on it.

https://www.phoronix.com/scan.php?page=news_item&px=Weston-4.0-Features