I can't get my 7800xt to work with Handbrake in Fedora (or any linux distro really)

xyguy · June 12, 2024, 1:10am

I feel like I am going crazy but I may also just lack understanding. I have a Sapphire Pulse 7800xt that I purchased over the 6800 mainly for the AV1 encoding magic. I have been running Fedora on my main workstation since I got it (replaced an R9 390, fresh install of Fedora).

OS recognizes it like a champ. I have no issues whatsoever running anything in steam or even running my ollama lab but I cant get it to show up at all for OBS, Handbrake or Davinci Resolve, (Resolve won’t even launch but that’s probably another issue.)

I have tried Linux Mint, Kubuntu and Pop_OS and haven’t had any better luck (with Handbrake anyway.) I feel like I must be missing something obvious. I’m bouncing back and forth to Windows to do my AV1 related tasks when I would like to stay in Fedora as much as possible.

If anyone has any ideas what I might be doing wrong I am all ears.

Currently running Fedora 40

Kernel 6.8.11-300.fc40.x86_64

It appears that Handbrake on Linux doesn’t support vaapi in any form, only AMF through the proprietary AMD driver. Since I’m on Fedora and AMD doesn’t provide a package for Fedora I am going to just search for a new frontend for FFMPEG. Thanks to exovert for pointing that out.

Also thanks to H-i-v-e for pointing out this guide for Fedora https://rpmfusion.org/Howto/Multimedia?highlight=%28%5CbCategoryHowto%5Cb%29 that involves changing default packages to better supported versions from Rpmfusion repos. Following this guide restored all of the encode options I expected to see within OBS. I have a comment with a capture of the solution page in case the link ever stops working.

exovert · June 12, 2024, 7:41pm

Whatever the distribution, this page is my first call on these kind of questions.

https://wiki.archlinux.org/title/Hardware_video_acceleration

what does vainfo say for you -

└ % vainfo
Trying display: wayland
Trying display: x11
vainfo: VA-API version: 1.21 (libva 2.21.0)
vainfo: Driver version: Mesa Gallium driver 24.1.1-arch1.1 for AMD Radeon RX 7800 XT (radeonsi, navi32, LLVM 17.0.6, DRM 3.57, 6.9.3-zen1-1-zen)
vainfo: Supported profile and entrypoints
[...]
VAProfileAV1Profile0            : VAEntrypointVLD
VAProfileAV1Profile0            : VAEntrypointEncSlice

VAEntrypointEncSlice says, at least VAAPI thinks it can encode (to me, and not on fedora);

wayland/x11 (probably the former on fedora, maybe doesn’t matter, maybe does)
if ‘vainfo’ seems to give a good answer - can you force the encoders to use va-api.

edit; maybe not handbrake, perhaps another.

https://www.reddit.com/r/Fedora/comments/156f5zh/handbrake_gpu_acceleration/

Handbrake has no VAAPI support, it’s the stupidest thing. They only support AMF and you need to figure out set up with the proprietary amd drivers.

https://trac.ffmpeg.org/wiki/Hardware/VAAPI

jxdking · June 13, 2024, 3:02pm

Nvidia’s Jensen: “It just works”.

xyguy · June 13, 2024, 3:42pm

Thank you so much for all of this. I think the key to my issue has been vaapi vs amf.

Vainfo is showing encslice for basically all the codecs i would expect including AV1.

Trying display: wayland
libva info: VA-API version 1.16.0
libva info: Trying to open /opt/amdgpu/lib64/dri/radeonsi_drv_video.so
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.16 (libva 2.21.0)
vainfo: Driver version: Mesa Gallium driver 24.1.0-devel for AMD Radeon RX 7800 XT (radeonsi, navi32, LLVM 17.0.6, DRM 3.57, 6.8.11-300.fc40.x86_64)
vainfo: Supported profile and entrypoints
      VAProfileH264ConstrainedBaseline:	VAEntrypointVLD
      VAProfileH264ConstrainedBaseline:	VAEntrypointEncSlice
      VAProfileH264Main               :	VAEntrypointVLD
      VAProfileH264Main               :	VAEntrypointEncSlice
      VAProfileH264High               :	VAEntrypointVLD
      VAProfileH264High               :	VAEntrypointEncSlice
      VAProfileHEVCMain               :	VAEntrypointVLD
      VAProfileHEVCMain               :	VAEntrypointEncSlice
      VAProfileHEVCMain10             :	VAEntrypointVLD
      VAProfileHEVCMain10             :	VAEntrypointEncSlice
      VAProfileJPEGBaseline           :	VAEntrypointVLD
      VAProfileVP9Profile0            :	VAEntrypointVLD
      VAProfileVP9Profile2            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointVLD
      VAProfileAV1Profile0            :	VAEntrypointEncSlice
      VAProfileNone                   :	VAEntrypointVideoProc

Of course Handbrake looks like a proprietary driver thing which unless there’s a nice easy way to switch back and forth or have certain apps use certain drivers I’m not super interested in fully transitioning to the proprietary driver.

In OBS, I messed with the settings a little bit more and found the option to use hardware encoding for h.264. This is using VAAPI also. I only have the option for h.264 acceleration but at least its something.

Looking a little deeper into the OBS docs it looks like by default hardware acceleration requires AMF and the proprietary drivers but there might be some plugins that enable things in a roundabout way. When I get a chance to test them out I will try to report back.

dawe · June 13, 2024, 3:56pm

I can’t speak to Fedora, but on Garuda Handbrake works on my 6900xt with the default amdgpu module.

Do you see “VCN” anywhere in the available hardware presets? Those are the ones that worked for me.

 ╰─λ vainfo 
Trying display: wayland
vainfo: VA-API version: 1.21 (libva 2.21.0)
vainfo: Driver version: Mesa Gallium driver 24.1.1-arch1.1 for AMD Radeon RX 6900 XT (radeonsi, navi21, LLVM 17.0.6, DRM 3.57, 6.9.3-zen1-1-zen)
vainfo: Supported profile and entrypoints
      VAProfileMPEG2Simple            : VAEntrypointVLD
      VAProfileMPEG2Main              : VAEntrypointVLD
      VAProfileVC1Simple              : VAEntrypointVLD
      VAProfileVC1Main                : VAEntrypointVLD
      VAProfileVC1Advanced            : VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointVLD
      VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
      VAProfileH264Main               : VAEntrypointVLD
      VAProfileH264Main               : VAEntrypointEncSlice
      VAProfileH264High               : VAEntrypointVLD
      VAProfileH264High               : VAEntrypointEncSlice
      VAProfileHEVCMain               : VAEntrypointVLD
      VAProfileHEVCMain               : VAEntrypointEncSlice
      VAProfileHEVCMain10             : VAEntrypointVLD
      VAProfileHEVCMain10             : VAEntrypointEncSlice
      VAProfileJPEGBaseline           : VAEntrypointVLD
      VAProfileVP9Profile0            : VAEntrypointVLD
      VAProfileVP9Profile2            : VAEntrypointVLD
      VAProfileAV1Profile0            : VAEntrypointVLD
      VAProfileNone                   : VAEntrypointVideoProc

xyguy · June 14, 2024, 4:28pm

I see those as options, but when I pick them it just sets my output settings to AV1 SVT. Even if i pick VCN H265 or VCN H264 as the “preset”.

I may try garuda to see what, if anything is different.

H-i-v-e · June 14, 2024, 5:05pm

No the presets are always there, the video encoder setting tells you if it is able to do hardware acceleration. You need to replace the vaapi and vdpau libraries with the freeworld ones from the RPU fusion. ave you followed the following page?

https://rpmfusion.org/Howto/Multimedia?highlight=(\bCategoryHowto\b)

xyguy · June 14, 2024, 5:34pm

I had not followed this guide.

I went ahead and did just now but still no luck in Handbrake.

That did get my h265 and av1 options to show up in OBS though. So at least there’s that.

H-i-v-e · June 14, 2024, 6:11pm

I did the reading for you and no the amdgpu driver can’t do it for handbrake. You need the proprietary amdgpu-pro driver and the amf-amdgpu-pro driver too. Otherwise it does not work. What dawe showed is missleading the preset is merely a preset it does not tell you if the support for it is actually there. You can always select the preset and it will fallback to a software decoder.

https://handbrake.fr/docs/en/latest/technical/video-vcn.html

xyguy · June 14, 2024, 7:12pm

Here is a screenshot copy of the original page referred to by the link in the main post just in case the link ever stops working.