[SOLVED] Compiling FFMPEG + OBS with NVENC support in Linux (Debian Sid)

I’m on Debian Sid, using an NVIDIA card with proprietary drivers from the Sid repos. I want to enable hardware encoding on OBS - however, the ffmpeg and obs-studio packages currently in the Sid repos were not compiled with NVENC support. This led me to trying the flatpak and snap releases of OBS, but both gave me different issues in detecting my GPU. So I took to compiling it myself with the nvenc flag. I was helped along a great deal by this script from Strider in the Lutris community , but I seem to have hit a brick wall at the last stage. This is why I come asking for help.

I successfully compiled FFMPEG with NVENC support, and confirmed it works as intended. I also compiled OBS with NVENC support, but I noticed I had problems whenever trying to record or stream anything with the hardware encoder - the muxer couldn’t be initialized, no file output would be created, and the only way to stop the “recording” or “streaming” was to kill the process. I discovered that selecting the “Custom Output (FFMPEG)” option in the Advanced Recording tab, choosing h264_nvenc manually and recording actually worked as expected! But there is still the problem of streaming. Since there is no “Custom Output” option for streaming, I’m stuck with just choosing NVENC in the Encoder menu, and not getting any data actually streamed to twitch because the muxer couldn’t be initialized.

What do you think is going on here, and how could I debug the issue? Clearly we’ve established the custom built ffmpeg w/nvenc support works, and we’ve established that OBS recording with custom output works, but the act of OBS calling on obs-ffmpeg-mux is what seems to cause the freeze. I suspect this is caused by a mismatched encoder name, but I’m not sure.

Thank you very much in advance.

info: ==== Recording Start   ===============================================
info: [ffmpeg muxer: 'adv_file_output'] Writing file '/home/ignacio/Video/2019-06-24_21-34-30.mkv'...
info: xcompcap: [window-capture: 'Window Capture (Xcomposite)'] update settings:
        title: OBS 23.2.1-18-g80e9bb4e (linux) - Profile: Untitled - Scenes: Untitled
        class: obs
        Has alpha: no
        Found proper GLXFBConfig: yes

info: ==== Shutting down ==================================================
info: Switched to scene '(null)'
error: X Error: GLXBadPixmap, Major opcode: 151, Minor opcode: BadLength (poly request too large or internal Xlib length error), Serial: 0
info: pulse-input: Stopped recording from 'alsa_output.pci-0000_00_1b.0.analog-stereo.monitor'
info: pulse-input: Got 17073 packets with 3306901 frames
info: pulse-input: Stopped recording from 'alsa_input.usb-Blue_Microphones_Yeti_Stereo_Microphone_REV8-00.analog-stereo'
info: pulse-input: Got 7494 packets with 3291058 frames
info: All scene data cleared
info: ------------------------------------------------
error: X Error: GLXBadDrawable, Major opcode: 151, Minor opcode: 26, Serial: 58595
error: Failed to make context current.
Couldn't initialize muxer
./data/target/scripts/obs.sh: line 4:  7711 Segmentation fault      ./obs "[email protected]"

By the way, since I’m new here - how do you tag posts? I’d like to tag this as “helpdesk”. Or is that something only mods can do?

1 Like

Just going to take a stab in the dark here, but what groups does your user belong to? You can find this out by typing groups into a terminal. You may not have the juice to mess with also.

[email protected]:~$ groups
ignacio cdrom floppy audio dip video plugdev netdev

Those are my user groups.

If what you mean by “the juice to mess with” is that my GPU is not powerful enough - according to NVIDIA’s website my card is capable of using the basic tier of NVENC encoding (GTX 760), and even that would be far better than software encoding for me.

Also, given my original post, I have confirmed NVENC works correctly with my GPU on Linux. The particular problem I haven’t been able to overcome is that OBS cannot correctly initialize its muxer when starting a stream or when recording on Standard settings. I’ve observed recording on OBS with custom settings explicitly telling it to use NVENC (and also directly using NVENC with ffmpeg) works perfectly.

If your system has an also group, you may need to add yourself to that, otherwise, it looks good. Take a look at this, https://gist.github.com/Brainiarc7/8b471ff91319483cdb725f615908286e, and this https://wiki.archlinux.org/index.php/FFmpeg#NVIDIA_NVENC/NVDEC

I done have an NV card so my help is limited.

Sorry for the semi-necro on this thread, but I’d like to help out anyone that comes across this thread on a search engine.

So it turns out that the work-around to streaming with NVENC ended up being very simple. Instead of using the Streaming option in OBS, we use the Recording option again, but we send the output to Twitch’s RTMP server. That’s all there was to it.

So in summary - use the script to get all the dependencies for you (except the NVIDIA SDK, you have to download that one manually and place it in the source folder), and then tell it to also compile OBS alongside FFMPEG. Make a profile for local recording and a profile for recording to the RTMP server when you want to stream. Done.

I slightly modified Strider’s script, you can find this version here:

1 Like

I found a solution to my problem and I hope anyone who googles this sort of thing comes across it here.

All you have to do to end your OBS w/NVENC woes on Debian Sid is just install them directly from the deb-multimedia repo, instead of compiling them yourself. They work as intended without any of the weird workarounds like recording to the RTMP server. Now it works out of the box, just as it should, with no crashes.


I did not know about the existence of this extra repo that you can add to your sources.list, so in case you didn’t, now you know too.