I’m on Debian Sid, using an NVIDIA card with proprietary drivers from the Sid repos. I want to enable hardware encoding on OBS - however, the ffmpeg and obs-studio packages currently in the Sid repos were not compiled with NVENC support. This led me to trying the flatpak and snap releases of OBS, but both gave me different issues in detecting my GPU. So I took to compiling it myself with the nvenc flag. I was helped along a great deal by this script from Strider in the Lutris community , but I seem to have hit a brick wall at the last stage. This is why I come asking for help.
I successfully compiled FFMPEG with NVENC support, and confirmed it works as intended. I also compiled OBS with NVENC support, but I noticed I had problems whenever trying to record or stream anything with the hardware encoder - the muxer couldn’t be initialized, no file output would be created, and the only way to stop the “recording” or “streaming” was to kill the process. I discovered that selecting the “Custom Output (FFMPEG)” option in the Advanced Recording tab, choosing h264_nvenc manually and recording actually worked as expected! But there is still the problem of streaming. Since there is no “Custom Output” option for streaming, I’m stuck with just choosing NVENC in the Encoder menu, and not getting any data actually streamed to twitch because the muxer couldn’t be initialized.
What do you think is going on here, and how could I debug the issue? Clearly we’ve established the custom built ffmpeg w/nvenc support works, and we’ve established that OBS recording with custom output works, but the act of OBS calling on obs-ffmpeg-mux is what seems to cause the freeze. I suspect this is caused by a mismatched encoder name, but I’m not sure.
Thank you very much in advance.
info: ==== Recording Start ===============================================
info: [ffmpeg muxer: 'adv_file_output'] Writing file '/home/ignacio/Video/2019-06-24_21-34-30.mkv'...
info: xcompcap: [window-capture: 'Window Capture (Xcomposite)'] update settings:
title: OBS 23.2.1-18-g80e9bb4e (linux) - Profile: Untitled - Scenes: Untitled
class: obs
Has alpha: no
Found proper GLXFBConfig: yes
info: ==== Shutting down ==================================================
info: Switched to scene '(null)'
error: X Error: GLXBadPixmap, Major opcode: 151, Minor opcode: BadLength (poly request too large or internal Xlib length error), Serial: 0
info: pulse-input: Stopped recording from 'alsa_output.pci-0000_00_1b.0.analog-stereo.monitor'
info: pulse-input: Got 17073 packets with 3306901 frames
info: pulse-input: Stopped recording from 'alsa_input.usb-Blue_Microphones_Yeti_Stereo_Microphone_REV8-00.analog-stereo'
info: pulse-input: Got 7494 packets with 3291058 frames
info: All scene data cleared
info: ------------------------------------------------
error: X Error: GLXBadDrawable, Major opcode: 151, Minor opcode: 26, Serial: 58595
error: Failed to make context current.
Couldn't initialize muxer
./data/target/scripts/obs.sh: line 4: 7711 Segmentation fault ./obs "$@"