[SOLVED] Mystery crash when streaming with NVENC (on Linux)

Software & Operating Systems
, ,
1

I am using ffmpeg 4.2 (compiled with NVENC support) through OBS to stream to Twitch at 720p, 60FPS. There are days when I’m able to stream games for up to 8 hours in a row without any problems or hiccups, but then there are other days when the audience sees a weird encoding glitch and the broadcast cuts off for them - at which point they notify me in chat that the stream died, and I have to stop the recording on OBS and restart it.

My question is, how do I go about diagnosing this? I’ve tried looking at command line encoder settings, bitrate, format, and I haven’t been able to narrow down the problem. This only happens when encoding with NVENC, and has never happened to me when doing CPU encoding. I have also been unable to replicate the problem in local recordings, I only seem to have this problem when sending the recording to Twitch’s RTMP server. Twitch Inspector does not report any instabilities in the broadcasts that go down, although I do notice that the bitrate seems to vary quite a bit - which is odd, given that I’m telling ffmpeg to use CBR encoding through the encoder options.

Here is an example clip of the type of visual glitches the audience sees right before the stream dies for them:
NVENC encoding glitch example

My GPU is a GTX 760, which while old by today’s standards, should be capable of handling this sort of task, no? Specially on non-demanding games. And besides, what bothers me is the inconsistency. Some days it’s fine for hours-on-end, others it crashes not even 10 minutes in. I’m on Debian Sid, and I’m using nvidia-driver 440.44-1 from the Debian experimental repo (And yes, I know experimental branch can be extremely broken, but I had no option left because Sid’s nvidia-driver is older than what the NVIDIA Video Codec SDK requires to compile ffmpeg with NVENC support. The whole reason I had to resort to compiling ffmpeg w/NVENC support myself is that the version of ffmpeg and OBS in the debian repos does not have this flag turned on).

For context on the compiling process and how I stream, I made this post a few months back detailing the problems I had and the solution I found:
https://forum.level1techs.com/t/solved-compiling-ffmpeg-obs-with-nvenc-support-in-linux-debian-sid/144288

2

Yeah, that’s a VERY old implementation of NVENC, and doesn’t use B-frames.

The driver may be too NEW for that NVENC encoder. But that driver also has a inflated shader cache issue so you could be running out of RAM.

Try the Vulkan beta instead. Failing that, you could look into the newer lower priced EVGA RTX 2060 KO to replace your 760 for those brand spanking new Nvidia drivers. Turing NVENC is FAR superior to Kepler NVENC.

3

Apologies for the necro, but I found a solution to my problem and I hope anyone who googles this sort of thing comes across it here.

All you have to do to end your OBS w/NVENC woes on Debian Sid is just install them directly from the deb-multimedia repo, instead of compiling them yourself. They work as intended without any of the weird workarounds like recording to the RTMP server. Now it works out of the box, just as it should, with no crashes.

https://www.deb-multimedia.org/dists/unstable/main/binary-amd64/package/obs-studio
https://www.deb-multimedia.org/dists/unstable/main/binary-amd64/package/ffmpeg

I did not know about the existence of this extra repo that you can add to your sources.list, so in case you didn’t, now you know too.