Windows? If so check that the place you are recording to has enough space and that the device which is doing recording has enough VRAM. if you’re using the replay feature.
Hardware Unboxed just recently did a thing on this, and AMD GPUs share resources with the shader cores when encoding, so it hangs on encoding when you have a graphically intensive game like Assassin’s Creed Odyssey. Nvidia GPUs have a separate ASIC section for the NVENC module, but still downclocks when entering NVENC mode.