[Ubuntu 22.10]Regular but seemingly at random graphics slowdowns on Xorg but not wayland

I’ve been having this issue ever since I build this AM5 machine and Im kind of at my wits end.

Particularly while playing java minecraft (but also in other games) the game will randomly drop to 2-4 fps for minutes at a time before returning back to the usual expected framerate(even in a void world). This happens 4-7 times every 10 minutes(or so) with no discernible pattern.
Needless to say gaming isnt really fun like this. During that time everything slows down immensely(desktop, firefox, task switching, all other applications…), when it ends everything else is fine too.
This does not happen at all when booting into a wayland session, which makes me think it may be a configuration thing?

Using mango HUD I have confirmed that temperatures, usage % and frequencies do not change notably when the slowdowns happen(if anything they appear to go down slightly).
This machine is capable of running the same minecraft void world at around 500fps uncapped when on wayland.

Being permanently in wayland is unfortunately not a solution either because nvidia and their old drivers make that an awful experience. And I really dont have the money to upgrade my GTX 770.

Im now on Ubuntu 22.10, but it was also an issue on 22.04(why I updated to 22.10 in hopes of newer kernel fixing things, but obviously that didnt happen)
Searching online yielded nothing.

My system:

  • Asrock B650E PG Riptide WiFi (new)
  • AMD Ryzen 7600x (new)
  • 2x Crucial DIMM 16GB, DDR5-4800
  • NH-D15 (from previous build, known good and working)
  • Seasonic Prime TX-750 750W PSU (from previous build, known good and working)
  • GTX 770 (from pevious build)
  • Mushkin SSD