There is a clear issue with Linux kernel where VRR disrupts GPU’s vram frequencies to where certain frequencies like 682 mhz will lead to a freeze or disruption of some kind. So if you’re playing a game and the GPU hits 682 mhz it results in your fps tanking all way down to 30 fps and lower, and vram speed will be locked at 96 mhz. Turning VRR off and on is a temporarily fix, though it introduces another (1000 mhz vram speed all the time), but unfortunately the issue will happen as long as the GPU needs to use certain frequencies, that kernel has a issue with for some reason (6.4, 6.11.8, 6.12). 6.11.5 seems to be fine.
If you use CoreCtrl and try set your vram speed to a frequency that is a problem it will lead to a black screen freeze (restart necessary). In games you will get black screen or general game freezes, because as you switch out and into them the vram speed fluctuates - more likely to hit the problematic frequencies.
Changing your refresh rate to higher than 60 will help, higher refresh rate = higher vram speed, but it DOES NOT solve the issue. 682 mhz is one of a myriad of frequencies that is causing a issue with VRR for certain of the kernels I mentioned.
I think what I described is related to issue you’re having. The frequencies that cause issues may variate from one system to another.