Also, consider using Wayland over Xorg. A lot of infrastructure is moving to Wayland which is, in general, have a much better architecture beneath the surface - but a lot of edge cases are not covered and some will never be covered.
Still, Wayland might give you better luck.
So I sat down with Ubuntu LTS (It uses X11 so I could use xrandr) and played with a bunch of different timings and resolutions.
Based on my testing, for some weird reason, Linux currently hates 1680x1050 as a resolution and to some degree dislikes 16:10 aspect ratios as a whole.
I tried CVT, CVT reduced and GTF timings, I tried shifting them around but nothing changed in any way. All 16:10 resolutions would fail to work on the monitor, and 1680x1050 still did not work on my main monitor.
So I played around with weird resolutions and fiddling with overscan (1080p looked pretty good but had parts just off screen).
Since I knew 16:9 worked, I adjusted to a 1872x1050 which matches my vertical resolution and it looked worlds better than any other attempt and my monitor auto-scaled it to fit on the screen. However, it destroyed the aspect ratio (obviously).
I was about the settle there as good enough but then realized “Linux hates 16:10”.
I threw in 1688x1050 the smallest difference off and it worked just fine.
Actual insanity
I was leaning towards Wayland, the only issue is it’s harder to find good info on how to easily play with custom resolutions outside of X11 and xrandr.
However, now I have something (un)reasonable nailed down for me to try throwing something together.
If you have any good guides to dealing with Wayland and custom resolutions I’m open.
TL;DR
Linux hates 16:10 but will take 16.08:10
This monitor doesn’t seem very sensitive to timings/resolution so something really weird is going on.
TL;DR^2
1688x1050 works and seems to have tolerable image quality.
Glad you have found a workaround you may be able to live with.
You never mentioned what kind of hardware you are working with, but I am starting to suspect some weird driver bug here. What GPU is powering this beast of a monitor? 
The only other thing I would check just for sanity’s sake is to use a different HDMI cable, and preferably one with a different revision number than the one you are using now.
I mentioned elsewhere but not in the OP (like I probably should’ve).
I’m running an RX 7800XT and I’ve tried at least 2 HDMI cords and 1 DP-HDMI cord.
Yeah, I’m leaning towards obscure driver bug too. Probably unlikely for many people, especially in the dev teams, to have a monitor like this with a card this new.
I have two old 1680x1050 monitors plugged into one of my docking stations that work just fine… probably around the same vintage as yours, maybe a hair newer. And steam deck is 16:10!
What an annoying bug!