Bought a Ryzen 5 2400G. need best non-bank breaking motherboard/ram combo for Media server

Entirely up to you obviously, however IMHO if you are going to be using an ATX case, and once built the case isn’t going to be changed - I’d just go ATX.

How about this option? Replace the entire server with a Synology or Qnap 4 or more bay NAS?
As long as it has the software I need? Or do those vendors stop updating the hardware (firmware/software/etc) too quickly?

My 2013 Synology DiskStation still gets updates just fine, that’s not a problem. The single core ARM chip definitely is getting long in the tooth however.

hmm. Im seeing these Celeron Quads like the Synology that Logan was touting. the DSG918 er 981+ or something.

I feel obliged to say that 2400 with ASRock Mini-ITX Motherboard - AB350 GAMING-ITX/AC and any other mobo most likely is unbootable under linux. 4.14 boots with fixed low res/refresh rate but freezes periodically. 4.15 freezes during boot, similar issue reported with 4.16 and 4.17 kernels too.

Does this apply if I run headless? Im on the fence if I should be getting a NAS box or buying the rest of the system components.

Yes same issues regardless of what’s on screen. See this thread for exact issues I encountered: 2200G and 2400G non-bootable under Linux

For now either wait or go with different hardware. NAS box may be fine depending on what your needs will be in the future. I outgrew the pre-built NAS devices quite fast.

Hmm thats kinda interesting.
Good note.