I’m planning on building a DIY router with 2-3 APs.
I haven’t decided if I should use OpenWRT or pfSense or something else (would love to hear your inputs on that).
If I want it to have mesh roaming capability, what are the things I need to keep in mind?
Get Wifi APs that support mesh capabilities. 
aah ok! I wasn’t sure if I would just be able to use any APs.
I wasn’t sure that, if I’m using a custom OS, I’d be able to use any AP to make it mesh, or I have to buy one that is capable of meshing.
The router OS matters not to the APs & mesh. The router is the router. The APs handle wifi.
For mesh wifi you can go either controller based (makes managing multiple APs easier) or stand alone as long as they have mesh ability baked into them.
Lots of choices for either path.
You have to do the research.
One thing to beware of, is some WAPs require some kind of controller (hardware or software) in place to work.
I recently picked up a (WIFI6) TP-Link EAP610 specifically because it has a standalone mode so I don’t need to fuss around with a hardware controller of docker container. Supposedly it can do mesh connections, but I only have 1 and have never messed around with that sort of setup.
Do you mean mesh or roaming? The word mesh gets used incorrectly a lot so it can make your life difficult if you’re looking for the wrong features.
ah! apologies. I just looked up to be sure, and I did mean “roaming”.
Aah! I see. I misunderstood something fundamental. Thanks for pointing that out
i’m currently considering TP-Link EAP773, and since you own one, I’m just wondering if seamless roaming will work with different models, but same brand (i.e. another Omada AP)?
I’m planning on getting 1 AP for the time being. And would that be enough for a 3500 sqft home?
Regarding the router itself, full sized diy pc a bit overkill here. No showstoppers, but why? All that perf will be mostly wasted as heat 
Consider using purpose build opnsense appliance (more expensive but turn-key and solid), or just buy small form factor computer like this example.
LOL I did notice how lightweight it is when my CPU usage was pretty low after doing pcie_acs_override and PCie passthrough. Makes me wonder how much I’d have been able to push my Jellycloud if OPNsense was also in there…
The total usage of my jellycloud server, Proxmox server, & 2.5Gb Mikrotik switch is around 150W at idle (according to my 1kW UPS)
Yes you should be okay perf wise, but unless space and power are critical considerations, I would go with modular approach and not cohost router and app server at the same device.
Its cleaner and more reliable to have separate host for each functionality. Same idea as all-in router+switch+ap vs separate router, switch and APs.
Imagine misbehaved app choking your all-in-appliance and hosing you entire home network by accident.
yea that crossed my mind. I was play pros/cons with myself for months. Only reason why I stopped was because my cousin was selling his old PC for about $150 (CPU & mobo). Figured I should go right ahead and snag it.