Its late for this, but as a thing I really want to do I want to build an OS for myself. Maybe more than me can make it because I know for a fact that if I do it alone it'll be 83 years before people see it. My essential idea is:
Based on OpenDarwin
Has a Linux Subsystem
OSX binary compatibility
HFS/+, XFS, JFS, EXT4, ZFS
I don't have much more than that. It was a spur of the moment idea 10 minutes ago but I have wanted to do something like this for something of.... Probably 5-7 years. However old the account is at OpenSuse Studio.
As far as I'm aware there's no OSX binary compatibility for Linux operating systems yet. Keyword "yet". They're working on it. It's not as simple as just sticking it on a bullet list.
Everything else seems do-able, I guess, though I'm not so sure how OpenDarwin and the Linux kernel are gonna... merge... Though I really don't know that much about either of them, so that's probably my lack of experience showing there.
AMDGPU should work with the Linux kernel from here on out.
Looks like OpenDarwin has been dead since 2006, how about PureDarwin or even FreeBSD. AMDGPU is in version 12. The best you can get right now for OSX binary compatibility is Darling which currently only works with CLI stuff. Debian/kFreeBSD may also be an option due to it essentially being a Linux distro with the FreeBSD kernel.
Its just that the .app packaging is clean and it would make the code base wider. So like hackintoshing minus OSX. Don't have to pay a billion dollars to dev on the platform when theres an open source version.
OSX also has some features I've never really seen before. Such as formating even if the drive is active. Even the boot drive.
Take a look at TrueOS. It's a FreeBSD distro with the latest kernel version, 12 and definitely uses AMDGPU, it works with my R9-380 perfectly. The Lumina desktop environment looks like it's from 2008 but you can install other DEs. Maybe that could be the place to start.
You can still use the open source parts of OS X (now macOS) to create most of an operating system. The hard part is not the kernel or the base operating system even, but the libraries. Apple provide a wide range of libraries with the operating system, upon which most applications are built. These include libraries for user interfaces, system interaction, DSP, image manipulation, and much much more. Most of the higher level libraries are kept closed source. These are where Apple really differentiate from the competition. Someone of low moral fiber might get away with borrowing the binary libraries from a donor Mac, but this is in direct violation of your desire for an open source solution.