My wife and daughter use iPhones, my son and I use Android.
We all currently use Nextcloud to back up our phones/tablets, however, I’d really like to get away from Nextcloud because 1) we increasingly only use it for backing up mobile devices, and that’s kind of a lot of threat surface just for that, and 2) On every device, Nextcloud has been flaky and unreliable in terms of backing files up.
Is there another good option for a self-hosted backup solution for iOS?
AFAIK Nextcloud is going to be it in a mixed iOS/Android environment
You could move out the Android devices to syncthing.
You can also remove the photos component of the Nextcloud to Photoprism (havent done this yet - this is a future project).
This way, you reduce complexity that Nextcloud gets burdened with if you are willing to add complexity to your own server by adding more services running. Unfortunately this also increases your threat surface by trusting more services, developers and apps involved in your own setup. But there is some sort of resiliency in your setup by spreading eggs in different baskets, not just in a single fragile Nextcloud instance.
Its up to you.
Have you considered using apple’s backup utility? Builtin to macos and available through itunes on windows. Linux options were too jank for me, personally, so while it’s not FOSS it does run on your machine.
It creates a whole device backup that can be used to restore to a new device. The archive is easy to find and copy to a central locations, and your family members can have their own password to encrypt it natively.
Only issue is a somewhat manual process, but I use this to backup my families iphones quarterly.
We use iMAZING to backup our iPhones. I run a small Windows VM for that and save the backups to my NAS as well as to my OneDrive.
Older versions of iMazing have been kinda unstable, but since around two years I hadn’t any issues.
License is lifetime and affordable.
“Manual backup processes” might as well be Latin for “things which won’t be done,” at least as far as my teenagers go 
Then have a look at iMazing. Does fully automated backup via WiFi
Syncthing?
I could be wrong but I do remember using Syncthing years ago on iPhone, however after checking on my wife’s iPhone I don’t see a client availble in the app store.
I’ve been using the mobius syncthing app and it’s good, but is limited by iOS. To be sure synchronisation happens quickly you’ve got to either open the app or put the iPhone on charge. I mean, it always gets there eventually, but if you’re waiting for something to sync it usually won’t just happen in the background.
The other option I’ve found, which I’m still using for iCloud photo sync, is to run Windows in a VM with the official iCloud Windows sync app installed. This will get your iPhone files into Windows, and from there you can do whatever with them now they’re in a less locked down OS.
Thanks for the input! When you say you have to have your phone on charge or manually open the app to get the sync to happen, is that by design? Or is that a quirk you’ve encountered?
Also, what “limitations” does iOS place on the app?
When you say you have to have your phone on charge or manually open the app to get the sync to happen, is that by design? Or is that a quirk you’ve encountered?
Yes, it’s by design of iOS. Apps running in iOS aren’t allowed to run in the background as easily as they can in Android, iOS usually puts them to sleep 10 to 20 seconds after you switch to another app even if they’re still processing something. This extreme power saving lets iPhones have smaller batteries and so they can be thinner, but is kind of annoying for background sync apps.
But because that behaviour is all about battery saving, when the phone’s charging iOS is a lot less strict and in that case I’ve seen MobiusSync report that it’s been able to stay running for hours at a time uninterrupted.
This is what I meant by being limited by iOS, just that it isn’t allowed to run in the background all the time.