OK so I’m moving into a new apartment very soon, and has a hobby project I’ve given myself the goal of self hosting as much has humanly possible. (Plex/Jellyfin, DNS,Router,NAS,HTTP,GitLab, Security Camera’s, etc etc)
Though what I’m wondering is, should I go for physical hardware for most of the services or should I invest in one massive virtualization server?
Kinda curious what any of you think so I can come to a better conclusion
I would suggest keeping your router as a separate box. That way, internet stays up even over reboots of your main server, and it also stays up if something (hardware or software) breaks on your main server.
I would also suggest keeping your NAS as bare metal. You could run other things (in containers, VMs, or also bare metal) on the same box, but having your NAS os as bare metal is probably best. Not to say that it is impossible to have a decent virtualized NAS, but to do that well you are getting into passing through a HBA or similar so the OS has direct access to the disks.
What I’d do is trying to give physical hardware to the critical components in the chain so the router and NAS should have their own machine. Then you can use one with multiple VMs/containers for proxy, gitlab, Plex and so on.
I started off with one 24 core 96GB ram machine thinking that I would just virtualize everything. I’ve slowly added more and more machines because it’s really hard to find a balanced machine that works for everything.
The biggest issue has been drives. I wanted SSDs for services and software, but HDDs for storage. Enterprise grade SSDs are expensive, so I opted to go with many small ones, but that are up drive capacity. 3.5" drives are cheap, but take up more space as well.
I am now at the point where I have one machine with a lot of RAM, 32c/64t that I both host services on in one VM, and use as a workstation in another VM. I’ve got another machine with 12x2TB drives that’s acting as a NAS, and I’ve got a 3rd machine with mirrored SSDs as a router, because I decided router redundancy was important.
Thanks for the response!
I will take this into consideration, the router I can see should be standalone, though the NAS would be interesting to do via a HBA card
+1 for storage and router on separate hardware. For services, you can:
A) virtualize/containerize services on the NAS
B) virtualize/containerize services on a 3rd piece of hardware
C) run services on small, dedicated hardware (raspberry pis or similar)
For option C, you’d still probably want to run Plex/Jellyfin on the NAS and maybe the surveillance NVR depending on what you use for that.
Really hard to gauge how much storage or compute you’ll need. Everything on your list, could theoretically run good enough for you on a raspberry pi; or you could find your self happier filling up a box full of disks.
If latter and you’re going for 6-8 drives and more, lookup LSI controllers in IT mode for ZFS on Linux… that way you get both docker and native container support without awkward VM shenanigans.
My setup has a dedicated hardware router, then a R720xd for Truenas and another R720 for VSphere. The 10GB nics for those are like $20 on ebay so you can have really fast storage access between the two, as well as local storage if needed on the R720.
I had a hard time getting the included raid controller flashed into IT mode so I bought preflashed ones on Ebay.
I also highly recommend Nextcloud for hosting your own stuff. It checked boxes for a lot of things for me. Bookmark sync, File sync, Password Manager. Plenty of other good ones too.
I personally run a Ryzen 5 1600 @home and it something around that should fit your setup perfectly as a NAS as well as some VMs. As every one else pointed out, do yourself a favor and buy a really small router sth with a Celeron or better and you’ll be golden with Opnsense or Pfsense (or even OpenWRT 
)
My NAS for example has 7 Linux Container (they’re like VMs without the overhead and their own kernel). One that does SAMBA, another with routhly 30 containers (yes container in a container ;P), another one for freeIPA, another one for backups, to name a few. Most of them have bind mounts to my BTRFS array, so that they have direct access to it without any translation. All that works pretty well in Proxmox.
Most VMs on the other hand are kubernetes nodes. And that all on a small little Ryzen. Just make sure to get enough RAM!!
I wanted similar to you. I went for an x86 box for my router (OpenBSD 6.9, Asrock Rack IMB-191, Pentium G4560, 4GB RAM, Intel NICs), Synology DS218+ for the NAS (I’d prefer self built but Surveillance Station for my 4x 4K PoE Hikvision cams won the day), and a Threadripper desktop for labbing/playing on *nix.
The NAS is only a low end 2 drive, but it self hosts almost our whole house in Docker. The router… routes, and handles firewalling (all hail pf) for our gigabit WAN. The NAS handles network shares of course (afp, ftp/sftp/ftps, nfs, rclone/rsync, smb etc), nginx reverse proxy for the whole domain/network, and lots of Docker containers including:
and anything else I want to play with. It’s not perfect but it’s basically flawless and it’s very easy to administer and backup.
I have had my server up and running for about a week now. Right now Proxmox is the base install with 2 VM’s running. One VM is TrueNAS with a PCI passthrough HBA Card and the second VM is a Docker container with Rancher that is currently hosting Jellyfin, OpenVPN, Pi-Hole, Minecraft and Heimdall.
I’ve yet to dabble with pfsense or opnsense or any other router type OS’s. I’m probably gonna wait until I get a dedicated machine for that.
The only things I need to fix is how to make jellyfin use the Quadro K2200 inside the docker container and how to set that up in Rancher.
also here is the neofetch from the other thread
WOW, sweet…cant see you needing to upgrade soon! Nice choices. Mine is similar but way less powered.
I ran my storage on proxmox via ZFS with a guide from here
I also then ran Plex in a Debian LXC container with a GPU passthrough that has been working out well for me so far. I may move to TrueNAS to make it a little easier to monitor my ZFS pools. It may be the easy way…but I can mess around more once I have some more knowledge.
I do pi-hole, with recursive DNS…I’m not sure what this is but I made it work. Im learning though.
Not nearly as powerful as your rig (which is awesome btw), but the CPU is up for a upgrade… I just have to decide if I want to put the 3700x, or the cheap 3900XT in it. I may do 3700x and use 3900x in my big rig with new gpu.
I do have the same goals as you. Im exploring the posts by @PhaseLockedLoop in this series he did.
I’m liking the push to self host. The next step I’d say in any learning is how to avoid Single Points of Failure. Which can be quote catastrophic. I kind of want to pic @Dynamic_Gravity’s brain on avoiding SPOFs
I do have high availability. A trully unlimited plus plan with T-Mobile. Full hotspot. Good for 350 GB before throttle… In case comshaft shittiest network in the history if networks ever. Goes down
How much do you pay for that?
The hotspot that is?
Extra 15 on my grandfathered 45 dollar a month so 60 bucks a month
I early adopted T-Mobile early earrrrlly in their uncarrier moves. And its been sooooo nice… As a long term customer
Damn, I doubt I could get that rate lol
Nope its an old rate and i stay and i upgrade my own phone … I dont do anything through them
Current phone:
Google Pixel 3 XL
OS: Lineage OS 18.1 self built - Titan M signed - No root
Gapps: Pico
Main Store: Fdroid
Case: Spigen Tough Armor
Screen Protector: THICC tempered glass protector
Very much handle my own deal my dude
Almost all my stuff is self hosted, this includes media and music streaming, my VPNs, my NTP server, My own full recursive DNS server (Top down). The list goes on. I dont even use the LTE towers for accurate time. My NTP server is GPS disciplined. (I use a high narrow band rejection antenna given it sits in an RF crowded area)
@Dynamic_Gravity do you know anyone on the forum who has implemented Secure NTP aka SNTP. Something I try and implement so nobody can tamper with my time
When I stop moving every year I want to start trying to self host it is just a PITA to do much other than a desktop and a few Raspberry pi’s here and there.
A self contained little box with everything you need for self hosting would be a interesting project though
