New home server OS recommendations

High-Performance Computing Workstations & Servers
1

I’m about to build a new home server and I’m not sure which OS to choose. I can’t decide between TrueNAS Scale and Proxmox.

I would like to use one ZFS storage pool for VMs and SMB/NFS/iSCSI shares. I’m not planning on using additional L2ARC or SLOG devices, I guess I have enough RAM and SSD performance?
I’m running some VMs with PCIe passthrough devices and some “normal” VMs. Also some Docker containers.

I have experience with Proxmox but not with TrueNAS. I like the idea of TrueNAS having bare metal access for maximum SMB/NFS/iSCSI share performance while also providing VM support and some apps that I would have to install in VM otherwise. But how good is TrueNAS Scale when it comes to VMs with PCIe passthrough and FreeBSD based systems?
Proxmox on the other hand would offer LXC support which would be nice to have but not a deal breaker if it’s missing. But I would have to install TrueNAS within a VM with virtual disks.

Some applications I’m using:

  • OPNsense
  • Frigate
  • SMB Shares
  • NFS Shares
  • iSCSI Targets
  • Home Assistant
  • piHole
  • NginxProxyManager
  • vaultwarden
  • Nextcloud
  • Wazuh
  • phpIPAM
  • checkmk
  • Random Linux/Windows VMs for testing
  • and more

Hardware:

  • CPU: AMD Epyc 8224P
  • MB: Gigabyte ME03-PE0
  • RAM: 192GB Micron RDIMM DDR5-4800 ECC
  • SSD: 4x Micron 7450 Pro 7,68TB (RAID-Z1)
  • NIC: Mellanox CX455A 100GbE (P2P to my workstation)
  • NIC: Intel X710-T4L (PCIe Passthrough to OPNsense)
1 Like
2

I think the reason to go with TrueNAS Scale are the kubernetes apps (if you want to go down that rabbit hole).
Here is how that could unfold for your apps:

  • bare metal
    SMB Shares
    NFS Shares
    iSCSI Targets
  • ootb kubernetes apps
    Frigate
    Home Assistant
    piHole
    NginxProxyManager
    vaultwarden
    Nextcloud
  • VM
    OPNsense
    Random Linux/Windows VMs for testing
  • Unknown, may run as custom k8s app or in VM
    Wazuh
    phpIPAM
    checkmk
    and more
2 Likes
3

the only good way to virtualize TrueNas is to passthrough a HBA. also to get max NIC performance you can even passthrough a NIC.

you are landing in that middle area where you may want to actually give a thought to having 2 seperate physical boxes. one for proxmox and a seperate for a bare metal TrueNas install. IF you do consider this as an option, you should use Truenas CORE as it will have better performance and you will not need to use it for any virtualization.

if you decide to go single box, and you use Truenas Scale, i believe you will find its virtualization a limiting factor at some point. maybe not right away, but it simply is not as robust as a dedicated hypervisor.

3 Likes
4

Ugh, forbidden router…

Just because of this reason, I’d say stay with Proxmox. The reason I don’t like TrueNAS Scale is because of its locked-down approach (AFAIK, I know that was the case at some point, but haven’t looked into it in a while), while on Proxmox, you can mess with the OS all day long.

You’re already doing PCI-E passthrough, so you install TrueNAS Scale in a VM and pass some disks, then use it as a backend storage for proxmox (you’d basically run proxmox and trunas from a single SSD, then everything else from the truenas pool).

I’d personally advise against that. You can run SMB, NFS and iSCSI on Proxmox. And you can even add something like Portainer straight on Proxmox to run containers, or you could run portainer in a LXC container or VM.

I can’t comment on passthrough, since I never used Scale, but as far as BSD VMs go, it should be the same as any plain Debian system. TrueNAS Scale runs on Debian and uses QEMU/KVM (same as proxmox, but I think truenas uses plain virsh if I’m not mistaken, while proxmox uses its own tooling).

But you’ve got an Epyc system. You can just slap however many VMs you want and not need to deal with containers.

The reason I’m a big fan of containers is because I use (and recommend) low-end devices for services and skip VMs altogether (more efficiency). Unless you run hundreds of services on your epyc, it’s probably going to be better you run everything as VMs (or most things).

As far as requirements for SMB, NFS and iSCSI, I’d say it’s more straight forward to do them on Proxmox itself, instead of a TrueNAS VM with storage passthrough. You have the resources, but I find it to be just a waste (and you have more management to do by trying to maintain both).

3 Likes
5

Thank you very much for your replies!

No really. :sweat_smile:

Yes, I want to avoid that. Because I would have to passthrough all my enterprise SSDs and would be stuck with consumer SSDs for my other VMs.

Working fine for me for more than a year now. :smile:

That’s probably what I will end up doing. I don’t want seperate storage pools for VMs and shares because I want both to directly profit from those enterprise drives. That’s why I considered TrueNAS Scale but @Zedicus just confirmed that I won’t be happy with the virtualization. I was already afraid of that. (That’s one of the reasons I want to switch from Unraid)

6

I’d revamp a few things, use FreeBSD and VMs for things like Home Assistant etc.

7

I would definitely not run truenas if you need that many applications. Truenas is great for a nas or a nas with some extra things. This is what i would do:
Proxmox with zfs
LXC Fileserver container for SMB/NFS shares (just pass through the zfs pools)
VM or LXC Docker for applications like home assistant and nginx proxy manager.
VM for OPNsense (dedicated cpu’s/hardware for routing)

If you have an LBA, it’s also possible to run Truenas in a vm and pass it through. People seem to have mixed opinions of it though.

2 Likes
8

That sound like a good plan! I’ll do it that way.

9

Not a fan of the VM:ed “router” solution either and you can also use FreeBSD jails instead of containers in many cases :slight_smile:

10

Eeew, why would you do that, instead of running a dedicated router appliance anyway? You want your router to be a router (and maybe DHCP and DNS, maybe NTP server, but it shouldn’t serve as anything else, making jails kinda… eh… I could see reasons for putting DHCP, DNS and NTP into jails, but they are small software, the potential of these being used as breakin points is really low).

You know I’m a fan of BSD routers, but I can’t get behind jailed routers (and very unlikely to get behind jailed “routing services” either). The reason being mostly about uptime: you don’t want your network going down unnecessarily. You can restart the router a few times here and there for updates (but given how rare the minimal installs of freebsd / openbsd are, that’s going to be a decent enough uptime time). But having your router (VM / jail) restarted because you need to patch your host OS is unacceptable in my eyes.

1 Like
11

Run everything dedicated, it’ll be better… /s

I’d go with proxmox, but I’d do cluster of two boxes + a quorum node.

12

I agree: infrastructure components like firewalls, dhcp, dns, should be on hardware you’re not going to feel guilty leaving on all the time. Consider something like a i5 fanless industrial chassis for being your router/dnsmasq server. You will want to tinker with your big system (change disks, change cards) and when you do that, you don’t want any of your other machines to lack internet, right?

1 Like
13

If I would host services for anyone but myself I would agree but with only one user affected I don’t care.
I’m running an OPNsense VM for a year now without a problem.

1 Like
14

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.