Looking for advice on Hypervisior/storage combination

Operating Systems & Open Source Virtualization
, ,
1

I am setting up a pair of servers in a small business. One of the machines would primarily be a domain server, fileserver, and host several Win 11 VM’s. The second server’s primary responsibility would be for AI and Video processing (having GPU’s). The second server’s secondary responsibility would be to take over services of server one in event of a trouble (Domain, file, and VM’s).

Domain services would be provided by a Windows 2025 VM.

I am attempting to select a core hypervisor/storage platform to accomplish the task. None of the platform contenders are functionally complete, each needing some CLI interaction and architecture “tweaking”. Under consideration are:

XCP-NG
Proxmox
TrueNAS
UnRAID

Microsofts HyperV is not in the mix for two reasons. One, I found the version that I tried (included with SBS2011) to be flaky in running anything but Microsoft products (ie Linux). Additionally, Microsoft has moved the product from free to a pay-to-play license model.

I prefer a drive partition model where partition 1 is a bootable part of a RAID1 array, the OS always boots and provides tools in disaster circumstances. Partition 2 is part of production storage, typically part of a RAID5 or RAIDZ array. Partition 3 is a rolling backup of the partition 2 storage system, data is always recoverable even if only one drive survives. Yes, I run ZFS storage in a partition model with ZFS drive members being partitions. I have found it does not impact performance in any tangible way (disputes will require benchmark data).

The selection process seems to revolve around how much extra work is needed to get me there. I have created a system using XCP-NG and it works, but requires a lot of cli labor (installing and configuring missing packages, samba, zfs…). TrueNAS seems to work fine, but warns vehemently against using the CLI in a way that violates their architecture (storage in a partition, loading packages such as pbzip2). Proxmox appears to want money in a ongoing unknown progression, so I have not really tried them as of yet in a working model. I just learned that UnRAID might work, but I have not tried them as of yet either.

Additional info: Server one is complete (if it has any bearing). Gigabyte MC13-LE3, Epyc 4565P, 96GB ECC, 5 Samsung 4TB 9100 Pro, 4 16TB WD WUH721816ALE6L

I am looking for useful input (information and data) to steer the projects selection of the hypervisor and storage provider.

Thanks in advance.

2

TrueNAS is a great NAS. Not so great Hypervisor.
Proxmox is a great Hypervisor. Not so great NAS.

Virtualizing TrueNAS in production is IMHO a stupid idea.

Window 2025 is… well modern Windows… Every day you will wonder what they fucked up next. I installed one a year or so ago and was pretty happy that there was still a 2022 DC and application server around.

I am not sure I understand your model 100%. Just a few tips for a working setup:

  • for blockstorage use mirrors
  • don’t but data onto blockstorage
  • use RAIDZ2 exclusively for large files
  • local Snapshots offer a great way of snapshotting a VM before installing a update or do any major changes
  • PBS is a great tool for offsite Proxmox backups

what are you taling about?

thinking about running Unraid in production

Seriously?

9100

BTW get different drives from different vendors. In your setup if all Samsung 9100 die at the same time because of a firmware bug (remember Samsung overheat bug?) your pool is gone. If you mix it with a WD Red or something, your pool survives.

Support is important if something goes wrong. Your plattform is pretty new and not much battle proofed. I would seriously consider getting a support contract from Proxmox.

3

Thanks for the reply.

I misunderstood the Proxmox licensing page. I originally interpreted it as the customer having to absorb a large reoccurring expense. I tend to design for long term, walk away solutions. As such I shy away from recurring expenses.

I am attempting to add resiliency at cost in this design. With more money, redundant storage would be presented via the likes of Fiber Channel (actually DAC’s) to redundant compute. That design would require minimum of four machines. That is becoming more attractive the more I look at it.

The thought exercise is to collapse the design into two machines, where they each provide compute and storage. Linked via 25Gbs DAC’s, data is shared/replicated between them. In day to day operations, some specialized functions such as AI and Multimedia processing could exist in one of them.

TrueNAS, Proxmox, and UnRAID are, at their core, a KVM Hypervisor paired with a ZFS storage. In a four machine scenario, your remarks seem to indicate having two TrueNAS and two Proxmox machines. Bringing the design down to two machines is the conundrum. The question becomes what is a better, less painful selection, TrueNAS, Proxmox, UnRAID, or XCP-NG.

In a rather dated machine(s), I have three drives partitioned up as follows (roughly) using linux md:

Partition 1: BOOT, RAID1 mirror. Any drive can boot
Partition 2: OS and Storage, RAID5 ext3. Around 1/3 of the drive.
Partition 3: ext3 filesystem. Used in a round robin backup of OS and data. Around 2/3’s of the drive.

Even in a two drive failure, you get to walk away with something, even if it’s 3 days old. With ZFS and XCP-NG, I have basically recreated the same scenario. I do like using snapshots feature of ZFS.

Using drives from different manufactures is a good suggestion. I have lived through a drive failure in an array while watching its siblings develop errors at the same time.

Windows Server has become a crutch. It is the easiest way of providing a management domain that Windows workstations can join and share resources. I very much miss Small Business Server. Windows Server is just a shell of what once was, and expensive at that.

4

IMHO HA is overrated, but Proxmox offers CEPH. So with a cluster of 3 nodes, you could run a HA Hypervisor.

Trust me, if you use UnRAID in that context, running a HA is above your current pay grade. That is why I recommended Proxmox with a contract to you.

But whatever you choose, real HA is a lot of pain!

If you don’t need HA and you don’t mind running two machines, a Proxmox and a TrueNAS node will offer you the best of two worlds.

1 Like
5

If you want to do hyperconverged XCP-ng, you would use XOSTOR which requires a minimum of 3 hosts with local storage.

TrueNAS can host VMs, but it should not be used for anything in production. It’s nowhere near feature complete or stable as a dedicated hypervisor.

Also, you should have a pair of active domain controllers. They can be configured for DHCP/DNS redundancy so everything on the network stays up when one DC goes down.

1 Like