Home server layout and resource management

Hello. I need help with a project that is well over my head as I am still relatively new to server management. But I have played around with my home network, Unraid and built every computer in the apartment. I have little experience with Linux and resource management. Any help for this is greatly appreciated, Thank You.

Server Specs;

CPU: I9 12900K

MB: Z690 ASUS Prime DDR4

RAM: 128GB DDR4 T-Force 3200mhz (4x32GB)

GPU: ARC A380

A PCIE HBA 2 mini SAS to 8 SATA and a NVME HBA to 4 SATA

Blu-ray Drive

Icy-Dock 5.25” to 6x2.5”

Dual 2.5Gb nic

PCIE to dual NVME card

Storage;

3x14TB Seagate Exos 3.5”

4x1TB Crucial 2.5”

1x1TB WD Blue 2.5” (old but unused)

2x120GB Inland 2.5” (planned for proxmox on atleast 1)

2x1TB NVME (1 Sabrent, 1 WD Black)

All in a old NZXT Hush 2.

My plan is to install proxmox as a hypervisor on one of the 120GB Inland drive and possibly have the second as backup. Atleast one 1TB 2.5 drives for VMs. One of the VMs for HexOS for for NAS software, using the 3 14TB drives for bulk, the 2 1TB NVMEs for redundant level 2 ARC, and I don’t know how many 2.5 drives I should use for video or apps I want to run in HexOS. Atleast one application is ARM, Automatic Ripping Machine and Jellyfin or Plex for video streaming with the A380. Other VMs under Proxmox are HomeAssistant, CasaOS (game server management), and other operating systems or containers to play around with. If there is any advise that can be given or better layouts, anything is appreciated. Thank you in advance!

Seems fine. But you will need to check to make sure your motherboard supports PCI bifurcation for the PCIE to duel NVME card, if needed.

I would use the NVME slot for NVME and not an NVME to 4 SATA. Use the PCI slot for more HBAs if more SAS/SATA are needed.

The 1TB 2.5” drives might not be worth the hassle unless you have a need for them. You could RAID them, then use them as the OS drives for the VMs.

Once you get ProxMox loaded, you can RAID the other drives (if needed) and then split them up for the VMs depending on need. Media/backups on the spinning rust, VMs on the 2.5”, databases and anything that needs performance on the NVME.

I have not used HexOS. Not sure if it’s recommended under Proxmox. Worth looking into.

On my server, it runs Proxmox with 1 main VM under it (Ubuntu Server), and that has all the storage on it and runs all the docker containers. No pretty GUI, but also no GUI to get in the way.

1 Like

From what I’ve seen the few people that have ran HexOS as a VM say it’s running fine. A few others before those say in theory it should work fine as it’s just a layered GUI on TrueNAS Scale.

I also can’t use the PCIE to NVME card as there is no slots available for it that are wired for 8x. I do have 3 NVME slots in total. I just wanted something easy to pass through one slot vs two. My design mistakes for failing forethought…

What are you planning to use HexOS for?

If it’s just file sharing and you are passing though the hardware HBA, then it’s a bit overkill and a box inside a box setup. If you need to access that storage from the VM/docker, then it’s going to be an NFS/block device mount across VMs and the network layer adding another layer of complexity and overhead.

Proxmox has ZFS support like TrueNAS. You can do your physical drive setup there and pass that to the VM and use NFS/Samba to share it out on the network, and any docker containers running on that VM have direct access to the files without overhead.

TrueNAS/HexOS/OpenMedaVault does provide a nice GUI for share and user management and is my choice for a standalone appliance or a business with lots of users and AD/LDAP. But most home setups have 1-5 file shares on the home network and a few user accounts. Less need to use all the features of a dedicated file sharing VM.

Nice homelab in the works, I see. I just did much of this, albeit with 2 TrueNAS physical appliances running on older Cybernetics MiSAN D v4 devices from 2011 :slight_smile: and 3 physical ZimaOS boxes made from Dell business tower ewaste. I also, after much frustration with ARM, built DiscoBall (GitHub - Jazsmoosh/DiscoBall · GitHub) as an intermediary layer between MakeMKV, HandBrake, and the files final resting path. It will in most instances properly name things with a higher success rate than ARM, upon which it is build with the ARM MIT open-license. An omdb API key is free and required for this to work. Otherwise, CasaOS is now ZimaOS. I would recommend using version 1.5.4 as there are a number of new USB and HDD-read issues with the update 1.6.* in the wild. Interestingly enough, in 1.5.4 there is a bug that when you go to pay then close the pay window, it just licenses itself. So there’s that I guess.

All of that said, my local model that assisted with all of my finer BASH needed to get everything working locally eactly how I want has this to say on the matter.

==========================================================================

THE FOLLOWING TEXT IS LLM GENERATED

==========================================================================

Your hardware is more than sufficient for this. The primary goal should be avoiding three different platforms all trying to manage applications, storage and virtual machines simultaneously.

I would structure it like this:

Proxmox VE
├── TrueNAS SCALE or HexOS VM
│   └── Bulk storage, snapshots, SMB and NFS
│
├── ZimaOS VM
│   ├── Automatic Ripping Machine / MakeMKV
│   ├── HandBrake
│   ├── DiscoBall
│   ├── Jellyfin or Plex
│   └── Game-server and other Docker applications
│
├── Home Assistant OS VM
│
└── Experimental VMs and containers

That preserves almost everything you wanted while giving each layer one clear responsibility.

Proxmox should remain the foundation

Install Proxmox directly on the machine and use the two 120GB Inland SSDs as a mirrored boot pool.

The second SSD should not merely sit unused as a theoretical backup. A ZFS mirror allows Proxmox to continue operating after one boot drive fails. You still need a separate backup of the Proxmox configuration and important VM data.

I would then use the two 1TB NVMe drives as a mirrored Proxmox VM-storage pool.

That NVMe mirror should hold:

  • The ZimaOS virtual disk

  • ZimaOS application configuration

  • Active ripping and encoding workspace

  • The TrueNAS/HexOS boot disk

  • Home Assistant OS

  • Other VM disks and containers

Do not use the NVMe drives as “redundant level-two ARC.” L2ARC is an optional ZFS read cache, not another RAID level, and it does not normally need redundancy. With 128GB of RAM and primarily large sequential media files, those drives are much more valuable as protected VM and application storage.

TrueNAS SCALE versus HexOS

HexOS is based on TrueNAS SCALE, so either can perform the NAS role. The disk-layout and HBA-passthrough advice is substantially the same for both.

For this specific design, I would personally use TrueNAS SCALE as the NAS VM.

That is not because HexOS is bad. It is because:

  • Proxmox is already managing the virtual machines.

  • ZimaOS is already providing the friendly application interface.

  • The NAS VM only needs to manage disks, ZFS, snapshots and shares.

  • Adding HexOS introduces another convenience and orchestration layer where plain TrueNAS is already sufficient.

If the HexOS interface is one of the reasons you are excited about the project, you can absolutely keep it. Just treat HexOS as the NAS and storage appliance—not as a second application server or hypervisor.

Do not install both TrueNAS and HexOS. Pick one.

Bulk storage

Pass the entire eight-port SAS HBA through to the TrueNAS or HexOS VM.

The HBA should be operating in IT/JBOD mode so that ZFS sees the actual disks directly. TrueNAS specifically recommends avoiding hardware RAID in front of ZFS and using HBA or passthrough operation instead.

Create the initial bulk pool from:

3 Ă— 14TB Seagate Exos

With the hardware currently available, that would likely be a three-disk RAIDZ1 pool:

Approximately 28TB decimal capacity
One-drive fault tolerance

That is reasonable for a predominantly replaceable movie and television library. Anything irreplaceable still needs another copy elsewhere.

If you eventually add a fourth 14TB drive and the data is important, RAIDZ2 would provide stronger protection. I would not delay the entire project solely because only three drives are currently available.

The TrueNAS/HexOS VM should export:

  • NFS shares to ZimaOS

  • SMB shares to desktops and other network clients

Do not create large virtual disks on Proxmox and present those to TrueNAS as the main storage pool. Pass through the physical HBA and let the NAS VM own the disks directly.

ZimaOS should own the application stack

CasaOS has evolved into ZimaOS, with ZimaOS adding broader hardware compatibility, disk-management improvements and an updated operating-system model. There is no reason to deploy a separate CasaOS VM alongside it.

Use one ZimaOS VM for:

  • Automatic Ripping Machine

  • MakeMKV

  • HandBrake

  • DiscoBall

  • Jellyfin or Plex

  • Game-server management

  • Other Docker and Compose applications

This keeps the complete media pipeline together:

Disc
  ↓
MakeMKV / ARM raw rip
  ↓
HandBrake completed encode
  ↓
DiscoBall identification and organization
  ↓
Jellyfin or Plex library

DiscoBall should watch the completed HandBrake output, not the raw MakeMKV directory. That prevents it from trying to process incomplete or unencoded files.

A clean ZimaOS path layout could be:

/DATA/AppData/makemkv
/DATA/AppData/handbrake/config
/DATA/AppData/discoball
/DATA/AppData/jellyfin

/DATA/Media/MakeMKV/output
/DATA/Media/HandBrake/output
/DATA/Media/Library

The application configuration and temporary working directories can live on the mirrored NVMe-backed ZimaOS virtual disk.

Mount the TrueNAS NFS media share at:

/DATA/Media/Library

DiscoBall can then move the completed and organized files into the NAS-backed library, after which Jellyfin or Plex sees them immediately.

This gives you fast local NVMe workspace while encoding without placing partially processed files into the permanent library.

Blu-ray-drive passthrough

The optical drive needs genuine low-level access from the ZimaOS VM. Passing it as an ordinary virtual CD-ROM may not be enough for MakeMKV or ARM.

ARM expects access to the optical block device and its related SCSI-generic device. These commonly appear as:

/dev/sr0
/dev/sg0

The actual /dev/sgX number must be identified on the running ZimaOS VM. ARM’s documentation specifically covers passing physical optical drives into QEMU/libvirt virtual machines, which is directly relevant to a Proxmox deployment.

The cleanest approaches are:

  1. Pass through a dedicated SATA controller containing the Blu-ray drive.

  2. Use a compatible USB optical enclosure and pass the complete USB device into ZimaOS.

  3. Pass the required SCSI devices through individually and confirm that both /dev/sr0 and the corresponding /dev/sgX appear inside ZimaOS.

Do not pass through a motherboard SATA controller if it also contains either Proxmox boot drive.

Once the drive appears inside ZimaOS, expose both required devices to the ARM or MakeMKV container.

Intel Arc A380

Reserve the 12900K’s UHD 770 integrated graphics for the Proxmox host and pass the complete Arc A380 through to the ZimaOS VM.

In the motherboard firmware, enable:

  • Intel VT-d

  • IOMMU

  • Above 4G Decoding

  • Resizable BAR

Proxmox requires the passed-through device to be in a usable IOMMU group. Confirm the A380 and its associated functions can be assigned together before building the final ZimaOS VM.

Inside ZimaOS, the Arc card can be exposed to Jellyfin, HandBrake and the associated media containers.

Jellyfin supports Intel Arc hardware transcoding through QSV and VA-API. Its documentation recommends QSV for mainstream modern Intel GPUs and notes that Arc A-series support under Linux requires kernel 6.2 or newer. Resizable BAR is also recommended for best performance.

Verify the ZimaOS kernel and Intel media drivers before assuming hardware acceleration is operational. Confirm with an actual Jellyfin transcode rather than relying only on the device appearing in the interface.

The remaining SATA SSDs

I would not assign every disk merely because it is available.

Get the basic environment stable first:

  • Mirrored 120GB Proxmox boot pool

  • Mirrored 1TB NVMe VM pool

  • Three-drive 14TB NAS pool

Leave the four matching 1TB Crucial SATA SSDs temporarily unused.

After the environment is stable, they could become a second fast pool consisting of two mirrored vdevs:

2 Ă— 1TB mirror
+
2 Ă— 1TB mirror
=
Approximately 2TB usable

That pool could later hold:

  • Game-server data

  • High-I/O application datasets

  • Proxmox backups

  • Additional media workspace

  • Frequently accessed files

The older WD Blue should remain separate as testing, scratch or temporary transfer storage. I would not mix it into the four-drive Crucial pool.

Home Assistant

Run Home Assistant OS as its own Proxmox VM rather than as another ZimaOS container.

That preserves the supported Home Assistant supervisor, add-on, backup and update experience. It also makes future USB passthrough for Zigbee, Z-Wave or Bluetooth adapters easier to reason about.

Suggested build order

Build this incrementally:

  1. Update the ASUS BIOS and verify the HBA is in IT/JBOD mode.

  2. Run an extended memory test against all 128GB.

  3. Install Proxmox on the mirrored 120GB SSDs.

  4. Create the mirrored NVMe VM pool.

  5. Enable IOMMU and verify the PCIe device groups.

  6. Create the TrueNAS SCALE or HexOS VM.

  7. Pass the SAS HBA into the NAS VM.

  8. Create and test the three-disk bulk pool.

  9. Configure NFS and SMB shares.

  10. Create the ZimaOS VM.

  11. Mount the NFS media share inside ZimaOS.

  12. Pass the Arc A380 into ZimaOS.

  13. Validate Jellyfin hardware transcoding.

  14. Pass through the Blu-ray drive.

  15. Install and test MakeMKV or ARM with one disc.

  16. Add HandBrake.

  17. Add DiscoBall after the ripping and encoding stages work reliably.

  18. Add game servers and experimental applications last.

The hardware is not the problem here. The real risk is allowing Proxmox, HexOS, ZimaOS and individual application containers to overlap in responsibility.

Keep Proxmox responsible for machines, TrueNAS or HexOS responsible for disks, and ZimaOS responsible for applications. That gives you a powerful system without turning every problem into a four-layer troubleshooting exercise.

My plan is to use Promox to isolate resources. Running HexOS as main NAS with simple install of media streamer and maybe a couple other applications. HexOS as a layer on TrueNAS let’s me still go behind the curtain if need be. I’m no command-line warrior by anymeans and like the one click installs or a walk-through install. Had issue with understanding that with Unraid.

CasaOS over a Debian install will give me a place to spin up gameservers to host Minecraft, SpaceEngineers, Ark and give me a different place to access with another web UI, I like it seperated.

And last is using Proxmox to let me spin up containers or VMs to play around with without (hopefully) messing with the other VMs configs or stored files. I want to say it’ll lead to better configurable space that can help give me a place to grow my skills even just as a hobby.

I would not virtualize a NAS, if something goes wrong, your data is probably instantly on the line.
Keep it simple, and you will be fine.

Since you’re looking at home server layout and resource management, you might want to look into transparent data tiering. If you’re planning on mixing fast SSDs with larger, slower HDDs or even SMR drives I’ve been contributing to a tool called HuskHoard. its on github

It allows you to use a fast tier for active data while transparently migrating colder data to slower storage without breaking your file paths. It’s particularly useful if you want to get SSD-like performance out of a bulk storage array.

1 Like