Advice on setting up my drives in new Homelab build

Operating Systems & Open Source
,
1

Hi all!

First post but a looooong time lurker :slight_smile:

Some drives, a GPU and motherboard died on my homelab server last week, in fairness it was over 10 years old and was secondhand hardware when I got it and it should have died years ago.

So after to much frustration last week, in a bit of “anger” I ordered hardware for a new Homelab and thinking to myself that this time it’s going to be good. I work waaaay to much overtime and have been saving for a while for this.

Anyways, I use my homelab for the following:

VMs (must have)
-Opnsense.
-Home Assistant.
-Windows install (I make tutorials for work and this is used for that, so it’s fresh everytime).
-TrueNas Core ( Plex, file storage).

VMs (would like to have)
-Windows vm for gaming with passthrough gpu.
-Linux VM (not sure distro) for learning more.

Here is the list of hardware I ordered (remember, I was angry and frustrated hehe).

-AMD Ryzen 9 7900X
-ASUS Motherboard X670E-CREATOR WIFI
-ASUS Phoenix GeForce RTX 3050 V2 8GB
-G.Skill Trident Z25 Neo RGB 32GB (4x16GB), didn’t order ECC, I know I should have.
-Dual Intel 1Gbit ethernet card.

Drives or related to.
-4 x Kingston NV2 M.2 2TB with ASUS HYPER M.2 X16 Gen 4
-4 x PNY CS900 SATA 4TB SSD

My thoughts was to use the nvme drives for all the vms and the SSDs for storage.

I’m just unsure how I should set this up or what would be the best way to do this in regards to ZFS, many different options here and ZFS is still very new for me. It’s one thing reading about it, but hearing advice helps so much.

And it’s not that somebody should tell me exactly what I should do, I like learning and reading. It is more advice and best practice I am looking for.

Btw it could be that I use the Nvidia 3050 for my “studio” PC and use a Nvidia 1050 in the server instead so I can use Nvidia Broadcast. Not sure yet.

And, you people rock! Insane how much knowledge is in here, it’s actually a humbling experience sometimes :slight_smile:

And sorry for a long post!

Edit: I forgot a question hehe.

Am I right thinking that a cache drive isn’t really needed with only nvme and SSDs?

2

It kinda depends on how complex you want this to be?

Update bios to BIOS 1602 (latest throws weird allocation errors)
Update firmware on your NVME M.2 drives (this is going to bit painful as Kingston only supports Windows and no EFI etc)

This seems to be similar for your PNY drives

Not sure why you got the Asus Hyper M2 card as the mobo already supports 4x NVME drives?

I would run FreeBSD 14.0 bare metal and go from that using bhyve when needed.

Not sure what functionality you need to I’d ditch opnsense + truenas core and run that natively on FreeBSD 14.0 (pf, blocky/unbound/dnsmasq/adguardhome whatever you prefer, odhcpd, samba, plex etc) using either ports collection, packages or setup your own Poudriere instance.

Run HA (Operating system) in a VM using bhyve
Run Windows in a VM using bhyve

As for setup, NV2’s in mirror pairs and PNYs as RAID-Z array (you might want to consider using some padding too)?

1 Like
3

Can you pass a GPU through to a windows guest for gaming yet?

4

GPU Passthrough works however nVidia might be troublesome

5

My plan was to use Proxmox, but you would recommend going FreeBSD and bhyve instead? That would be a new world for me and it would be fun to learn, just not sure I can do that at the moment because of time sadly.

And the Asus card just was one of those things I have been wanting to play around with for ages and because I wasn’t thinking straight when I was frustrated, as I wrote :slight_smile:

But even if I did go do the FreeBSD road, my original question would be kinda the same I guess (ZFS seems more built into FreeBSD).

I am not sure what ZFS Raid I should go with and if I am thinking correctly regarding nvme drives for VMs and SSDs for storage. I know you answered some of that, just not sure of pro and cons yet…but now I have something to look for. Thanks!

Edit: and also my question regarding cache drive.

6

While initial setup might be take a bit more it would likey save you a maintenance time and overhead.

Regarding storage
It depends on what kind of redudancy you want (if any) and performance to some extent I guess

2x NVME + 2x NVME (mirrored pairs), 1x boot + 3x RAID-Z, 1x boot + 1x “scratch disk” + 2x NVME (mirror)
4x CS900 RAID-Z, 2x + 2z (mirrored pairs), JBOD

In general I don’t think you need to bother with a ZIL drive with your current setup

1 Like
7

Thanks man!

I’m gonna start reading about FreeBSD a bit more. Only contact with FreeBSD so far for me has been pfsense and it’s forks. And you don’t really touch the OS there so to say.

I’ve been working with PC’s the past 30 years but mostly building them. It’s the last year really that I have been moving away from Windows, I have seen the light! Hehe

Huge thanks for the suggestions!

8

The handbook is a very good start to be honest, FreeBSD Handbook | FreeBSD Documentation Portal

1 Like