Hi all, I am configuring my homelab/NAS device and need a sanity check and some advice. This is my current HW setup
Motherboard: MSI X570S PG Riptide
CPU: AMD 5700X
SSD: 2TB NVME on M2_1
RAM: 48GB 2666Mhz ECC
HDD: 4x Seagate 10TB directly connected to the SATA controller
GPU: 2x RTX A4000
NIC: Intel x520-DA2
I want to run Proxmox as the Hypervisor and then run TrueNAS scale as a VM on Proxmox for my NAS. My goal is to run a bunch of VMs and containers like Ubuntu, Windows, Ollama, Nextcloud, PhotoPrism, Plex, Radarrr et al, a couple of webservers with Traefik and the likes on Proxmox. Not all of them would be running all the time (would be used for testing/learning things like cybersecurity and data engineering). Also there wouldn’t be very many users for the Webservers/Plex.
I wish I could separate the NAS into its own thing, but because of space and noise and budget constraints I really can’t. I also don’t have a lot of flexibility in changing the hardware.
Since all the PCIe x16/x8/x4 slots are taken up by my GPU, I also don’t have any to spare for HBAs. I have a couple PCIe 4.0 x1 slots available for expansion if necessary.
Now my questions are:
I’ll be passing the entire SATA controller to the TrueNAS VM. So do I partition the NVMe and use it for installing Proxmox and TrueNAS?
How much RAM should I allocate to the VMs? Is it possible to do dynamic allocation?
Should I use NFS or SMB for accessing data on the TrueNAS VM from my Proxmox VMs/containers. I read about how NFS uses synchronous writes and would be slower. Would SMB be better in that case. Do I need to get a separate SSD for SLOG if I use NFS? What speeds could I get, in theory, in either case?
I’m planning to use RAIDZ2 on my pool. In the future if I wish to expand my storage, would be possible to add another vdev with just 2 more HDDs in mirror mode? I do not plan to expand beyond that for now as I don’t have such high data needs. Even 20TB is way too much for me (famous last words).
At some point I plan to add two more SSDs in mirror mode to act as “special devices” that would be used for storing metadata which in theory should make my pool run faster. But since I’m out of ports, is it a good idea to use a PCIe x1 to M.2 adapter like this (https://www.amazon.com/GLOTRENDS-Adapter-Installation-Bandwidth-PA09-X1/dp/B09P3HY3P3)
If I plan to do some video editing directly from the TrueNAS on my Macbook Pro, what’s a good idea to increase read speeds? This is not something that I would be doing frequently so not a high priority. Except for increasing RAM would reserving some space on the NVMe for L2ARC help?
How can I back up the Proxmox installation and settings? Incase the NVMe fails or something. Is it possible to back it up inside the TrueNAS VM. How I recover the TrueNAS VM/Proxmox VE in that case?
Adding to the previous question, can I back up the other VMs/Containers running on the Proxmox VE, inside the TrueNAS VM?
Any other general advice/tips would be greatly appreciated. I know its a lot and I would be very grateful for any inputs.
P.S: I’m an ML Engineer with plenty of terminal experience so I’m not worried about my hands dirty, but still a noob in networking/virtualisation. I would like to (try to) setup everything as optimally as I can from the beginning so as to not have too many headaches in the future.
Instead of truenas you can also setup the zfs pool from proxmox. This allows you to pass them into VM’s directly. I would do a ssd mirror to install proxmox and other vm’s on, because you do need random write performance for that.
Doesn’t have to be big, just for their os and any write intensive stuff like torrents and swaps.
Then you can use the fileserver lxc container to handle that.
You should check my homelab thread, I ave a somewhat similar setup but with a seperate nas and proxmox system.
On my 3x16tb zfs zraid1 I can easily saturate a 2.5gbe link with sequential data like video. But lots of small files are not as fast. But I have a 4tb ssd mirror that I run.
What kind of data are you storing? Big files are not going to benefit from metadata or slog, you are better off using those Ssd’s for a fast mirror pool.
Instead of truenas you can also setup the zfs pool from proxmox. This allows you to pass them into VM’s directly. I would do a ssd mirror to install proxmox and other vm’s on, because you do need random write performance for that.
I did consider that but I read on several forums that TrueNAS must have direct access to the hard drives for best performance (I’m paraphrasing here).
What kind of data are you storing? Big files are not going to benefit from metadata or slog, you are better off using those Ssd’s for a fast mirror pool.
It’ll be mostly photos, videos and data for my VMs (ollama, webservers, databases)
But I have a 4tb ssd mirror that I run
Unfortunately I’m out of PCIe lanes on my motherboard, I have a couple of PCIe 4.0 x1 slots I can use, but not sure if its reliable with adapters for SSD. I only can use 1 NVMe SSD If I want to have 6 SATA ports. Although I’ll only be using 4 SATA devices now, I’d like to keep the last two available for a later upgrade. I noticed that the latest TrueNAS release candidate (electric eel) supports expanding vdevs, so I would like to expand my current vdev in the future (with 2 more HDDs).
You would setup proxmox as the zfs filesystem and not use truenas at all. It’s going to be the exact same zfs file system but without the ui of truenas and instead you use the proxmox ui. Any filesharing of samba and NFS can be setup in a lxc container.
I am considering that now. What benefits of TrueNAS do I lose in the process? I’m evaluating my options between installing Proxmox as Hypervisor and also for shares vs using TrueNAS as hypervisor and for shares
Proxmox will be a better hypervisor and truenas will be a better nas. But if you have a single zfs pool for 1 user there is not all that much to setup on that front.
I’ve tried truenas but I didn’t know what i was doing and really don’t like how docker containers work in truenas (k8s). So i just have a debian install in which i setup zfs/nfs/smb by hand with docker, and i use cockpit as the UI.
Someone else on this forum can tell you more about truenas.
But you save a lot of headache by leaving it out.
Like passthrough of devices, double ARC, linux bridges hindering your performance, the list goes on.
Don’t do it. Proxmox is a nice hypervisor, TrueNAS a nice NAS.
For a “NAS with some services” I would rather use TrueNAS plus some rudimentary VMs and Docker than the other way round. But that is just my personal taste.