Hello! I currently have a single Proxmox server with an HBA and 6 disks attached. They are presently in a RAIDz2 pool in a TrueNAS VM with the HBA passed through. I have Proxmox currently connected to an NFS share hosted by that same TrueNAS VM. I then have a large virtual disk passed through to my main Ubuntu media VM. I know this is inefficient and overly complex.
I am looking to rebuild the server and I am wondering the best method of using those 6 disks with the Ubuntu media VM would be. I have seen a few recommendations to use an LXC with Cockpit and there are a few guides out there, but I want something that makes SMB, NFS, etc. sharing and permissions easy and want to set it up right the first (second) time. I also want automated scrubs and SMART tests similar to TrueNAS. I could also use the command line, but I’m not nearly familiar enough to get it working without a step by step guide. I need these shares to be accessible by other VMs as well as external Windows hosts. Any and all advice is appreciated!
this is a lotta shit to unpack, also welcome to the forum…
SO you have a Proxmox hypervisor:
TrueNAS VM with HBA passthrough (nice)
ProxMox mounting NFS from TrueNAS VM which hosts media for Ubuntu VM
Sounds like you want to make a virtual switch and assign a virtual interface to the TrueNAS share and virtual interface for the Ubuntu share.
This keeps filesystem virtualization to a minimum as it becomes a blazing fast local network share only limited by TrueNAS drive speed (should be near line with the HBA passthrough)
Beyond that, only recommendation is a dedicated NAS and separate hypervisor box.
Thanks! Yes that’s correct. I would also like to change the virtual disk on the Ubuntu VM to a general NFS share so that the Linux ISOs stored on it will be accessible to the other VMs/clients. I was under the impression that Proxmox did that automatically, but I haven’t tested/verified that to any extent. I’d love to do a dedicated box, but that isn’t in the budget at the moment…
So I would create a new Linux Bond and then assign that interface to each VM as the “virtual switch”? Would I need to manually assign IPs to each VM for that interface?
A network bridge on proxmox can provide vm to vm communication at bus speed and also still connect to the external network at interface speed. So you can have dhcp from an external source and not have to manage multiple nics in each VM. Make sure the VMs are using VirtIO.
TrueNAS is a great NAS.
Proxmox is a great Hypervisor.
For a NAS that hosts mostly large files that sequential reading and writing patterns, RAIDZ2 is fine.
For a Hypervisor that has mostly VMs with small and random reading and writing patterns, you should use SSDs in mirrors. RAIDZ2 isn’t feasible for this, not only because of the speed, but also because of how RAIDZ and the fixed blocksize can lead to fragmentation and amplification.
You can also run zfs in proxmox and manage it there. It saves you from needing truenas running and essentially depending on truenas and proxmox at the same time.
You would manage zfs and nfs in proxmox, and any other file sharing like samba in a lxc container or different VM.