Hello all.
I have my server with ESXi and a RAID5 done with mdadm inside a VM passing the disk with RDM. I will move to Proxmox in the coming days and I was ready to start to use ZFS. The idea was to follow a similar setup, but I saw some people doing the pool in the Proxmox itself and mounting it in the VM through NFS. Which is the best practice? The pool will be used only by one VM, the one that has the Plex and such containers.
Thanks!
ZFS on Proxmox host, shared to multiple containers (Plex etc). I recommend you do containers instead of a single VM with everything crammed into it, kind of defeats the purpose of virtualization…
I will have multiple VMs (proxy, monitoring, VPN, etc…) but only one for managing the media in the HDDs. Why containers (I guess you refer to LXC) over VMs?
Also, single VM hypervisor has one purpouse, act as a KVM when the VM hangs up.
According to most forumers the best practise and easiest solution for data storage in Proxmox is a ZFS pool through proxmox and the data is stored in a VMs disk image. You get the virtualisation benefits (fast backup through Proxmox GUI; snaps shots, migration) and of the ZFS dataset (cache, compression, deduplication if needed). You then share the data through another protocol NFS, smb, however you like. Within the internal network on the server your transfer speed is as fast as your r/w speed to the source.
This is most likley the way I will handle it myself in the future for my main machine. Currently I use a lxc (container with a virtual disk). But backing up is a pain because the backup process in Proxmox will skim through the files and copy them over, which is horribly slow. Disks of VMs are read as block device instead during backup.
A third option would be to create a VM and pass through a HW controller for the disks, and let the VM handle the data management (for example running a TrueNAS VM which handles the ZFS pools for data storage).
This is what is Proxmox built for. ZFS pool and Zvols as VM disks. Proxmox is pretty much designed for block storage. If you need file storage, people like to make a Samba/NFS VM (in whatever form, Cockpit GUI comes in mind) because Proxmox isn’t a fileserver and installing nfsserver and stuff on Proxmox kinda defeats the purpose of a dedicated hypervisor.
TLDR: Bare-metal ZFS pool and zvols for your VMs. Done. Fileshare via dedicated VM/Container.
You can also store files in a ZFS dataset directly and mount it in a lxc container. Its nice if you plan to physically move the disks somewhere else and import the ZFS pool. Its a pain if you want to manage it within Proxmox (I guess you could “zfs send” the dataset / snapshot). Another disadvantage is that you need a privilidged container which you don’t necessarly want if you have outward facing services and if accessing parties / network are untrusted (i.e. the internet).
the pool that your VMs and containers reside on, and the pool that your media and other data reside on, should be separate. preferably entirely separate disks for performance reasons. then considering ProxMox is not a file server, makes the rest of the design pretty easy.
Maybe I didn’t explain it correctly. I will have the VM disks in a NVMe SSD, but I will have a 2x8TB HDD setup to store media files. That media storage will be used in only one VM. I will use ZFS. I don’t know if it is better to do the ZFS pool with that 2x8TB HDDs inside the VM passing through the disks or do that in the Proxmox itself and mount it in the VM.