Understanding Proxmox and ZFS

Operating Systems & Open Source
, , ,
1

Hello

I am in the planning steps to migrate my current homeserver setup into something more modern now that the hardware is passing a decade in service.

What do I have today
I have two machines. A NAS running TrueNAS Core and a box running Ubuntu Server. The linux box has slowly taken over more and more tasks from the TrueNas box so today it is more of a pure NAS + backup target for the Ubuntu box and two TrueNAS systems belonging to/located at family/friends houses. The Ubuntu box run these services

  • Nextcloud (Bare metal)
  • Homeassistant (Docker)
  • A personal Website (Bare metal)
  • Vaultwarden (Docker)
  • A gitlab instance (Docker)
  • LMS (Docker)
  • Plex (Bare metal)
  • Unifi Network Application (Docker)
  • Some misc thing to play around with (Mostly docker)

What do I want to achieve

  • I want to consolidate the two boxes into one to save power and complexity.
  • I want to migrate away from TrueNAS Core as it is being de-prioritized as a solution to host applications. God knows what happens with it long term
  • I want to get a more performant setup
  • I want to increase reliability

Requirements

  • I need a “compatible” ZFS replication target for remote TrueNAS hosts. Dad need to be able to setup replication jobs in the TrueNAS GUI (he is going the be migrated to TrueNAS Scale)
  • I want to keep the principle “data is in ZFS datasets, accessible on file level” that TrueNAS core had with Jails. I like the separation between data and application
  • I need to be able to host VMs (Homeassistant and Nextcloud at a minimum, probably more)
  • I need to be able to host my Docker containers. These have their data on ZFS datasets
  • The ZFS setup will be 2 pools. One “speedy” NVMe pool and a “slow” HDD pool for bulk storage and backups from the NVMe pool
  • Be able to replicate backups of the slow ZFS pool to a remote TrueNAS systems.
  • Maybe in the future throw a GPU in to accelerate what can be GPU accelerated
  • Good enough performance. I do not have a datacenter and nobody is paying for what I host. But I would like that the upgrade to NVMe is an “upgrade” over the spinning rust I have today

Now to what I need help with.

I am trying to understand what I should do to achieve this. I think I want to use Proxmox. Where I have trouble is understanding how to string everything together without completely tanking performance or grow even more grey hairs. My understanding is that my options are

1 - Proxmox handle everything
This seems easy to setup. But I understand that my options for how to manage storage are a bit limited. I either use VMs with mounted virtual disks for storage but then loose having file level access on my ZFS snapshots. Or I can use containers and mount datasets directly, so kind of like the jails I am used to(?). How to get remote replication target for the other TrueNAS systems working is a bit unclear, same goes for pushing my replication jobs to a remote system (Sanoid should work with an unknown amount of headache as I understand it)

2 - Proxmox but passthrough and let a TrueNAS VM manage disks
This would allow me to use TrueNAS basically as I am doing today, just virtualized. But here I do not completely understand how to share datasets with the VMs/containers in Proxmox. NFS shares? But how is that in terms of performance/ease of use? Something else much more elegant? And are there other caveats in general?

3 - Proxmox but not as I think
Option 3 is Proxmox but I am utterly wrong about how to use it

4 - Something completely different
Option 4 is that I am wrong about Proxmox and should use something else

Thank you for your help

Edit: Fixed messed up formatting

1 Like
2

I guess reading this im asking the question why migrate to proxmox when you can do all of this on Ubuntu Linux, and you have most everything working. Ignoring the pre made ease of proxmox’s ui, it seems quite stright forward to move everything to ubuntu, and then just migrate the os drive and zfs pools to newer hardware when that happens. Ubuntu with the right software can and will do everything Proxmox can do. On ubuntu 24.04 lts you can install zfs 2.2. you can export your zfs pools and re import them on any machine running the same or newer version of zfs.

Ok, so take this coming from a person who has not used proxmox, but i feel like your getting way from truenas because you want a more custom setup, and not be locked into a certain way of doing things, and going to proxmox is going back to this very thing. A pre-built environment with a very cookie cutter way of doing things, unless you go the extra mile.

idk just sounds like your wanting to try proxmox. If you just looking for a management ui, idk if you are using or have heard of Cockpit web console, works great on top of Ubuntu server to give it a nice web management interface. Docker being the lacking part of Cockpits web interface.

1 Like
3

Thanks for your reply. I guess the reason to not just continue with Ubuntu are mainly 2

  1. I want to move Homeassistant into a VM (managing the dockerized setup is a slowly expanding pain as it is not the officially supported way), and managing VMs on a headless system is something I have never done. And I kind of presume this issue with how to share ZFS datasets into the VM will be true even if I do it more myself
  2. I need to do a OS upgrade quite soon because of Nextcloud dependencies. This is not super tempting right now as updating major LTS release twice will probably break something and I need to repair/rebuild it. My current life situation does not really allow for me needing to spend a day and a half to fix it. Having the possibility to mess around with a parallel VM/docker image 20 -30 minutes a time while upgrade and then migrate my real data/setup over once done is quite tempting. Kind of “nuke and replace” without loosing access to everything during the rebuild.

My idea is at least that using Proxmox or similar solution is going to make that easier. That being said I am sure there will be Ubuntu Server VMs running, at least if I can achieve what I want with the ZFS storage.

4

Ok, so what im hearing is its all about vm managment.

Lets start with, making sure you do understand that proxmox os just gives you a pretty gui to kvm the same thing you would be using on ubuntu to run virtual machines. Which leads back to my cockpit suggestions because it also has a kvm plugin to manage vms. In terms of which gui is better probalby proxmox, however your using the same base to run vms with either, which is kvm. Rember via cli you have the power to do anything on either even if the gui doesn’t give you the option.

So from my stand point its again back to do you like the gui and customizations a certain distro ships with(Proxmox) or do you wanna diy it on ubuntu server. Proxmox like ubuntu is a debian based linux distrobution.

I’m not trying to tell you not to use proxmox, rather make sure you understand your getting a gui and a bunch of things pre installed or removed compared to ubuntu server. How zfs kvm and all the underneath stuff works pretty much the same, assume they both have them installed.

Edit: Side note you actually can install proxmox’s web ui on ubuntu or any debian based dsitro with apt.

5

Thanks again. So now it is pretty clear to me that it is a GUI and tools on top of what I could do myself. My big issue for understanding how to proceed is this part

I am not sure how these parts would interact to achieve what I want. My old setup was either TrueNAS (FreeBSD jails) or bare metal/docker on the Ubuntu machine. Virtualization is pretty new to me and I have not really found much info on ZFS dataset level sharing to VMs

6

Ok so i realized i left out one important detail in my last post proxmox has its own vm maanger on top of kvm where normally you would use something like libvirt. libvirt and proxmox configs are not directly compatible but can be migrated between.

Can’t speak on freebsd and truenas jails, more than there a form of virtualization.

Ok so docker is a container based virtualization environment, proxmox uses lcx which is another container based virtualization environment. You can run any Open Container Initiative(OCI) base image like docker container images in either.

Again proxmox is a debian based linux distro i have never used proxmox so idk how to tell you to get to its comand line, however assuming you can get there you should be able to do anything you would on a debian based distro(weather you should may be a different question).

Ok so back to zfs and virtualization, so assuming you have picked proxmox or ubuntu, and weather your using libvirt and kvm or proxmox’s vm manger and kvm we get back to more straight forward concepts. Basicly zfs is a file sytem you create a vitrtual hard disk on your filesystem (in this case it will be zfs) in proxmox or libvirt then assign that virtual had disk to a vm. How this might look is you have your zpool, with a dataset calld vms , and you amke a new virtual hard disk HA.qcow2 (you could pick a different virtual hard disk format here) on your dataset, and assign a vm to use it. You can do one vhd per data set or put them all in the same. You can snapshot the vms data set as many times as you like. obviously spreading out your vhds into their own datasets would allow you a finer gain control of snapshoting specific vms.

Proxmox i believe also allows for assigning a dataset directly to the vm since datasets could be treated similarly to partitions. You could do the same with libvirt.

How you store your vm is up to you.

7

Thanks again

First of all, I am going to take a look a Cockpit, forgot to say that the last answer

This part with creating virtual disks is what I would like to avoid, as the simple file level way of exploring snapshots (to restore files etc) would be lost when it is used as a block device by the VM (as I have understood it).

Under TrueNAS Core, datasets is mounted into jails which mean it is handled on file level. Worst case the jail can be destroyed and it is simple to manage the dataset and it snapshot directly from the host OS. No need to mount a old version of the virtual disk to restore old files, just explore the snapshot like any hidden folder. I have snapshot history going back years for datasets where deletes are supposed to be unusual (like photos).

The only way around this that I can think of is to pass though the disk to a TrueNAS (or I guess Ubuntu) VM and then share the datasets with NFS or similar. I have however had a hard time finding out if that is smart or stupid from a performance standpoint

8

I think unraid makes some amount of sense for your application of a hypervisor that also does ZFS. As of unraid 7 they allow you to ditch their ‘not-a-raid’ solution and use ZFS for the whole show. It takes some getting used to but I see unraid as a good middleground for proxmox and truenas. Like if truenas scale was actually good. :grimacing:

Of course I understand if the pricing model is a turn off. its not for everyone… but I think it saved me a lot of time and effort. GPU passthrough is a check box and a drop down menu.

Just another perspective :wink:

9

Oh no, are you one of those people who are doing qcow2-on-CoW filesystem? That’s terrible in a number of ways.

10

Well the $100+ charge sure sting a bit, but I would not say it is a complete dealbreaker. The economicly rational choice is to just not bother with any of this and pay google/MS or somebody else for cloud storage :rofl:

I will take a futher look if it seems like a good choice. Thanks

1 Like
11

There is a trial you can use for 30 days with all the features just to see if it makes sense. I’ve tried to do the $DISTRO + cockpit route… I tried TrueNAS scale, this is probably the most viable free option right now for what you’re after… I tried proxmox, its fine but you have to tinker it a lot in which case its not much different than $DISTRO + cockpit for what you’re doing…

I think TrueNAS scale is worth checking out right now. They just ditched k8s for container/vm management which urked some people but IMO it was the right move. Its not going to be as full featured on the VM side as unraid, but all the parts and pieces should be there.

12

Yeah it looks like there will be some experimentation with the software setup. Reading about the state of running VMs etc in the TrueNAS forums is not super encouraging right now. Having parts of the documentation saying a method to share data with VMs is the “prefered way” and then “unsupported” somewhere else does not really scream stability. But maybe they are on the right path.

13

My preference is virtiofs however for quick and dirty, mounting network shares is viable and usually ‘good enough’.

14

You can still restore an individual file in a virtual disk on a snapshot. Remember the zfs snapshot mechanism is snapshoting changed blocks, it is not looking at the file as a whole, or even vhd as a whole, rather only blocks in the filesystem(zfs not the filesystem of the vhd) that have changed. Your not losing anything here in terms of snapshot detail, this rather is adding the step of mounting vhd in the snapshot-ed point you want to look at before you browse.

Ok, again im not familiar with truenas jails, but that sounds like its just passing through a zfs dataset as a block device which i believe i mentioned you can do with both proxmox and libvirt, on kvm.

This is just my opinion, but i like the ability to transfer the vm to other devices easily by copying the config and vhd and you just launch the vm on any other device(this has a lot to do with my use case for vm’s)., I agree with you that objectively what your saying is easier, however it is a simple mount command extra to have to mount any vhd weather its in the hidden snapshot folder or its the current version.

Well i said this because qcow2 is the default qemu/kvm option in a lot of cases, however its true that many of my vms stored on my server are qcow2 . Yes it is terrible and i’m aware, it stems from how i use vms. It should be noted my server while i occasionally run vms on it from time to time, is more become a dump point of old vms, that should i ever need to run an old vm and said vm doesn’t need some pass through hardware not on the server, i run it on the server rather than move it back off. I have moved to running everything that runs 24/7 to containers, way less overhead. When i did run vm’s 24/7 all of them got their own physical drive device or partition(really need the iops of a stand alone ssd or drive for performance), and just backed up to my backup server like any physical computer. I use vms a lot for short periods as specific sandboxes for testing of running software on specific versions of os’s with specific versions of software. Then they get archived in case i have to revisit that case which does happen from time to time.

im not sure i understand why you would have to do this, or want to? ZFS can run on any of the os’s named in this topic so far, so why create the overhead of running a second os and pass-through derives to a guest os just to run zfs there instead of on the host, and only passing vhds or datasets through to guests. This is just adding complexity and extra overhead.

Ok tbh unless your on a all ssd zfs pool your pool is going to be pretty iops constrained on the number of guest os’s you can run off a single pool. Which leads back to my preference of containerization over vm’s, why waste iops running multiple os’s if it can be avoided. Also why i didn’t choses to run 24/7 vms on zfs rather on a separate drive.

Again this is just my preference, but i like to roll the docker image of home assistant. Let me say Ii totally get why people want to run home-assistant os, and not the docker image it is quite convenient the way they have packaged it, but what is really happening is your running a os that is running a container orchestrater, that is then running the almost the same container image of homeassistant, that has some extra buttons to then pull other container images and run them so you don’t have to manually do this to get graffana running for example, or any other add on. and why they don’t just put the hacs store in the docker image by default is silly since you can just put it back by getting the integration off the ha github, and putting it in the right folder. Really why run a second os just to launch containers there instead of directly on the host os, then have the whole now i have to manage a whole virtual network of vms thing.

15

I’ll describe my experience with Proxmox and why I decided against it.
Like you, I had an Ubuntu server to run bare-metal Nextcloud and a couple of Docker containers. I didn’t have a TrueNAS server but wanted an OpenBSD system as a router for my home network and to switch from Ubuntu to Slackware for better control and time efficiency. Having everything in one box seemed appealing to me initially. However, this got complicated quickly when I wanted to use different disks, NICs, and a RAID 1 array for various VMs and purposes. Proxmox and VMs seemed like significantly more work and effort compared to my previous setup. I experimented with Proxmox for a couple of days, but it didn’t work the way I wanted. Maybe I’m just not cut out for Proxmox, but I eventually settled on two separate boxes:

  1. A Slackware server with Nextcloud installed on bare metal and about 10 Docker containers.
  2. An OpenBSD router built with energy-efficient PC parts I had lying around or purchased cheaply on eBay.

This setup is simple, effective, and energy-efficient (~78W most of the time, with rare spikes to 130+W). It avoids the overhead of running VMs for different tasks, so the hardware doesn’t need to be overly powerful, which helps keep costs down. Maintenance is straightforward: it takes me maybe 15 minutes a month for checkups and maintenance, plus weekly backups to an external drive that I leave overnight to complete.

Most importantly for me, one hardware failure won’t bring down the entire setup, and I am using very conservative OSes, free from corporate greed and frequent policy changes.

2 Likes