Proxmox / XCP-NG / Unraid / Ovirt ? for homelab with nvida gpu passthrough?

Thanks for the input
on proxmox for storage pools for your VMs are you using ZFS or local storage for your VMS?

1 Like

I am not using proxmox currently, but I did use it with zfs in the past and would recommend doing so unless you really like lvm.

2 Likes

what are you running currently for your home lab ?

1 Like

Eh, I’m running an (admittedly small) side business, so my “lab” is 1/4 rack in a datacenter where I’m running oVirt on 3 hosts connected to a SAN. oVirt is nice, but there’s too much learning curve and complexity there for a single node homelab. Only reason for you to use it would be if you want to invest time learning the RHEL (or Oracle) ecosystem.

3 Likes

Yeah i was looking into it and it seems like if i wanted to get into RHEL certs ovirt is the way to go but for now ill try put proxmox and see how i go thanks for the help!

might also try out XCP-NG very interesting but seems like gpu pass through is bit awkward on that platform might post a follow up once i have it all set up

1 Like

I gave XCP-NG a go initially. I like their web UI better than Proxmox’ but they (or CentOS/RedHat) appear to rather aggressively prune drivers for older hardware from their kernels, which ended up causing trouble with my Infiniband network since only ConnectX-3 and up drivers were properly available and most of my systems have ConnectX-2 cards (I don’t know about you, but for me 40Gbit/s is still ample :wink: ).

Other than hardware support, the main issue with XCP-NG is setup, since it doesn’t include a UI by default first time setup can be a tad painful. @lawrencesystems has a bunch of tutorials on their channel that’ll get you going though.

Proxmox, being Debian based, appears to have much broader hardware support, and it is what I’m using now. I honestly don’t know how XCP-NG would handle installing other stuff on the host, but I’m (ab)using Proxmox as both Docker host, and NAS currently and that works just fine even through upgrades.

Currently don’t see any reason to move off of Proxmox (though I am in the process of splitting the responsibilities off of that single machine) but should XCP-NG switch underlying distro to something with better support for older hardware (or improve said support themselves, not sure where that decision lies) I might give it another look at least.

3 Likes

I would say, it depends on what you desire from your home lab. If you don’t plan to go into making a cluster (and even if you do), I suggest Proxmox, just because it’s the easiest to setup from the ones I tried (IMO). Note that I haven’t tried oVirt (although it should be very similar to OpenNebula to setup), Unraid, nor XCP-NG (but I did try out XenServer with XenCenter on a Windows PC). However, you could just run any Linux distro on your hardware (CentOS, OpenSUSE, Oracle, Ubuntu, maybe Gentoo if you’re into that etc.) and just use KVM and virt-manager. You don’t need a GUI on the PC itself for virt-manager, as you can install virt-manager on another linux box / laptop and connect from virt-man’s GUI (via ssh) to manage the VMs and storage (I used to do that). It’s basically “oVirt light,” but without clustering (although if you run similar KVM versions, ie stuff like not combining CentOS 6 and CentOS 7 or maybe other distros, you can still live migrate between hosts if you require).

Proxmox is usually a more “whole” experience, but running a barebones distro and libvirt may spare you some (negligible) resources, maybe enough for a few containers.

Note: Proxmox uses QEMU/KVM, but in the back, it uses its own tools / APIs to control KVM (it’s called qm). oVirt, virt-manager, OpenStack and OpenNebula use libvirt to control QEMU/KVM. You can migrate from Proxmox to any of those that use libvirt and the other way around, but it’s an absolute royal PITA (ask me how I know) and I’m not sure how you’d automate that (it’s probably doable, but in our case, we just split around 280 VMs into 3 people and migrated in a month or 2 to Proxmox, one host at a time).

2 Likes

The main focus i want to do here is for building different Kubernetes clusters,

So what I want to do is use a hypervisor to manage the storage and guest VMS and also pass through GPU for some workloads to my containers,

Is it possible to create a pool of storage to for guest vms and use the same storage pool to present that to Kubernetes clusters as a storage class if I run NFS on-top of that pool in Proxmox ?

Short answer, yes.

Longer(-ish) answer, yes, you can. It’s probably easier with ZFS, but you can do it with LVM / ext4, but you have to delete Thin-LVM and expand the local LVM and use that for NFS. For both ext4 and ZFS, enable NFS on the local storage and basically connect Proxmox to the NFS server running on the server.

With 3x disks, you can only do RAID-5 / RAID-Z or 3-way mirror (I’d say go for RAID-Z, since they’re SSDs) and install Proxmox on ext4 on the boot SSD (to avoid some headaches with / on ZFS, although it’s not that bad, but since you don’t have 2 ssds for raid1, just go with ext4).

Edit:

Should work great, since you just make VMs and use them as nodes for K8s.

Should also be either trivial or easy. USB Passthrough can be done through the GUI without any configurations. For GPU you may need to use VFIO to prevent loading the GPU drivers on the host OS (Debian), then you can easily passthrough the GPU.

1 Like

Isn’t there some magic you have to do to trick the nvidia cards when they’re in passthrough?

1 Like

Yes, just edit the qemu config (they are the same for libvirt and qm). Solution discussed here:

(Exabit’s answer).

Edit: ok, I was wrong about that, they don’t appear to be similar.

2 Likes

Running kubernetes with Proxmox works reasonably well. Since there is a great project on GitHub that lets me use Terraform on it, I used it to provision some nodes from a cloud init template and since most cloud images are made for OpenStack they work great on Proxmox as well. The rest does Kubespray and the nfs client provisioner for me. If you you’re on Proxmox you can and propably should use ZFS. This will give you the option to use the democratic CSI driver: GitHub - democratic-csi/democratic-csi: csi storage for container orchestration systems

3 Likes

Makes sense I think I will go this route seems like the most straightforward approach,

ZFS looks very interesting way to go although i have zero experience with it from friends tell me it can be expensive in terms of memory overhead but I guess I will need to get it setup and see how it performs

1 Like

Very cool project I will do some more reading into this but looks very useful for my purposes here thanks a lot for posting it

2 Likes

Yes I think it’s the way to go with not al lot overheat as well as a lot of info online.

Btw ZFS memory usage can be tamed by changing the kmods cmdline. This will reduce caching and therefore potentially make it slower but that way it won’t eat your RAM for breakfast

1 Like

I’m glad we’re having this discussion, it has been a while since I looked into a qemu-server VM conf, not sure why I thought it looked the same as libvirt. Proxmox doesn’t even use XML.

@Blondiee , I jumped the boat too fast with Proxmox. oO.o was right, you need to jump through some hoops with nVidia passthrough and I personally have no idea how you’d do that in Proxmox. According to this article, the nVidia doodads is easier solved in Proxmox than my ordeal with virt-manager (although that was a fun experience):

But as always, YMMV. Try Proxmox and the first thing you should do is passthrough the GPU to a VM to make sure you don’t get funny behavior. Then you can proceed further with other stuff.

Depends on what you’re doing. In your case, you should be fine. Heck, I’m fine* with ZFS on 24 GB of RAM, out of which around half is used in my running VMs. ZFS is smart, it caches however much RAM it can gobble up, but if the OS says it needs RAM, it frees some RAM for the OS to use. You can get away with basically no RAM on ZFS, but the “standard recommendation” is to have 8 GB of RAM for the whole system (you can get away with 2GB of RAM on a system running ZFS and only NFS or Samba, but obviously not recommended).

1 Like

@ThatGuyB Your input here has been very helpful for me it seems that Proxmox is for sure the way to go about it to reduce some head aches!

Yeah for sure the first thing I will try once I have it installed is the GPU pass-through with the NVIDIA GPU make sure that works the way I expect then power on ahead with my k8 clusters!

@ThatGuyB @modzilla

Are you guys saying that you can nerf it to a MAX ram size like 6/8GB or should I just let it dynamically allocate for the time being and let it figure it out on it own?

1 Like

Glad to be of help. Just in case Proxmox doesn’t work, you can try virt-manager with libvirt, as that one is known to work (I see no reason why Proxmox can’t work when others tried it and did it).

I suggest you don’t mess with ZFS settings unless you absolutely know what you are doing. Default settings are sane and they have been chosen for a reason. =)

1 Like

Great I will give it a go and see how i get on with Proxmox & ZFS =]

1 Like

Although I tend to agree with @ThatGuyB, setting the zfs module to only use up to a certain amount of RAM should be fine.

Proxmox even as an entry in their wiki ZFS on Linux - Proxmox VE

1 Like