Proxmox VE for enterprise use, any thoughts?

Hey Everyone,

I'm looking into open source alternatives for Vmware ESXI. My company has a few of the free and paid licenses for Vmware ESXI. We're considering the purchase of vcenter, but I would rather spend the money on new hardware than on vcenter.

Thus, looking at alternatives. I have experience with Ovirt and it's ok, but the cluster died on us during an upgrade and we lost everything. We had backups, but couldn't restored and had to import the disks from backups. So a lot of pain there, it worked! But still a lot of pain. Especially when I had to redeploy the Virtual Hosts almost every time I updated them from the Ovirt manager. sigh...

Anyway. does anyone have experience utilizing an open source VM infrastructure in production?

Why not use ESXi? What is your company gaining by going open source?

Hey,

Well, ESXI just doesn't do much without Vcenter and Vcenter can cost about $5K to purchase with around $2K every year for maintenance. That's a lot of money to get high availability, scheduled snapshots, disaster recovery, etc...

So yeah, I feel that KVM does everything ESXI does if you don't have Vcenter. Vcenter provides most of the functionality.

vsphere/esxi are free to some extent I believe, I think it's like up to 100 servers and like 2 processors? I don't quite recall, but if you're wanting support and maintenance that's going to cost you, and depending on your industry unless the software you choose has support you won't be in auditing compliance. I'm not sure of any vendors that support any opensource hyper-visors out there, but I'm sure they exist. (@zoltan would surely know this)

Hyper-v is also an option as it's 'free' as long as you have the required server licenses for the VMs, the bare metal doesn't need a license just to run hyper-v.
https://technet.microsoft.com/en-us/hyper-v-server-docs/hyper-v-server-2016

Edit: but yeah the DR for ESXi is insane... I think we pay like 150k a year for our licenses for it.

Hey Yockanookany,

What about Ovirt? or Proxmox? They are both fully open source and have support via vendors (RedHat and Proxmox Enterprise). But the vendor support is only secondary and believe me I'm 100% on board for paying for support.

My hang up is the limited usefulness of ESXI compared to open source alternatives. Ovirt doesn't have CPU/Socket limitations based on licenses and you can make templates out the gate. Same with Proxmox. We're a pretty small shop. Most of the VMs are web hosts, thus the HA requirement.

I started using proxmox as a VM server for testing / pre-prod at work. It has worked great.

is your setup just a single server? or a cluster?

thanks for sharing!

P.S.

Any pain points?

I've been running proxmox in a single server lab setup for over 2 years now. I can say it's pretty awesome. I haven't delved too much into it but i can say there is alot of granularity to it's control over resource pools, user groups, and VMs/containers. Proxmox does come with out of the box support for clusters of servers, from reading online some people run fairly massive clusters on proxmox. Backup/restore works flawlessly on proxmox. Recently migrated all my VMs to an NFS share when switching form a 2950 to a t610. Backing up each VM was about two clicks, and restoring them was two clicks, no headaches. Only gripe i have so far is that there is no out-of-box support for backing up directly to tape from the web GUI, but it can be done relatively easily from the command line.

Thanks mrpopo. You've always been a good friend to Goku (if not to anyone else...) :smiley:

lol, thanks for sharing. If you're running that in a test lab what are you running in your production datacenter?

@jebbaxley Before I can recommend Proxmox VE, I need to know about your hardware. Proxmox clusters like to have an odd number of nodes in a cluster and on top of that, they like to have external (shared) storage to support live migration. Proxmox supports a Ceph backend, but I would recommend against it because of how resource intensive it is. If you're interested in using Ceph, you're better off building an openstack cluster and having an external Ceph cluster built for Glance.

That said, proxmox is an excellent option assuming you're using less than 350 or so VM/containers. Once it gets above that, the cluster tends to fall over.

1 Like

Hey,

yeah this is a small/medium deployment. I have enough spare hardware for the 3 nodes and NFS/iSCSI storage. Vmware and Ovirt have the same NAS/SAN requirements, so no change there. I'm surprised by the 350 number for Proxmox VE. Have you tested that or is that in the documentation?

It's my testing.

I run an openstack cluster with 85 compute nodes. Before we had that, we were running a 23 node proxmox cluster. It worked well. It required feeding every now and then, but that was only really during major updates. I forget if the cap is documented officially anywhere, but I've definitely hit that wall a few times. It's always right around 350. That was proxmox 2 or 3 though (can't remember) proxmox 4 may be different. (higher or lower)

proxmox is pretty much libvirt/qemu with a nice GUI.

you can in thoery just set up libvirt and virt-install on any linux server and use that for virtualization.

its a pretty manual set up but it would get the job done if you don't mind learning a few things.

I'm going to defer on these suggestions as my only experience with proxmox is tinkering, and none in Ovirt.

Cool. Thanks for sharing.

yeah, that's one of reason's I'm interested in it - less crapware to mess with.

thanks bud!

Any time.

As far as HA goes, as I realize that I've missed that, proxmox HA works fine. Last I remember, it did take a bit of work to get it going, but again, that was on V2 or V3, so V4 may be different. Give a look at the HA docs.

As far as comparing it to ESXi, I'd say that Proxmox and ESXi are very close to feature parity, so I'd not be worried about missing out on a feature.

Proxmox is a nice front end for KVM. I've personally deployed it to dozens of small businesses around my city. I would absolutely love to tell you about a company I worked with a couple of years back who chose KVM over ESXi and never looked back. Unfortunately due to NDAs, I can't really go into details. But I can tell you that the company in question has literally thousands of stores across the US.

I'd personally stay away from clustering and focus more on service restoration time, but that's me. You may have more stringent SLAs, and/or the ability to create a fully HA solution end-to-end. Remember, your virtualization nodes are only one piece of the puzzle. Don't forget about shared storage, and the network connecting your clients to the servers.

On a final note, I would highly recommend not installing libvirt on Proxmox. Proxmox doesn't use libvirt, and so it's a crapshoot as to whether or not their web interface will detect VMs made with libvirt, or vice versa. Proxmox is a turn-key solution. By adding another management method to it, you're undermining the very thing that Proxmox brings to the table. If you want to go the libvirt route, which is a perfectly valid thing to do, grab the latest Ubuntu Server LTS and go hog wild.

1 Like

Hey Levitance,

Thanks for the input. We're pretty set on our internal HA and we aren't looking for Vcenter or the alternatives to provide HA for those systems, but it would be nice to have the VMs at least restart on running hosts. Ovirt and Proxmox both do this reasonably well, but I believe Ovirt can failover with 2 nodes where Proxmox typically likes at least 3.

Cheers!

1 Like