Proxmox is a very powerful tool, which I have used for many years. It even supports the use of live-migration of VM's (and containers) between other Proxmox nodes. What is Proxmox?
What is Live Migration?
Live migration is where the VM being run is migrated to another Proxmox node whilst it is still running. (There are some requirements, see below)
However, this functionality requires that the user manually migrates the machine between nodes. As a result of this I have started working upon allowing Proxmox to auto migrate a machine depending on what it is doing.
So my goals:
- Migrate VMs between machines without user input.
- Depending on VM machine load status (currently only CPU usage)
- Not to use any external package that are not already installed within Proxmox
Future Goals:
- Allow load balancing of VMs depending on Proxmox machine load (not just the VM load).
Why do I not want to use any external packages??? Well the problem with being reliant on other stuff is that it may go out of date, stop being developed, have additional security issues etc.
How have I done this: (Currently limited to only Linux based VM's)
Using BASH (woooo bash) I have created a small script that SSH's into VMs (using a pre-shared key, so that password input isn't required (can be added)) and simply runs the command "cat /proc/loadavg". This returns the VMs current load over the last minute, five minutes and fifteen minutes.
When the VM starts to exhibit high load over a minute (more than 1 core usage on a dual core VM), it starts warning and monitoring the five minute average. When the 5 minute average exceeds more than 1.5 core usage, it migrates it to what would theoretically be a machine under less load or a Proxmox node with faster processors (the example I use here)
The machine is then migrated back to the other node when load over the 15 minute load average drops below a half a cores usage.
A silly video: (That uses game servers as an example)
www.youtube.com/watch?v=zNz4YMZIFYg
Thanks Tek Syndicate for the music.
Requirements:
- 2 or more Proxmox nodes setup into a cluster. (See here)
- An NFS system to host the disks (You can setup this on one of the nodes if you really wish to)
- A system to use as a "Heartbeat" monitor (aka the one running the script)
- The machines being monitored have an SSH pre-shared key setup with the monitoring system. (How to)
Code: The variables which define when a system is migrated can be adjusted as needed.
- #!/bin/bash
- echo "CPU load script by zanginator"
- echo "Monitoring Ubuntu"
- echo ""
- declare -i low=50 //low setting
- declare -i high=100 //warning setting
- declare -i caut=150 //migrate threshold
- declare -i loc=1 //machine location, used for loop variable so it is known which proxmox node to message.
- while true; do
- while [ "$loc" = "1" ]
- do
- result1=$(ssh user@host "cat /proc/loadavg | awk '{print \$1*100}'")
- if [ $result1 -gt $high ]; then
- echo "CPU load is high"
- result2=$(ssh user@host "cat /proc/loadavg | awk '{print \$2*100}'")
- if [ $result2 -gt $caut ]; then
- echo "CPU load is critical!"
- echo "Machine is now high requirement"
- echo ""
- echo "Starting Migration"
- exec=$(ssh root@proxmoxhost1 "qm migrate 100 proxmox2 -online")
- echo ""
- echo "Machine Moved to Proxmox2"
- loc=2
- fi
- fi
- sleep 30
- done
- while [ "$loc" = "2" ]
- do
- result1=$(ssh user@host "cat /proc/loadavg | awk '{print \$3*100}'")
- if [ $result1 -lt $low ]; then
- echo "CPU load has dropped"
- echo "Machine is now low requirement"
- echo ""
- echo "Starting Migration"
- exec=$( ssh root@proxmoxhost2 "qm migrate 100 proxmox1 -online")
- echo ""
- echo "Machine Moved to Proxmox1"
- loc=1
- else
- echo "CPU load still high"
- fi
- sleep 30
- done
- done
The code is pretty self explanatory if you know basic programming (or at least functions)
I'm sure the code could be cleaned up, although this was more a proof of concept I wanted to share. Expect this to be updated in due time. If you make any modifications, please share them. (Just credit me for the original idea/code)
Hope some of you find this useful.
-zanginator