I work for a small software company (five developers).
We develop a java/postgresql application and for now all our servers are running on D2s ( 2 vCpu and 8GB ram) and D4s (4 vCpu and 16 GB ram) vms in Azure.
For our production environment, this works just fine and we find that the price we pay to Microsoft is fair. We don’t worry about servers failing, uptime and stuff like that.
But we keep expanding our testing environment, and at the moment it looks like we are going to be paying almost $2k every month just for testing VM’s
So now I’m contemplating if it would be a good idea to move our testing environment from the cloud, down to a on-site server to reduce costs.
My plan is to buy a 1U server with a 16c/32t to 32c/64t cpu, 64GB-128GB ram and 1TB ssd’s.
Then move all our testing environments into docker-containers and run those containers on the new server. All our configurations are in Ansible and migrating from Azure to on-premise server would be a breeze.
Seems to me that I could save a lot of money by doing this, but I’m afraid I’ll spend time managing the new server.
Although when it is up and running I hope that it will be as easy to manage as our cloud-vms.
Does this sound like a smart thing to do?
What are the pit-falls that I’m not seeing?
Are there other cloud-technologies that I should consider for this use case?
I realize that this is kind of an open question with a lot of variables.
But I would love hear your opinions and experiences.
if you want a testing environment as close to actual Azure as possible you can get it with on-site-servers IIRC Microsoft have some Azure stack you can run on your own hardware. but otherwise on-site is better for a fast developer cycle IMO
Dev Vm’s in Azure - do you use run the Dev VM’s 24/7 or could you set a schedule to shut them down at weekends and overnight? If you only ran them 12 hours a day Mon-Fri that would be quite a saving. You can create scripts that use Azure Automation to shutdown and start all VM’s within a Resource Group.
You mention using docker on-premises, if you used Docker on the Azure VM’s would that reduce the overall number of Dev Vm’s needed in Azure?
How much VM power is going to the PostgreSQL databases? PostgreSQL as-a-service in Azure is really cheap for basic tier databases. These can also be re-sized dynamically, so you can scale them down at weekends etc. and Scale up when load testing. This should help cut the number of VM’s.
Since your workload is portable (nothing too Azure propritary by the sounds of it) you could also do a price comparison with Digital Ocean. In my experience DO is a lot cheaper than Azure and also now has PostgreSQL as-a-service. The equivilent of a D4 v3 will probably be about $100 cheaper per month if you runb 24/7
Both Azure and DO offer managed K8’s clusters, if you do containerise the dev workloads checking on the running costs for these could be worthwhile.
On prem servers, can be cheaper, but you can’t scale them as quickly and if you need to access them from outside the office you need to configure a VPN, and then you have all the management and patching etc. The premium price for cloud can be worth it, if it rids you of needing to become the defacto admin of the Dev Servers.
Azure does have discount schemes if you commit to run a VM for 1 or 3 years. This might not be available for all agreements, but can save a lot if its a plan that works for your company.
Overall cloud tends to be best when you are using the Platform-as-a-Service type solutions (that tend to lock you in to a single vendor). If you are just using it to host VM’s the cost savings are not there unless you can be smart about how you do it.
Most of the time the VM’s just idle. They are used for manual testing and automated tests buy our build server triggered either by a cronjob or developers pushing code through git.
We tried shutting them down in the evening, but this was annoying for the developers who are working flexible hours. So for now we are keeping them on 24/7
I’ve also contemplated if we could start the servers by command, either using a git-hook or by talking to some slack-bot. Then we could keep the servers off at all times when not used. My biggest concern is boot time before the testing environment is booted and ready for use. I think this would be the best option if we could actually get it to work.
I could run a DS15v2 with 20 vCPU and run docker on that one, but that is also roughly $1300 each month, so I think I could do better with on-premise server or some of your other suggestions.
This is something I’ll look into, but our database load is very small.
I also like to be somewhat independent of our cloud provider.
I want linux to be our platform.
A good tip, I’ll look into it.
I’ll look into this as well.
Management and patching worries me. We already have a VPN for users working remote. The Azure VMS are only available from our network (site-to-site VPN with Azure). There are no public ip access to our servers.
This is also something I’ll look into.
For now Linux seems to be the best platform for us. Neither our software or our organization seems to fit the PaaS solutions. At least for the time being.
The nice thing about PostgreSQL as a PaaS service is that it doesn’t tie you to PostgeSQL on Azure. The only bit you do different is set your connection strings for a remote server, and one secured with TLS. The contents of the database can easily be dumped and imported back to an on-premises instance no bother.
You can also use the service endpoints to block access to anything except the VNET/subnet that your Linux VM’s are on. This optimally routes the traffic and keeps it all internal within the Azure data-centre. It’s a good feature of Azure.
If you can stick with Linux ( or at least Docker containers) as your platform you won’t get stuck being restricted to a single vendor, if needed you could probably also build DR in another cloud and be able to fail-over should Azure ever be offline for an extended period.
Infrastructure-as-a-Service is what you’re looking for, then. CenturyLink’s cloud offering will probably save you money hand over fist.
Back in 2013 I was working with an organization that was weighing whether or not they should move from CenturyLink’s environment (which they had been on since before CenturyLink bought it) over to Azure. The price difference was astronomical. For rather small virtual machines, we would literally pay 1/3rd the price that it would cost in Azure.
And on top of that, Azure’s disk performance (at least at the time) was godawful terrible. I remember running a test on a VM template Microsoft put out for SQL Server. It involved striping together as many virtual disks as the VM could be configured with (iirc, it was a total of 12 disks, 8 for data, 4 for log). It got on the level of like 3,000 IOPs, where CenturyLink was putting out 20,000 without any special configuration.
Might be worth checking them out to see if they’re still kicking Azure’s IaaS.