Does a SAN make sense?

So I startet working at a local community.
Their IT equipment is outdated as hell. I mean EOL for all servers and network devices. Luckily the firewall still had service.
We are now tasked with updating the entire server and storage side to new systems.
We have a couple of Virtual Servers. Nothing fancy. Two AD servers, exchange. Couple File servers, two sql systems and that’s about it apart from some support hosts to manage the voip system and so on.
All in all let’s say about 10 TB of VMs and Data. Since we do not need enormous uptime and a system can be down in case of error for a few minutes or hours. Do we really need an expensive SAN to run our virtual machines from? ATM all our VMware hosts run on local storage so no vmotion in case of error or anything. If the VM hosts goes down that’s it. Please start from backup .
In that case would it be ok to migrate the virtual machines from local vmstorage to a NAS. License VMware essential plus to enable vmotion in the cluster and run the machines from the NAS with sufficient bandwidth.

Also we could enable replication between two NAS systems to enable redundancy even if the data is slightly behind on the passive system.

Thanks for your help.

Do you have some type of backup plan? That would be my first step. (yes some sort of san could do that)

Backup from all maschines daily, weekly and so on a different NAS and off site as well

1 Like

No but you could do some cheaper stuff, you dont need to buy the new Dell / HP etc.

You could go with a different virtualization to save costs if money is an issue and you have time to invest in learning.

Do you have a budget / IO needs, is 10TB pretty accurate in current use? What is your projected growth?

if your hypervisors are using local storage… how much are you able to add drives to them until you run out of slots?

1 Like

Not really since we a few older colleagues who really have problems with keeping up with new stuff and finding competent it personal is hard in our area.

10 TB is accurate growth would be about another 10 TB in the next 5 years since we outsource all hungry applications.
IOPs is another question. Any tips how to figure those out with all this spinning rust and local vm storage?

So… what a san will get you:

  • shared storage between VM hosts to enable things like live migration for vmware.
  • possibly: improved performance vs. local disks
  • possibly: improved resiliency vs. local disks
  • possibly: enterprise features like replication and snapshots

What it also causes you:

  • increased complexity: VM+SAN (or NAS) is more complex to configure and administer
  • increased cost
  • other associated possible cost increases: for decent performance you’ll need either 10+ gig ethernet (preferably 40-100gig) or fiber channel. even if you don’t need the speed for network throughput storage needs high bandwidth for IO command latency as well as throughput (i.e., you can only send so many storage commands down a network connection due to framing overhead, etc. - it isn’t just bandwidth throughput for the data you’re transferring - the speed of your network will impact the command rate you can send.

Is it worth it ?

If you can afford it, it is nice. But it depends on the business appetite for cost vs. reliability, performance requirements, etc.

it aint cheap unless you DIY but then you’re adding all the complexity, points of failure, etc. without a decent level of support - its on you. You still need to be pretty aware of a lot of moving parts and how to diagnose/troubleshoot performance issues or faults as they occur.

edit:
Oh the other thing with SAN/NAS storage. Generally you’ll run out of throughput before capacity unless you’re talking archive style workloads. VMs? You’ll hit throughput bottlenecks most likely first.

2 Likes

Already all full. If we run out of storage in a host we have to options. One migrate the vm to another host with more storage the old way… two … move old data to archive partitions on other systems.
Yes that’s fucked up i know.

You can see it under the monitor tab then performance and overview for the datatstore you are using.

1 Like

Will check that thanks.

Well money is an issue so is adding complexity. I can document and configure a load of stuff and get them running at decent speed but what if i hit a tree. None of my colleagues are technical advanced enough to understand what that should do even if it’s documented well.
I setup netbox for system and network documentation…. The didn’t understand a the basics and how to use it.

This is a valid concern.

Also its easy to get in over your head.

What sort of storage workload are they doing? File? DB? Mail? etc?

edit:
Oh the other thing. If you’re a VMware shop, last i checked you don’t get vmotion/SAN/NAS attached storage for free. No more ESXi, you need a full vCenter deployment which isn’t cheap either.

You could go with HyperV instead and then fuck about with clustered volumes but that’s a whole other layer of complexity with Windows clustering services.

Unfortunately for the VM side it isn’t quite as simple as “add SAN” - the licensing costs for vSphere alone will hurt. If you go that way.

There’s always cobbling something together with open source, but it aint mainstream yet and you’re far less likely to get fired for going mainstream than DIY and having shit go sideways :smiley:

2 Likes

Like I said in the opening. Nothing fancy. Mail, Exchange, AD, some file systems that’s about. Oh also we are about 70 users

it does NOT sound like a SAN is beneficial for this infrastructure. here is what i would build.

1x 4u or tower server with local storage. this would be you VM host.

1x TRUENAS NAS box. i would NOT use this for VM storage directly. i would connect this and use it for a disk to disk backup solution. you could also use it for shared drives from the VMs so that windows ‘previous versions’ of files are available with the benefit of ZFS.

this configuration is easy to manage, and way less than a SAN as far as ‘learning curve’ goes. plus the cost is less. (the TRUENas can be configured with SAN like features if you need them at some point)

I was going to suggest something similar but you beat me to it.

I wouldn’t say you’d get no benefit from a SAN for VM stuff directly, but you are a bit small to justify that sort of deployment. Simply from a cost vs. complexity vs. not so heavy workload perspective.

As above i’d focus on using some sort of NAS (i.e., file shares) level storage for more frequent disk to disk backup rather than over-spending on full VM level shared storage, etc. You’ll get better bang for buck i think.

Can you add a JBOD to one of the esxi server? or both to give you more disks on the server, probably your cheapest option (would allow you to have backup versions on the VM on the both if you added a jbod to both.

Also probably lowest technical knowledge needed for this expansion (Assuming you have the rack space)

But If that one hosts goes down we would be doomed. That’s why I would like vmotion with at least two host systems.

Yeah more than 1 host would be prudent. Ideally you want say, 3 (or more ideally)… otherwise with 2 you inevitably end up tanking performance massively (i.e., 2x workload on 1 host). with 3 hosts, you’re only overloading the remaining two by 1.5x (assuming they’re all loaded - which will happen eventually) which is a little less… shit.

Without the heavy uptime requirement though vmotion might be overkill - do consider how long a restore will take though, copying a few TB around takes time.

edit:
also worth noting that if you do ever plan on doing windows clustering across hosts, there are some rules regarding how many hosts need to be up for the cluster to work.

The other way to skin this is to virtualise your storage across your hosts (hyperconvergence!) but the commercial solutions for this are either

  • expensive (e.g… nutanix, vmware vSAN)
  • somewhat flaky (vSan, last i heard, which was admittedly a while back)
  • combination of both

The non-commercial solutions? I’d run in lab but… i wouldn’t risk my career on experimenting with them at work in production.

edit:
and yet another way to skin this is just buy more internet and stick it in AWS/Azure/etc.

2 Likes

If the VM performance tanks because one host is down that’s ok until we got our hands on a new host. If all systems are down for a couple of days we are doomed

1 Like

Now… free alternative… that i’d run in a lab and would “work” but its a lot of effort and “you’re it!” - but maybe workable for your situation but comes with associated risks

  • 3x Linux hosts with a bunch of RAM and disk
  • KVM for VMs
  • ZFS for VM storage on 2x hosts
  • third host for backups (also running ZFS) via ZFS-send snapshot destination - possibly in another campus

zfs-snapshot 2-3 times per day on both your VM hosts, send the snapshots to the backup host (with presumably more, slower disk in it)

host breaks?
zfs send snapshot from the backup to the other host, load the VM from the recovered dataset on the other host.

cheap? yes
you own it? yes
your problem if it breaks? also yes

1 Like