A noob setting up all the IT for a small business -mistake?

Hardware Enterprise Gear
, ,
1

I work for a small-sized empolyee-owned business and we’ve bought ourselves out from or mother company and so we’re in need of our own IT infrastructure.

The requirements are fairly simple.

  • Storage server with backup and remote backup (TrueNas most likely)

  • VPN and client solution for employees to access file server and maybe software licenses (we’re not that many so might just have local licenses )

  • Networking for the office, Wi-Fi for most and wired for a few workstations.

Alternative addons to consider are

  • Email hosting

  • Website hosting

I’ve considered trying to set this up my self since I’ve got 4 months before it actually needs to be working I can take my time fiddling with it.

Is it a mistake to try and DIY a company system like this?
Are there any obvious pitfalls to avoid when it comes to creating a secure and independent system like this?

I’ve never done it before, but I think I could set this up for personal use without major issues (I watch a lot of L1 techs >_<) but setting up the IT for my livelihood is not something to do on a whim, but it could be a huge cost saver (or cripple the company if it’s unsecure and buggy).

So far I’d be looking at buying Ubiquity networking gear (sorry wendell, there are limited options in Norway) and a bunch of drives to set up a server and if it all fails we can hire people to set this up for us, might even save money on the hardware this way.

(Questions in bold)

2

4 months to fiddle with it? Not enough time. Hire a company.

6 Likes
3

Wot he said. Even in Norway there are small IT consultants that won’t charge you an arm and a leg. What you can do, is wiring everything yourself, that’s usually the biggest cost as it takes the most time, so a big saver for your emerging company. Cat 6 is the minimum, but preferably wire for fiber. Terminating the fiber-optic cabling should be done by professionals, for Cat6 you can do it yourself. The FibreNinja YT channel has good examples of what (not) to do when wiring your IT, regrettably the chap hasn’t posted in years due to personal circumstances.

5 Likes
4

^^^ What he said. If you need cabling and wiring done, get the supplies and do that yourself. It will save your company a metric butt-load of money not needing a company to pull cable and terminate it. Just be sure to plan ahead for the amount of drops you will need and WHERE the switches are going to be located.

5

I second the motion of getting a consultant involved. Here are some things to think about though;

  1. Get a proper rack. This could be a 10 unit half height rack that fits below the desk of an employee, but it is enough to get you started.

  2. Get a proper 4u enclosure that can fit a ton of 3.5" drives. Do not fill that to capacity, start with 4 drives or so.

  3. Do not go for threadripper/epyc/xeon right now. A 5950x on a b550 would be more than enough to get you started. Upgrade to epyc as business needs grow.

  4. Do invest in industrial grade firewalls. You can get those for $300 or so, which is nothing, and will save you a ton of money and electricity.

  5. Have a single file server/ web server/virtualization machine and do virtualize the services separately.

  6. Get separate NICs for your file server and everything else as that link will be saturated.

  7. Wire up your office properly. Make sure every office desk has their own port and leave room to grow.

  8. Give each wifi access point their own dedicated cable and group them logically in their own little network, separate from your wired network. Either only allow specific MAC addresses on the wifi, or treat it as an untrusted network.

  9. Get a consultant, discuss, learn, implement.

6 Likes
6

Yeah, you’ll likely get it working but overlook some detail that will cause either small or enormous problems much later. Network is especially difficult for an uninitiated person to wrap their head around.

If you have to set it up yourself, I would get 2 Synology’s (active/backup), backup to backblaze or similar and use Tailscale for VPN. Use consumer network gear that you fully understand. Don’t get Ubiquiti or pfsense or Mikrotik if you have no networking background. A small company can run off of a Linksys or ASUS or whatever consumer wireless router you prefer until you can hire a professional.

Avoid at all costs.

2 Likes
7 
Get a proper rack. This could be a 10 unit half height rack that fits below the desk of an employee, but it is enough to get you started.

Get a proper 4u enclosure that can fit a ton of 3.5" drives. Do not fill that to capacity, start with 4 drives or so.

Do not go for threadripper/epyc/xeon right now. A 5950x on a b550 would be more than enough to get you started. Upgrade to epyc as business needs grow.

We had about 10 TB of storage before I joined the company, now that I’m working with 50-300GB data sets the first server would have to be over 100TB (we need to store all the data for a 1-2 year period as it would be unreasonable to expect them to pay for us to collect a new set if they need something else).

The new office will have a dedicated server room and will most likely have a 30U rack (for expandability).

The goal is to have a secure yet easily accessible storage server with 1-2 clients having local hardlined to the storage for transferring large datasets. (security risk?)

The rest of the machines would VPN into the storage server in the office or at home.

I was hoping to do as much as possible in house to avoid expensive subscriptions and cloud storage solutions in the long run. This whole train of thought was born from the fact that our business would be killed if we suddenly lost our data so we would need to follow the backup+offsite backup rule.

Breaking off from a larger company and shrinking in size by three orders of magnitude, I guess it just means our IT costs will go up.

Thanks for your feedback guys. I’ll keep looking into this in-house project and professional help concurrently

1 Like
8

That changes your needs quite a bit then, I was under the impression you were a startup that just started to hit the wall. :slight_smile: Still, if you need 100 TB of storage, that is around €350 per 18 TB drive, so make it 8 drives instead… Of 60 total drives. I’m talking then about servers like these: Storinator™ XL60 - Affordable, Big Data Storage

Don’t be afraid to bring in an IT consultant for the first few months, make sure you work with him on designing the network and learn from it though. A one-time expense is much easier to swallow. The idea is that the consultant will train you and teach you to be self-sufficient, then cut loose but with a small support contract of say, 5-10 hours a month. With any luck you might even be able to hire the guy. :slight_smile:

1 Like
9

This is your basic set of requirement:

  • Dedicated rack for storage/network gear, presumably where your interenet connection will be as well
  • 100TB of redundant storage, space to grow
  • VPN concentrator to allow for remote workers
  • local LAN for local users

Out of the 100TB you stated 50-300GB for datasets, is that a single user dataset and how much of that dataset is used by a remote user ? That amount of data accessed randomly would require a beefy pipe in …

Your next step, before deciding whether to hire or to DYI, is to set a budget, total or broken down into the various parts:

  • Storage
  • Backup
  • Local Network
  • Internet connectivity

Budget does not need to be strict, but it will give you/us a ballpark to direct over best balance between having all your requirements met and costing you a fortune and having the bare minimum going but probably force you to compromise …
While a storinator with 60 drives of total capacity is an easy win, a refurbished R73-0XD with 12x18TB drives would get you going for the first couple of years, and with the budget for a new storinator without drives you could probably get two of them with 100TB of raw disks each …
Also, it will give us an idea of your/your company real appetite for the diy vs the ‘enterprise’ approach … it all starts from your budget …

10

Yeah its a mistake. A mistake that will keep on giving problems over time.

If you’re starting up the IT for a small business you probably shouldn’t be buying server hardware at all in 2022. At they very least you should IMHO be thinking cloud first for manageability/security reasons.

I’d be buying them an Office 365 subscription; this will both give them storage/mail and licensed copies of office and manage them totally cloud. Set them up with multi-factor auth, 365 hosted mail, etc.

Use Onedrive/Sharepoint for their storage (or at least: azure files). This way if they lose/kill a machine they can carry on from a tablet/any random PC, browser, etc.

VPNs? Fuck that. Try to focus on using cloud storage/cloud apps (obv cached locally).

Unless you have legacy on-prem infrastructure or are a hobbyist, trying to run shit on prem in 2022, especially if you’re new to this stuff is most likely a mistake. Unless you have some very specific app that has an on-prem requirement… just don’t if you can avoid it.

2c - i’ve been doing on prem for 25 years, but given a green-fields set up i would not willingly choose that path today. And thats WITH my 20+ years of experience in networking, enterprise SAN/NAS, virtualization, etc.

The amount of money you’re going to spend on hardware, time, fucking around and software licenses to get them off the ground would probably pay a couple of years of 365 for a 10 user company quite easily. And that’s before you even think about backup software, tape drives, off-site DR plans, etc. The sub costs may (? its all relative) look expensive up front, but to do what you get included in a 365 stuff (or google docs, whatever), on prem simply is not cheap either.

All that said, you’re probably better off hiring someone who knows what they’re doing to help set THAT up as well. You won’t need much, just some assistance to make sure you don’t fuck up the initial tenant setup and record the relevant security details properly; you should have a break glass account that is NOT used for daily admin, but saved somewhere safe.

Trying to get an on-prem environment sorted out and backed up, etc. properly in 4 months by yourself as someone green is just a recipe for disaster. I’d definitely start cloud and if necessary add some on-prem stuff for the “can’t run in 365/azure” stuff as required.

Otherwise its going to be death by 1000 paper cuts of patch management, backups, troubleshooting performance problems or worst case recovering from data loss due to bug/cryptolock, etc. If the power goes out on-prem? Guess what - you don’t have infinite UPS capacity. Internet goes out on prem? etc. Cloud provisioning gets you resiliency for free.

Try and reduce the shit you need to deal with down to local switching/routing and administration of their tenant / local PC/mobile device management imho - otherwise you’re going to get overwhelmed with it and need help pretty quickly, even with skills there’s simply way too much to do, to run a secure 100% on-prem environment for anything other than a trivial dinky little home setup.

Also remember this:

  • if you set them up for the cloud, you’re forward thinking and going with the trend for the entire IT industry, getting them best-practice managed services, global reach, etc. The subscription costs are spread over time and require little up-front capital expenditure (shit, just deciding what hardware to buy for some unknown future workload they may have is a mission in itself). Adding new capacity is trivial and requires minutes of effort.
  • if you set themselves up yourself on prem you maybe save them a buck but if it all goes to shit you’re toast. If you buy the wrong gear? Find a glitch with the way your backup software handles Truenas? Gear blows up due to poor/insufficient power protection (or plain manufacturing defect)? etc.
  • billing for cloud consumption is trivial. what if person/dept X in the company complains about server capacity but it isn’t in the budget for a new company server. who pays for it?
1 Like
11

‘This’ is the story every IT manager (or would be IT manager) is trown at from eager Cloud ‘Solution Architects’ in fancy powerpoint decks who don’t bother for one second to listen to ‘your use case’

There’s no clear cut answer as to whether for new deployments it is economically more convenient to stay on-prem or to go cloud native, it is always a big ‘it depends’, except for standardized services like emails (google, 365, whatever, it makes no sense to keep them on prem)

The OP doesn’t require compute, serverless, containers, ELB or stuff, he needs shared storage to dataset files, the size of which is estimated at 100TB, no need for remote processing, no need for AI backed compute.
And he doesn’t want to spend a truckload of money, and is presumably aware of the compromises that would eventually entail.

For the stated used case, assuming the OP is willing to move from a central CIFS shared, on prem hosted storage solution, to a set of S3 shared AWS buckets, assuming he can accept data soverign relaxation (he’s in Norway, the closest AWS region is Sweden), assuming his workload can accept the perfromance hit you take when working with S3-fuse, the cost for 100TB of s3 storage before optimizations with tiers would be about 2400USD/month - 30KUSD year, 150K over 5 years, for S3 storage only

150K for a 5 year on-prem solution involving 2 servers and 250T of RAW storage is more than doable, even when factoring in the costs of an external IT company managing it for you , and if the risk appetite is there for going with refurbished/second to last generation hardware, that 150K budget over 5 years may as well include all the networking/connectivity/remote VPN and probably a hypervisor or two to allow people to remote in and work locally on the datasets.
The OP did not expand on how many users would be needing to remote in, assuming the usual 10-20 the cost of a VDI solution alone in AWS to support workstation workloads would be another 20-40K USD per year … these are not trivial amounts, especially for a workload that is known and doesn’t need to scale other than on the storage side …

3 Likes
12

Questions for doing this on prem:

  1. How do you plan on backing this up? You included that in your 150k right?
  2. Is the hardware fault tolerant? What about fail-over location if the office burns down, hardware is stolen, etc.? You included this in your 150k, right?

Its easy to spin up storage with no thought for DR, high availability, etc.

And the OP isn’t just after 100TB of storage. He’s wanting mail and website hosting as well.

Even if you can do it and meet the resiliency that cloud offers, you’re spending 150k up front plus someone to set it up and maintain it, only to have to forklift upgrade it and find someone to fund it when someone needs say, 50TB beyond the current server capacity (and power it, cool it, etc.) instead of pay as you go. Vs. just add some storage to the monthly bill.

If you’ve got a couple of hundred grand in the budget to get the project off the ground and an appetite for risk having a new guy set it all up… and can actually get all the hardware right now in this pandemic supply chain disaster, sure, go for it i guess?

Me, having spent the past 25 years doing this shit and dealing with the politics and funding through multiple upgrade cycles… i would not do it.

Now… if the user requires sharing massive data files between users collaboratively, then maybe some shared high speed storage is required. But that wasn’t clear from the use case presented.

And the question was whether or not getting someone who’s new to this stuff to set it up was a good idea.

Covered that.

1 Like
13

Well, the 150K in S3 do not account for backup/versioning/remote copies … do you want your 100TB backed up to a different region? Add the cost of transferring the data and storing it on a lower tier … say 15K/year … so add another 75K to your budget …

I am not saying he should go one way or another, I am saying that depending on the use case, and the budget, cloud native may not be an option …

Is the harware fault tolerant?
Two scenarios …
Oracle ZS7-2 storage appliance

  • 24*14TB RAW storage
    • 206 TB usable storage
    • 2x 11 disks RAIDZ2 VDEVs
    • 2x hot spare drives
  • 2x 3.8TB readZilla SSDs
  • dual controller heads, each with 256GB ram
  • 4x 25Gb network interfaces
    that will run you around 75KEUR+15K/year for support and gives you local, redundant, resilient, hot spares, and double the capacity needed … with space to grow to a couple petabytes if needs be …
    ZFS based, snapshots capability, NFS, CIFS, compression all form the most evil corporate provider …
    Add to that backup to Oracle Object storage archive for your offsite backup (260USD month/3200USD year for 100TB archive storage)
    and you have your on-prem top of the line storage solution with local redundancy and offsite backup in case of extreme catastrophic failure of your office …

Want to go cheap and less ‘corporate safe’ ?

2x Dell R740XD refurbished

  • 256GB RAM
  • 16x 16TB nearline SAS drives
    • 160TB RaidZ2 usable with 1VDEV / two hot spares
  • 2x M2 ssds
  • 2x 1TB intensive READ SSD
  • 4x25Gbps network cards
  • Additional HBA to connect external enclosure for expansion
  • 2x Xeon Bronze 3106 1.7GHz 8 Core

Truenas replicating snapshots from one server to the other (that would require spending money for an external contractor to keep it running/monitored)
Same backup to OCI object storage archive using S3 layer compatibility for offsite
Total purchase cost, including 5 years warranty on parts:
22KUSD/server → 44KUSD for two servers
3200KUSD/year for offsite cloud native backup/16K over five years
that leaves us 90K over 5 years to pay for maintenance/monitoring/setups, estimating 30K for the nitial setup and 12K/year for monitoring/maintenance we get to the 150K over five years
Is it the perfect solution? No, because you are taking on risk with refurbished hardware that may or may not last you five years (but most likely will last well over that) and yes, I am not accounting for power/infrastructure costs, but once you add AWS costs for transfers I may be surprised if they don’t come up higher than the cost of running your infrastructure on-prem

Agreed, and I am not telling the OP that he should just try that and pray all goes well, I am trying to be ‘open to various possibilities’ :slight_smile:

As for email and website hosting, 100% agree, these should go somewhere in the cloudy pastures

I hear you, and have been doing the same, probably on the other side of the fence (IT contractor :slight_smile: ) and have to deal constantly with either people not wanting out of their datacenters when their use case is 100% cloud native, and people that have been told by the CIO they need to move to cloud native in 6 months and their use case is 100% datacenter … it’s a weird world indeed.

14

There are plenty of instances where on-prem storage not only makes sense but is the only feasible solution. If your performance requirements exceed available WAN bandwidth and/or latency, cloud storage is inherently not an option. Based on the data requirements from OP, I imagine this is the case here, although I think cloud for off-site backup/archive would probably be feasible and recommended.

I don’t recommend it but I have seen tech savvy small businesses effectively configure their own Synology’s with back ups. I have never seen one configure a TrueNAS system correctly and I have never seen one configure network correctly unless they were using a consumer wireless router.

For the record, I have a side business as an MSP for small businesses…

4 Likes
15

100TB to me is a quantity that is already too much for Synology SHR/btrfs/md/custom raid config
The power of a ZFS deployment in that storage range comes from snapshots, and the ability to ZFS send them to a backup unit either local or remote without too much effort, also the integration of said snapshots with Microsoft VSS for accessing them through a samba share is something that is way underestimated …
Also, the ability of ZFS of handling the snapshot on a per dataset basis would fir this use case particularly well, IMHO, supporting different tiers of archiving/different models of backups and resdundancy …

I was a smart ass in my estimate and when I mentioned truenas I also included budget for an external it firm to do the setup, 30K would be a more than generous business to cover a two node hardware install/storage setup/identity setup and backup setup test and validation, I never expected the OP to get it going on his/her own as a first truenas deployment …

In general I agree with previous posters in saying that for this use case getting it done and working properly without prior commercial experience is going to be way too risky when compared to the cost for the hardware … but still I feel like the OP deserves to be explained what the challenges are and he/she needs to realize on his/her own whether it is something that can be done, eventually, or something that needs to be outsourced from day one …

16

I agree but would take an acceptably configured Synology over a poorly configured TruNAS.

2 Likes
17

Given the 100TB storage requirement, I’d suggest investigating the Nimbus Exadrive offerings. Top of the line is the 100TB all SSD Exadrive, but it comes at a cost (40kUS$ a piece), fortunately lower capacity drives are also available at significantly less then the linear relationship suggests :wink: Linus did a tear-down (of sorts) on it on the main LTT channel in either 2020 or 2021. Mind, those are SATA drives!

HTH!

18

Can do on-premise, is a PITA to get right though.

Put that on some hosting service externally, far away from the systems you are in charge off.

2 Likes
19

Community - Zentyal Linux Server is a nice solution to look at.

20

Hey guys, been super busy as expected so I didn’t reply here for a while. Thanks to the info I got here I had bidding round with a few different vendors and ended up with a vendor that let us buy the hardware we needed and they set it up at our location with a tunnel to their data center for backup. They’re great guys, however most of their competition are completely inexperienced with non-cloud based services.

From the 8 vendors involved only 3 could actually deliver a reasonable solution! :money_mouth_face:

4 Likes