Starting small civil engineering firm with large compute needs

Hardware Hub
1

My civil engineering firm (coastal, water resources) is separating from our large parent corporation to start a small business. We are developing a plan for our IT infrastructure which we can build from the ground up.

Unfortunately we have to go with SharePoint and MS Office for office work, communication and collaboration. The managers and EITs would revolt otherwise. However, we have fairly serious compute needs to run simulations and provisioning this is up to the senior engineers.

Right now two plans are being floated.

  1. Everyone gets a weak laptop and engineers get a > 24core desktop. Our workloads can sometimes benefit from more cores and for these cases we would use some cloud provider to spin up > 100 core VPSs. We also need GPUs for some models, but nothing too serious.

  2. Everyone gets a weak laptop and we lease servers to run our simulations.

My question is, can option 2 ever be as cost effective as option 1?

Secondly, if we go with option 1, what kind of CPUs/ builds are the best for parallelized simulations (usually implement with MPICH2)?

2

Civil Engineer who got advice from an IT guy.

He suggested option 2, using a basic PC but renting AWS for the heavy compute work. That probably makes sense for a business with multiple users, because you avoid the cost of constantly updating your equipment to the current standard.

I went with option 1.
A 20 core workstation with a Nvidia CUDA GPU, to accelerate my CAD (PTC Creo). As an individual it made more sense for me to have a kick-ass PC 100% of the time, rather then renting AWS for the few times I need that much compute.

1 Like
3

Everyone gets a laptop / weak laptop, whatever - needed for meetings and what not + non engineers can hook up a 4k screen .

Engineers additionally get a mid range workstation (e.g. 7980 threadripper based).

You might need local network gear for IT and so on.
Use AWS/GCP/Azure by all means, but use them as dumb VMs and/or for hosting k8s with containers. i.e. try to be as cloud agnostic as possible by relying on the commodity features of clouds. That makes it easier to jump clouds or use smaller vendors for non-specialist workloads, which ends up being dramatically cheaper.

1 Like
4

work out how much its gonna cost to provide the pc’s and cost to run them.
excluding the cost of personnel.
take the amount of data they would potentially crunch in a year, and ratio it against the cost of the system. so if they cost you 25k and your crunching 1000tb or 100000000 gigs works out at 0.4 bux a gig (note not real numbers)
thats your comparison base.

then you way that against the cost of running your work externally which will depend on what hardware you use and the amount of data you process, and its cost to process it…
look up a few plans.

if it costs more per gig to do it remotely then forget about it and get the pc’s and laptops

i know that sounds like RTFM…
but it really is something only you can figure out because you have your numbers and projections for the amount of work your company will process.
so while we can help with the hardware its up to you to decide if the cost/benefit is worth it either way.

2 Likes
5

The other thing to consider is how secure your data is vs how secure your data needs to be.

Always have a copy of your data locally if you decide to crunch it on the remote servers!

6

Former Chemical Engineer/PE/MS/MBA here; who also used to teach at a biz school; and did the IT consulting bit…

How many people are we talking about here? When you say “small firm” I think less than 12, 20 max.

How many owners vs non-owners?

How many engineers vs non-engineers?

Do people need to work remotely? Who? How many? For what jobs? VPN needs?

Off-site backup? Not just to cloud. If AWS shuts you down for any reason does your company disappear like Parler? Remember them?

My personal bias - the less you have to rely on other companies for mission critical stuff the better. Even if you do all the compute in the cloud, it still may be usefull for you to locally have a bunch of threadrippers and epycs on hand. Even if only for smaller jobs and test runs.

Can you get by with used equip? College surplus or other large company surplus? I know a guy here in Phoenix who grabs over 1000 Dell precision laptops & desktops everytime American Express decides to upgrade something. People like that are not hard too find. I found him on Craigslist. My dad has a really nice precision laptop now.

Finally as a current small business owner, who has been in biz since the 90’s, if the OWNERS say linux/ libre office / nextcloud (or whatever) and “The managers and EITs would revolt otherwise” fire them immediately. They will fight you on other things down the road. Too often. It’s not worth the trouble as you deal with those other issues.

I’m assuming by EIT you mean “Engineer’s in Training”? If they’re not willing to learn new systems what kind of engineers are they? Or will they become? Every new job and project you’ll have to learn something new. New simulation software, programming languages, etc.

Remember - they will get unemployment. YOU as the biz owner(s) will not. All the risk is yours, none of it is theirs.

Good luck and have fun with it. Seriously it is immensly rewarding.

2 Likes
7

You didn’t mention which operating system you are using, but if it is windows, you may consider getting windows multipoint server. That lets you use a single higher end computer with several workstations.

The easiest way is to assign each workstation 1 gpu and 1 usb hub then share the ram and cpu cores as needed.

The server I recommended at this link, btw this was written before thread ripper was released, and that may be a better fit for your needs, but you may need to check with your models on the balance of ram bandwidth vs cpu clock speed.

cost $11k for 96 cores and 768GB of ram. The equivalent computer from dell was $57,000.
The 16 core 96GB version of that build is $3500 and you can scale up as needed. the dell equivalent was $22000.

if you get:

then mount 2 of your GPUs to the side, you can fit 4 GPUs in it.

You can fit more with a:
some of the devices from:

but it isn’t as clean.

allows you to fit more GPUs in the space as they each use less space physically.

8

Any recommendations on a vendor for the workstations? We probably need ~10.

9

Right now, we are planning to minimize overhead and work from home. The multipoint server might work out when we get office space.

Is it easy to reconfigure the provisioning? I’m thinking about allocating all the resources to a single job when we need to run large models.

10

Yes, 12 - 15 people max.

Everyone there at day one will get some ownership, but three principles will own 70% of the company.

Everyone is potentially an engineer. However, some will focus on biz development/management and others aren’t power users (yet!). I’d say 7 - 8 engineers need access to the most capable machines that we are able to provide.

Offsite backups might be required by our clients. A good thing to look into, thanks!

I’d be interested in used hardware. You say just look through craigslist? Were based in TX/CA.

They could learn another office suite/cloud provider, but I don’t want the responsibility for that decision. I imagine I’d be on the hook for all the training/troubleshooting. I don’t know how much trouble that would be, but I’ve got other things to do… unless there is significant cost savings.

11

OK, So with that ownership structure the “Genghis Khan” management methodology would probably be counter productive. :stuck_out_tongue_winking_eye:

A more collaborative approach for these decisions may be needed.

12-15 engineers, atm there are no other support staff? And I’m guessing you will be the relevant geek/IT (in addition to doing engineering)?

This is looking familiar.

Yes, I am putting affiliate links here…they are listed AFTER the direct link to the service

Hardware follows software…

Starting with Office software

Remember - you can always switch to paid software. And if something runs on multiple systems (like LO) who cares what they run

Accounting: Unless you need the features, stay away from quickbooks. They litterally will NOT let you export your data, on their help pages they explicitly say that you “don’t need to back it up” because “they do that for you” since it’s all online. So unless multiple people need live online access stay away. Your accountant/payroll/benefits company will obviously play into this. Side note: I’ve never had one complain when I sent them reports or info in spreadsheets or gnucash. Gnucash runs on both Windows and Tux

Basic Office Suite: Obviously I’m biased towards LibreOffice (Google Docs without the spyware). I started LO when MS wouldn’t let me format a report the way I wanted. LO did. And at the time I was an MS developer. For what it’s worth, everyone I installed LO for (from 20 yr olds to 80+ yr olds) was happy. You may already have copies of MS laying around. Most people I found (1 exception) were happy with free over monthly subscription. And it runs on everything. Even if you end up using google docs, install LO on the local machines regardless.

Company email: Keep your email separate from personal!!! My reccomendation is fastmail (https://fastmail.com || (affiliate) Fastmail). Good features and apps, reasonable pricing. Plus if you need encrypted email it works great with desktop applications like Outlook, thunderbird, evolution, etc. (see hosting below for more on this)

Cloud Storage: Personally I like pCloud. Synch apps are good, runs on everything, lots of storage, stuff that you share with other users doesn’t count against their storage. So get 1 BIG storage account and everyone else gets the free version. You might be able to use this as your “offsite” while the big 3 partners each keep a local copy - especially since your not in an office.
https://pcloud.com || Affiliate 4 personal: pCloud - The Most Secure Cloud Storage || Affiliate 4 biz: pCloud - File Security Made Simple | pCloud

Hosting/email/storage/collaboration/etc: Google/outlook/fastmail you pay per user. At a company like Pair (https://pair.com || Ref code for 20% discount: pairref-VuSmVqzA) you get a crap ton of email accounts, plenty of space, webhosting 24/7 PHONE SUPPORT (!!!) I’ve been there for about a year and they’re great.

Engineering/Project Management
The tools you require on the engineering side will dictate what you’re doing there - probably NOT running windows on aws for example. Same thing with CAD, project management or anything else. But you already know this. Are you creating your own models in C++? Using a finite-element analyis package? THAT will determine your hardware requirements for the woorkstations.

If you’re working on a treatment plant for San Diego and the city engineers say they have to run it after you’re done on Windows XP (or unix or whatever) that’s what you’ll do it in regardless.

Now onto hardware:
First off: https://www.dellrefurbished.com/
And Lenovo: https://www.lenovo.com/us/outletus/en/

Don’t ignore craigslist/facebook marketplace etc. Last year here in Phoenix a graduating Mech Eng. PhD student was selling his threadripper (64 core system). $20k worth of hardware for 10 grand. On craigslist no less.

Office work you can realistically run everything. One company in LA uses gaming laptops for just about everything. They’re powerful, look pretty, staff feels their getting good tools so morale improves and they don’t need “certified” systems.

I would imagine you will need “certified” systems. Especially if you’re going to put your stamp on it. Assuming you guys are active in your local engineering community (professional orgs such as AiChE, etc) you can network there to find out who’s replacing equip.

Hopefully this wall of text helps

2 Likes
12

I know puget systems, never used them because i live accross the pond, but they tend to build systems themselves without the bullshit sales tactics that dell/hp use.

What software are you running? I’m in mechanical engineering but if i hear civil then revit and that stuff comes up for me. But that doesn’t benefit from a lot ofr cores. What simulations are you running?

13

Google workspaces might be a good alternative. but sheets is still not a first class experience compared to excel.
for those few who need it office 365 subscription could be a option.

14

These are a hard requirement imho regardless of your clients needs / wants.

No one wants to get the 3 am call that the server crashed / the company you were hosting your data went out of business / internet is down and you need the files for a 3 pm meeting / natural disaster… yada yada yada!!!

1 Like
15

Sorry, they announced that they are discontinuing the feature after windows server 2019.

16

AWS has bill shock and AWS accounts need a decent amount of management, which is expertise and time you’ll need to pay for.

If you just create an AWS account for the team to use, someone will spin up an expensive GPU instance and forget about it. Or you’ll end up with private data sitting on a public S3 bucket. So you end up needing someone as an AWS person, to be in charge of keeping costs and security under control.

Since you have to issue all your employees a laptop anyway, a beefy workstation option doesn’t add much extra work for anyone.

I mention this because it’s very hard to quantify AWS costs ahead of time, many a time people proposing will assume your AWS setup is the best case situation - that in effect you aced your AWS setup for free. But it’s hard, a thing sitting on your desk is much easier to reason about. There will be less token bucket exhausted throttling of your IO, less virtual network engineering to avoid egregious NAT gateway frees, oh how I could go on…

2 Likes
17

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.