Server for a small company

Hardware Hub Build a PC
1

Hi All,
I am preparing a shopping list for Black Friday for a small DIY server for my company. This is not for a critical infrastructure, if something breaks, I would have a backup solution in cloud (both for VMs and storage) that would be in sync and I do not have to replace parts fast to keep SLA.

What do I need this server for:

  • Proxmox as a main system
  • TrueNAS Scale (VM) - accessed via LAN only
  • W11 (VM) with Remote Desktop - accessed via LAN only
  • Ubuntu (3 x VM) - services accessed from outside (heavy use)
  • Some docker containers - accessed both via LAN and outside (low use)
  • Storage/data rsync with other servers, daily
  • HomeLab kubernetes testing
  • In the future storage for IP Cameras Surveillance and another 3 Ubuntu VMs

So far my list looks like this:

  • Case - Node 804
  • Motherboard - ASRock Rack X570D4U
  • CPU - AMD Ryzen 9 3900 OEM (12c/24t, 65W)
  • Cooling - some low-profile Noctua (I want to get something on sale)
  • RAM (ECC) - ‎2 x KSM32ED8/32ME to be run without OC @ 3200MHz (in the future I will add another 2 sticks and run all 4 @ 2666MHz since that mobo only allows that speed with 4 x DR sticks)
  • Boot Disks - 2 x 120GB SSD (Proxmox RAID1)
  • NVMe - 2 x 970 EVO Plus (2TB) or 980 PRO (2TB, PS5 edition with radiator) - for VM Pool, mirrored
  • HDD - 6 x 8-10TB, ZFS RAID-Z2 (passthrough to TrueNAS VM), probably HUS728T8TALE6L4 or some other enterprise HDDs, Seagate maybe
  • PSU - EVGA SuperNOVA 750 G3 (fan should be off when system idles or under small/mid load)
  • UPS - some simple full-sine UPS, only to be able to shut down server cleanly

I chose Node 804 due to a cube size and the fact that I can add additional 6 HDDs for a second ZFS pool. Server will be placed in a ventilated space, so there should be no issues with temperatures. There is also a space in front door for 2 x SSD, so I would place there Boot Disks.
I need IPMI and 24/7 use (non-critical infrastructure), so not to break a bank, I chose X570D4U. It has most of the features I need: 8 SATA (6 x HDD + 2 x SSD), 2 x M.2 - one via CPU and one via chipset, but both PCIe 4 x4 without compromises (if not considering one going via chipset as a compromise), max 128GB ECC RAM and PCIe4.0, so I can add 4 x M.2 and/or HBA for storage upgrade. I do not need more LAN ports and 1Gb is enough, since I am limited by the internet connection speed and current LAN hardware infrastructure. If that changes in the future, I can add 10Gb card via PCIe or M.2.
I want to set CPU in Eco mode, so when idle, power consumption is lower. 12c/24t should be plenty.
NAS would be accessed rarely by up to 5 users + rsync processes, so I am planning to give max 4 vCPU + 16GB RAM. I do not want to use Special Disks, ZIL, SLOG or L2ARC (this NAS does not to be very fast and also I can always add extra 16GB RAM to this VM). Other VMs would be run with remaining vCPUs and RAM and stored on mirrored NVMe disks - I am estimating that I would need max 1TB, so I still have some buffer before hitting 75% capacity on NVMe.

I still have some questions and doubts:

  1. Will I be able to run W11 without GPU - no gaming of course, only for remote desktop for simple apps? I know that mobo has ASPEED AST2500 GPU, but I am not sure how this works: if it is only for IPMI BIOS or also available for CPU?
  2. As for Boot drives I want to use mirrored (RAID1) consumer 120GB SSD drives. I know I would be better with some DC SSDs from Intel, but do you think if I install two SSDs from different vendors, different TBW (<100TB though), that could be a safe-ish solution? I am OK with replacing these SSDs every couple of years and rebuilding mirror. I would also have on-site backups on other NAS, so even if 2 SSDs die at the same time, I can handle the server being offline for 24h, replace both SSDs and restore from backup.
  3. Wouldn’t it be better in my scenario to use my NVMe disks both as Boot and VMs storage? Saving 2 SATA connectors would be grate (I could have hot spares for HDD without adding HBA), but I am worried that system logs would kill NVMe fast - they would also be consumer grade ones.
  4. Since TrueNAS would be in VM (on NVMe), but HDDs would be passthrough to be put into ZFS without other cacheing than RAM, do I need to configure it somehow, so all ZFS logs would be on HDD ZFS Pool rather that VM Boot Disk (NVMe)?
  5. Do you see any issues with hardware I have chosen, based on my needs?

I would really appreciate your input. Thanks!

2

Yes, you (qemu) can emulate a GPU - it’s not pretty but works.

As for the rest, install proxmox onto a ZFS mirror on nvme, and

  1. make sure you have backups
  2. make sure you’re monitoring hardware metrics for drive wear and tear
  3. you have some alerting mechanism when wear and tear goes into the red - so you can swap the drives

Clearly, lots of people run various OS that write all kinds of things, logs included onto nvme drives, proxmox isn’t the biggest offender in that category. (just don’t run it off of a crappy USB stick and you’ll be fine).

If you want to you could get drives rated for higher DWPD (daily writes per day) and/or drives that include power loss protection. e.g. a couple of PM9A3 drives maybe.

3

You may want to have a closer look at the Asrock Rack boards EPYC3451D4U-2L2T2O8R and/or the EPYC3451D4I2-2T

Both have tons of connectivity (breakout cables required!), including dual 10Gb networking and dedicated IPMI ports, PCIe/M.2 etc. Read the spec’s in the links above. The AMD 3451 SoC is a 16c/32t EPYC CPU with up to 100W TDP.

Downside is the price: for the EPYC3451D4U-2L2T2O8R I found an Newegg entry for 2k USD earlier this year (mid-late summer, IIRC) so well out of reach for home-labbers. But for business expense, quite feasible. I suspect its smaller counterpart isn’t actually that much cheaper.

HTH!

1 Like
4

IMO HP Microserver gen 10 plus with upgraded Xeon processor and extra ECC ram is the best no-nonsense SOHO server I know.

1 Like
5

Thanks for your replies!

@ulzeraj I was thinking about this, but too little expandability, especially for storage. I want to put everything into one casing and still have some options. No external HDD shelves.

@Dutch_Master This looks great, especially on the power consumption side. Also, lots of storage available without adding HBA (at least 12HDD + 2 hot spare) - I do not mind extra cables and ext cards. The price is high, though. I can get X570D4U for $400 and 3900 OEM + cooling for $300. I need to re-consider my budget :slight_smile:

@risk OK, thank you. Maybe you are right, NVMe mirror as a Boot Drive and VM storage (definitely better endurance of drives). I am thinking out loud here … if I do not need an extra GPU, what if instead of using 2xNVMe mirror (2TB drives mirrored, taking into account 75%, that gives me 1.5TB usable storage) I would install 4xNVMe in RAID-Z1 (1TB drives, 2 in M.2 mobo slots + 2 with a PCIe 4.0 card using only x8 from top SLOT - would still have option to connect something x8 in the bottom SLOT, i.e. HBA). Probably with the same price (higher power consumption though) I could get ~2TB usable storage (3 Data Drives + 1 Parity Drive, accounting for ZFS and 20-25% free limit, also only 1 drive can fail, but better R/W speeds). Do you think this approach would be better than mirrored drives?

6

I’m not entirely sure, it depends a lot on your growth rate and your “exit strategy” and opportunities for upcycling and recycling hardware.

For a homelab type setup, I see myself utilizing zvols and iscsi sharing features of TrueNAS - so that I could have block storage over the network, and from that perspective I’d maybe consider deploying additional flash to help with the performance of TrueNAS, and would minimize Proxmox storage to only store Proxmox itself and I’d force all other VMs even on the same box to use TrueNAS iSCSI shares.

I’d also see myself adding more boxes into this network at some point (and more network booting of things).

As an actual small business, I wouldn’t want to depend on a single server anyway - would prefer at least a pair of boxes with an ability for end users to switch between the two.

7

Amen to that. I generally tend to avoid HPE, but those MicroServers (and their ProCurve switches) are dang good for their price.

@gelmi Both the HP MicroServer and your DIY build should be plenty, I would even say overkill. for just 5 VMs and some containers, a 12 core CPU is a lot. I ran more stuff on more underpowered computers, but anything 6+ cores should be fine.

Not questioning the hardware, but I would ask you to reconsider the software. Why do you need TrueNAS Scale as a VM? I can somewhat understand Core, but not Scale, at least not on Proxmox. What does a TrueNAS VM offer you that Proxmox can’t offer as it is or with a few addons, like installing say Portainer on it, preferably in LXC (nested containers)? For storage, it can handle ZFS just as well (both of them are based on Debian after all).

From experience, generally speaking you want to manage as few VMs as possible and reduce resource consumption from virtualization. This reduced power consumption and the costs of the servers.

I used to manage 370-ish VMs with oracle DB servers with 2GB of RAM and 2 cores each, with 1 DB per VM. Well, in those 370, there were some web servers, but far less. We planned an upgrade, cut back some of them and combined 4 to 10 VMs in a single one and increased the CPU to 4 cores and RAM between 4 and 8GB. We cut about 30% of RAM usage by decreasing the number of VMs. We couldn’t quantify the power consumption, but I would think we decreased it too.

Well, in your case, the number is low, but you can optimize your costs by finding more efficient software solutions. I would drop to an 8 core. Can the Ubuntu VMs be converted to LXC on Proxmox? In my eyes, the only thing that would require to be a VM is Windows, which yes, should run without a GPU via RDP. Just that you won’t have GPU acceleration. We used to run some software for multiple people that was licensed per PC instead of per user (later they changed their licensing model, but we still had the old version) and we ran it on Windows server with RDP multi-user access licenses (cheaper than buying the software for everyone, lmao).

For the ZFS RAID, yes, you can put the OS and VMs on the same SSDs. My home server and gaming PC is a TR 1950x with 2x 2TB Intel 670p QLC SSDs in RAID1. These are mid-range SSDs and should perform worse generally than the 970 EVO and 980 PRO, about on-par with 970 QVO I think. And they can still pack a lot more than what I am running on them (well, I used to run 20+ VMs on a RAID10 of Samsung 860 PROs, those are SATA and they barely flinched).

I would suggest that you put the OS partitions on the SSDs and the data drives on the spinning rust. And if you are worried about SSD endurance with logs and swap, you can always put those on the spinning rust too.

Just putting it out there, but Pi-KVM is pretty good if you plan on having a single server. Even with more servers, if you have a KVM switch that can switch by a key-combo, like the obligatory shilling for the Level1 KVM, you can use Pi-KVM on multiple boxes. Not sure how you’d deal with the power button presses, I would think you need a dedicated controller board (arduino or something?) to have certain GPIO pins make contact to provide N volts to a relay switch to close the connection on the power button inside a server. When you are not dealing with entire racks of servers that would need to be hacked to get some power button leads to outside of the case, this is a good solution.

1 Like
8

@ThatGuyB Thank you. Regarding hardware, I know that 12c/24t seems plenty. Currently, I ran all my things on the Ryzen 1600 (6c/12t) and I still have some resources free. However, the VMs are for services that are getting CPU and RAM hungry, i.e. Mautic for newsletters with more than 10k emails or IoT injector gateway, storage (postgre and cassandra) and visualisation that gets data from >5k devices. During peak times 1600 is getting sweaty :slight_smile: Probably it would be OK to get some 8c/16t Ryzen, but I want this server to last 5-10 years, so that is why I am hunting for 3900 which I would put in Eco for now in BIOS. There is not much of a difference between 8c and 12c in price.
Regarding LXC, you are right, however I am running my services from ages on VMs and I would like to stay that way, since it is easier for me just to convert the VM and run the same, i.e. on Unraid as last resort backup (I know this is not ideal).
Regarding TrueNAS, I had Core as a standalone initially, after some years moved it as virtualised VM and stayed that way. I got used to using TrueNAS apps (just like Unraid apps) - very easy. I guess I could run all of them as LXC or even run one VM to rule them all, but old habits die hard :slight_smile: I may do that eventually as a learning process. I may end up just using backup of my current Core VM and restore it in Proxmox and just move also drives - I am not dead set on Scale, since I would have Proxmox to use. I just think that they are spending more resources on Scale development than on Core development nowadays.
I just want this server to run 24/7 with ability to be pushed to the limits in peak times (i.e. when 10k emails sent is being scheduled or self cleaning database records starts). If something fails, I have a less resourceful server in other place that takes over and if that fails I can spin up cloud VPSes from stored images/backups to take over for time being - it would take some time, but as I said this is not critical infrastructure. That is why I am trying to build it based on low-end server/workstation/prosumers hardware and not to bankrupt my wallet. But still, I want to do it in a smart way.

But thank you for your insights, I feel that I need to think some more about this before Black Friday. If you have any more ideas for me, I would really appreciate it.

1 Like
9

tbh i cant see anything wrong with what you want to build saying as you have a defined use case.
the cpu is power efficient even more so that its newer counter part but just as performative.

plenty of storage
plenty of scratch space on your boot drives.
i might consider adding high bandwidth nic’s for the lan.

but that would be about it.

fair warning on the ram though make sure you get the same make model and type.
often manufacturers change components and brand it the same as the original product because its similar enough.
but similar isnt the same and similar brings issues.
so be very specific when you but a second kit. make sure its the same make model and chip type.
along with the same speed and timings.

1 Like
10

Ok, you had me at Cassandra.

Anything over 6 years is running on borrowed time and is likely to use more power than worth its salt. I would plan for a 4 year upgrade cycle, using a smaller server (big corporations usually upgrade every year, medium-sized businesses, every 2-3 years, so 4 is an ok-ish number - I had servers run for more than 7 years, but I was seriously afraid those would fail one day, so I tried decommissioning them).

But then again, your customer doesn’t mind a 24 hour recovery time, so it should probably be fine. Since it’s no big production server and recovery can be slow, then probably the risk to run it longer may be worth it.

1 Like
11

Servers are expensive and their power consumption is relatively moderate these days. Dell estimates 3MWH/yr on a heavy workload for their fully kitted-out R740 servers.

Most areas in the US charge well under 20¢ per KWH. That would be $600/yr. A newer server likel won’t cut that in half or double your performance, you can hope for maybe 25%, which will take quite a while to show a return on the investment. And if your server isn’t under such a heavy workload, the power consumption will be quite a bit lower and the payback period much longer.

I’ve had servers running for 10, 15, and close to 20 years. The more of the same model of server you have, the more easily you can keep a couple spares around. Just do a drive swap and be back up and running in a few minutes after a hardware failure.

12

Is it really that bad? I know it’s written in PHP, but I’m pretty sure you can render 10k email templates within a few seconds on e.g. a pi zero.

… AIM for 2 years.

13

I agree that the projected lifetime of the system seems very optimistic overal and given some of the hardware selection. I wouldn’t project consumer SSDs and/or HDDs overall to last that long especially in a server environment.

@gelmi
Only the 3900 PRO is 65W
I would be concerned about cooling for all of this in such as small case and I would highly recommend that you fit as large cpu cooler as you can.
HDD: Get largers ones, 16TB+, Toshiba MG08/MG09 or Seagate Enterprise. Less is better

14

Please forgive me when I say this thread is a bit confusing to me because its not clear you want a server for a company or a sandbox to play and test things. Both are genuine uses but I believe you should never mix “x for homelab” with “server for company”. You can test kubernetes with minikube, mikrok8s or k3s on your own machine and those are as proper k8s as k8s can get. You can test an actual kubernetes cluster with inexpensive PCs.

There is nothing unnatural about a business running their own infrastructure despise what some cloud loving people might say but this might be the beginning of a story where you and the people who work with or for you might end with a bad taste in their mouth. Depending on your choices you might lose time and money dealing with issues and tunning things instead of focusing on the actual business so I instead recommend you to not overengineer hardware, software and network infrastructure.

Sorry to sound sour but I think its the best advice I can give.

6 Likes
15

Thank you all for replies and hints. I will take that in mind and rethink my hardware strategy before Black Friday. Thanks again! I really appreciate it.

16

8 or more VMs? Sounds like you need cores and plenty of them.
99+% uptime not a major concern?
Sounds like a job for used server gear.

image
image989Ă—852 193 KB

image
image962Ă—922 211 KB

image
image1520Ă—883 127 KB

I have an earlier revision of this board. Not hard to get it to fit in a Rosewill Rise case. UEFI video mode seems fucky (UEFI video and UEFI storage are separate options on it), I disabled the BMC because of fan behavior , other than that… been fine/issue free for months. And this is the later revision. Maybe, MAYBE, they fixed some of that stuff. Of course there are other options too. Getting a board and putting it in a case would be the hardest part; the rest is every day available.

17

I was sitting here thinking the same thing. If this is for business use, especially if OP is providing the system to them as a service, then they should be spec’ing something like an HPE ProLiant MicroServer available from CDW, not doing a home build in a Fractal case.

18

It is intended for our company, but if I can add 12c instead of 6c with only additional $100-$150, I thought I could also be using this machine as run some tests scenarios as I would normally do on a homelab machine.

19

Ideally you should be buying two. One for production and one for development and testing.

2 Likes
20

I know, however if I have the cpu and RAM resources in that machine, there should be no harm to run extra test VMs if that would not jeopardise NVMe I/O and network bandwidth needed for primary services, right?