Need help with home network/server setup

[nutral]

I’m looking into updating my home setup. including servers and also my home networking.
So a list of what i want to get into:

• 2,5gb networking for my computer, laptop and servers.
• Splitting critical items like home automation from media and nas tasks.
• Redundancy in things i always want to have work. this is home automation, web services and document availability.
• More bulk storage and more ssd storage.
• Energy efficient. i pay 40 cents per kwh and can’t write that off my business. but i can write off any hardware (so that gets about 50% cheaper.

I’ve made a list of what i want to buy but i have no idea if i’m going to run into issues for setting it up.

My idea is to use proxmox and run proxmox, all the vms and my document storage on 2 2tb nvme drives mirrored.
It would then have proxmox, all the vm’s, the config/local data of docker containers, and as a file share for small documents or things like photos.
i’d like to have it at a point where the hard drivers are normally spun down, only when watching a movie or needing to get at archived data would it need to spin up.

I don’t really know if this is going to work that well, right now everything is on the same machine, so i can just pass any folders that are needed for each container. but i don’t know if that works well.

I want to use my test server as a backup server (see thread below). That means i would run the services on this server except the 16tb file storage. I want to put the 8TB drive in this, and have the main computer backup here. Ideally it would take over everything except the media, which is not as important. Documents would run off the 8tb instead of an issd, but that’s fine.

What is the best way to run it where it is automatically redundant and allows me to update things without bringing it down? I’ve seen proxmox vms, kubernetes and docker swarm used for this.

Building a testbed small IoT / PLC interface computer

The mechanical engineering world has been a bit behind on the whole automation thing. So often enough outside of the PLC, not a lot is done with the data. I’m testing some services, to sell to smaller companies to gain insights from their data with opensource software. I wanted to build a really small but powerful pc that i can run multiple servi…

Intel Core i5-13500 Boxed € 250,47
ASRock B660M Steel Legend € 142,30

3x Seagate Exos X16 SATA (Standard model), 16TB € 248,70

TP-Link 2.5 Gigabit PCIe Network Adapt € 28,96

TP-Link TL-SG108-M2 € 157,46

G.Skill Value F4-2666C19D-64GNT € 91,75

Corsair RM550x (2021) Zwart

2x Samsung 970 Evo Plus 2TB € 103,99

Total about 1700 euro

My current setup is that everything runs off a skylake pc with ubuntu server and no redundancy (mergerfs).
Specs:
My old gaming pc. MSI Z170 ITX,
Intel 6600k, 16gb ram (quicksync!)
500gb nvme pci-e drive
14tb hard drives (8 + 4 + 2)
Coolermaster V650 (one of the more efficient atx power supply at very low load)

Power use: 33W with drives always spinning.

Software running:
Ubuntu server LTS (21 i think).
Docker with about 34 containers running at the moment.

Management

• Homer (for static landing page)
• Portainer
• Unifi controller
• Nginix Proxy manager (also for reverse proxy)
• Authelia for single sign on
• Grafana with Influxdb, Telegraf and varken
• Duplicati for backups to onedrive
• Code server for editing config files (still easier for me than ssh in with nano)

Home

• Home assistant
• Nodered
• Dahua vto2mqtt (to connect my doorbell to home assistant)
• Mqtt broker
• paperless-ng

Media consumption

• Plex (video)
• Booksonic (audiobooks)
• Ubooquity (comics)
• Calibre (application for managing library)
• Calibre-web (graphic interface to acess books)
• Restreamer (for streaming content to my friends)

indexing, downloads and others

• Teamspeak (for my friends)
• Sonarr, Radarr, Jackett
• Sabnzbd, Qbittorrent
• Ombi, Tautulli

Obligatory dashboard

[diizzy]

I would imagine that running that many VMs and docker instances won’t scale all the well etc but that seems to be your choice…

AM5 is by far more energy efficient than Intel’s offerings, given that you want to do some multimedia related stuff you might also want to have AVX512 instructions available. Downside is that you won’t have Quicksync out of the box however the entry level Intel ARC cards are cheap however I do not know how mature support is yet.

Asus TUF Gaming X670E-Plus - ~280 EUR
AMD Ryzen 5 7600 (or 7700) - 230 vs 330 EUR
Toshiba MG09 - 16GB ~240 EUR (270 for MG09 18TB)
Intel i225/226 NIC (I’d avoid Realtek like the plague) - ~40 EUR for a dual port NIC of Aliexpress, ~70 for QNAPs variant
Zyxel’s switch seems like a much better deal overall, https://store.zyxel.com/mg-108-zz0101f.html (119,90 EUR right now). Unless you need all ports to be 2.5G I think https://store.zyxel.com/xgs1210-12-zz0101f.html would be a better deal overall especially since it’s managable (VLAN capable9. 144 EUR of Zyxel’s own store right now
The Asus board actually supports ECC which is very nice and pricing of such memory isn’t that bad.
Micron 32GB DDR5-4800 ECC UDIMM 2Rx8 CL40 | MTC20C2085S1EC48BR | Crucial EU
You can of course go for non ECC, in that case you can also look at other brands than Asus regarding motherboards.
The Corsair isn’t bad but it does not offer a very good price/performance ratio, you can probably shave off ~20 EUR looking a bit more for a decent 80 Gold-rated PSU.
It makes littles sense to not grab the Crucial P5 Plus over the 970 Evo Plus given the very minimal difference in price
Crucial P5 Plus (zonder heatsink) 2TB kopen? - Prijzen - Tweakers

Your build:
DDR4 2666 is really slow, you want 3200 DDR4 as it will help overall performance noticeably
You might want to consider a somewhat better tier motherboard, I would highly recommend that you get one that supports more than 16x and only 1x PCIe slots in case you want to upgrade later on or add a HBA etc.
MSI Pro Z690-A WiFi DDR4 looks like a pretty good candidate in that regard

[wertigon]

Since you want two builds, one NAS and one virtualisation, here are my recommendations for small-but-powerful hardware:

NAS

If you don’t need very much storage (e.g. ~10TB or less) I would invest in an Asustor Flashstor 12 bay m.2:

Asustor Flashstor 12 Pro FS6712X - 12 Bay Reine M.2-SSD-NAS Netzwerkspeicher...

Start filling with cheap 4TB m.2 drives like the Crucial P3 m.2. This is probably the way to go for future storage NAS boxes. It does cost around €950 but I’m pretty sure you can’t get anything more energy efficient than that and it will last you until 2035 or so.

If you still need SATA drives, I would suggest a Node 304 build based on an Intel Core i3 13100:

PCPartPicker Part List

Type	Item	Price
CPU	Intel Core i3-13100	€148.97
Motherboard	ASRock Z690M-ITX/ax	€216.03
Memory	G.Skill Aegis 2x16 GB DDR4-3200 CL16	€55.99
Storage	Samsung 980 500 GB M.2-2280 PCIe 3.0	€27.99
Case	Fractal Design Node 304	€86.27
Power Supply	be quiet! Pure Power 11 400W	€61.89
Custom	ORICO M.2 PCIe M-Key auf 6 x SATA RAID Adapterkarte	€49.99
Total		€647.13

This should be enough to power 6 SATA HDDs.

Virtualization Server

The virtualization server is slightly more tricky. I would invest in the following build, YMMV.

PCPartPicker Part List

Type	Item	Price
CPU	AMD Ryzen 9 7900	€423.99
CPU Cooler	Noctua NH-L9A-AM5	€50.00
Motherboard	MSI MPG B650I EDGE WIFI	€234.27
Memory	G.Skill Ripjaws S5 2x32 GB DDR5-6000 CL32	€221.89
Storage	Western Digital Black SN850X 2 TB M.2-2280 PCIe 4.0	€130.89
Storage	Western Digital Black SN850X 2 TB M.2-2280 PCIe 4.0	€130.89
Case	In Win Chopin MAX w/200 W Power Supply	€148.61
Total		€1340.54

This should be more than enough cores for most of your virtualisation needs. If you need a PCIe expansion card in the future, swap the Chopin for a slightly more expensive Node 202 + an SFX power supply.

Also, keep in mind the above are suggestions, a starting point. Feel free to make alterations if I have misjudged your situation.

[nutral]

I was more looking for software and storage recommendations. I’ve already researched the hardware quite a bit.

Efficiency is important, because i pay 45-60 cents per kwh. so every 10W of use is 40-50 euro per year.

Intel is still more efficient when talking about idle power and that is more important for a home server. The APU’s are pretty good but i don’t really have a lot of numbers on and cpu’s. and there is good data on intel in a german forum. -Die sparsamsten Systeme (<30W Idle)- - Google Sheets

I do have a 5800x3d in my game PC

Most efficient systems use 1 1,2v dimm, so i upgraded to a dual channel but not full 1,35v. Again, no real measurements but i don’t need that small extra speed boost. The CPU is overkill as it is.

The motherboard is also chosen because it is proven to be efficient, asrock boards generally are more efficient. Z690 is more power hungry anyway (and not to real benefit)

I’ll take a look at the switch. i selected this one because i found measurements that it is at 4W of power use where a lot of 2,5 or 10g switches go to 10/20W, which is a bit too much.

[nutral]

Thanks for the suggestions.

the corsair power supply is actually quite a bit more efficient in very low power idle running (around 10W). a different power supply might get close but gets more expensive over the running time. (also because the server is a tax write-off but electricity isn’t)

I want to run both storage and virtualization on the same server. With a second backup server for just the important services and testing. Also the 7900 system uses way too much power compared to what i need.

I also like that services then have high speed access to the hard drives and ssd’s. But for good performance i would need to pass the whole hard drive controller into truenas, maybe i should install a PCI-E one?

[diizzy]

There might be some edge cases but in general Intel isn’t all that great at idle not certainly not during load. Most of that you liked to is 3+ year old hardware which isn’t relevent to new hardware.

AMD 65W Zen 4 Review: Ryzen 7600, 7700, and 7900 CPUs Tested

AMD Ryzen 5 7600 processor review

ASRock has shown several times that they’re doing weird things on the Intel platforms such as
PSA: Don't Buy This Asrock Motherboard | TechSpot so I’d be rather cautious getting ASRock for Intel platforms. But yes, they certainly will be more efficient by effectively power limiting the CPU to begin with which you can do on pretty much any board.

2666 vs 3200 sets you back ~10% or even more depending on workload, if power consumption is that important you might as well look for slower hardware (which will be less power hungry) to begin with.

Best RAM for Intel 12th gen Core i5, i7, i9: Memory Guide, DDR4 vs. DDR5

[adman-c]

I still saw the idle power difference when building a current-gen workstation. To be fair I was comparing a 13600K to a 7900X—so the AMD chip had a second chiplet and double the full-size cores as the Intel. But with the same cooler, ram, and ssd, the AMD chip on a X670 board idled around 50W in Linux while the 13600K on a W680 was closer to 25W. YMMV of course, my tests were not that extensive. And the 13600K certainly is no power sipper when you ask it to stretch its legs.

[wertigon]

Let us take a step back here, what do you want?

If you need a high performance NAS and have modest storage reuirements (e.g. less than 20 GB), get the Asustor for storage. It does this with 12 drives in less than 30W idle.

If you have bigger storage needs, build a NAS. The one above can comfortably fit 150TB over SATA. This system will idle at 25-30W but six HDDs will push that figure up to maybe 50w.

Do not combine NAS and virtualization. The two use cases are different and will in all likelyhood interfere with each other, leading to unnecessary stutter and performance bottlebecks.

Now, maybe you are thinking of a virtualization server with local storage capacity, at that point the 7900 with two mirrored 4 TB drives and 64 GB are way overspecced, on the other hand cutting down that to something more in line with what you need would only shave $300 off the price, for half the capacity. The 7900 system idles at 30-40W and you can turn off that when not using it. Local HDD storage at this point is pointless, since transfer speeds are sub gbit and latency is in hundreds of ms. At that point a NAS storage, even with HDDs, makes more sense.

Now, one final advice; given your power bill, if you have the possibility, invest in solar and a home battery. For ~$15k, you can invest in a 10kW installation + 10kwh battery. This will produce roughly 11 kWh a year, at $0.30 per kWh that is $3.3k of savings a year. After 5 years you are making a profit. Only if you have the possibility though, $15k is a lot more than $2k and not everyone own their home.

TLDR; above two systems will cost you ~$2k-$2.5k, draw a maximum of 25W+50W idle and max 300W total at bemchmarking loads, and do everything you would like it to.

Feel free to alter anything above, but I stand by my statement that you want a two system setup here - especially as this allows you to power down your homelab when not in use.

[nutral]

I’m going a head with a 2 server setup. with my old server as a backup (but usually turned off). That will make things easier to setup. as the new NAS would just run Truenas. and then the small server i already have will run proxmox and run any containers.

I’m also going to upgrade to 2,5gbit, as that is not too expensive to get, and the energy consumption for things like switches is actually pretty low. about 4W for the Mokerlink switch i want to get.
This is the list of parts i’m going for.
The asus NIC is for the virtualisation server and the TP link adapter is for my main computer. My work laptop already has 2,5gbit only the dock hasn’t. (maybe i’ll upgrade the dock at some point)

Storage will be 3x 16TB in Zraid. so 32TB usable.
SSD storage is 2x 4tb in mirror, this will also have my personal files and photos.
The PM9A1 is for Truenas because i don’t like running on USB sticks.