My current homeserver is being pushed a bit too hard and I’m in the market for an upgrade or replacement. I am a bit out of date on hardware though so was hoping you guys could provide some pointers
What I plan to do is have a single virtualization (kvm) server that;
Runs a NAS VM with host pcie sata controller passed through to it (currently cannot do this due to unlucky iommu groups) - I’d also like to add a 2nd pcie sata (or maybe nvme) card later
Runs a VM for k3s that runs; home assistant, plex, gitlab, frigate, etc; with (i)gpu for plex transcoding and usb passthrough
Runs another VM for infrastructure services like, postgres, influxdb, minio,
Runs a pfsense VM that has a pcie 2 port nic passed through
Has enough left over horse power so that I can alternate between running additional desktop VMs (windows, linux etc) - but these VMs will not be always on - maybe with a dedicated gpu pass through to try out virtualised gaming - but I haven’t given that much thought yet. I’d also like to use the left over horsepower to occasionally try new things - eg, cluster some arbitrary software over 5 VMs or something similar
Right now I’ve got points 1,2,3 & 4 covered with a struggling-to-keep-up server made up of;
Intel i5 6500 (4 cores)
32GB RAM
1x pcie sata card (4x HDDs are passed to VM as block devices though)
1x pcie 2 port nic
1x onboard nic that has vlans/bridges set up so that the hypervisor and (non-pfsense) VMs can communicate with the rest of the LAN
all in a fractal design atx case
I hope to cannabalise as much of that as possible - I’m picturing a new cpu, mobo, ram & probably PSU (mine is a no-name inefficient thing that’s kept me awake at night for too long) - but I’m open to other ideas & options. I’m not keen on rack mounted or ancient enterprise hardware for energy, floor space & general loudness reasons - but if y’all know of a xeon/epyc/threadripper tower server that ticks the boxes then I’m totally not married to the prospect of consumer grade hardware
In terms of budget…I’m not in a huge urgent need so while sub $700 is probably the ballpark I’d hope for, I don’t mind waiting a couple more pay days if there’s a better option slightly out of reach
I was looking at some of the ryzen cpus because them seem to offer a good balance of cores/thread count, power efficiency, pcie lanes, cost while still having decent max clock speeds - but I don’t know much about their igpus (for the ones that have them) and how good they are for plex transcoding of 4k hdr content and the like. If I went that path, would I need a dedicated gpu for plex, and then maybe a 2nd gpu if I wanted to try out virtualised gaming?
I’d also like the motherboard to have sufficient capacity for >=64GB RAM, and pcie slots I can pass through to my VMs without the iommu groups trying to also pass through the onboard nics or other hardware at the same time
Anyway - thanks for reading my ideas & ramblings, I look forward to learning from your replies!
Lots to consider in here. Good luck with the project. Sounds fun.
I’ve just retired my old x79 based system for Ryzen and I was amazed at how much power, heat and system responsiveness improved in just a couple of generations.
If you are buying used, I’d suggest getting an older CPU but newer motherboard, or just get new (newish) everything. I used to be the “old enterprise all the things” guy but it simply isn’t worth it now.
Full size ATX motherboard, even b450 to save cost, will be a step change for you, just lookup the PCIe slot layouts as cheaper boards cut back on those. Pcie4 won’t matter to your use case so don’t waste money there.
On ram, prices for older ECC ddr4 are not too bad if you keep clocks low (2666 or below). That may be important to you. Again, you don’t need to overclock and the Ryzen CPUs don’t play nice with top spec ram.
You have the hba and nic so no specific recommendations there. Case is a good choice. Make sure you get a beefy air cooler and lower the fan rpm
Spend more on your PSU than you want to, you won’t regret the investment.
Not really needed for your setup above. New Threadripper is great but only if you want all the PCIe and have all the $€¥ but is going end of life for first to third gen. Epyc is probably overkill for your use case and out of your budget.
If you are going used, a deal on a used Xeon may be an option but given the confusing array of options intel now gives its bronze / silver / gold / Xeon w naming conventions, they make it hard to recommend to non-expert buyers. Do shop around though.
Happy to look again if you find something. Good luck.
Thanks for the insight, I really appreciate it - I’ll take a look at some ryzen options. Different retailers seem to have prices all over the place at the moment, so hopefully I’ll get lucky a bag myself a deal. The downside with ryzen is it seems even those with integrated graphics are not supported by plex’s hardware transcoding - so I’ll either need to re-encode my media ahead of time, or, chuck in a basic dedicated gpu whose sole purpose is transcoding - maybe that’s OK but I’ll have a look into the prices of ryzen + dedicated gpu vs intel with supported igpu
Regarding the motherboard and b450 - I ran into some issues with my current intel with iommu groups where I essentially couldn’t pass a pci slot to a VM because it was in the same group as the onboard nic which I didn’t want to pass - short of using a custom patched kernel, there didn’t seem to be much I could do about it.
Is there a difference in iommu grouping for different chipsets and/or price brackets for motherboards? Or is it just down to each motherboard model’s unique configuration?
Xeon may be an option but given the confusing array of options intel now gives
I’m glad it’s not just me that finds xeon naming unnecessarily confusing
IOMMU passthough on my AM4 X470 MSi board is fine. I have TrueNAS Core running as a VM with a real HBA and 8 drives. I’m not sure this was a good idea but it works.
I would not recommend running your router as a VM, it’s the heart of your network and you want it separate from the mess that’s your main server. There are some excellent and cheap small Fujisu computers you can set as a router or simply buy a powerful enough router.
