Homelab/router help: virtualize or multiple PCs?

Hardware Hub Networking
, ,
1

Howdy guys. I’ve been tearing down and rebuilding PC stuff lately and I think that it’s finally time to build up my homelab and start self hosting apps as well. My problem is that I was going to try to run this all on one system (epyc or the like), but I’ve heard that virtualizing this stuff is much more of a pain than it is to simply build multiple systems. So I guess my question is how many systems do I need? Could I virtualize most of it? Let me run down what my system reqs and what apps/services I’d like to run.

I have a gaming pc. It will be running 2 GPUs on an x570 motherboard and I plan on picking up a 40gbe NIC as well. It is going to run linux (manjaro gnome or ubuntu probably) and run windows 10 in a vm with GPU passthrough.

Next up is a NAS or NAS functionality. This system will need an HBA to support my 8x SAS-12 hard drives. I’d like this one to have 40gbe as well.

After that is routing. I need this one to have multiple 40gbe connections (probably 4) and I also plan on adding a quad 10gbe card.

After routing is the self-hosting/VM functionality. I need enough cores to run a decent amount of apps and services.

Lastly, the firewall, and the reason I’m writing this post. I guess I’m wondering how this would fit in with the other services. I understand that it would probably be better to build an independent system or use an open appliance for firewall usage, rather than to virtualize it on the system that might house the router or NAS.

So for my network/lab, can I get away with putting the routing/NAS/and self-hosting apps on the same (epyc or TR) system and then having a separate firewall? Should the routing be done on a separate system apart from the NAS? Should the apps and services be done on another system as well?

As far as self hosting services and applications I know that I’d like to run:
pi-hole
reverse-proxy (traefik?) (certificates as well?)
plex/jellyfin
a book library host
grafana
self-hosting for a website
self-hosting for an email service
kubernetes ( need to learn more about this)
docker/rancher (I need to learn more about this. Are containers that different from VMs?)
Data syncing, backups, and file sharing (syncthing?)(nextcloud?)(photoprism?)
battery monitoring (nutserver)
Password management (bitwarden/vaultwarden)

I have a pi-kvm that I can use as well. So my question is mostly along the line of what systems/services should be separated and how much can I or should I combine? I already have the components for the gaming PC. I also have an extra b550 motherboard and 3800x CPU. Can I get away with 2 systems? One for a firewall/vpn and the other one for the NAS/Router/services? How difficult is it to virtualize these processes instead of building 3 or even 4 systems?

on a small sidenote: how different are the threadripper pro CPUs and epyc? I understand that epyc is probably the better choice if I need that many lanes, but is the main difference just clock speeds?

Thank you for your time, and for reading my ramblings.

1 Like
2

Common sense says troubleshooting your router while tethered over your phone is no fun… - so your router should be a separate system/appliance that you don’t mess with…

… however, if you’re learning then troubleshooting while offline is fun, and you should virtualize / containerize routing.

Everything you listed as your workloads in the list of apps is very light weight, I suspect windows would probably be your only VM… everything else, including routing/firewalling you can do in a container.

I’m thinking install Ubuntu LTS (Ubuntu because of ZFS) on your x570 and just start setting things up.

What internet connection, and what HBA and disks do you have?

2 Likes
3

Epyc or TR Pro for a homelab? Way too overspec!

For all of the listed services, I probably wouldn’t go with a system with more than 64GB of RAM and even that can work with just 32GB, especially if you use containers for most (LXC or OCI containers).

In all honesty, my own preference is to have a separate router. I don’t know what you are planning with 40G, even some video production places struggle to saturate 10G, unless they have 10+ editors. For all of your services, I wouldn’t go more than 2.5Gbps, but you can get a 10G connection if you want to set up your house (or at least your lab) for fiber.

Pi-Hole can run in a container. So can a reverse proxy, grafana, the book library, website, email, nextcloud and nut. If they don’t have OCI versions, they can easily run under LXD. Rancher / k3s, or Portainer / microk8s, or just docker, can run in LXD as a nested container, then you can have the rest of the containers, like vaultwarden (which from what I remember, only had an OCI version) inside that containers container.

That only leaves Jellyfin, which you should be able to run in a container, but I don’t know how easily you can add a GPU to it, for on-the-fly encoding. People had luck with it in Proxmox. If you don’t need to encode the video stream to other formats and you keep everything under a single format, you don’t need GPU acceleration. That seems to be the most intensive one and probably the hardest to setup as a container if you need the GPU. So it can be a VM, VM PCI-E passthrough is easy.

Other than that, if you want a BSD distro for your router, then that would need to be a VM. So would Windows. The last intensive workload would be the router’s IDS / IPS. I personally don’t run one, it’s overkill for a homelab. If you want, you can install one and test on a low-traffic subnet, just to learn. This would allow you to have a lower spec router, like a quad core Celeron, a Ryzen 3400G or an i3 12100 as the router. Actually, the later 2 should be fast enough to run snort or suricata on the internet-facing traffic.

I prefer my router to be a separate device, because rebooting your hypervisor shouldn’t impact the rest of your physical network. It’s fine if you lose access to samba or jellyfin or something, you plan the reboot when your household isn’t using it. But the router reboot will impact stuff harder, because it’s one thing to not watch a movie and another one to completely lose internet access (I know, people can’t be offline for 5 dang minutes!).

My router used to be a Pi 3 running Alpine for 8 months or so. I could have ran OpenWRT, but I preferred a full Linux distro (and I liked Alpine’s diskless install). Now I switched to a RockPro64. I don’t have a PCI-E 2.5Gbps NIC yet, I plan on getting one, the rkpr64 should be able to handle that. I only use its built-in NIC, it does fine, it was an upgrade from the Pi 3’s gigabit working at only 300 Mbps (USB limits) for LAN, although my VPN is nowhere near fast enough for me to notice a difference.

A RockPro64 would not be able to route 10G, let alone 40G, but I’m mentioning it as an example. I would personally use something akin to this if I were to build a faster homelab and have the “fast LAN” on a separate VLAN, not going through the router. Kind of like a dedicated offline storage area.

With a 6 cores CPU, you can probably combine all of these on a box. Probably even with 4 cores if you tame your expectations. I will personally run a few of these services on ARM SBCs in containers, I’ve been trying to do this for a while now, but I keep hitting some walls every time I try to get things set up. I hope once the next order arrives next week or so, I can finally start setting things up.

We have a few fellow forum members who used to run productions on way underpowered computers (we’re talking pentium III with 50+ users). While I don’t think people can get away with that level of compute with today’s CPU intensive, RAM-hogging software, I still think most people don’t need more than an 8GB RPi to host 90% of their homelab needs, with 9% only having to add a single more beefy SBC like an Odroid N2+ or Jetson Nano for some intensive workloads, with just 1% of people actually needing something better than a quad-core Intel or AMD CPU.

The reason I recommend people build x86 builds is because it is way easier to get into, albeit more expensive and people don’t have to smash their heads against their walls to set up the OS and services on aarch64, which is not exactly easy (well, I make it hard on myself, because I don’t run officially supported OS, like Armbian or especially not Manjaro).

tl;dr if you want a separate router, like I’m suggesting, get a quad core AMD or Intel CPU and 16GB of RAM if you plan on running IDS / IPS. Or get a quad core celeron if you don’t plan on IDS / IPS, with 2GB of RAM.

The hypervisor can be a 6 - 8 core consumer CPU with 32 / 64GB of RAM, which should handle all the services you enumerated to throw at it, including a VM router if you fancy that. The only advantage for a “forbidden router” concept is that you save money on the switch, if your NAS, hypervisor and router are in a single box.

4

of course I’m not planning on getting anywhere near saturation levels for 40gbe. If you design your network around what you saturate though then you aren’t leaving much room for growth. I planned on using 40gbe because used enterprise nics are fairly cheap and 40 gbe is only marginally more than 10gbe. I know that you can run things like routing on a pi, but good luck fitting a pi with a used 40gbe nic. Most arm systems have little room for growth as well.

Thanks for the advice on the other two systems though. You’re suggesting to run 2 systems then? 1 for routing and 1 for nas/virtualizing? My plan wasn’t to go with epyc as much for virtualization as much as it was for all of the pcie lanes. An HBA, a quad 10gbe nic, and 2 connectx3 40gbe cards would require a good amount of lane bandwidth. I recognize that most virtualized apps are small and low on processing power. I’m more worried about these 3 things: the firewall, the router, and the nas. You’re telling me to go with 2 systems? One for routing/firewall and one for the nas and virtualization?

Thanks for the advice! I’ll look into that.

1 Like
5

I was not suggesting you go with 10G on ARM SBCs. I just pointed out that most times people overspec their systems, creating waste. Waste that they pay for with higher prices for their systems and higher power consumption. There is a reason why businesses first plan their needs out and only then buy the servers. :slight_smile:

That is probably 56 to 64 PCI-E lanes right there. If you add a GPU, you easily jump to 72 to 80 lanes and that excludes 4 to 8 lanes for NVME, so you can go to around 88 lanes total. Funnily enough, that is exactly how many lanes a TR 3rd gen has, like a 3960x, which is what I would recommend you get in case you go with a single “forbidden router” build. TR Pro is way too expensive, but of course, if you want to spend the money, go for it, I’d just think it’d be a bit (a bit more) of a waste.

Now, suppose you build everything in a single box. Router, NAS and hypervisor. Like Wendell calls it, forbidden router. With everything on one box, you probably could get away with just 2 connectx3, a gigabit NIC for WAN and a switch. Or you could realistically get away with gigabit, as most communication will happen on the box itself, not having to go through the physical layer out on a switch, if you plan things right, but that’s besides the point. Maybe arguably a single 10G or 40G connected to a switch.

My recommendation is for a separate router build, because that is my bias. It gives you some redundancy, if your hypervisor fails, it doesn’t take your whole network offline. While the scenario isn’t exactly likely, it does happen that hypervisors need restarts more often than a router, especially a BSD router (with these speeds, you’d probably want FreeBSD or Linux).

I would still say that a quad core with very high frequency, like the i3 12100 / 12300 would run as a router very well. 40G cards I believe have some internal ASICs that speed up routing, but for IDS / IPS, you still need fast cores. I doubt you can get to 40G when doing that, but as mentioned, you can (and should only) do it on the WAN side, so those kind of CPUs will be plenty powerful. You need the RAM to save some signatures and cache some traffic in the RAM, preferably as fast as you can go with it (3600+).

As for the system itself, if you go with 2 builds, I’d think you only need 1x connectx3 (as the other one will be on the router) and link them through a switch. That way, you can set up VLANs and other stuff on both ends on those NICs and have the switch ports open for other things on the network, leaving the built-in gigabit port on the router build to serve as a WAN. And then you wouldn’t need the 4x 10G on the hypervisor, as everything will happen through the 40G connection.

Meaning you can lower the amount of needed PCI-E lanes down to about 24 lanes, maybe 32 if you add a GPU in a x8 slot. With just 24 lanes, you can easily use consumer hardware. If you need more lanes, I personally wouldn’t go past a 1900x, but you could go 2nd gen TR with a 2920x if you want the higher frequency. 3rd gen TR only has 24 core variant and up, complete overkill for what you plan to run (and arguably a waste of electricity).

As an annecdote...

I have ran a production infrastructure with 3 racks on gigabit switches (they were 5 racks, but weren’t full). On the hypervisors I had 2 ports in balance-alb and later in LACP in the dedicated storage network on separate switches and other 2 ports in the same config split in vlans as the management, server subnet, untrusted subnet and a few other subnets. The NASes only had 2x gigabit ports in LACP and 1 port for management.

They ran fine. Initially had about 370 VMs, later on decreased to 240 - 250 or so after I started combined VMs, to squeeze more out of the VMs, as a lot of RAM was wasted on OS. The only time I felt the need for more was when running Oracle backups at night on probably 180 VMs, sometimes they couldn’t finish backing up in time, before rundeck killed the jobs because the work hours started.

But during work hours, nobody ever complained that VMs were slow (well, besides the Jenkins VM that I wanted to convert into a TR build server, because it needed a faster CPU, but that’s besides the point, building Java programs is CPU intensive and the servers only went to about 3.3GHz at best - still, that is 1 out of 200+ VMs and it wasn’t slow because of the network, it needed a better CPU).

I wish more people would comment though. I know I have my biases against forbidden routers, but maybe there are counter arguments. And maybe someone agrees with you that you can justify ponying up the :heavy_dollar_sign: :heavy_dollar_sign: :heavy_dollar_sign: for a 3960x or a TR Pro build

In all honesty, if Zen 4 Ryzen had boards that split the PCI-E 5 lanes into a bunch of PCI-E 3 and 4 lanes, I’d recommend a build around that, because of the high frequency and good IPC, but there’s nothing like that out there, so I can’t in good faith suggest that. 24 PCI-E 5.0 lanes split means about 96 PCI-E 3.0 lanes, or 48 PCI-E 3.0 + 24 PCI-E 4.0 lanes (if my math is right).

1 Like
6

I’d be more likely to go with epyc than threadripper, as I believe there to be more epycs on the used market than threadrippers. I also think that splitting out the lanes from 5.0 to 4.0 and beyond is difficult at best. At worst even when you have a pcie 4.0 x4 connection and switch it down to 3.0 My understanding is that it doesn’t switch it to 8 lanes. Maybe someone with more pcie knowledge can comment, but I believe that the amount of lanes doesn’t change, only the speed that they are set to.

I’ll keep my eyes on 2 systems. It would probably be a good idea to keep the router on a separate system. Can/should you run a firewall on the same system as the router?

7

Threadripper systems that are on 24/7, even at idle, will consume quite a lot of power. Do the math for the cost of the power consumption and reconsider what you want vs. what you need.

Spend a few mins watching this video:

I operate a home server for >15 years and had to learn all the things painfully myself. I have a 40G switch and NICs but only turn them on when I am actually using them. My 1G switches consume ~3W each.

3 Likes
8

Yeah, I’m not going threadripper. Power consumption-wise though I’m not super concerned about it. Power is cheap where I am. I do think that it’s worth trying to cut down on power use, but maybe not so much that you are forced to sacrifice pc performance.

9

PCI-E switches do increase the amount of lanes.

You can plug a x16 PCI-E 3.0 GPU into a x16 PCI-E 4.0 and it will run at x16 3.0 speed. If you plug a x8 PCI-E 4.0 GPU on a x16 PCI-E 3.0 slot, it will run at x8 3.0 speeds. But if you have a PCI-E switch, the amount of bandwidth that you give it, is the amount of throughput that it can have. You could put a PCI-E switch into a x8 lane and have it spread to 16 lanes (fan-out), but you will be bottlenecked by the upstream connection.

I am not aware of any plug-and-play device that you plug to a PCI-E slot, those switches are bought in bulk and installed on motherboards or SBCs to provide more lanes on devices that would otherwise lack the necessary quantity of lanes for certain applications that require a lot of them, but don’t necessarily require their full speed at all times (say that you do intensive stuff on 8 lanes, then you move to doing other things on other 8 lanes and so on, but you only have 8 actual lanes to the CPU, but you need to connect multiple things to it).

There are other options, like PCI-E bifurcation (turning a x16 into x8 x8, or x8 x4 x4, or 16x x1 or other combinations), but this would limit the bandwidth available for any devices you plug in, while a PCI-E switch will give the full bandwidth of the upstream channel if no other devices need the bandwidth.

10

Almost always, that is the case. Doing the routing also has to do with blocking the traffic. pfSense, OPNsense, OpenWRT, all have firewalls built-in. If you choose to DIY using a generic Linux distro or one of the BSDs, you still pretty much get the firewall preshipped in your distro (iptables / nftables, or the wrappers around them like firewalld, ufw, awall etc. on Linux, or on Open and FreeBSD, pf).

Very rarely you would have so much traffic that you want a separate firewall box (that basically just analyses traffic as a man-in-the-middle between the router and the switches), but that is enterprise level and even there it is rarely implemented.

1 Like
11

I kinda knew this part “You can plug a x16 PCI-E 3.0 GPU into a x16 PCI-E 4.0 and it will run at x16 3.0 speed. If you plug a x8 PCI-E 4.0 GPU on a x16 PCI-E 3.0 slot, it will run at x8 3.0 speeds.”

What I’m more curious about is whether you can go from something like x4 4.0 to x8 3.0.

Looks like it’ll be 2 PCs.

1 Like
12

If you can do with 2 40gbit connections, a Brocade ICX6610 will give you layer 3 switching / routing at that speed and for a fraction of the price it would cost you to setup your version of a router…

13

EPYC Naples is probably the cheapest way to get 128 PCIe lanes in a single system, which is nice for running multiple NICs/HBAs/NVME (can actually save time because you don’t need to go into BIOS and fiddle with PCIe switch settings like on Xeon boards, because it doesn’t need them).

1 Like
14

If I’m not mistaken, the reason that broccade’s switches are so cheap is that a license is required to get them to run at full speed and the license is several thousand dollars.

15

You are technically not mistaken, but since they are eol, a very kind set of individuals have provided the means tu unlock it’s full potential without requiring a license:

I have bought a 7250 (no 40gbit, but 700w of Poe) a couple months ago and couldn’t be happier :grin: .mine switches 40/50 gbit between all my homelab with CPU at 1%. I do not use them for routing, but I am pretty sure it could handle 50/60gbps of aggregated routing without breaking a sweat…

16

Are the brocade switches super loud?

17

they are datacenter switches, so some level of hacking is needed to quiet them down, depending on the model it can be a simple fan swap or a more complex hack…
The OP did non give constraints about noise AFAICT…

18

It’s almost like he knew. Granted I’m going with an asus X570 Workstation motherboard instead of intel, but almost everything else is spot on. Sliger rackmount case, fancy nics, virtualizing the router. I hope we see more on this series in the future. I’d love to know what I can or should run in containers in the future. Wendell is a hero amongst nerds.

1 Like
19

If you like to tinker, you really want to keep your network and core services box separate from your tinker box.

1 Like
20

The one thing I tend to do is spec something conservatively or accurately, and then scope creep pushes things further than I intended. All I have to say is buy as much HW as you can when you get going and you will end up using it. Full Stop. It gets expensive buying more hardware a second time.