Jaskij's homelab journey

Community Blog
1

So, I just did my first actual step (beyond planning and theorizing) in building out a homelab.

I ordered a 7302P on eBay!

So, since it’s official, I have decided to document my journey.

First, a little bit on the hardware.

The background: this will be my everything server. NAS, virtualization, firewall, software builds, everything. I don’t plan on getting a second one in the foreseeable future.

I was looking at various solutions, T630, W680, AM4, all missed something. Be it PCIe slots, bifurcation, something was always missing.

Then a friend nudged me: there are EPYC 7302P chips from China on eBay for 270 USD. Damn. Everything I wanted and it doesn’t break the budget. Not to mention the great upgradeability (even if I can’t upgrade to Milan, I can still look for a 64c chip in the future, as compared to my 16c 7302P). I’m sold.

Then came three days of frantic research and and searching. The motherboard was settled fast - Supermicro H12-SSL. Although halfway through I decided to go from a used one from the same seller as the CPU to buying a new one - the price difference wasn’t enough to offset the lack of warranty.

RAM was easy to find - currently you can get 8x16 GB 2133 ECC RDIMMs for 200 euro.

What came next - case and cooling - were agony. I don’t trust a 92mm 4U cooler to be quiet enough for my needs. Nothing bigger fits in 4U. Do I even want it rack mountable? In the end this issue was settled by a friend offering me a free case. For the cooler I decided to go with Noctua because they guarantee compatibility with the motherboard. U9 TR4-SP3 to be precise, check if it isn’t too load, keep or replace with U14S.

As for PSU, I decided to move the HX850i from my PC - great PSU, high efficiency, lots of connectors, power monitoring - and get an RM850e for my PC.

Other than that - I’m going all flash. Six Kioxia Exceria G2 1 TB for data, and two TBD 250 GB NVMes for the OS. Altough I’m also trying to do some fishing on eBay for used enterprise SSD drives, but don’t expect good results.

TLDR for the specs so far:

  • EPYC 7302P
  • Supermicro H12-SSL-i
  • 8x 16GB 2133 ECC RDIMM
  • Corsair HX850i
  • Dark Base 900
  • 6x Kioxia Exceria G2 1TB for the data array (they’re dirt cheap over here)
  • Asus Hyper M.2
  • NH-U19

Some of you might’ve noticed the lack of any sort of NIC - the network infra I have right now is, basically, nonexistent. ISP-provided everything-router and that’s about it. So I need to build the whole network because I can start thinking about NICs.

Next post will be about the software I’m looking to use.

1 Like
2

A word of caution to you. I hope that you did your research on the used 7302p you bought to ensure that it is not vendor locked. Otherwise have fun with your build.

1 Like
3

I did, and the eBay listing even had “no lock” in it’s title. So if it’s vendor locked I’m refunding.

1 Like
4

So is this on the list? Have you considered a Pfsense or OPNsense box?

5

Unless you have a workload that demands it, like doing CAD or video editing or something, I doubt you can make full use of 16 cores, let alone 64. And if you know your workloads, you would be getting something beefier anyway. :slight_smile:

If you are going to do a forbidden router, then you probably don’t need much connectivity besides the connection to the ISP, maybe arguably a 1x or 2x 1Gbps NIC: WAN and LAN for a router VM. The WAN is mandatory, but you can get away without a LAN if you just do a virtual NIC (or more if you want to split into different subnets, but without actually using vlans, just differently connected bridges) to connect your virtualized infra. The second NIC would be if you want to connect a managed switch to it in the future.

It is an advantage of the forbidden router, but I still prefer a separate router, switch, NAS and always-on services host (all low-powered, maybe the NAS and container box combined, but splitting them this time around).

^

While I don’t like the idea of rebooting the hypervisor taking away your internet, if everything is going to be living in it, then all good. OP will still have the ISP router anyway, so his household won’t complain.

6

ahhh he’s building a forbidden router. Yeah I never embarked on this route. Everything is separate and purpose built but if you are virtualizing I think… OPNsense might be slightly easier? Its more modern. Newer and more up to date. Not sure though as I never tried to virtualize it

2 Likes
7

Probably about the same, I ran pfsense inside proxmox inside virt-manager (helping a fellow forum member a month ago or so). I doubt either makes a difference, just goes down to preference (supposedly more stability, or following FreeBSD a bit closer). I would suggest going straight to OpenBSD or FreeBSD, but that is just me. I believe jaskij has some knowledge, so should be fine with anything. :man_shrugging:

1 Like
8

Oh yeah I know jaskii from discord and Matrix/element I know he has the knowledge for that. It just depends how lazy you are. These days I find myself liking things to be more turnkey … Up to him though. If you are going the straight to OS route… OpenBSD is the way and its also pretty fricken easy to install

1 Like
9

Going forbidden, virtualized.

The TLDR on this is that 11th gen SMCI boards are hard to get, and 7302P was the cheapest one that would work in a 12th gen SMCI. I know my workloads, and I will be running a build server - 16c is more than enough for my needs.

Motherboard has a two-port BCM5720. I’m still working on the situation, pricing stuff out, considering upgrading home network to 2.5G. Or getting an LACP switch and just throwing a multi-port card in there, preferably one with SR-IOV.

Me to. I’ll happily sit in the terminal whole day digging through configs, if it’s something I care about. Was thinking pfsense/opnsense in this case. My main issue is that I don’t want to waste brain capacity on learning stuff I’ll use very rarely, so a nice web GUI is neat. I was very terminal-first, but recently realized that sometimes it’s just better to be lazy.

1 Like
10

“Easily solved” with a backup router, e.g. a switch and a wimpy secondary router running VRRP (e.g. with keepalived). … , or even an OpenWRT running router where you just map two ports to WAN instead of one.

11

Considering my setup physically, everything but my mom’s PC is within three meters of each other. So, yeah. Worst case I can replug some cables, move the LAN side cable off the firewall and to the ISP router or whatever, and be done.

1 Like
12

So, I finalized my specs and placed the orders. Had to order from five different places to optimize a little.

Server:

  • EPYC 7302P (used from Ebay)
  • H12SSL-i (new)
  • 8x16GB 2133 ECC DDR4 RDIMMs (new)
  • 2x256GB ADATA SX8200 Pro for OS (new)
  • 8x1TB Kioxia Exceria G2 for storage (new)
  • two Asus Hyper M.2 cards to mount the storage (new)
  • Corsair HX850i (moved from my PC)
  • Noctua NH-U9 TR4-SP3 (new)
  • bequiet! Dark Base 900 (hand me down from a friend)

The cooler is somewhat tentative in that I don’t know if it’ll be quiet enough - if it won’t, I’ll return it and buy an U14S.

Network

This was somewhat unplanned, but I started on building the network. TP-Link Omada is an SDN that’s getting popular lately. I’m guessing it’s probably because the stuff is good enough and definitely cheap. I decided to bet on it as well. For now I’m going to virtualize the controller, but I’m not ruling out getting a controller in the future.

What I bought:

  • TL-SG2210P - PoE switch
  • EAP610 - WAP
13

the 7302P memory controller supports 3200Mhz, 2133 is ok for Haswell or Broadwell-E

14

Budget constraints. 2133 is easy to get used, 3200 I would’ve had to buy new and likely wouldn’t be able to fill all lanes. Between faster RAM or filling all lanes I decided to go for filling all lanes.

15

So, a small update, my T340-2 (which is meant to hold me over until I go 10G networking), arrived today.

Thinking I’ll pass this one through to the firewall/router, and use the BCM for host vSwitch. I know that driver situation on BSD isn’t the best, so it’s better to use Intel there.

16

Almost everything is here…

DSC_0290
DSC_02901920×1440 172 KB

And I have Friday off cause national holiday!

2 Likes
17

So, been a long time, work has been super busy. That said, the server is up and running, in a provisional case. I have a Grandia 09 on the way though. Another thing on the way, I found a decent deal on dual-port Mellanox ConnectX 3 - dual port ones for slightly less than I’d pay for a single port one locally.

DSC_0305
DSC_03051920×1440 163 KB

Interestingly, the thing has super low idle power. I move my HX850i there for the extra connectors and power readings. With nothing but Proxmox running this is what I get (reading in micro watts):

root@pve:/sys/class/hwmon/hwmon11# for i in $(seq 1 4); do cat power${i}_label power${i}_input; done
power total
52000000 # 52W
power +12v
38000000 # 38W
power +5v
11000000 # 11W
power +3.3v
5500000 # 5.5W

Currently running some drive benchmarks - fio driven by Phoronix Test Suite. ETA is just shy of five hours, will update then.

A sneak peek of the results:

    Type: Random Read - Engine: Sync - Buffered: No - Direct: No - Block Size: 1MB - Disk Target: /flashtank:
        5478
        5469
        5533

    Average: 5493 MB/s
    Deviation: 0.63%

    Type: Random Read - Engine: Sync - Buffered: No - Direct: No - Block Size: 4KB - Disk Target: /flashtank:
        63700
        63900
        64900

    Average: 64167 IOPS
    Deviation: 1.00%

Small update:

Power generally hovers around 110W during random read test, jumped to 220W during random write 1MB for a moment there.

18

Great build! I’m looking at something similar for TrueNAS or Unraid running a bunch of VMs and tons of storage. I was leaning towards AM5 but I’m worried the limited PCI lanes will be a problem for future expansion.

Any regrets on going with the 7302P?

19

Work’s been so busy, I haven’t had the time to set up the software properly yet.

And yes, AM5 has quite limited PCIe bandwidth, personally I’d rather look into LGA1700 - depends if you care about ECC. If you do, there’s only several motherboards on the market, and they’re hard to find. Only the Gigabyte Wendell did a video about has ECC and IPMI but AFAIK there’s no stock.