New home lab: moving to mini pc's to build a Proxmox cluster

I don’t know your living situation, but if you own your house, or have a back yard that gets sunshine during mid day, you may want to consider getting solar panels and a battery.

It always pays until you are generating more power than you are using. At that point feel free to mine crypto as the power is free.

How you ask?
Look at some of the projects here:

You can either buy new or used panels. Panels 40 years old are putting out more than 80% of their rated wattage.

In general roofing companies don’t install panels, so when someone gets a roof leak when the roof is under warrantee, the roofing company will fix it (and uninstall the solar panels). Then the insurance company will return the solar to its previous condition before the warrantee repair.

Around here the groups who salvage the perfectly good panels that got removed from someone’s roof advertise on facebook marketplace. The panels sell for around 20% of retail usually,

2 Likes

Blockquote
It isn’t about crypto, it’s about economics. Paying collateral without dividends or interest is a bad deal. You are lending someone else money for free. I get better return in buying federal bonds and not running any servers that use power or generate write-offs etc. And especially now where interest rates are on the rise and inflation is a thing.

What has to do with the Flux project?

It looks like a beast!

Of course, the 2.5Gbit port is a bit of a waste on a system this powerful, I wish they’d start putting ACQ113 controllers in these boards, that require the same single PCIe lane (at v4) that all the other onboard NICs need but for the full NBase-T range: I just don’t understand the hold-up there, the chip’s been around for years now…

Sure you can use an M.2 slot for that, too, with the older ACQ102, but I’d rather keep that around for storage.

64GB RAM also can become a bit tight on a system this strong, I wonder if these non-binary sticks would work, but then I’d also want ECC, which is much less guaranteed if not actually disabled on these SoCs…

On a more compute oriented node I’d probably want to keep that sole PCIe x16 slot free, just in case I’d want an L4 or something similar later, but dual ports NICs can be very useful on hyperconverged setups to keep the control and the data plane truly separate. Alternatly you could just solve that issue with far more bandwidth, e.g. 100Gbit (or 25/40) or even use a quad, to sort out live-migration and some IP based storage as well.

At work I run a HCI cluster (currenlty on Oracle’s brand of RHV 4.4) using ConnectX-5 adapters in VPI mode, that is a switchless setup using six direct-connect cables. Originally it sounded as if that would also work with Infiniband (the NICs are dual personality), but driver support was only put in for Ethernet, probably at the behest of management…

There the hosts are big Skylake Xeons and use quad NVMe boards for plenty of HCI storage bandwidth.

Unfortunately those ConnectX-5 VPI cards are still quite expensive, even refurbished.

yes, interesting board @abufrejoval!

So your HCI cluster at work consist out of 6 servers? The ConnectX-5 25Gb is affordable. The ConnectX-5 100Gb is indeed expensive. Cool stuff!

I also checked the Intel X710 a 4 ports SFP+ 10Gb card. To connect the servers 10Gb switchless, and an 10Gb connection to the storage server. Looks interesting too because I already have Unifi Aggregation switch.

Thanks for sharing.

Solar panels in on the list to investigate, and indeed an option!

I’m still in the process of making plans how to proceed with this project. Did some investigation and experimented with Proxmox.

If I chose to go the desktop hardware route I’m leaning to use a Ryzen 7900 because it has a lower tdp. People advised to use this mobo → ASUS ROG Strix B650E-E Gaming WIFI.

I also came across the MS-01 from Minisforum. I’m now running an Minisforum UM773 as an experiment. It runs okay for a couple of months. One downside is the maximum 64Gb of memory. A reason not to choose for Minisforum MS-01 is the delivery problems some people complain over at ServeTheHome. The compact case (heating issues?). Not officially supporting 96Gb of memory.

I used Proxmox the last couple of months and watched videos on YT. My goal is still to build a 3 node cluster and I’m thinking of using ceph. I also like to have a power efficient cluster. Not that is needs to use the minmum power a possible but just power efficient. That is also a reason for me not to use older hardware. 10Gb (SFP+) network speeds is wanted. I have Unfi aggregation switch. I understood SFP+ is more power efficient as the ethernet variant.

Case only supports mATX, I’d suggest that you look at Fractal Designs R5/R6 or possibly R7 depending on your requirements

You might want to consider ECC

There are 10 EUR cupons available and free shipping

I would recommend that you also go for a ATX 3.0 PSU
Examples (no idea if they’re good or bad)

Thanks!

I wasn’t aware of ATX 3.0. What’s changed?

ECC memory is something to consider…

Fractal has indeed very nice cases. Interesting option.

Lead-acid batteries don’t like being discharged more than 50%, it’s bad for them. So you’d have only only about 420 (haha) Wh before you have to charge it or shut down the PC. Which would only amount to about 4 hours for 100W per hour draw. Which is still a lot, compared to the average 7Ah UPS battery (10 times larger). With a picoPSU, I’d at least assume the average power consumption would be somewhere around maybe 30W, so that’d put it at 13h of runtime. Even if we are generous and say the power draw is 15W, it’s still 28 hours of runtime.

I wouldn’t run a proxmox cluster on anything less than 5 clusters (due to past experience). Getting a node out of the cluster, when your cluster is down is a royal PITA (if even possible). You have to start your VMs using a command to disable quorum on each startup.

I don’t remember, what was the purpose again? For home usage, single servers with SPOFs are completely fine, if you don’t want to invest in both hardware and upkeep costs. Of course, if you can get away with running containers and use SBCs, that also helps with power consumption.

1 Like

https://seasonic.com/atx-3-0-standard