PHASE 1 COMPLETE: Low Power, Low Cost Home Lab for Virtualization Practice

[jglink]

First off I want to thank the folks over at this thread:

How to create home lab network

I am creating a small homelab with a PC I am building and some tinyminimicro nodes. However I have no physical ethernet connections in the studio I am rented at. The landlord has ATT gigabit fiber, but we are provided with just the SSID and password for the ATT router/ap. I have offered to run an ethernet cable to our studio as the modem and route…

and this thread:

Hardware Recommendations for Low Power, Low Cost Home Lab for Virtualization Practice

I know I might be asking for a lot here, and I have researched the forum and the web for the past few days , but I am looking to create a home lab on which I can learn about virtualization in general, and VMWare, ESXi and clustering more specifically. I would also want to learn Hyper-V as well, although I am not sure if the information would transf…

I got lots of awesome advice and guidance on type of hardware to go with and the types of set ups that would be ideal for my situation, along with a ton of useful links and information to sort through.

Over the last few months of planning and scoping out Ebay, Craigslist, OfferUp, r/HomeLabSales and PCPartPicker, I was able to acquire the parts necessary to build out a home lab on which I could set up a cluster of mini PC’s using shared storage, and get internet to the set up wirelessly with no cable drops. Below is my parts list:

• 3 x HP Eli​teDesk 800​ G2 mini (i5-6500t, 16 GB RAM, 128 GB NVMe SSD) - $360
• Ubiquiti Unifi Switch Lite 8 PoE (8 port POE switch) - $125
• Mikrotik SXTsq 5 ac (wireless CPE router) - $59
• Qnap TS-451+ NAS (16 GB RAM, 4x3tb WD Red’s, flashed with TrueNAS Scale) - $500
  Grand total: $1,044

I was elated to be able to come in under my price expectations with everything included. I had to scour eBay for the mini PC’s, memory, and the Ubiquiti switch. The NAS was from r/HomeLabSales, and the Mikrotik CPE router was from Amazon. I actually came in about $200 under my initial plans which allowed me to get good protection for the setup:

• APC BR1350MS UPS (1350VA Sine Wave UPS) - 200

Once I got the last of my parts last week I instantly became a mixture of giddy and nervous. I was hopeful everything would come together nicely.

I got to working as soon as I got the APC UPS. As I was cabling everything, I was able to confirm that everything fit together nicely in 1 Kallax Cube, along with the PC I built recently in the second Kallax Cube.

The MikroTik CPE router was placed in a higher up location next to the wall dividing us and our landlord’s so that it could get the best link to the ATT Combo Router/AP.

After being satisfied with the cabling set up I set out to configure all of the networking side!

With the Mikrotik CPE router I was expecting a REALLY painful process but using WinBox I was able to quickly log into the device using it’s default IP, and WinBox recognized it was a CPE Router by default in the QuickSet pane. There it was just a matter of finding the SSID I needed to connect to and entering the WIFI password, adjusting the position of the actual antenna to get the best signal strength, making the device the IP I wanted, and setting a strong admin password. About the only hiccup I encountered was that I connected the POE injector into the wrong side between my computer and the CPE. (Glad I didn’t fry my PC )

With that out of the way, my next step was to configure the Ubiquiti switch. Oddly enough it took me longer to configure the Ubiquiti than the Mikrotik. I had issues figuring out how to manage the switch because the included manual only had instructions for how to control the switch if you use the Ubiquiti management appliance. I figured out of course that you can just run the Unifi network management software on your computer, so of course I did that and then ran into the next snag, adopting the Unifi switch. Apparently the previous owner had not adopted the switch so I had to factory reset it myself and do so. Once I did that, I adopted the switch, configured the right IP’s, passwords, and port configurations.

Now, the moment of truth, when I connect the MikroTik CPE Router to the Unifi, I would get an uplink and huzzah! … Unfortunately, nothing happened when I connected the 2. I scratched my head, and let out a frustrated sigh. After about 15 minutes of google-fu and unplugging and replugging cables, I figured out the issue:

APPARENTLY, the MikroTik SXTsq 5 ac accepts PASSIVE POE as input, while the Unifi switch outputs 802.3af (standardized POE). So I had to plug in the Mikrotik’s included POE injector into the Unifi switch to provide the right type of power to the Mikrotik CPE. I bought an inline POE converter that will convert the Unifi POE power to passive POE power without needing an extra power injector (Ubiquiti INS-3AF-I-G) and expect to receive it tomorrow.

Once I took care of that mix-up, I connected my PC to the switch and BAM! I had an uplink and a 400 Mbps with 4 ms ping uplink at that (used SpeedTest). Everything was stable and the first phase is complete!

I was extra happy that I got it working because most info about my type of network setup that I found online said that it was illogical and sub-optimal. So proving that wrong gave me extra satisfaction.

The next phase of the project is to set up the mini cluster along with the shared storage. The NAS is already flashed with TrueNAS and is just blank at this point. I plan on setting up the mini-PC’s with ProxMox and running them in an HA cluster.

Any advice for the next phase of my plan? Potential pitfalls and recommendations in the relationship between the TrueNAS shared storage and the ProxMox cluster? I also want to virtualize the Ubiquiti network app and put it on the cluster, so any advice on that would be appreciated. And finally, I would like to implement some logic so that when the APC UPS goes into battery mode in an outage, the shared storage and cluster shut down gracefully. Any advice on that too?

As always, all of your help and advice are much appreciated!

[risk]

@jglink

APPARENTLY, the MikroTik SXTsq 5 ac accepts PASSIVE POE as input, while the Unifi switch outputs 802.3af (standardized POE).

You have a converter on the way… but, there might be something you can try: if you click your way to devices > select your switch > settings > ports > click on a port > manage port profiles… you might be able to create a port profile for passive POE:

me looking at switch port profile settings on unifi controller on my phone's browser

There’s still 2 or more flavors of 24V passive POE … some devices use inner wire pairs, some use outer pairs of the ethernet cable… Mikrotik wiki used to have a table for each of their devices…

@jglink

I also want to virtualize the Ubiquiti network app and put it on the cluster, so any advice on that would be appreciated.

I’m running mine in cloud behind nginx using Tailscale…
… I guess a fun way of running it for you would be to perhaps use wireguard between RouterOS and Linode to support L1 techs. … or maybe Oracle Arm VM (because free), i use an always free VM on GCP because of inertia.

If you don’t want to run the controller on host os (java, updates, yuck) , here’s a docker container: Docker Hub . You can use portainer to set it up as well if that’s what you want, you can keep it up to date with watchtower.

[felixthecat]

@jglink

And finally, I would like to implement some logic so that when the APC UPS goes into battery mode in an outage, the shared storage and cluster shut down gracefully. Any advice on that too?

Connect the UPS to one machine and set up a https://networkupstools.org/ Server on it.

Then set the other ones up as NUT clients.

[jglink]

Ooh this seems interesting, I will check it out, thanks! Would it be possible to virtualize this tool and have it one of my VM’s?

[jglink]

Thanks for the info! You know I didn’t see that in the port settings for the switch. Not sure if maybe the Switch Lite 8 doesn’t have that config but I will check it out when I get home.

[felixthecat]

@jglink

Would it be possible to virtualize this tool and have it one of my VM’s?

Absolutely, or in a Linux Container since you’re going to use Proxmox.
TrueNAS Scale also has it built-in.

Just keep in mind you’ll have to pass through the USB device.

[jglink]

So I checked the settings when I got home and it turns out I only have the option of POE OFF, AND POE ON in the profile override section. I will have to check if I can create a new profile with the right POE settings. But I did get the Ubiquity adapter and it works like a charm so I can thankfully take the POE injector off.