Hello again, boy, a lot has happened.
Mentioned in the last post I bought a Asus Hyper M.2 PCIe expansion card, and it turns out the Z97-a motehrboard I was using doesn’t support it, bummer. It requires PCIe Bifurcation to work aaaand… I got a X299 motherboard and a Intel 7820x to upgrade my server to support this… This mobo does support PCIe bifurcation… I totally did not spend a lot of money on something I would barely use! I promise!
Well, I hit a few snags, I bought the CPU and mobo second hand, and I could not get it to boot… After days of troubleshooting and borrowing parts I discovered the motherboard was dead sadly. So now I’m waiting for a new motherboard… Money go poof.
So, am I going to use the PCIe bifurcation and M.2 expansion card when the new motherboard arrives? The almost sole reason for this upgrade? Probably not… Because the motherboard has 2x M.2 slots that uses the chipset PCIe lanes, and that means I can fit a GPU(One day) for Plex encoding! I have decided to make the two M.2 drives a vdev/raid mirror for ZFS metadata instead of read/write cache, cause I/O in some edge cases on this raidz1 setup goes through the roof, aaaand a Level1tech video said it was a good idea.
Next up: pfSense and networking, yup, another rabbit hole.
My old Z97 and 4790k server that was being replaced, I decided would make an excellent pfSense router! I had heard IDS/IPS required some horsepower and my 250/250mbit line would probably be fine with that CPU. I found out later it wasn’t the CPU I should have worried about, it was the Snort/Suricata IPS tuning I would have to do, oh god… Still working on that.
But the X299 mobo being dead on arrival and all, I had to put that on hold. I did have though, an LGA775 motherboard and a Intel Core 2 Quad Q6600! If anyone remembers that thing, it was a beast back in the day that would OC like crazy. I also bought a Intel i350 4-port gigabit NIC that everyone recommends for pfSense.
Related to this, I also bought a Ubiquiti US-48 switch. This has 2 SFP+ uplinks and 48 gigabit RJ45 ports. I was using a MikroTik CRS305 4-port SFP+ switch that I only bought a month or two ago that this now replaced .
This brought another issue, being a Unifi switch it wants a controller, I initially set it up only using a desktop app on my windows machine to initially set it up and closed the controller, worked perfect. But I wanted a dedicated controller. I though of buying a cloud key, using a raspberry pi, but what I really wanted was for the pfSense box to host a controller. How hard can it be… Well, really hard, googling this meets you with a lot of “no not possible” and “just buy a cloud key”. But I persisted in my googling and I did find a solution! Someone has posted an install script for this exact solution on github!
Link to that: https://github.com/gozoinks/unifi-pfsense
It’s not without it’s flaws, it’s a bit fiddly, but it works. All I had to do was admin SSH into the pfSense box, go to the FreeBSD console and fetch the script, easy. Took a few tries, but I got it working. I was actually really happy with this solution as the pfSense being a dedicated router, that thing will mostly be up, and I saved myself a hundred or so dollars on a cloud key. Nice.
So, we put the Q6600 system up and running with pfSense. If anyone tells you that pfSense has a bit of a learning curve, that’s an understatement… It’s hard. I spent about a day and a half getting it somewhere where I wanted it, a lot of that spent on IPS/IDS stuff and just learning as I go along in general.
So some people online, while researching this, said the 4790k would be too power hungry for a pfSense router. So I bought a wattmeter to do some checking, and oh man am I glad I did.
The 4790k server I was running draw a surprisingly low 45 watts on idle, this system has 3 HDDs in a raidz1 that are always spinning, 1 SSD, 4 fans, a connectx3 10gig NIC and I’m probably forgetting something. I just expected more. The CPU is runnign completely stock so it has all of it’s power saving features enabled, which I think is a good thing for a mostly idle server.
Now, wattage on the LGA775 Q6600 pfSense router… With a PCI display adapater(Not PCIe) and an Intel NIC, ON IDLE… drum roll… 160 WATTS! With spikes to 200w on load! I have a 200 watt router… Yeah, I’m not gonna keep that hardware running for long. I’m expecting the 4790k to draw around 30 watts on the pfSense, so that is much, much better.
Talking about watts, UPS Time!
So I decided I wanted a UPS. New ones are expensive, so I decided to look up the used market. Most of these UPSs use standard form factor batteries, so replacing them is no issue if the price is right.
I found one… an Emerson Liebert GXT3-6000VA… This thing is a monster! WAAAY more than I need. It was sold as “broken”, me knowing what deceiving bitches lead acid batteries can be I decided to buy it, ehem… I bought two of them…
So, I get both of the 6000VA UPSs home. I try to boot them and there is error codes all around, but they do turn on. A LOT of manual reading later I find emergency switches I need to check and a laundry list of other things.
But, it still won’t function, but the only error code now is “battery missing”
There is 2 banks of 10 batteries in each UPS. So I have 4 battery banks in total, 40 batteries… I check the voltages, they should read 120 volts each, 2 of the banks are dead as can be, but two of them are reading 120 volts! Success! Or so I though, as soon as I put a load on one of the two 120 volt banks it tanks…
So, I take apart the failing bank that still has 120volt, I check all the voltages, all batteries are reading 12 volts, except one reading 9 volts, aha! I grab a car light bulb 55 watts, to do some load testing, and the battery reading 9 volts tanks to 3 volts when hit with a load. No light in the lightbulb, I have found my culprit.
I buy a new battery, series them all up again, assemble the UPS again and voila… I now have a 6000VA UPS… for a 45 watt server!
I haven’t been doing much course studying later, but I feel justified in having been busy troubleshooting motherboards, fixing dead UPSs, learning pfSense.
This post is LONG! Sorry about that
But it is coming together and it is exciting times!