[Build Log] Home server and old Xeon : it sounded like a great plan

So, after watching L1T videos and reading forum posts, and having no reasonable arguments to build a threadripper / epyc workstation, I decided to build my first (serious) home server.

The Before Times

Actually, this is not my first home server. I had a Raspberry Pi 3B (not the B+) running OpenMediaVault for a couple of months as a NAS. But, with the poor network capabilities of the RPI and having only 2 HDD connected through USB the performance was below what I was willing to endure. I was also not conviced by the OMV UI.

Why am I doing this again ?

With my home server, I would like to solve a couple of issues. I have multiple free-tier online storage solutions (google drive, dropbox, FAI…) that I would like to consolidate in a single place, under my control. I would use it as a backup solution and also provide storage to the family if needed. It would be a useful piece of kit to synchronize my data between multiple devices. It will also be a good platform to host other services, and learn Linux stuff on the way.

Requirement engineering is a full-time job for a reason

I have simple requirements :

  1. I want my server to be cheap
  2. I want good performance
  3. I am not sure what I want, so better have expandability in terms of functionalities
  4. ECC is good, so I want it
  5. I need it to be reliable, and low-maintenance
  6. I don’t know what I am doing, so I need to pick solutions with good documentation
  7. In terms of applications, I know there will be Nextcloud and Navidrome to start with, and probably other stuff will come up on the way
  8. It will be accessed from the outside world
  9. It need to be reasonably secure

Nothing complicated you see !

Trying to be cheap

For the CPU /Mobo combo, I settled on a chinese X99 motherboard (HUANANZHI X99 8M) and a Xeon E5-2630L v3. The L variant is a low-power variant without sacrificing functionality. Adding 8GB of DDR4 Registered ECC and a Snowman cooler, it cost me around 140€, a really good deal !

I was tempted by going Ryzen but a CPU/MOBO combo would cost me at least double and I was afraid of lacking PCIe connections for the expandability side.

For the case and power supply, I recycled a home computer in a Antec mid-tower case and the no-name power supply in it. On the drive side, I reused most of the stuff from the RPI OMV trial, so 3x1TB HDD and a 128 SSD for boot.

The Antec case :

Putting everything together, I realized that the cooler is slightly too high for the case :sweat_smile: So one of the side panel was bending a little bit…

Connect everything together, boot Ubuntu and off we go !

Save a penny today, loose a dollar tomorrow

After playing around a bit with the system, I realized I had a couple of issues :

  • Although RAM is working, Linux does not see the ECC protection
  • The case cannot fit more than 4 drives total without extensive modification
  • The case can only be cooled using a rear 92mm fan
  • If more drives are required, I will probably fall short of PCIe connectors
  • The bending panel FFS

But I am happy and confident enough with the whole project to spend a little more money on the thing.

Fine, I am opening the wallet again

Looking at the (un)expected limitations of the first build, I accepted my fate and bought a couple of new parts : a Fractal Define 7 XL to store enough drive, a 550W gold PSU, and a new motherboard => 400€ total for the upgrade, reusing the CPU, the RAM and the drives from the first build.

The “new” motherboard

I did not want to switch CPU as I was quite happy with the performance I had, so I found a used Z440 motherboard from an HP workstation, bought it with PSU adaptation cable and an IO shield for 100€.

Testing the Z440 motherboard :

HP proprietary shenanigans

Before buying, I was aware of the non-standard PSU cable for Z440 motherboard, and I anticipated by directly buying adaptor cables. What I did NOT expected was the FANS having proprietary connectors too :face_with_symbols_over_mouth:
and also the front-panel connector has an undisclosed pinout :face_with_symbols_over_mouth: :face_with_symbols_over_mouth: :face_with_symbols_over_mouth:

On top of that, if the motherboard does not detected a CPU fan it will refuse to boot. So time to heat up the soldering iron

The CPU connector has 6 pins. Grounding the 2 extra pins seems to do the trick and the CPU cooler is working and detected. I wanted to avoid any modification of either the board or the connector in case I need to re-purpose them later. No 6-pins connector were in stock when doing the mod so I used 2 3-pins connectors I had in stock to do the adapter.

The other fan connectors on the board also are non-standard, but it is the same pin-pitch and wiring as the standard one, juste without the offset plastic bit. I made an adapter for those too but you can simply break the offset/foolproof bit on either the fan or the board side to plug it in.

After some sweat and tears the system is willing to boot ! Hooray !

I still have an issue where the BIOS is reporting a missing front IO connector, so I have to keep a keyboard plugged-in to press enter everytime the system reboot :sweat_smile: but I can power on the system using the switch on the rear IO panel of the motherboard. At least I have that working…

The beginning of the end ?

I still need to give another go at the front panel IO connector so I don’t have the BIOS message preventing booting sequence and so I can use the case power button and LED indicators, but the system seems to be working, it even survived a power shut-down in the neighbourhood.
I plan to open the system again to upgrade to 32GB of RAM anyway so…

The good, the bad and the ugly

So that was for the hardware part. Even if I ended up replacing most of the component of the original build (I only kept CPU + RAM), I think it was valuable to learn about my options and how the whole system work/could work. If you want to build a small home server that can be a cheap route you can follow as long as you keep your expectations in check. For me, realizing that the motherboard did not handle ECC correctly was the real deal-breaker, I could have probably lived with the limited BIOS and the expandability for a while.

I do think the Z440 motherboard was a mistake. I am lucky enough to have found workarounds for most of the issues for now, but I would advise anyone to avoid trying to reuse these boards and pick a regular motherboard from maybe SuperMicro or Asrock Rack. I cannot comment on reusing a full Z440 workstation though.

I am really happy with the Fractal case. It is heavy and expensive, but I know that my drive needs will be covered and I can put whatever I want in it. the build quality looks really good.

For now I am happy with the CPU. It does not consume too much power and if I ever need something more powerful I have plenty of options in the Xeon v3/v4 family to upgrade to.

Next time I will talk about the software side of things… stay tuned !

7 Likes

Pick your poison : the OS

For the operating system I hesitated between Ubuntu Server, RHEL, TrueNas Core and TrueNas Scale. I planned on using docker container for the multiple applications, so that basically put TrueNas Core out of the competition. I was hesitant with TrueNas Scale as it is relatively new and I was afraid of lack of documentation and weird issues. Finally, between RHEL and Ubuntu, I selected the latter given the abundance of documentation and information, and it also seems like you are more likely to find a .deb package than a rpm. So Ubuntu Server 20.04 LTS it was !

6 Volts above ground

Storage

To manage the storage, I tried using LVM. The benefit is the ability to expand a partition that is already existing when adding a drive. Thus, when I will realize that I under-estimated my data hoarding addiction, I can add a single drive at the time and spread the cost of new storage over time.

When rebuilding the system with the Z440 motherboard, I switched to ZFS for data storage with a single pool and a single vdev of the 3 drives in raidz1. I have the benefit of data protection but if I want to expand the storage, I will have to install a full new vdev (so at least 3 drives for raidz1). I am sure there is a way to get parity with LVM, but ZFS seemed an easier solution.

Apps

To manage the multiple applications I planned to put on the system, I went with docker container and portainer to create and manage them. Containers felt to me having enough isolation between them so that if one screws up badly it does not take the whole system with it, without having to set-up full VMs for everything. I need some of the apps to communicate together to some extent so docker can help with that.

Remote access

To access my system from the outside world, I created a domain name (for free) with duckdns, also providing a small script to update the IP address. My domain is therefore MYDOMAIN.duckdns.org and each app will have its own subdomain (like nextcloud.MYDOMAIN.duckdns.org). This is handled by a reverse proxy running in a container : Nginx Proxy Manager . So I have to

  1. Open port 80 and 443 on my router
  2. Forward these ports to my server (also done on the router)
  3. Check that those ports are open on the server
  4. Add a proxy host in NPM, with a self-signed certificate from Let’s encrypt (done automagically by NPM)

For this to work you need all your containers to be on the same network (can be checked in portainer). That way only ports 443 and 80 need to be open and Nginx is doing the triage of sending the good packets at the correct place. Note that you can still access the apps directly on the local network with their normal port (such as 32400 for Plex).

Ubuntu strikes back

When rebuilding the server with the new case and motherboard, I switched from Ubuntu Server 20.04 LTS to the latest 22.024 LTS. When reinstalling the software stack it looked like docker applications were blocked by apparmor. This did NOT happen with 20.04. Trying to kill containers got denied by apparmor even being done with root ! It was really weird but what got me on the way was seeing messages in apparmor and dmesg logs. I am not sure why or how it happens, but it seems like at least some part of docker was running from the snap docker package and not from the regular .deb package. A good sudo snap remove --purge docker solved this issue. I now have a reason to hate snaps like everyone else.

Apps like in a candy store

At that point, it was pretty easy to deploy everything. Simply use portainer to create stacks for each application, and pay attention to having them on the same networks.

Nextcloud

For Nextcloud, make sure to map all the volumes to permanent ones like so :

    volumes:
     - /home/USER/nextcloud/custom_apps:/var/www/html/custom_apps
     - /home/USER/nextcloud/config:/var/www/html/config
     - /home/USER/nextcloud/html:/var/www/html
     - /tank/nextcloud/data:/var/www/html/data

In my case the data lives on spinning rust with ZFS keeping an eye on it, and the database is on the SSD. I hope that this config gives me better performance but I am not sure if it makes a difference.
You will have to add your subdomain in the NEXTCLOUD_TRUSTED_DOMAINS environment variable.

Also if you have an issue of the desktop nextcloud client complaining about using http instead of https, you need to add 'overwriteprotocol' => 'https' in the config.php file in nextcloud directory.

You can check the security of your nextcloud installation here.

Navidrome

I simply followed the official documentation for this one. First, I wanted to have Navidrome directly reading my music folder from the Nextcloud directory, but when using the same UID:GID for Navidrome than Nextcloud I got an error when running the container. So I made a cron job for sudo to copy the music folder from Nextcloud to the Navidrome directory…pretty inefficient. I will probably remove my music from Nextcloud in the future anyway.

Plex

I used this container for my plex server. Note that you need to pass a token the first time (doing it in the docker-compose did not work for me). To do that, I went to https://www.plex.tv/claim to get the token, then tunnel to my server through ssh with ssh turing -L 8888:localhost:32400 -N and then by connecting to http://localhost:8888 I was able to start the plex server.

Minecraft java server

For Minecraft I used this container. Note that there is also a bedrock server from the same person if you want. I was NOT able to get it through Nginx for this one, so I needed to also open and forward port 25565 on my router. This is an annoyance for my because it kinds of break the whole model and organization and it also prevents me from simply creating a new subdomain and a new container for a new minecraft server. Duh…

Always look on the bright side of life

For now, the server is up and running. It even survived a power outage in my area. I have Nextcloud synchronized between desktop, server and phone. I played with my friends on the minecraft server. I was able to remove my music library from my phone and listen to it remotely with Navidrome (thanks FFMPEG for format conversions btw) and the Plex server seemed to work for me.

Life is a piece of sh*t, when you look at it

…But there is still a couple of software tweaks that I would like to do :

  • I don’t have an external backup. My server sits in my flat. If the building burns down, I loose the server, the desktop and my backup drive.
  • I do not have incremental backups over time. I have no way of checking back to the previous state of a file.
  • I don’t have a good way of monitoring the server. I tried Cockpit and I like it, but the fact that is gives you a terminal and thus bypassing the ssh key that I have set up is a bit of a concerned for me. It is available using container but the status is weird (looks like it is not supported for docker, but only for podman). So wait and see on this one.
  • I feel reasonably secure for non-targeted threats on the internet, but given that the server is on the wifi network, I would like something a little bit more robust than just password protection to access all the services (especially portainer and nginx with its certificates).

See you on the next one :wink:

3 Likes

For monitoring, check out LibreNMS. Pretty easy setup and it works well, and will run in Docker

1 Like

Ex-Xeon X3450 Lynfield user here. That thing was doubling my power bill. I’m glad it broke down, although I wish it didn’t go out the way it did. Now I’m a fan of SBCs and containers.

In your place, I would have installed Proxmox and ran VMs and containers on it. Bonus for built-in ZFS support. With LXC, you get the benefit of running any OS you want without the penalty of virtualization. You can more easily find tutorials for GitLab, databases and other stuff for CentOS / RHEL, so you can do a Rocky container, while for things like plex and other stuff, you can run Ubuntu.

For my personal use, I prefer to not mix and match, but when you are a beginner with not a lot of VMs or containers, it’s fine.

1 Like

I was not aware of LibreNMS, thanks ! At a first glance, the alerting system looks nice, but I need to look further into their docs

Power consumption was also on my mind, that’s why I chose the 2630L v3, but it is probably worst than a recent Intel or AMD chip. However, given the overall cost, I think a more recent and power-efficient system would have cost me a lot more to build than the savings in power it would have brought.

1 Like

Depends on how long you plan to run it. I would say that if a device will stay powered on for more than 5 years and the difference in both idle and load are not marginal, you’ll probably make up for it in electricity bills, especially if you live somewhere where electricity is expensive.

I now go with ARM SBCs because I want eventually to power everything by solar and in cloudy days, a small generator (still not sure what I’d be using).

1 Like

Update

Hardware

On the hardware side, I found a good deal on another set of RAM and increased to system memory to 32G. I don’t think it was needed at all, but could not resist the good deal.

I also played around with the front IO power connector. I managed to find the power switch connector on the board and a way to power the system LED but I still have a “Front Panel not connected” error message during boot. I still need a keyboard plugged in the machine to get it to boot.

Here is what I have so far for the front IO connector :

LED+ LED- X 0 3.39 3.52 3.52 2.52 0 0 0 0 5.0
3.3 1.69 PWR SW PWR SW 3.42 3.52 3.39 2.52 5.0 5.0 0 3.29 2.39

The X is for the missing pin on the board connector. The values are the measured voltage on the connector.

Software

Aside from the original Java server, I deployed a new minecraft Paper server, with Floodgate and Geyser. This allows a PC bedrock user to join your java server. Also deployed using a docker container. I only had a quick try with it, so it appears to be working but did not investigate further. It does not work with the console version of Minecraft.

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.