Hi everyone, instead of making topics i wanted to go with blog topic where i share my homelab setup and continuous upgrades.
I run my own business as a freelance mechanical engineer and i like running my own servers to automate things around my house.
I’ve shared parts of my homelab before. for the last 10 years I’ve had a download server, it used to be an old laptop with external hard drives and windows. This has continued 4 or 5 years ago when i turned my old skylake gaming PC into a server with ubuntu server and a couple of hard drives.
Current specs:
My old gaming pc. MSI Z170 ITX,
Intel 6600k, 16gb ram (quicksync!)
500gb nvme pci-e drive
14tb hard drives (8 + 4 + 2)
Coolermaster V650 (one of the more efficient atx power supply at very low load)
Power use: 33W with drives always spinning.
Ubuntu server LTS (21 i think).
Docker with about 34 containers running at the moment.
I’ve always just had disposable storage with no redundancy and only part of it backed up. But i want to move my important data out of the cloud and also on my own server.
I’ve had some advise from this forum to separate storage and services. So i’ve gone ahead and ordered all the components to build a new Nas to run Truenas.
New NAS:
Fractal node 804
Asrock B660M Steel legend
Intel 12400 (will take it from the service pc, and i bought a 13400)
2x16GB 3200 Memory
3x Seagate Exos X16 16TB (RAIDZ1)
2x Samsung 870 Evo 4TB (MIRROR)
1x Samsung PM9A1 512GB (for Truenas and storage services)
PicoPSU XT160
192W adapter (the switch and edgerouter will be also connected to this to save power)
VM server (i already have this):
Asus Pro H610T D4-CSM (Thin mITX)
Intel 13400 (upgraded and 12400 is used in the NAS)
Noctua NH-L9i-17xx
Silverstone PT13 case
2x 16gb
Crucial MX500 500gb (for Proxmox and VM’s)
Samsung 980 1TB (for storage of services)
Seagate 8TB drive (this will be used as a 2nd backup location for the 4TB mirror)
I’m also going to upgrade my network to 2.5 for those that can use it. 2 cables i’ve ran in my house have been glitchy on 1gb, so i will replace them with cat6.
Switch: Mokerlink 8p 2.5gb
NAS: already has 2.5gb
Gamepc: 2.5gb PCI-e card
Work laptop: already has 2.5gb
VM server: USB3 2.5gb card
My idea is to move the server from the attic to the electrical room. This will be with custom brackets and the 804 should just fit with the door closed. the 804 should also be large enough to house the VM computer. so both will be protected by the dust filters in the node 804 case.
Expected extra services:
Photoprism for photo’s
Nextcloud or filerun for files/documents
Something to record the front door camera
Uptime Kuma with notifications so i will actually know what is down.
Testwebserver for my company application
Recipe server (tandoori.dev or something else)
For now i’ve ordered all the parts i need. I can design the bracket and see if everything fits. and then the final build and software installation will be on my vacation from 1 okt.
Duplicati → Syncthing
paperless-ng is end of life and unmaintained since February this year
You probably want to Owncast instead of Restream or simply nginx with a rtmp module
I personally be concerned about exposing some of these services online without filtering and very close monitoring.
I don’t really like paperless-ng handling of files. Except the search handling. So I’ll move to something else.
I don’t really use restreamer anymore so that gets dropped.
All the services that are available online are through a reverse proxy and SSO with authelia. With some apps having their own login on top of that. And a SSL wildcard cert on the domain I use. What kind of danger am I still exposed to ?
As long as you keep on top of authelia and nginx security updates, not much unless one of the authenticated accounts get compromised (e.g. someone gains access to both its pwd as well as its MFA ) - then, potentially escalation of privilege attacks at worst.
Personally, I’d take a look at paperless-ngx before abandoning it - its a fork maintained by the lsio team, with some fairly substantive updates.
Ah I’ll be moving nginx and authelia to a different vm than the rest of the system. with more frequent automated updates than some of my other docker containers (because they sometimes break…)
My biggest gripe with paperless-ng is that it didn’t rename files or put it into a different folder structure. but looking at paperless ngx it does have that feature. I’ll try that for sure!
Probably the most interesting thing in the setup is the picopsu. Essentially a small board that sticks into a 24 pin and connects to an external 12v adapter. It’s splits off 5v and 3.3v and manages everything a powersupply does. But more efficiently, computers with 5-6w idle usage should be possible with it.
This is the largest 160w model that can power this whole system.
I built the NAS and installed Truenas. did some testing but the power usage is kind of dissapointing. This is after applying drivers in windows and Powertop optimization in truenas.
Windows 10 without drives: 22W
Truenas 22 5.x kernel: 45W
Truenas 22 5.x kernel drives off: 35W
Truenas 23 6.x kernel drives off: 30W
Compared to that, the VM server with the 13400 is around 12W min.
I’m not convinced my power meter is very accurate, so i’ve also ordered some shelly plugs that i can connect up to home assistant.
So the RC1 version of truenas scale 23 is more efficient, but also kind of buggy (not a stable release). I’m thinking of stepping off of truenas and just running zfs in a proxmox cluster. Ideally just a docker compose that gets pulled off a git.
The other issue is that k3s is not very efficient and a pain to setup for any file related services. I’d rather run things like nextcloud or photoprism on the nas instead of a different computer.
Here is the setup running in the corner of my office.
I also installed the 2.5gbE adapter in my game computer. The GPU really effectively covers 5 slots because the slot next to it would obstruct the fan You can see the NIC on the bottom left in the last slot.
I assume that’s TrueNAS Scale, given the kernel versions? Bit odd, since in my experience Linux tends to be quite a bit more power efficient than Windows when mostly idle (too much random background jobs on Windows). Might be something about the way iXsystems is doing things…?
It’s mostly that the hardware is a bit new and only got support in kernel 6.2 especially because intel alder lake with the P and E cores was quite a big change. (even though this CPU only has P cores)
I can see in powertop it supports better CPU states on kernel 6.2
Nice! I was thinking of doing an apu build with a pico-PSU myself (probably a 90 to 120W model), especially since 12 to 19v can be drawn from USB-C PD, that’d make for a funny build.
What’s the case you are using? I’m looking for a cube-ish case for my next build myself. I found the silverstone sg11 to be a good candidate, but I’m open to other microATX suggestions that are as compact as possible.
I’ve been playing around with getting the power usage lower on both the VM and the NAS. I have Shelly plugs now that report the power usage for both constantly.
For the proxmox server VM server the (completely idle) results are very good! What I’ve done:
Powertop --autotune
Set in grub GRUB_CMDLINE_LINUX_DEFAULT=“quiet consoleblank=15” This stops showing the console and makes the graphical output sleep (so the GPU also goes into sleep)
Installing non-free-firmware by adding it to the repo and installing. This did want to uninstall proxmox for some reason, so i stopped it. and i don’t know if it did anything.
This brought the completely idle power use from 18W to ~4W ! with lows at 3,4W That is pretty much the same power usage as my 8p 2.5gb switch!
I haven’t been successful in my truenas setup with the drives spundown and powertop i’m at 32W. There might be some driver issue? I’m thinking of just installing proxmox and setting up zfs in that. and testing if i can bring the power down.
I have a really weird issue. I’ve ran new cat6 shielded cables to the 1st floor and attic. and the 2.5gbe switch is installed in the electrical cabinet.
From my laptop (11800H gigabyte with 2.5gbe) i can saturate the 2.5gbe on the smb ssd pool or the smb hdd pool.
From my computer (x570 / 5800x3d with a 2.5gbe pci-e card) i can read the full 2.5gbe but writing to the ssd or hdd pool over samba is only at 130mb/s. Is this a windows thing? I have full link speed and even a file bigger than the amount of ram on the server gives the same result.
I’m feel like i’m mostly driving myself insane. the number of times i reinstalled something on my nas to test it:
Truenas scale 22: high power use and i don’t like k3s
Truenas scale 23: lower power use but still k3s and buggy (not a stable release)
Proxmox: had to remake the ZFS and installed an lxc to do samba. low power use. but i couldn’t figure out NFS sharing. i was hoping for an easier solution.
OMV: pretty easy install, but setting up zfs required all kinds of extra stuff like installing the proxmox kernel and installing extras.
Debian + cockpit: works allright although there was a lot of manual setup and me having to install everything and solve stuff. had to recreate the zfs pools because i couldn’t access them.
So now i’m at a point of, i can get cockpit to work, but it’s all just a debian install and i would have to do most stuff manually, the chance something breaks when it’s in full use is a lot higher. I should have just sticked with proxmox and installed an nfs server.
This list is mostly for issues i had and to help those googling.
Jackett
Failed to create CoreCLR, HRESULT: 0x80004005
Wrong permissions on the temp folder. after chown 1000:root and chmod 775 it was fixed.
Unifi container
Fatal exception has occurred. program will exit. invalid initial heap size
in the config data change file system.properties and change unfi.xmx=1024M to 1024
Qbittorrent
Unacceptable file type and with a black screen
this is an issue with alternate webui. close the container and in qBittorrent/qBittorrent.conf change WebUI\AlternativeUIEnabled to false
This allows you to fix the ui.
Sonarr
System.UnauthorizedAccessException:Access to the path is denied
I forgot to setup nfs shares in fstab. Just mounting will unmount in a restart.
nginx proxy manager
errno 13 permission denied
This was a permissions issue in the mysql database for me.
Going into the database (in portainer you can just click console and the switch to bin/sh and use the other commands based on the location of the mysql folder)
docker exec -it proxydatabase /bin/sh
cd /var/lib/mysql
chown -R mysql:mysql npm
exit
Docker LXC
error mounting “/etc/localtime” this is part of my docker compose files. I setup Alpine docker LXC images, but alpine images don’t have localtime. I fixed it by going for a debian docker lxc image.
Database passwords
i had an issue with photoprism where the database connection was lost because i changed the password in the docker compose file. Once the database itself is made, you can’t change the password! To change the password you have to go into the console of the database container.
I used this link to change the password. https://linuxconfig.org/how-to-change-mariadb-user-password
the command in step 2. use mysql; refers to the name of the database, so check what the name of your database is and put that in command (use databasename;)
Docker compose passwords
One thing i had a hard time figuring out is storing passwords outside docker compose files. I’m using a connection to git for the docker compose files, but i don’t want to put passwords in there. So how i set it up is:
In the docker compose files, replace passwords with ${variable_name}
then in portainer when creating a stack you can point to the git compose file and below in the enviroment variables you can put them in as
variable_name=“password”
I’ve had some errors in zfs that caused a drive to be put offline. But errors where on all drives. I had a feeling it has something to do with power.
I’ve turned off the powertop --auto-tune service to see if that fixes it. and it does look like that was the issue! I’m going to keep testing, but I think the motherboard doesn’t properly support sata link power management (which is also why it is turned off by default in the motherboard…).
It could also be because of the power supply just not being sufficient and causing issue’s. i might still replace the pico psu with a real power supply.
You can see the NIC on the bottom left in the last slot.
