Home Server Goals and Issues

I’m slowly/manually/painfully rebuilding my knowledgebase from scratch. That’s going to take a while, but, and I’m going to trigger all the engineers here right now (you know who you are):

Arbitrarily deep nested note trees is best note trees.

So:

  • Trilium’s up
  • Ombi’s up, but it’s going to need some further setup before it’s useful.
  • The SSL is improved drastically, but I think nginx is still breaking the site for some (mostly mobile) remote usecases. (I’ll be poking PLL to see if he knows what’s specifically wrong on that at some point. Maybe tomorrow if he’s free.)
  • The gateway seems to be primarily functional, however.
  • I have my list of containers to implement again, so progress will go well.
  • We are working on the task of splitting the composes into ‘type’ folders, so there’s no monolithic docker stack. Better for troubleshooting.

Actually, Trilium in particular is fascinating because it’s so easy to set up. Most of the notes options that are out there require a second linked database, and other requirements. Let’s look at what Trilium requires in the Compose file, with some extra documentation that sheds some light on my practices during this project:

    # Trilium - Personal knowledgebase / notes server sync. 
    trilium:
        image: zadam/trilium
        container_name: trilium
         # It's really nice that these can be arbitrary, but for a simple homelab 
         # for production/use purpose, the service name seems best. 
        restart: unless-stopped
        networks:
            - default
            - t2_proxy
            - tools
         # The first two of these are required for Traefik and Socket Proxy to play nice.
         # The last one, however, is me isolating containers from eachother by purpose. 
         # (I have another one called 'content' for example). 
         # There's still traefik and DSP talking to everything, but I see this as hardening. 
        ports:
            - "8078:8080"
        env_file:
            - ./defaults.env
      # This is one of my favorite things about the compose, above 
      # (and secrets, which isn't relevant here). 
      #There's no need to clutter the compose for a given stack. 
        labels:
            - "traefik.http.routers.trilium.rule=Host(`trilium.$DOMAINNAME`)"
            - "traefik.docker.network=t2_proxy"
     # Traefik's labels seem intimidating, but follow certain rules (if I understand them correctly)
        volumes:
            - $NOTESDIR:/data
      # Simple directory declaration based on the .env mentioned above 
        security_opt:
            - no-new-privileges:true
      # This should be on everything.

The only real issue with Trilium, to me? It’s single-user as far as I can tell.

1 Like

Let’s talk about a change we’ve made as of last week.

I’ve dropped both Vikunja and Planka entirely. GTD is now handled purely by Wekan, with work, server, and personal, handled by different boards and different permissions. There’s no access for non-household users, though – I’d spin up a different instance for that if it were requested. However, it’s still not great on the goal of data separation, but I’m coming to accept that.

Why have I done this?

Two reasons.

First, both Vikunja and Planka have had some major stability issues in my experience thus far, plus they were rather unreasonably difficult to set up in the first place. This seems to be a common, but not universal theme of some dual-container services (it’s often app+db). But Wekan is the same way, without difficulties implementing or (thus far) maintaining it. So I put the first reason down to coding/container quality.

Second, frankly, is that I’m finding that keeping the container count lower means more to resource usage and to overall stability as a result than data separation means to me in my field.

I’ll include the wekan parts of the work-stack compose down below for reference, since it seems to work well.

  wekandb:
    image: mongo:4.4
    container_name: wekan-db
    restart: unless-stopped
    labels:
      - "traefik.enable=false"
    networks:
      - wekan
    volumes:
      - wekan-db:/data/db
      - wekan-db-dump:/dump
  # Wekan, workflow board.
  wekan:
    image: wekanteam/wekan
    container_name: wekan
    restart: unless-stopped
    networks:
      - wekan
      - t2_proxy
    labels:
            - "traefik.http.routers.wekan.rule=Host(`wekan.$DOMAINNAME`)"
            - "traefik.docker.network=t2_proxy"
    env_file:
      - ./defaults.env
      - ./wekan.env
    environment:
      - ROOT_URL=https://wekan.$DOMAINNAME

So it’s time for a fairly major update on how the server project is going. Let’s start with a look at Docker.

As you can see, the (listed following in case you can’t see the image) containers (and any associated dependencies like the […]-db containers) are running. It’s been a long time since I made a major progress update, but this it. We’ve made some real progress.

  • Wekan
  • Invoice Ninja
  • Traefik
  • Wireguard
  • DSP
  • Gitea
  • Organizr
  • Fresh RSS
  • Firefly III
  • Vault Warden
  • Jellyfin
  • Trilium Notes
  • Shiori
  • Ombi
  • Grocy

As @TheCakeIsNaOH noted … almost a year ago (gee, I’ve been working on or planning for this a long time now, haven’t I? well, lucky me that I’m the primary client here I guess), multiple compose files are much easier. Especially with Gitea, since the files are smaller and easier to navigate. After about 8 or 10 containers total, this way is just better.

We currently have 4 stacks (grocy was implemented in the wrong stack at some point and fixing that is most definitely on the list). The containers in the ‘work’ stack primarily have to do with my business, and will be transferred to their own domain that’s more appropriate for that in near future – and I have obtained a 3rd domain for the gameservers and matrix homeserver as well.

Databases, even without multi-tenancy, are still our largest single problem. Anytime a container requires another container in some manner (outside of, for example, everything on here requiring traefik and traefik requiring DSP, since that’s all contextual), it creates first-boot problems. Even with .ENVs and configuration directly off of the creators’ github pages, something always breaks with that first boot. However, eventually we get there.

SSL is coming along well, and now I think SSL basically works across the server for both local and remote users (waiting on inevitable bug reports to come in… eventually). Unfortunately the rsync method (recommended by @PhaseLockedLoop) of ensuring that the gateway server and the local server always have their SSL files synced is unlikely to be implementable in the near future.

Hardware acquisition is very much in the research phase. There is a lot to learn about the 5-10 year old “complete units” (that is, server case plus PSU(s), boards, drive bays, CPUs, and sometimes even RAM) available on ebay for $300-$500 a pop or less, and I have no foundation in this area. I do know that I don’t need all that much in terms of performance, but of course that’s not an objective, binary evaluation is it?

There’s still a wordpress blog hosted elsewhere, distinct from the server, and I’ve been considering whether it’s worth transitioning it over (probably including converting it over to Ghost). For now, I’ve concluded that the answer is probably not – but later in the process, potentially.

As you can see in the list, we have Gitea running. Not only that, but with Git it’s managing the actual live server folders, so we have direct VC in prod. That’s probably a scary statement to some of you. The repository is not currently public, because of secrets having to live in ENV files currently. While there is a likely at least two to three workarounds for this, the primary purpose of gitea for us is VC, and it’s doing that – so further modifications are relatively low priority. We’ll get there.

Grocy’s been a bit bothersome to implement, and we’re not actually 100% there yet. I really need to get a cheap chinese nonsense tablet for the kitchen for it, something that I can deny internet access to so I don’t have to worry about actually trusting it. That will help a lot, so long as the device can actually run both Grocy web-interface and/or mobile client, plus the barcode buddy thing that really, truly, should not be a separate piece of software. It’s functionality is core to what Grocy is trying to achieve, after all. However, Grocy is already proving quite useful regardless. The only problem is that it doesn’t seem to want to store it’s data in the volumes that I’ve provided it – but that’s an issue for a later pass, when we implement container+data automated backups. The household members haven’t really complained much about it yet, but don’t really have the full habit of using it yet either.

Firefly III isn’t useful yet. It runs, and the functionality we’re looking for is absolutely there – for the most part, anyway, and certainly above the minimum bar. Many of the alternative options fall short for a number of reasons, chief among them, ironic or not, being monetization. What Firefly needs is time and use to fully come into it’s own – rather like Grocy in that.

Vaultwarden, meanwhile, is useful right out of the gate. I’ve had no major issues with it once we got it running. I cannot recommend Vaultwarden enough, and I mean to everyone. And that’s why I’ll pick Vaultwarden for our code snippet for this post. It’s fairly basic, but that it’s requirements are so simple is a good thing.

    vaultwarden: # Alternative bitwarden server
        image: vaultwarden/server:latest
        container_name: vaultwarden
        restart: unless-stopped
        networks:
            - t2_proxy
# Usually I'd put ports right about here, but with Traefik they're unnecessary. 
        env_file:
            - ./defaults.env
# No need for an env for Vaultwarden - or any db container, for that matter. I call that a well-maintained image, myself. 
        labels:
            - "traefik.http.routers.vaultwarden.rule=Host(`vw.$DOMAINNAME`)"
# Typing out the entirety of 'vaultwarden' as the subdomain seemed a bit much when I'm trying to encourage better security habits in users. 
        volumes:
            - vault-data:/data
# Single volume requirement - nice! 
        security_opt:
            - no-new-privileges:true

Invoice Ninja ended up replacing Akaunting for a number of reasons. Both were difficult to get running, but Akaunting’s locking of several basic features behind paywalls and integration with central servers was too much for me. Invoice Ninja does have monetization problems of it’s own, but they’re less important for the most part. Besides, Invoice Ninja seems to be much more of a direct Wave competitor, so I’m already comfortable with many of it’s preconceptions and assumptions. It also has time tracking, and trying to find a container with just time tracking was a nightmare. So that’s nifty.

FreshRSS would make me happier if my RSS feed collection – along with the books, music, and audiobooks – weren’t lost during that several month break of working on the server. However, it’s working, so I’m building that feed collection back up.

Anyway, that’s where we’re at. More progress is continuing, and we should have a synapse container for a Matrix homeserver soon :tm:,

2 Likes

Okay before I read the rest I’ll tactically deal with my @

Why is this hard? Or difficult to implement in the near term? Rsync is by far the easiest way

1 Like

Passphrase. That’s all I need to say.

2 Likes

Lol no. You rsync from the external to the traefik. Store it in tmp. Move it from tmp based on a systemd trigger running as root to the traefik servers letsencrypt location or ssl location.

Alternatively you could create a high privilege user for those folders with the same name on both machines and just rsync that way

There are many ways to skin the cat

1 Like

No, I mean there’s a passphrase on the cert which causes problems.

2 Likes

Why in the world do you have a passphrase on the cert? This isn’t standard practice

Because I assumed it would improve security. :stuck_out_tongue:

1 Like

Not to any measurable degree on PKI. I’d suggest removing it but if it’s the way you need to operate it then idk if I have the solution for moving the cert and syncing it

Give me some reference on how it’s not measurable lol

1 Like

Okay well do you understand how PKI works. Im going to brush the surface here. The certificate generated by Certbot and used by your webserver is not password protected - it only contains non-secret material and can be shared widely :slight_smile: The private key generated by Certbot that goes alongside the certificate is also not password protected and for good reason it only need to be protected by filesystem permissions and shouldn’t be shared. This is up to you. There is no good reason to share it. There is no sound reason to encrypt it within the permissions it already sits in. If your system is compromised to this level your data is already done for with or without this cert being protected.

You might wonder why the private key doesn’t have a password on it - the answer is that your webserver and any other software that needs to use the private key would have to ask you for the password first when it was started up, or restarted. I’m not aware of any webserver software that supports that type of handshake easily or without you again sharing that plain text password with people accessing the services or the software itself which would make the point of the password quite moot. (the software could be compromised and boom the cert is) It would also make automation very difficult and that’s a primary goal for Let’s Encrypt and Certbot so placing a password on the cert needlessly complicates things for no realistic benefit in day to day operations. It also shoots yourself in the foot like you are figuring out

The whole point of PKI is that public keys authenticate against the private key which doesnt need to be shared or moved or protected since only the public key is used in exchange.


Basics of PKI:

Public Key Cryptography designates the class of cryptographic algorithms that includes asymmetric encryption as well as its little brother (key exchange) and digital signatures. There are two operations that correspond to each other; namely being… encrypt → decrypt, or sign → verify … with the characteristic that one of the operations can be done by everybody while the other is mathematically restricted to the owner of a specific secret.

The public operation uses a public parameter called a public key and the corresponding private operation uses a corresponding private parameter called a private key… The public and private key come from a common underlying mathematical object, and are called together a public/private key pair. Hence what you created or rather what certbot created… now the true magic of asymmetric cryptography is that while the public and private parts of a key pair correspond to each other, the public part can be made, indeed, public, and this does not reveal the private part nor compromise security. A private key can be computed from a public key only through a computation that is way too expensive with existing tech and particularly if you are using chacha20 with AES in your key exchange and have a diffie helman. Your quantum resistant to a degree as well

The problem, now, is one of key distribution and the Public Key Infrastructures is the solution for that problem. Essentially:

The goal of a PKI is to provide to users some verifiable guarantee as to the ownership of public keys.
The means of a PKI are digital signatures.

In that sense, a PKI is a support system for usage of public key cryptography, and it itself uses public key cryptography. Thats how we define PKI and that is what you are using when you make a certificate.

The core concept of a PKI is that of a certificate. A certificate contains an identity (say, a server name) and a public key, which is purported to belong to the designated server. The whole is signed by a Certification Authority. The CA is supposed to “VERIFY” (like a notary for example) in some way that the public key is really owned by the named entity, and then issues the certificate… and the CA also has its own public/private key pair. When you use it the browsers can go to say letsencrypt and verify its valid by comparing against their private key as well. This also means if letsencrypt chooses to they could technically invalidate your exchange :wink: you are beholden to them. and guess what… A signature does not make something trustworthy. When a message X is signed and the signature is successfully verified with public key XKp, then cryptography tells you that the message X is exactly as it was when the owner of the corresponding private key XKpriv computed that signature. This does not automatically tell you that the contents of X are true… Now moving on to what the certificate does … it moves the key distribution problem: initially your problem was that of knowing the server’s public key right? I said this earlier :wink: … and now it has become one of knowing the Authorities public key, with the additional issue that you also have to trust that CA which in your case is letsencrypt. You are putting a ton of faith in them :wink: .

The main reason why SSL servers have certificates (NON ENCRYPTED ONES) is because clients cannot possibly know beforehand the public keys of all servers: there are too many of them, and new ones are created with every passing minute particularly as you go through your change and double that if you have DH parameters… Now usually servers know all their users which is why most can use a simpler password-based authentication mechanism. SSL client certificates are thus rather rare in the other direction from server authing to client instead of client auth with server… because the main advantage of certificates is not a feature that most servers want nor is it any easier to implement. There is a protocol thats an example of bidirectional certs. Wireguard. Its not an easy setup. They meet half way instead of having a server client infrastructure :wink: . Thus you normally only see different Public/Private Key pairs on each side in peer to peer systems.

Is PKI more clear? Do you understand how the password provides little to no if any additional protection?

PKI is not unclear (nor was it before reading this). Sorry, not helpful.

1 Like

Then I dont really know how to help you with your cert issue (being the encryption of it with a password is only needed on transport of the private key through external means). It will be interesting to see how you deal with it.

If it was “clear” then you should very well understand the password does nothing. False sense of security. Every guide and their grandmothers kitchen sink tells you not to encrypt the PrivKeys with a password for a good reason.

Here’s the problem: literally everything is a false sense of security.

1 Like

And thus youve learned the major lesson in going to far with security. Its diminishing returns Barlow. Thats why you dont go to these levels. You go to what is practical. “Standard Practice” and “Security First” both apply this mindset and neither encrypts the private key. It is so rare to see this. It doesnt even occur with wireguard (the first quantum resistant tunnel).

You take the engineer approach “good enough” because all else is taking the 10 mm and pointing it at your gut. Ive learned this the hard way. Made the same mistakes. It becomes damn near impossible to manage because time is a valuable resource and the time it would take to do everything manually instead of automated with the stuff you are running, well lets just say you might as well forgo any hobbies, walks, hikes or a personal life. Im not exaggerating. Highly recommend you rerun the cert without it. Im surprised you havent encountered issues with client systems. Usually it would mandate a password for the exchange with the direction you took.

Its not that your idea was “bad” its that consider what you are protecting against. A physical attack. Since the password would not protect against a virtual attack where they compromised the memory of your system where the unencrypted key would be stored during handshakes :wink: ( a point I previously forgot to make). That goes for all x86_64 systems. Particularly the older they get.

“Diminishing returns” - is usually not an argument I pay much attention to. Think about what I use to hunt and from what distance and you’ll begin to understand my perspective.

Regardless, we’ll be going ahead and replacing the key. While I do wish to protect against all sorts of attacks - absolutely - our alternative to rsync already failed it’s test!

1 Like

what was your alternative? Im curious

You can store your private key and at startup generate a related key to keep the main private key in escrow. Perfect forward secrecy comes to mind

Escrow sounds a phenomenal alternative. I’ll look into that - thanks, Wendell.