Better than TrueNAS for containerized apps

Operating Systems & Open Source Open Source & Web-Based
, , ,
1

Hi! I’ve been using TrueNAS for years and love how great of a system it is for easily running apps.

Issues

I was thinking of building a separate app server (I already have), and wanna look at alternatives to TrueNAS for running apps:

  1. TrueNAS’s app selection is limited.
  2. The TrueCharts guys who solved this app problem moved to TalosOS which sounds like a pain in the butt to setup and maintain.
  3. I’ve never heard anyone say that TrueNAS is the right solution to use when running containerized apps.
  4. TrueNAS requires a minimum of 4 drives if you wanna run apps, two for the OS and two for the apps. That sucks. I’d rather have 2 partitions on each drive and only worry about 2 drives total.

OS Options

I’d like to find a web-accessible OS to use. One that’s like an appliance where I can export a config file similar to TrueNAS or Home Assistant. I like these purpose-built OSes because they’re less maintenance for me.

What are my options? TrueNAS, TalosOS, Proxmox…?

The Apps

Plex

Right now, the main app I want is Plex. Honestly, it doesn’t need to be on this beefy server either (Epyc 16-core, 256GB RAM, etc).

Pi-Hole

Pi-Hole would be nice, but I can move that to a Raspberry Pi. Probably better that way to prevent whole-home network downtime.

Cameras

In the future, I want to do cameras myself rather than relying on Nest or even UniFi.

I keep going back and forth on a good solution, and having a dedicated app server would make it easy to store camera footage. This server has a GPU and can send compressed video files to my main NAS.

For AI processing, I’d need Coral something, and there are free USB ports and PCIe slots. One issue is that Frigate and others aren’t available on TrueNAS. That’s one of the problems I’ve run into simply getting started.

2

Proxmox is not purpose built. It’s just a hypervisor on top of Debian.

You might want to try Linux server like Ubuntu and setup the Docker containers yourself. You can use portainer to manage the containers and once you know how to use Docker and set it up once you can easily move your config folder + Docker compose to any other server.

2 Likes
3

So just run the Docker container itself?
The available Apps are a convenience, nothing more. You can run any Docker container you want and with the Electric Eel release it got even easier then it was before.

TrueCharts was questionable at best anyway.

Just because it’s not the first thing people think of doesn’t mean it’s not viable.

You… don’t?

You can have 2 pools with 1 drive each and it works just fine. And the boot pool can be a USB stick.

Technically it’s even possible to do everything on one single drive, although that’s not supported.

2 Likes
4

I’ve been asking myself a similar question to the OP for quite some time.
Wouldn’t Proxmox make more sense to containerize apps? You can have an OS specialized in running VMs or LXCs and within those as many containers as you fit, to my mind that makes more sense and gives you more flexibility than having one OS (which has one purpose) and putting containers and potentially VMs within it.

1 Like
5

So it sounds like TrueNAS is the only way to go about this? There’s no other appliance that can run Docker containers (and optionally VMs)?

Docker containers in TrueNAS, I haven’t messed around with. I converted one new app server to Electric Eel that I was testing with, but I haven’t upgraded my other for fear that it would break my existing TrueCharts containers. I will move off TrueCharts when I figure out how to port my data to the new format.

Why I dislike using TrueNAS

Also, I installed two apps in the app server on TrueNAS, and as usual, they won’t update to the latest version for who knows what reason even though I am not using them. They’re there for testing only:

image
image1335×204 23.6 KB

This has happened in the past with TrueNAS containers; although, it didn’t happen with TrueCharts containers (until they stopped supporting them). Seems like, for whatever reason, TrueNAS containers will get in an “unupgradable” state, and this will always happen no matter what you do.

1 Like
6

Yeahhh… good luck with that is all I can say.

Getting to the data was already not easy when they still supported TrueNAS but the TrueCharts devs just p**sed off and let people sitting in the dust with no documentation on how to continue.
In their announcement post they were spouting the standard nonsense of “we’re working on blablablablabla” and nothing ever came of it as far as I can tell.

The way they used to give you access to the data was via their “HeavyScript” (I think it was called) which was run in the TrueNAS shell or over SSH, but I don’t know if that’s still available anywhere.

I’m glad I never got invested into TrueCharts apps. I either use TrueNAS’ default catalogue or set up the Docker containers manually. I only have 2 TrueCharts apps running and luckily they aren’t super important.

I assume you hit the Update button? The Update is not automated, you need to actively do the update.
That being said, I do get a bug with this occasionally, but I’m fairly sure that is only visual, as in I update, the container shuts down and comes back up and then it’ll still say update available, which just goes away at some point. No idea why and it only happens with specific containers.

1 Like
7

I’ll have to get my data off with HeavyScript then (which isn’t supported) or use that VSCode server (way easier). I need my Plex data. That’s #1.

How do I get data into a TrueNAS container though?

How do you create Docker containers? I messed with the server on Electric Eel and didn’t figure out a way to do it.

Yes, I click update, but then it fails. And this happens often with TrueNAS containers for whatever reason. But it never goes away for me. Stays there forever. These two have said they needed updates for weeks now no matter how many times I’ve clicked it.

8

By mounting the host-path with the data into the container, or putting the data into the containers directories in the ix-volume. But since TrueCharts’ data directories aren’t exposed you’ll have to get the data out first.

I haven’t upgraded to EE yet so I can’t say how it works there, but in Dragonfish it’s under Discover Apps > Custom App.
Documentation for EE is here:

You say it fails, but does it actually give you an error? When it fails then it tells you it failed and why it failed. If it goes through and then just continues to show the “Update available” icon, check the actual version that’s being deployed and compare it against what’s on DockerHub.

1 Like
9

IDK if this is the kind of “appliance web-based OS” you had in mind, but I used to run my homeserver off CasaOS. It’s pretty easy to migrate and it’s all docker-based nowadays (used to use their own json based config files in the beginning).

Another one I hadn’t had time to tinker with, it’s UmbrelOS, similar concept.

I ended up climbing the learning curve cliff and stuck with Proxmox + HelperScripts. At first I hated that each LXC was its own monolithic blob on my drive and had difficulty accessing my files, then I realised I don’t need to do that, I can just ssh into each LXC if needed, but keeps the backup of data pretty trivial, since it’ll do a scheduled backup of each container as a single file to an external drive.

1 Like
10

That’s the crux of the issue. Okay thanks! Gives me something to work with. If I can figure out how to mount the data directory, I can get stuff out.

I’ll tell you next time it happens. I just removed that App server and moved around things for my existing TrueNAS install, so it can remain the app server.

Interesting solutions!

Exactly along those lines is what I’m thinking!

I’m skeptical of the data privacy of a Chinese OS though.

11

The way they used to give you access to the data was via their “HeavyScript” (I think it was called), which was run in the TrueNAS shell or over SSH. I don’t know if that’s still available anywhere.

I am the developer of HS. It wasn’t part of Truecharts—it was just a script for managing applications, no matter their source.

However, since TrueNAS SCALE moved from Kubernetes to Docker, I archived the repository. The script only works on TNS versions running Kubernetes.

My Transition to Talos Linux

I’ve moved to Talos Linux.

I used to run Talos in a VM on TNS via their KVM but later migrated the “cluster” to bare metal.

Even though it’s a single node, I still prefer Kubernetes over plain Docker due to tools like:

  • FluxCD: For GitOps-style management.
  • External Secrets: For secret management.
  • SOPS: Encrypt secrets, making it safe to expose my configuration publicly.
  • Volsync: Backing up PVC data to a bucket.
  • Vast customization: The flexibility Kubernetes provides.

The overhead of Kubernetes isn’t what people exaggerate it to be.

FluxCD allows me to keep my configuration public for others to use as an example, and it lets me know exactly what is running on my server at all times. Flux continuously reconciles everything based on my configuration in my public repository on GitHub.

Learning Kubernetes

The learning curve is harsh, but there are resources to help:

Bootstrapping

  • Truecharts “Cluster Tool.”
  • onedr0p’s Cluster Template (This is what I used.)

Setting Up New Apps

  • You can search on GitHub, but KubeSearch is the best way to find app setup examples.

My Personal Take

If you’re moving away from TrueNAS for your applications, there aren’t many (if any) downgrades out there. TrueNAS is a great NAS, but as far as applications go—since launch (I’ve used it since beta)—it’s been a mess. I’ve never experienced so many breaking changes in my life.

3 Likes
13

Thanks for taking the time to response @Heavy! Do you have a guide on setting up TalosOS and running it similar to how I used Kubernetes in TrueNAS?

14

It’s been a while. Here’s what I did to get this working:

Sticking to the older TrueNAS:

  1. I used HeavyScripts to copy out my configs. It’s easy. Just run heavyscript after downloading it, and it has a UI menu in the CLI.
  2. I deleted the old containers (copying all the config information with [CTRL-A][CTRL-C] and pasting that into a text editor).
  3. Then I installed TrueNAS versions of all those containers.
  4. After configuring the TrueNAS versions of those containers, I upgraded to Eletric Eel.

It’s been working fine since! No upgrade issues either surprisingly.

But even though NVIDIA is now possible as a passthrough GPU, I haven’t got it working. It fails to transcode any videos.

My Plex app is still limited to CPU-transcoding for devices that don’t support 4K HDR :frowning:. It sucks and doesn’t work well especially because I paid $400 for a low-profile NVIDIA RTX 4060.

15

Slightly off-topic, but you might wanna look into an Arc A310:

There’s also an entirely passive card from Matrox (Luma A310), but I don’t think those are available to consumers.

1 Like
16

Debian running portainer and optionally virt-manager (or install portainer on proxmox). Someone mentioned CasaOS, which is also a good option.

2 Likes
17

Hey my bad, thought I would get notified on mentions. I need to look into my settings on this site. If you are interested in setting up Talos in the future I can help you out with it. Just shoot me a message on discord. My server doesn’t need anymore work, so I have time on my hands if you plan on making the switch.

Either way, hope things are going well.

1 Like
18

Not sure if this is the right place, but here goes.

As mentioned above I only had 2 applications left that came from TrueCharts (Podgrab and Scrutiny), and I’m currently trying to migrate them to a regular “Custom App” before upgrading to Electric Eel.
Obviously I want to keep the data, so I used the aforementioned heavy_script to mount the volumes of these two apps and copied the data out. So far so good.

For Podgrab (I know it’s unmaintained, but it works for what I need it) I was able to set up the bare docker container just using the hostpath mounts and everything is fine.
However, with Scrutiny I’m running into an issue. I took a look at TrueCharts’ configuration (since they took the repo offline all I have is what it in ix-applications):

% pwd && cat ix_values.yaml
/mnt/tank/ix-applications/releases/scrutiny/charts/11.0.8
image:
  repository: ghcr.io/analogj/scrutiny
  pullPolicy: IfNotPresent
  tag: v0.8.1-omnibus@sha256:66a65d1d7f2bf330a55e0bb073a3b2496a7b61dc6414c8c53550bc0c3f6885dd
service:
  main:
    ports:
      main:
        targetPort: 8080
        port: 10151
persistence:
  varrun:
    enabled: true
  config:
    enabled: true
    mountPath: "/opt/scrutiny/config"
  influxdb:
    enabled: true
    mountPath: "/opt/scrutiny/influxdb"
  udev:
    enabled: true
    type: hostPath
    hostPath: "/run/udev"
    mountPath: "/run/udev"
    readOnly: true
portal:
  open:
    enabled: true
securityContext:
  container:
    runAsNonRoot: false
    readOnlyRootFilesystem: false
    privileged: true
    allowPrivilegeEscalation: true
    capabilities:
      # RawIO is for HDDs
      # SYS_ADMIN is for NVMEs
      add:
        - SYS_RAWIO
        - SYS_ADMIN
    runAsUser: 0
    runAsGroup: 0
workload:
  main:
    podSpec:
      containers:
        main:
          probes:
            liveness:
              path: "/api/health"
            readiness:
              path: "/api/health"
            startup:
              path: "/api/health"
          env:
            COLLECTOR_CRON_SCHEDULE: "0 0 * * *"
            COLLECTOR_HOST_ID: "TrueNAS"

I was able to enter everything in there into the Custom App interface (except varrun, no idea what that refers to), but when trying to create the container, I run into an error:

[EFAULT] Failed to install App: WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml Error: INSTALLATION FAILED: execution error at (ix-chart/templates/workload.yaml:9:7): Invalid hostpath /run/udev. Path must be a valid path under a given pool e.g /mnt/tank/somepath is valid whereas /mnt or /mnt/tank are invalid examples.

Not sure what that warning is about because I don’t think I have control over that, but my real problem is the invalid hostpath afterwards.

Soooo question is: a) do I even need this given that the container runs in privileged mode and just created it without the hostpath mount and the shell from inside the container says: ls: cannot access '/run/udev': No such file or directory b) if I do, any idea how I would get it to accept that hostpath? :thinking:
I’m not sure if a symlink on my pool would be enough since it’ll follow the symlink and then probably end up inside the container? just tried it and the symlink doesn’t even show up in the Host Path selection:

% pwd && ls -l
/mnt/tank/configs/scrutiny
total 10
drwxrwsr-x 2 root apps 3 Jan  3 00:00 scrutiny-config
drwxrwsr-x 3 root apps 6 Nov  5  2023 scrutiny-influxdb
lrwxrwxrwx 1 root root 9 Jan  4 01:47 udev -> /run/udev

image

edit:
Alternatively, does anyone know how the TrueNAS upgrade handles these orphaned TrueCharts apps? Does it just leave them be? Does it try to convert them to a custom app/compose file? Or does it just refuse to upgrade?
Since they are using Kubernetes’ PVCs I don’t think converting them automatically would work anyway…

19

@mihawk90
The SPARKLE looks great! 1-slot, and it has multiple video transcoding stream support!

Keeping with TrueNAS

I ended up sticking to TrueNAS and only a single NAS host rather than one for data and one for apps.

If I ever wanna split it up again, I’ll do something in Ceph, but that has so many gotchas, that it doesn’t really make sense for me right now.

I ended up doing some funky stuff to get this all working, but both Storinator XL60s are now JBODs (freed up 2 PCIe slots), and the server is in a 2U SuperMicro chassis with 24 x 2.5" drive bays.

If I change out the 4060 with the SPARKLE ECO card I just bought, that gives me one more PCIe slot for any future upgrades (of which there should be none :laughing:).

One other thing I just remembered, I’ve been doing some AI stuff on my 4060 for Home Assistant with Ollama, so that 8GB of VRAM is way better than 4GB. So I guess that SPARKLE card won’t work for me. Just canceled that order :confused:.

One benefit of using TrueNAS Electric Eel is the ability to use Docker-Compose files. That got me through a lot of projects that didn’t support it. I know have 17 app containers running. I keep finding new uses for this thing :+1:.