Some apps refuse to start after migrating Apps pool

OK, so as I mentioned in my blog post I am trying to move my TrueNAS Scale apps from my HDD data pool to an SSD.

The new apps-pool I made is a secondary partition on the boot-SSD (see link in the blog post). I know TrueNAS doesn’t support this, but the partitions don’t seem to be the issue. The secondary pool imports just fine, and TrueNAS doesn’t seem to actually care that it’s on the same physical device. The Apps screen lets me choose the apps-pool and move the data, and it seems to work for the most part, but for whatever reason 2 of my apps refuse to deploy and it doesn’t make sense to me.

What I did:

  1. I updated all my apps
  2. I stopped all apps
  3. Imported the (empty) apps-pool
  4. Rebooted
  5. Started the migration

After the replication for the migration was done it was already showing 2 failures on two of the apps, with the same error I was getting last time I tried this, but it doesn’t make sense to me.
Unfortunately I rebooted again and I didn’t realise TrueNAS empties its task manager history when rebooting… but suffice to say everything succeeded except those 2 failures. So I think the move itself was fine.

When trying to start these, I get this:

[EFAULT] Failed to update App: WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
Error: UPGRADE FAILED: failed to create resource: Deployment.apps "deluge" is invalid: spec.template.spec.containers[0].volumeMounts[3].mountPath: Invalid value: "/downloads": must be unique

I checked the config, and the /downloads path is only in there once:

[EFAULT] Failed to update App: WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
Error: UPGRADE FAILED: failed to create resource: Service "podgrab-ix-chart" is invalid: spec.ports[0].nodePort: Invalid value: 10093: provided port is already allocated

That port is not already in use:

$ netstat | grep 10093
$

And the config only has it once too:
image

I checked the Kubernetes config file and it is indeed group readable:

% ls -l /etc/rancher/k3s
total 14
-rw-r--r-- 1 root root     761 Jan 21 06:25 config.yaml
-rw-r----- 1 root netdata 2961 Jan 21 06:25 k3s.yaml
-rw-r--r-- 1 root root     144 Jan 21 06:25 kubelet_config.yaml

Note sure why… but I also never fucked with that. And besides, if this was actually an issue, why does it work for 9 other apps? That just doesn’t make sense to me.

Should note that all these apps have been running perfectly fine for a year. But both times after changing the pool they stop working.

There is also a third app that gets stuck in a Deploying state. I’m not even getting an error from it (in the task manager history the redeploy is “successful”) and when I click the Logs button it’s telling me this:

No Pools Found
At least one pool must be available to use apps

But this is also my last remaining TrueCharts app so it might be an issue with that… not sure.

Anyway, none of this makes any sense to me, so anyone got an idea?

So since my apps weren’t all working I already restored the config backup a couple days ago and they were working great. Wanted to do some further testing so I spun up the ISO again and recreated a new empty apps-pool on top of the broken one (it should overwrite the old, correct?).
After reboot however two apps didn’t come up… again. Another reboot and now one of them was working, but not the other… Figured something broke and restored the config (again), and they would still not come up… IDK what’s happening. The rest also came up immediately after reboot even though before I made the backup I stopped all apps and if I’m not entirely tripping on every restore they also kept being stopped until manually started… so IDK if the start/stop state is not actually in the config backup and I just happened to stop them everytime before the restore or what.

But anyway… I knew that in my past tries a reinstall and config import always worked, so I went ahead and did that again. I wanted to see first whether non-migrated apps could actually be started from the apps-pool.

So… further testing:

  1. Reinstalled TrueNAS (again)
  2. Didn’t import the config this time
  3. Only imported the (empty) apps-pool
  4. Installed Deluge, and it works (well, starts, I didn’t actually try downloading)

So, partition isn’t the issue.
Also interesting, the k3s config is group readable by default:

$ ls -l /etc/rancher/k3s
total 14
-rw-r--r-- 1 root root     761 Jan 26 08:50 config.yaml
-rw-r----- 1 root netdata 2957 Jan 26 08:51 k3s.yaml
-rw-r--r-- 1 root root     144 Jan 26 08:50 kubelet_config.yaml

So, next try:

  1. Reinstalled TrueNAS (again again)
  2. Imported config backup
  3. Tested if apps were starting
    • One of the apps didn’t come up again but after properly stopping it and rebooting, it went back to working…
  4. Imported apps-pool, rebooted
  5. Apps screen → Choose Apps pool → apps-pool → without migration
    • that removes all apps, which is somewhat expected because there can only be 1 apps pool at a time, the dataset stays though

My goal was to set up a fresh install of the app to verify that this pool could indeed be used to start apps in the first place, and therefore this was not an issue with the migration.

At this point when grepping through the chart’s config files (to copy the paths), I noticed something that I have no idea how I hadn’t before. When looking at the screenshot provided above, there is a “Deluge Downloads Storage” configuration option, which defaults to an ixVolume (as everything does), and which mounts to /downloads. But as you can see I also added a Host Path volume mounted to /downloads.

This explains the Invalid value: "/downloads": must be unique error, but I am baffled how this has ever even worked in the first place. I dug through the charts’ commit history and this option was always there, and always defaulted to ixVolume:

How has this worked then? No clue, but it is what it is.

Once I do the migration again I can just remove the Host Path and set it on that setting instead and it should work fine.


Anyway, this still leaves the podgrab container’s Invalid value: 10093: provided port is already allocated which I can’t find an explanation for.


As for the third app (the TrueCharts one), I just learned that’s now in TrueNAS’ apps catalogue too, so I’m not too worried about that anymore.

OK, solved this, and it a weird one…

I did the migration again and went to change the /downloads volume path. Except… that “Deluge Downloads Storage” setting can’t be changed once it’s set… so for a start I removed the Host Path mount I had set, then saved, started the container for a test, then stopped it, added the Host Path again and that works…

It seems that whatever TrueNAS or k3s does there, on first creating the container it checks for conflicting paths, but if the container had already been deployed once it allows overriding one of the default paths with a custom volume maybe? IDK. Seems weird but it works lol

So just to solve the issue with the podgrab container now.
Actually that one just… works now. What? I didn’t even change anything.

Also regarding this one… I found the cause. Since I most likely wasn’t going to be able to migrate this app to Electric Eel anyway I wanted for one last time to copy out the contents of its settings using Heavybullet’s heavy_script and when trying to mount the PVC, it wasn’t able to find it. So I guess when migrating the apps pool it doesn’t take PVCs with it because while that’s a Kubernetes feature, it’s not used or supported by TrueNAS. So I changed the app pool back (without migration obviously), and then I could mount and copy the data, and then changed it back to my new SSD apps-pool again.

1 Like

For posterity:

After finishing the Apps pool migration I finally got around to upgrading to Electric Eel.
After the upgrade I ran into the same issue again, and Deluge would not start due to duplicate paths. I somewhat expected that, however the workaround mentioned above does not work on Electric Eel anymore.

The Downloads Storage path remains immutable, and it seems the only way to “change” it is by flatout reinstalling the application:

Luckily there seems to be a new(?) option to not delete the associated ixVolumes, so if you wanted to reuse your config volume, you could.
However it seems that iXsystems actually doesn’t recommend using the ixVolumes despite them being the default (go figure…), and I opted to migrate my Deluge config directory to a new apps-config dataset within the apps-pool instead.