OK, so as I mentioned in my blog post I am trying to move my TrueNAS Scale apps from my HDD data pool to an SSD.
The new apps-pool
I made is a secondary partition on the boot-SSD (see link in the blog post). I know TrueNAS doesn’t support this, but the partitions don’t seem to be the issue. The secondary pool imports just fine, and TrueNAS doesn’t seem to actually care that it’s on the same physical device. The Apps screen lets me choose the apps-pool
and move the data, and it seems to work for the most part, but for whatever reason 2 of my apps refuse to deploy and it doesn’t make sense to me.
What I did:
- I updated all my apps
- I stopped all apps
- Imported the (empty)
apps-pool
- Rebooted
- Started the migration
After the replication for the migration was done it was already showing 2 failures on two of the apps, with the same error I was getting last time I tried this, but it doesn’t make sense to me.
Unfortunately I rebooted again and I didn’t realise TrueNAS empties its task manager history when rebooting… but suffice to say everything succeeded except those 2 failures. So I think the move itself was fine.
When trying to start these, I get this:
[EFAULT] Failed to update App: WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
Error: UPGRADE FAILED: failed to create resource: Deployment.apps "deluge" is invalid: spec.template.spec.containers[0].volumeMounts[3].mountPath: Invalid value: "/downloads": must be unique
I checked the config, and the /downloads
path is only in there once:
[EFAULT] Failed to update App: WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /etc/rancher/k3s/k3s.yaml
Error: UPGRADE FAILED: failed to create resource: Service "podgrab-ix-chart" is invalid: spec.ports[0].nodePort: Invalid value: 10093: provided port is already allocated
That port is not already in use:
$ netstat | grep 10093
$
And the config only has it once too:
I checked the Kubernetes config file and it is indeed group readable:
% ls -l /etc/rancher/k3s
total 14
-rw-r--r-- 1 root root 761 Jan 21 06:25 config.yaml
-rw-r----- 1 root netdata 2961 Jan 21 06:25 k3s.yaml
-rw-r--r-- 1 root root 144 Jan 21 06:25 kubelet_config.yaml
Note sure why… but I also never fucked with that. And besides, if this was actually an issue, why does it work for 9 other apps? That just doesn’t make sense to me.
Should note that all these apps have been running perfectly fine for a year. But both times after changing the pool they stop working.
There is also a third app that gets stuck in a Deploying
state. I’m not even getting an error from it (in the task manager history the redeploy is “successful”) and when I click the Logs button it’s telling me this:
No Pools Found
At least one pool must be available to use apps
But this is also my last remaining TrueCharts app so it might be an issue with that… not sure.
Anyway, none of this makes any sense to me, so anyone got an idea?