TrueNAS Scale: Ultimate Home Setup incl. Tailscale

Hello, I am setting up a new system and was having issues similar to others in the thread.

I think there was some confusion on one of the steps after you add the nfsdckr user:

mount 192.168.1.1:/mnt/tank/NFSdocker /nfs

should become

mount 192.168.1.1:/mnt/tank/NFSdocker/nfsdckr /nfs

Needed to mount the home folder of the truenas user versus trying to mount just the NFS share. Small oversight on my part but seems to be functional now (touch hello_world works)

Right now looks like I’m having the same issues as others by deploying nextcloud and getting permissions errors. Not really interested in using no_root_squash

A dumb question but how do you folks paste into VNC/QEMU? I’ve tried SHIFT+CTRL+V and pasting into the small box the pops up to the left but haven’t found anything that works. From googling around it seems to be a VNC related issue but not sure how to get around that.

@wendell or anyone else

I’ve gotten to the point of trying to access portainer but am unable to connect to it. This is the output from the previous step:

If I keep running docker ps it seems to show that portainer is always restarting, which seems like an issue that has occurred in the past.

Looking at the logs I’m getting this:

Which would make me think permissions are not setup correctly. I have them set as below:


And I was able to run touch hello_world in /nfs successfully.

Any shots of permission settings or advice would be appreciated.

The ports column is a clue.

I’ve noticed recent versions of truenas keep clobbering /etc/daemon file that controls docker networking. That’s the issue. Afk rn but maybe some kind soul can give you specifics. One you fix that file in /etc restart docker and the container and the ports col won’t be empty.

Gotcha, I’ll look around and see if I can come up with some info on that.

Also, I found you can dig back to the top of the logs with docker logs --until timestamp portainer and found some extra bits. Not sure if it adds anything but for completeness:

Edit:

Messing around I noticed it did apparently got online briefly:

I’m not sure if that adds any insight but thought I’d mention it.

Any fixes or clues coming down the pipe on the Bridge interface nonsense?
It’s working fine for me with my primary NIC but any attempts of bringing in my second 10G NIC has it functional for the 60s “trial” period then all networking goes down.

I really appreciate the simplicity and solid base Truenas provides but it has me wanting to flip back to Proxmox :cry: (especially given I’ve been having nothing but pain (generally slow / DB locks) with sqlite backed apps on the shared NFS root)

2 Likes

I’ve installed the whole package on TrueNAS Core in the VM ( one good thing with Core is that you don’t have to do this bridge nonsense ) and everything works expect chown on the client side in the VM. I can’t edit /etc/exports to add no_root_squash becuase its TrueNAS.

Some containers require to chown like Organizr or Openproject .

Any hints how to make chown work ? I tried maproot / mapall with root:wheel nothing works…

So i would like to make sure im not screwing myself down the line.
First @wendell thanks for the amazing post, after 3 days of losing my mind (but learned A LOT) i think i got portainer and nexcloud reading, writing and making files with no issues so far.

Now the issue is adding SMB Shares to nextcloud, i was able to get it working when i used iocharset=utf8,file_mode=0777,dir_mode=0777 in my /etc/fstab file to be able to mount and also be able to actually use it. Without it i can only see files and thats it.

So TLDR: is having iocharset=utf8,file_mode=0777,dir_mode=0777 safe when mounting smb shares?

Anyone else figure out the solution to nextcloud permissions with NFS storage?

It’s nothing but an uphill battle with nexcloud, constantly seeing these errors in my portainer logs:

Initializing nextcloud 24.0.5.1 ...

touch: cannot touch '/var/www/html/nextcloud-init-sync.lock': Permission denied

I thought it was a host VM issue so redid my fedora-server VM with debian and encountering the same issue. I originally was not adjusting permissions or ACL but I had to in order for the database portion to initialize. I’m very confused, because it looks like the permissions within the container’s data do not maintain the nfsdckr ownership and/or are not compatible with NFSv4? Bit too new at this to fully understand what’s going on there but will continue to troubleshoot.

2 Likes

Is the kernel source code available? I’d really like to do the ACS patch for me to passthrough some hardware.

You can add pcie acs override downstream to the boot line to override the grouping. And that’s how you make it permanent. Search for acs override truenas

Should work without having to do a manual patch. Just a kernel param

1 Like

I’ve been stuck on the same issue for a week or two now. I’m finding it is because portainer is creating as root and not as the user. Not sure how to get portainer to behave differently.
As a workaround back on truenas you can change the maproot user and group in the nfs share to ‘root’ and all the permissions behave. Not sure if this is best practice though but it does provide a work around. Would much rather figure out how portainer can behave as a user and create as ‘nfsdckr’ in this use case.

1 Like

I ended up getting it to work!

Though, like you have mentioned, there’s a lot more to learn, especially wrt to why we do it one way vs. another and the actual consequences of not doing it correctly. Reading online about not using “no_root_squash” kind of scared me into not doing it that way but I doubt it’s actually worse than what I have here:

image

image

I found that by setting the ACL (recursively) for my vm-persistent-storage to these settings actually gets everything to work.

(I just realized @everyone is in there twice… whoops)

I believe nfsdckr had to be explicitly added in order to make changes. I had to add www-data to the ACL in order for nextcloud to serve the webpage (after the first initialization completed).

I agree with you though, I’d rather docker just utilize an account - neither portainer or tailscales had any issues initializing.

Of course, now my nfs folder has these permissions (from the debian VM):
image

nfs folder from truenas:
image

Here’s the nextcloud folder from the debian VM:
image

Here’s the nextcloud folder from truenas:
image

If I could figure out how to allocate “999” as a user on the ACL and/or change the user that the mariadb runs as then I could probably remove my @everyone full permissions and everything would work.

I guess I’m looking for advice on how to clean this up b/c though it’s functional it certainly doesn’t seem sane. One path I previously explored was adding users to my nfsdckr group on truenas but I figured that would only work if the exact same users in the VM exist on truenas.

Last note for now: it also seems like NFS security relies on a device having the correct IP and knowing the path - is there not any authentication like in SMB?

1 Like

Any reason you chose NFS over just mounting a new zvol? Seems like a lot of work for NFS. Seems alot easier to add a zvol and mount it.

I added a new zvol

Then I added a new device on my VM

[email protected]:~$ lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
vda    254:0    0  100G  0 disk
├─vda1 254:1    0  512M  0 part /boot/efi
├─vda2 254:2    0 98.5G  0 part /
└─vda3 254:3    0  976M  0 part [SWAP]
vdb    254:16   0   50G  0 disk

Then I formatted it using sudo fdisk /dev/vdb

I chose n to add a new linux partition.
Then w to write the changes to the disk

Then to format as ext4

sudo mkfs -t ext4 /dev/vdb1

make the mount point

mkdir -p /docker/portainer

edit fstab - added the following to the bottom

/dev/vdb1       /docker/portainer               ext4    rw,relatime    0    2

then ran sudo mount -a

[email protected]:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            5.9G     0  5.9G   0% /dev
tmpfs           1.2G  492K  1.2G   1% /run
/dev/vda2        97G  2.1G   90G   3% /
tmpfs           5.9G     0  5.9G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/vda1       511M  5.1M  506M   1% /boot/efi
tmpfs           1.2G     0  1.2G   0% /run/user/1000
/dev/vdb1        49G   24K   47G   1% /docker/portainer

and now I can add portainer like normal using this path.

edit for permissions

add yourself to the docker group

sudo usermod -a -G docker rwendt

change ownership of the folder

sudo chown -R root:docker /docker/

give group rw on this location

sudo chmod -R g+rw /docker

test permissions

[email protected]:/docker/portainer$ touch test
[email protected]:/docker/portainer$ ls -al
total 24
drwxrwxr-x 3 root   docker  4096 Sep 27 22:55 .
drwxrwxr-x 3 root   docker  4096 Sep 27 22:29 ..
drwxrw---- 2 root   docker 16384 Sep 27 22:32 lost+found
-rw-r--r-- 1 rwendt rwendt     0 Sep 27 22:55 test

$$$profit$$$

I wanted the filesystem to be accessible on the host as well as other possible VMs. Different VM, maybe, for music, vs media, even if it is all in one volume or spread across one filesystem. Or nested zvols.

this is nice though.

Yeah that makes sense. Decoupling the storage from servers / services.

Right now I’m running on an old fx-8350 with not much to work with. Anyhow thanks for writing this post up. Its been really fun to mess around with Truenas scale.

Since portainer defaults to /var/lib/docker/volumes when creating volumes, I tried symlinking the volumes folder in /var/lib/docker to a directory in /nfs.
[email protected]:/var/lib/docker/# ln -s /nfs/volumes volumes
In this way I can use portainer to create volumes without setting up NFS volumes. It seems to work, just wondering if this is a good idea.
Any thoughts?

I was having a tough time getting volumes to go where I wanted regardless of where I was running with, till I found out in the composes that you can specify WHERE your volume is.

Example of my PhantomBot compose/stack

version: '3.2'
services:
  phantombot:
    container_name: phantombot
    image: gmt2001/phantombot-stable:latest
    ports:
      - target: 25000
        published: 25000
        protocol: tcp
    restart: unless-stopped
    volumes:
      - PhantomBot_data:/opt/PhantomBot_data
    environment:
      PHANTOMBOT_USER: "Very funny secret"
      PHANTOMBOT_CHANNEL: "super secret"
      PHANTOMBOT_PANELUSER: "a user"
      PHANTOMBOT_PANELPASSWORD: "SECRET"
      PHANTOMBOT_RESTARTCMD: "/opt/PhantomBot/restartbot-docker.sh"
    networks:
      - privatenetwork	  

volumes:
  PhantomBot_data:
     driver: local
     driver_opts:
       type: none
       o: bind
       device: /mnt/pond/appdata/phantombot

networks:
  privatenetwork:
    external: true

No need to bother with symlinks etc. which can often be a pain with backups and moving stuff around. This way you know EXACTLY where your data is and should me. I tend to want everything from a particular app in the same place… but that’s just me.

Not sure if this is the accepted practice, but find it easier than symlinks.

EDIT. PS. I am running portainer DIRECT on TrueNAS box, not through a VM, but should work fine with VM and making sure your volumes are on your NFS share.

I had a performance issue with tailscale when traveling - to move large files or stream things from home I had to enable NAT-PNP (booooo!)

Otherwise it would fail to move large files from phone to NAS (Synology) or stream from home to hotel as it was relegated to routing all traffic through one of their… DERP relays.

Otherwise, loving tailscale and not having to fully VPN in at ODS times.

Question for y’all. So, I am in the process of setting this up. I unfortunately don’t have the hardware storage requirements for TruNAS, so I just have a debian machine running Portainer, so everything on Portainer that you did I followed along with (NextCloud and TailScale). I set up the advertise-routes and in tailscale I have it set to route subnet. When I try to connect to NextCloud using the 100.#.#.#:8080 IP or the local 192.#.#.#:8080 IP I still get the “untrusted domain” message. I did a sanity check restart on the machine running Portainer but did affect the issue.

NOTE: I’m pretty new to this and since I am not using TruNas, I should have probably changed the volumes for the stack compose files but didn’t. I don’t know if that is a factor because the machine is on TailScale and I can access NextCloud locally.

Any ideas?