Long story short. I started with Proxmox on my Server running Containers and VMS for my applications (arr and game servers). This also included a windows VM (blech) for those annoying window game servers. No acceleration etc. Also using MDADM array for my drives. Self hack in Proxmox.
When TrueNAS scale came out, it sounded like a great AIO environment with a much better file system (ZFS and @wendell glowing endorsement). It had VM’s and docker/Kubernetes. Kubernetes was a pain in the arse and hating every second of it in truenas, so found work arounds to use docker natively (until they took docker out) then to a systemd container which was even better and was working with NVIDIA acceleration. Being the home-user hostile dicks that the trueNAS team is, they removed/are removing the systemd container support. Their VM support I felt was also sub par. Same hardware, linux VM’s worked ok if there wasn’t a bug in the system which prevented (happened on more than one occasion) but the Windows VM was abysmal. I couldn’t get any windows game servers to run at the speeds I was under proxmox. All this on same hardware.
Now Unraid has ZFS support, and has docker, and VM’s support as well. I have downloaded a trial and been messing with it a bit, but the drive setup for ZFS seems a bit weird. ie, you need to have one drive in an array and then you can make a pool of ZFS drives. Kinda need to wait till more experienced people can do some youtube videos on converting from TrueNAS to Unraid.
Currently I am running Proxmox again with TrueNAS as a VM this time (SMB share is eating up CPU for some reason which is annoying), but all this got me thinking…
Am I mad for thinking of doing this?
Since proxmox is already running ZFS as its primary system, why not just dump all the files on there and use native SMB and NFS shares directly from there. Currently I am running docker within a container, but having issues with nvidia passthrough for this, so why not run docker natively as well since dockers are already containerized? This would probably solve a few of the issues with nvidia in a container and docker. Also, can finally get TrueNAS off my hardware forevever.
Thoughts, advice, slap in the face with a large salmon?
Cheers
Talung
TrueNAS Scale is meant to be an Appliance, the Boot Volumes are completely interchangeable, the configuration is stored in the System dataset that lives under one of your ZFS Pools.
Yes, one can interact with the underlying host, but it makes sense things will break during updates - this is meant as a set-and-forget Software Solution for enterprise customers and not one for fiddling around in a homelab.
Their Bug Tracker and Community reflect this as well…often hostile if you’re trying to color outside of supported lines.
On the contrary, Proxmox is just a collection of useful tools and a fancy UI that you can install on top of any Debian System, so feel free to mess about.
It’s completely up to you whether you prefer the very easy nature of TrueNAS Scale with the downside of the limitations it presents or would rather interact with the system directly even though that might be more difficult.
I’d personally always choose the latter in a non-critical home environment.
No reason at all, except convenience and a slightly higher learning curve.
Docker in LXC can be quite the hassle…either run it in a VM or directly on the Host. Up to you.
Actually got docker running extremely well in an LXC… except for 2 minor issues. NVidia passthrough is going through to LXC, all active and visible there, but causes some hiccups with docker itself. Other one is docker containers like Calibre and Blender that seem to run a “desktop” environment inside the docker… Can’t get those to work. I would imagine the same thing with pterodactyl wing which I was meaning to do in a VM to get around that.
I can do everything in scripts etc, but gui to handle things helps… although only for initial thing tbh, once its running you really don’t need it anymore
Since my data is safe on a ZFS volume, and external backup, now is definitely the time to sort things out.
Your thinking is right on the money. By going down the TrueNas route you are limiting your latency and speeds to the limitations of the network adaptor and protocol. If you emulate a network adaptor you are going to get 1Gbp/s speeds unless you have a 10, 25 or greater adaptor and the drivers for it. If you Para virtualize it say run virtio network adaptor you can get speeds of around half your RAM speed.
Yes going through ProxMox is going to be way faster especially if you’ve enabled caching. With ProxMox you can end up with two different types of caching that can give you RAM speeds.
The first is your ZFS ARC which will read cache in RAM if you set it and allow it to pass to through to guests. I would recommend reading the settings as there is a safe way and unsafe way.
The second is using virtio drivers. These drivers need to be installed on both the host and the guest and then set to active. They effectively map the host to the guest giving near passthrough performance.
Your NVidia passthrough desktop issue might be for several reasons, the passthrough might require both a dgpu and igpu which I’ve seen to be the snag in a lot of guides or it could be because the LXC image actually doesn’t have a desktop environment installed.
For example since I like GUIs after I’ve installed ProxMox I install xfce4/lightdm to give me a desktop. This way I can also access ProxMox from the same machine, research cmds and copy and past if I need to.
Since this post, finished everything including backup strategy yesterday, I have fully converted to running everything on Proxmox… not a TrueNAS in site!!! Yeah, it took me a little while to get snapshots and replications working as well as the zfs disk layout using CLI, but I got it done.
Docker is running natively and has access to all the NVidia runtimes for encoding etc. They also have direct access to the data, so no need of annoying shares and fighting permissions etc. (I realize this might not be ideal for a corp, but home user with just me on the systems, its fine!)
I do have SMB shares running for my windows boxes to dump stuff on using robocopy. Using Sanoid and syncoid, I am getting all my snapshotting done automagically and replication of the things I need via crontab running syncoid processes.
And just when I finished, Proxmox 8 released! Said screw it, and just did a standard proxmox upgrade process… and not a SINGLE issue!!! TrueNAS updates used to leave me hanging as to what it was going to break… there was never an upgrade that didn’t break something or remove something I was using.
I am over the moon with my current setup! I haven’t even needed to use any containers yet and only VM I fired up was to test my Nginx Proxy manager stuff.
