The goal is now to move to ProxMox as a Hypervisor, run FreeNAS in a VM with all of it’s previously used hardware passed through, run a secondary VM for Steam OS/ Debian for games.
This way I can experiment with Virtualization, and keep my primary storage box using FreeNAS!
/EDIT
Hi Everyone!
I’m looking for advice/guidance on moving from FreeNAS towards ZFS on Linux.
As much as I’ve gotten to love FreeBSD (Jails are f***ing amazing), I’m getting the itch to game again and want to start playing with Steam in home streaming in the short term and use CUDA acceleration for GPU compute for some AI research/ experimentation for class.
I’m running 2 quad core Xeon’s, 144GB of ram, a 256gb NVMe for L2Arc and Slog.
I have 2 pools, each composed of a single RaidZ1. One with 33tb drives, one with 310tb drives.
Wait, this user was registered in 2013? and first post today?
Hats off for some serious Lurking!if legit, not some old credentials stolen off the internet…
Nice setup though.
FreeNas pools should import fine into Linux.
When you do, the status will Probably say you can update… don’t upgrade the pool until you know you’re gonna stick it out, as importing back to FreeNas from an upgraded pool might not work so well.
As long as you don’t have dedupe, the pools should run pretty fine, not sure if you are using zvols for the jails, and how they might apply to virtual machines, but I guess you can tell us?
Yeah! been around for a good long time! I signed up after Wendell did a post about a passively cooled its xeon board, wanting to jump into the conversation. everything I was interested in had already been answered, so stuck around to see what else was up (spend a lot of time on r/datahorder r/bsd and the LTT forums.)
Currently I’m thinking of grabbing a flash drive with either Pure Debian or OMV with the ZFS plugin, having them import the pool as read only and seeing how that works for a few days.
the 3 by 3tb pool does have dedupe (currently saving about 500GB of space because of it). does ZoL not support dedup?
the Jails are running on a dedicated NVMe. My VM’s are running off the same NVMe. (have to love those ssd iops drools)
Any recommendations/advice on how to proceed? for that matter, any ideas on how to set the pools to read only at first so that nothing get’s hit too hard while I test? I have backups of everything, but rebuilding data is such a PITA.
So your use case is to move away from FreeNAS on bare metal to linux as a hypervisor and run a load of VMs, one of which will be your NAS, one will be a Windows partition for gaming and one will be another partition for development. Have I got that right?
I’d recommend ProxMox rather than Debian as your host. That way you can have easier VM setup and still have a debian client. Hell you could even run a FreeNAS VM and leave your pools on FreeNAS. What I don’t know is if Proxmox will mess with the disks, but in theory you can pass through the controller to the VM and it will run as normal.
Of course, goes without saying, BACKUP.
Also depending on what you are using your NAS for data wise, it may be advantageous to run your steam machine on an SSD, you will have a better time.
That is a lot of hardware for a NAS… good to see it stretching its legs.
I am exploring the possibility of moving away from FreeNAS to Debian as well. I run FreeNAS on ESXI at the moment which is a bit of a pain due to having to pass through an HBA to the VM and I’m sure I’m not getting the best performance with overhead of both a hypervisor AND FreeNAS.
What really helped me to actually UNDERSTAND what ZFS is and to get over the fear of managing ZFS on CLI was this video (if you have a couple hours some day…)
I’ve been playing with ZFS in Ubuntu server ( any Debian based distro should be similar) and really starting to get comfortable managing ZFS all by myself. FreeNAS made it really easy with GUI clicking a few things but I feel like after learning about under the hood, I believe it won’t be a difficult move once I iron out everything else.
If you have the time and capability (as in spare hardware), I highly suggest messing around with a couple disks and a light weight debian distro to build a ZFS pool, dataset, zvol and learn how to make a share via NFS or Samba to get comfortable with it.
AS others have stated, right now, ZFS is a one way trip to Linux. Look open OpenZFS for some details on how that may change in the future, but for now, ZFS on Linux adds things that make it incompatable with BSD (or at least most of them). I have moved my arrays from freeNAS to a number of Linux hosts, and can say that it’s been a painless experience to move around once I realized that I could not go back to BSD. I’ve had no issues on Proxmox, Void, Arch, or even Gentoo with this array. I tend to agree with others on using Proxmox for what you are doing, as long as you have a backup, and are willing to put in a bit of work. Dedupe SHOULD transfer just fine, or at least it’s designed to. As far as going read only, there should be absolutely no issues as long as you don’t upgrade the array on Linux as that will add the features that prevent you from going back.
That’s close to what I was thinking. I’ve been told to avoid freenas as a vm as much as possible, but if I have the blessing of the forum, that might end up being a very clean solution for my needs.
reading your post a few times over, that seem’s like it may be a better solution for avoiding potential issues with ZoL vs ZFS on BSD
All the disks are off a single HBA, so passing it through should be fairly rudimentary. I Think that would mean that I could keep my current FreeNAS configuration as is, jails and all.
For the Steam Machine, I was thinking of hosting it off of an over provisioned cheep 128GiB NVMe in a PCIe to M.2 adapter (things are ridiculously inexpensive these days). I would then have it mount an SMB share from the same FreeNAS machine for storage. ProxMox seem’s to be quite good at PCIe passthrough, so that should mean passing a gpu into that VM is drag and drop (knock on wood)
After reading AirStripones post above yours, I’m looking into if ProxMox might be better suited to this application. In terms of overhead I’m not too worried, I can easily carve off 4 threads and a 100 odd GB of ram for the FreeNAS VM, and pass everything through. Even with overhead I’d still be pushing near 10Gbe from the ARC and L2ARC. Would also mean not having to mess with ZFS on linux at all, something that’s handy all things considered.
In the end I’d end up installing a gpu and 2 more SSD’s, one for Proxmox and 1 for the steam machine (iops for OS and all that.)
Yeah, my current thinking is use Proxmox for my OS, install FreeNAS in a VM, passthrough everything my current bare metal install is using and upload my config, therefore everything is as it was before.
after that carve off 8 threads and an additional SSD for the Steam Machine, add the GPU, mount an SMB share for the steam library and off to the races for in-home game streaming (knock on wood)
AMD gpu’s tend to be very easy. Nvidia like to fight you. Google “Code 43” I believe is the error they throw if they detect a VM.
None of us are saying that moving the array is a bad idea as far as I can tell. It’s more the point that you have to be ready to not go back. Proxmox will do literally everything that you want, but it will take some work and time to get it there since it’s not plug and play.
There is still overhead there, so be ready to lose some speed possibly. Not saying your plan won’t work, just that you have to consider that. Seems like you have enough hardware at least.
Seems reasonable. I use a 4k dead header (forgot the name) for the same thing. One note I have found is using NFS generally gets me better perf, and full support just hit the steam beta client without the hacks I used to have to run, so that would be my recommendation personally.
Yeah, currently have a spare 670 so wanted to start their before I invest in too much more hardware.
Fair enough on overhead. at worst I change my CPU’s to 5670’s, 50% faster clocks, 50% more threads per cpu should get me in the right ballpark.
not quite sure what a dead header is unfortunately. Is that a device that tell’s the graphics card it’s connected to a device without actually having to be? Is such a device needed?
I’ve never played around with NFS, since my main machines are windows and MacOS, so could be nice to play with something new.
EDIT: with steam on linux with the Proton emulation layer having gotten a fair bit better in the last few years, should I run a linux vm instead of a window VM for in home streaming?
As of Windows 2000 NFS has been supported in Windows. ZFS can actually do exports natively on datasets for you, at least on ZOL, I can’t speak for FreeNAS.
I have lived in *nix for the last 12 years and haven’t touched Windows. I will say check out protondb.com for the games you want to play and make sure that users have gotten them working. I play games that run well on Linux (proton included) but that means that some games I just won’t play. If you have to have the ability to play any game, and don’t want to do research first before buying, then Windows is still needed. That’s up to you.
Thanks for the link on the Headless adapter. Had been thinking a KVM might have to be involved, but the adapter seems better suited to this task.
Looking at the proton database, it has everything I’m looking at playing, so should be fit for purpose. Any idea if the Nvidia drivers through a fit if they’re in a linux VM? from the Nvidia site they do seem to have unix drivers, just a question of them supporting this use case. at worst it seems relatively trivial to trick the VM into thinking it’s running on bare metal. (source for the Gpu passthrough for anyone that might come across this thread in the future http://mathiashueber.com/windows-virtual-machine-gpu-passthrough-ubuntu/#Removing_Error_43_for_Nvidia_cards)
I’m installing Proxmox into a VM as I type this, so will take a look and see what is what
is the link you meant? and yeah, he’s got the right idea.
Vendor id has to be 12 characters.
When he says virsh edit, it might use one of several apps, depending on the local editor.
the CTL+X then CTL+y is for Nano or similar (default on 'buntu.)
Virsh sometimes opens in vim or vi.
if so, you’ll need colonwq
Literally
:wq
then enter
Just in case the editor doesn’t work with the CTL+x CTL+y
in vi/vim, colon escapes the text area and takes the cursor to the bottom, w will save out on exit, then q is for quit.
I guess if you are in Vi/Vim, you probably need to press the Ins (insert key) te be able to edit text, and when everything to your liking, you can press the Esc key to stop being in “insert” mode.
I’m not really a Vim user, sorry
If that is your plan… might want to take a look at a topic I posted about recently…
One of the big reasons I may end up ditching FreeNAS all together is to greatly reduce complexity. I have also researched VLANs to further reduce complexity on the network hardware (save a bit of power too) Proxmox (and other linux distros) are perfectly capable of managing the array and with some research, you can do pretty much anything FreeNAS offers and more. Maybe just not quite in the nifty GUI.
Friends don’t let friends get trapped in VIM.
Even new internet acquaintances…
My dog is no longer a fan of my being on this forum, just laughed for a good minute at that one.
I’m also scared to admit that Vi is my editor of choice
Alright, I’ll keep that in mind.
Also update on the testing of Proxmox in a VM. looks like it required a VGA device
Have Class early in the morning so won’t be dealing with that this evening. might pull down virtual box and try it that way.
