Convergence - NAS and Virtulisation

Operating Systems & Open Source Linux
1

Hi all,

New here, so I've posted in the wrong place or made any other faux pas then please let me know!

The background:

Brought a new flat and I'm doing a full renovation on it - it's a lovely place but the electrics were positively archaic. So while I put loads of holes in the place, I might as well network it/automate the hell out of it! It's a Victorian property and very large (by UK standards, probably tiny by Texas standards...). This means it could cost a fortune to heat and run so the automation is a pretty cool way of making it cheap to run/eco friendly.

I am building a new house server to control everything - networking, AV, lighting, heating etc. The whole lot. The cost of running this should be recovered from not heating rooms unnecessarily etc. The server stats are:

  • Xeon E3-1260L V5
  • 32 GB ECC DDR4 (unbuffered)
  • SUPERMICRO MBD-X11SSH-LN4F
  • Toshiba E300 3TB *6
  • Crucial SSD (can't recall the model/size off the top of my head)
  • Seasonic 400FL

The problem:

What O/S to run... I keep researching and I keep changing my mind! Here is my want's and needs:

  • High reliability - Need
  • Run OpenHAB in a VM - Need
  • Run PFsense in either a VM or container (I will offshore this to another machine in the future but at present I need to run in the main host) - Need
  • Operate as a NAS - Need
  • Fault tolerance - Need (don't want my wife being stuck with no heating because a disk died and I'm 500 miles away at the office
  • Protection against bitrot on the NAS - Want
  • Ability for the NAS to rebuild a disk from a hot standby drive - Want
  • Prefer not to spin up every disk in the array for every little read/write operation - Want
  • Probably a ton of other requirements that I have in my brain but can't think to put here right now...

Thoughts so far...

  • Proxmox - this can set to use ZFS but I'm not sure if there is an easy way to expose this as NAS.
  • FreeNAS in a VM - host could be almost any hypervisor
  • OpenMediaVault with SnapRAID
  • Do my own set up where I do a Debian or Ubuntu Server install with ZFS on Linux (or SnapRAID) and then add Proxmox on top - may add Webmin to make management easy/convenient.
  • OmniOS or SmartOS - never played with either but I've got an idea that these might fulfill some or most of my requirements.

Any thoughts guys? I like FreeNAS because it makes ZFS easy to manage from a Web GUI but I can't run it as the bare metal OS because PFsense won't operate it BSD jails. I think I am at the point where I've over researched and have begun to confuse myself more so your thoughts would be great!

Sorry for the long post and I'll post a separate thread for the house build/automation if any one is interested...

Cheers,
Pix

1 Like
2

FreeNAS never recommends to run it in a VM.

3

Working on basically the same thing minus the home management.

4

Thanks guys for responding - you can virtualise FreeNAS, it's not recommended for production systems but as I can do a back up to my fully dedicated FreeNAS box once a week or early enough once a week then I'm not too concerned.

@Superfish1000 - that's cool. Did you run ZFS under Proxmox or inside a VM? I'll dig up your build log and have a look too.

5

I tried to use BTRFS for the RAID for Proxmox but I think it's still not mature enough to use for this. When I tried to configure it I kept getting errors. I'm not sure it liked my 32TB of storage, not really sure though. I opted to go with ZFS because it seemed to be an easier solution. I was hoping some people would suggest things but I haven't gotten a lot so far. The only VM I currently have running is for a Minecraft server. Still need to sit down and load up some to handle a caching proxy, media storage and network backups.

6

If you're still having problems with this, stop trying right now. Historically Proxmox has not supported booting from USB thumb drive (a "feature" which has irritated me to no end since the very early days of Proxmox), and I don't think they've made any changes to support it.

Back to this thread, though...

It's really as easy as can be expected of any Linux distribution. Proxmox ties back to the Debian repos, and has a repo of its own for paying customers. If you're comfortable configuring Samba via the command line, you'll be right at home in Proxmox. BUT! Last I saw there were some bugs with ZFS on Linux. One of which, and this was killer when doing a couple of Proxmox deployments earlier this year, is that ZoL ignores /etc/modprobe.d/zfs.conf, which you want to edit so that ZFS doesn't consume all of your memory.

Also, if you do go through with this, never ever ever ever ever ever be fooled by the installer into installing the OS to a ZFS partition. It's an option. Don't use it. Your system will slow to a crawl in no time.

As has been mentioned, the folks at the FreeNAS forums never ever recommend running FreeNAS in a VM. That being said, it's doable. But you make damn sure you:
1. Memtest
2. Pass your NAS disks to the VM rather than presenting FreeNAS with virtual disks
3. Hook your server up to a UPS that Proxmox (or the FreeNAS VM at the very least) can read from

Memtest, because you want to make sure that shit is solid. Passing disks through because ZFS is worthless if you start putting layers between it and the disks. And the UPS because with layers between FreeNAS and the memory, you don't want power going out suddenly. Clean shut down every time.

If you go this route, I'd highly recommend just sticking with libvirtd+KVM. I find it easier to deal with than Proxmox's management tools.

I don't know anything about OMV, SnapRAID, Omni or SmartOS, so I didn't comment on those.

All that being said, I'd personally go the way of exploring ZoL on a basic Debian or Ubuntu Server install. If you've only got 1 machine to work with, and you need ZFS and you need pfSense, this is the way to go. Really, really, really experiment with ZoL before you commit though.

7

Gonna put it on an SSD. It didn't like booting until I made some changes to the startup script and I wasn't able to make a cluster so that I could migrate machines so I'm admitting defeat on that one and throwing it on an SSD.

1 Like
8

I think the solution is to go with my own install and load what I need on top... Is there a Web gui that makes zfs on Linux easy to manage like FreeNAS? I don't want to have to keep logging on all the time to check for failed disks etc.

Can ZoL manage a hot standby disk and start an automated rebuild on disk failure? I am sometimes away for work so this is a beneficial feature of FreeNAS.

Any recommendations on adding more NAS functionality to the ZoL install? FreeNAS has some great features that might be useful to me.

One more question... Can proxmox go on a Ubuntu server install? I am more familiar with Ubuntu server than base Debain but of if I have to learn then that's fine too...

Thanks for all the input guys!

Cheers,
Pix