I currently run a little NUC as a general purpose home server. It hosts Plex and a few other small services. It works fine but I want to expand and keep the hardware to a minimum.
Here are my goals:
NAS
Want to be able to host servers (teamspeak, possibly a small game server or two, etc)
Run various personal services (owncloud, plex, etc)
Want to be able to run a couple of VMs (linux and win)
I am not sure how I want to do this. The NAS+hypervisor is my big hangup… they both, ideally, should be running on bare metal.
The common solutions for these things are all basically independent operating systems. Proxmox is debian, FreeNAS is FreeBSD … you can’t* install these side by side and run them at the same time. And that is the crux of my issue.
*I’ve read very conflicting info about whether or not it is a good idea to virtualize NAS… and quite frankly, I’m not terribly comfortable with the idea. Maybe I’m off base here but I feel a little uneasy about putting an extra layer of complexity around the platform responsible for managing my data.
I can run my servers (games/teamspeak) in a VM, so that is not a great concern. The personal services (plex/owncloud) can possibly run as a plugin for whatever NAS solution I end up (FreeNAS/OMV)
FreeNAS 10 is going to include support for hosting VMs with bhyve … and seems like a great solution to my issue, but it’s anyone’s guess when that’s going to be ready.
I just don’t know how best to do this. I’ve researched the shit out of this and I’m stumped. Any suggestions?
You are better off running BSD jails then bhyve at this time, though with the advent of the IT mode controllers, there is no real need to have the NAS or services running on bare metal as you can just pass through the JBOD arrays and use them as a ZFS array or similar from the KVM VPS running on a Linux host. This gives you the best compatibility with all software and OS and not limited to the restricted ecosystem of BSD and the software supported there. As for hardware it would come down to what sort of budget and form factor you are desiring and your total array size that you wish.
Thank you for that. I'll look into it.. do you have any more information about the IT mode controllers/JBOD passthru that might be helpful? Also.. I was looking into Proxmox but that would require that I virtualize my NAS .. which was something I was hoping to avoid. But if it isn't as big a deal as I think it is.. maybe I need to revisit that.
Budget isn't a huge deal, I have something planned out to suit my needs and it's not too expensive. This is the part list I've been toying around with. Not set in stone but something in this ballpark. http://pcpartpicker.com/list/vfYQVY
ESXi is probably the one thing I haven't looked into yet. I skipped over it because I figured it would lead me to the same roadblock I'm already at: requiring me to virtualize my NAS
Is there perhaps something about this that is different than running Proxmox with FreeNAS running as a guest VM?
You can go with lots of things for full virtualization, but VMware is the 800lb gorilla. I've used ESX and ESXi for years, and have used VMware's other virtualization products for over a decade. While ESXi is only free as in beer, the ESXi is the same stuff used on the largest virtualization installation that exist. It's fast, well supported, compatible, relatively low footprint, and extensible should you ever want to go whole hog VMware.
My next choice would be something like Xen, but thats considerably more fiddly.
And yes, you can totally run FreeNAS in a VM. What you do with the disks is important, but it's doable.
So if you want to screw around with VMs and what not, use whatever you want. But if you want something that just works and is well supported, I'd recommend ESXi.
This is the kind of thing I've been reading about virtualizing FreeNAS.. and it's why I've been a little hesitant to do so.
Virtualization vs. Bare Metal FreeBSD (the underlying OS of FreeNAS) is not the best virtualization guest: it lacks some virtio drivers, it lacks some OS features that make it a better behaved guest, and most importantly, it lacks full support from some virtualization vendors. In addition, ZFS wants direct access to your storage hardware. Many virtualization solutions only support hardware RAID locally (I’m looking at you, VMware) thus leading to enabling a worst case scenario of passing through a virtual disk on a datastore backed by a hardware RAID controller to a VM running FreeNAS. This puts two layers between ZFS and your data, one for the Host Virtualization’s filesystem on the datastore and another on the RAID controller. If you can do PCI passthrough of an HBA to a FreeNAS VM, and get all the moving pieces to work properly, you can successfully virtualize FreeNAS. We even include the guest VM tools in FreeNAS for VMware, mainly because we use VMware to do a lot of FreeNAS development. However if you have problems, there are no developer assets running FreeNAS as a production VM and help will be hard to come by. For this reason, I highly recommend that FreeNAS be run “On the Metal” as the only OS on dedicated hardware.
I wouldn't be using a RAID controller... but this is still somewhat of a concern. I understand it's doable.. just makes me a little uncomfortable. Notably this part.. as these are all out of my control.
it lacks some virtio drivers, it lacks some OS features that make it a better behaved guest, and most importantly, it lacks full support from some virtualization vendors
OpenMediaVault is another option but I am not as familiar with it, or btrfs
Yeah that's the classically referenced info whenever talking about virtualizing FreeNAS.
Are you planning on running a huge file server with RaidZx? If so, bare metal and lots of RAM are what you need (about 1GB RAM per TB per FreeNAS docs). If you are just doing disk mirroring or JBOD, none of those performance constraints matter.
You can run KVM under linux and still host your file services with it too. You don't have to run FreeNAS.
Not "huge huge". My NAS would be for my media, as well as encrypted storage for personal files on my home computers. Movies and music is the bulk of it, served up from Plex. If you look at my pcpartpicker build above, I have chosen 32GB of RAM and plan for 4x4TB. I've researched FreeNAS to death and understand ZFS's requirement for lots of memory.
My choice to use FreeNAS basically just boils down to ease of use and redundancy (thinking of going with mirrored vdevs over parity). The plugins and useful web interface are a nice perk
you can cut over 100$ off if you go atx. have enough leftover to get a ssd for os's and cache, and use a higher rates psu. http://pcpartpicker.com/list/GRYCLD the 1230 is cheaper and should meet your needs, but they are the same socket so you can switch if you want that 2.5% improvement in clock speed. also i would suggest a non stock cooler as you can "overclock" your xeon to use turbo as based clock, and it's quieter.
@fredrich_nietze I like that system except that motherboard doesn't support ECC memory. I just checked the manual
@beer Spend a little on a boot SSD and just run ESXi. Pass thru the SATA drives and everything will run great. RaidZx or mirrored. If mirrored you can cut FreeNAS way back on memory and have more left over for your guests.
The one thing I don't like about FreeNAS is how difficult (impossible) it is to add to an existing drive pool. RaidZx feels like a house of cards. I have a FreeNAS at work for archival backups, but I just mirror drives.
If you put another windows machine on top of that it would need a license, but ths bare 2012 Hyper-V server does not. It'll run anything you want as well pending you have the RAM/CPU to accommodate. I ran a Linux machine on it at one point just as a proof of concept (Debian.)
It may work for you. It will satisfy your hardware NAS requirement. You can set it to use BTRFS (Butter FS / Better FS) which has come a really long way.
It also has Docker containers for your little services like Plex / own cloud.
And you can setup multiple VM's for whichever OS you like.
Make sure your Hardware supports VT-x (mandatory)
and VT-d (optional) if you want to give your Guest OS's direct access to PCI devices.
There is likely a better setup than what I suggest, but I'm in the ESXi camp as well and my reasoning which may not apply is as follows: - when I entertained unRAID and proxmox, a coworker who is an IT Architect suggested to use a solution that is widely used in the enterprise for resume sake- that advise is now echoing in my head whenever I have any software choice at hand.
it does the job. I got a used z800 on craigslist (hindsight I'd get a rack mount unit on ebay for a lot less), I flashed the really slow LSI RAID chip on the board to IT mode, in ESXi put it for pass-through use and now have Rockstor as my NAS with a Plex "rockon" (docker), a share, and VMs up the wazoo for whatever I'm up to. Recently I spun up a CentOS7 instance that runs as my Splunk server, starting splunking my home router and found my poor little Raspberry Pi was part of a botnet using a wordpress exploit. The z800 has two NICs and I'm working on getting Splunk stream running as well. The key to running a NAS on top of virtualization is lots and lots of research and understanding how they work, buts its not explicitly impossible or bad practice.
It can't hurt to try ESXi IMO, and the free version is plenty good for home use IMO. The issue would be hardware, its risky trying it on on 'whitebox' equipment, but again its free, if it doesn't pan out, then you have other options out there.
There are things I would do differently, but not the selection of ESXi, but when I go to revamp it, I would put it on a USB stick, and use a hardware RAID card I have for the data store, and software RAID/NAS as well. You can do USB stick ESXi and still have everything else on a software RAID I beleive, you just need to setup the bootup sequence of VMs in ESXi properly, and shutdown of course-- and openvmtools will be your friend.
Proxmox might work. It doesn't have any kind of NAS GUI built in, but it's a heck of a hypervisor, and it supports ZFS, so you might be able to install other packages on it to make it more polished. It's just modded Debian so pretty easy to screw with.
Update to my post, Rockstor has lost everything in the share, I don't know where to begin on trying to restore it, non of the snapshots are working as well. I'm not mad as I have everything backed up and knew this was experimental- its just an FYI.
I have a hardware RAID card just sitting here, I think for simplicity I will build on that this next time around- ZFS, BTRFS- all requires too much command line ninja skillz for me when it doesn't want to work right.
You should just install a FreeNAS VM and use that for storage duties. I'm using it on top fof ESXi and it's been solid. I even had to move the 8 drive array to a different controller and that work flawlessly.
I tried the BTRFS thing for a while and it was too flaky.
Here's the post I made a while back about the build; planning to update the post soon to reflect how it's currently setup but the hardware and premise are still the same.
I had installed and tried FreeNAS before Rockstor, I just couldn't get it to work right so thats why I gave Rockstor a shot. I'll try FreeNAS again, and am also looking into Ubuntu with ZFS.
I do want to put that hardware RAID card to use though haha, a great piece of server grade hardware just sitting there.