Hi everyone,
Have been busy lately planning my latest server upgrade, considering the old one has been in place for over 12years, has done me well, so it’s time to let it retire.
The hardware is a massive step up, going from essentially an old P4 QC Xeon setup w/16Gb RAM & 4x250Gb SSD’s running from an Adaptec 6405E, to a Dell R730 8LFF. Hard drive configuration with ZFS (which is the filesystem i eventually decided on) is where I’m at now, trying to work out the NVME drive allocation for the array
Nextcloud i’m intending to run via the following drives which i’ve procured:
Seagate EXOS 6Tb x 3
Now here comes the dilemma, i was planning to run one of the 512Gb NVME drives i have here as a SLOG device for the array, and purchase another one for L2ARC. But that’s where I’m kind of stuck scratching my head atm. I know there’s many different caching options, and considering this array is just for Nextcloud, it’s going to be a mixture of things like small files, photos, videos, documents for myself and my family & friends
The L2Arc Cache drive i could probably see as worth it, but it’s the SLOG device I’m not 100% set on yet. For general fastpool I’m going to be adding 2x1Tb SSD’s for everything else that’s not nextcloud.
So my query would be, would you go for the extra 512Gb NVME as a SLOG device for the Nextcloud array, or, a metadata drive? Maybe both? That’s what I’m not 100% sure on. Any assistance to nail down this last little bit of the FS config will be appreciated
no RAID controller and ZFS. golden rule. Don’t do it.
If the controller supports IT-mode, then this will be fine. But using hardware RAID controller for ZFS vdevs will end in tears and trouble.
I would put both NVMe as L2ARC and set sync=disabled (I was too cheap to buy a drive for LOG). Or get more memory instead and run everything on a single pool with a single 512GB L2ARC. If ZFS has good caching, separate pool is just more work and probably not much faster. ZFS excels at making HDDs not feel like HDDs, ARC is HDDs best friend.
I ran all my VMs and stuff on a HDD pool, ZFS made them go DRAM-fast. Just needed enough memory, L2ARC/special and some tuning.
Hi Exard3k,
That was my old setup, which had the raid card, not the new one. I’m still conversing with the company i’m buying the server from the final configuration of it, but atm it’s looking like:
Dell R730 8LFF
2x E5-2699v3 2.3Ghz 18C CPU’s
384GB Ram (12x 32GB DDR4) ← Looking to upgrade this to 512Gb
8x 3.5" HDD Bays with caddies
H730 Raid Controller
IDRAC Enterprise
Quad Port 1G rNDC ← Hoping to change this to 10Gb
Dual PSU
I originally got the quote for 384Gb so that i could dedicate 128Gb for ARC. But when the place im buying the server from offered me a decent price to update to 512Gb, i would have to say it’s left me tempted
So the L2Arc as a Striped setup you think, rather than mirror? I’m aiming for RaidZ1 for the 3 Drives (Found a good ZFS calculator site to give me SOME idea on numbers to expect, better than nothing at all lol)
I could always run them in that configuration, but what about running 1 for Metadata & 1 L2ARC? As I’ve just sent emails to ask about the upgrade for the memory, so could easily allow for between 128-256Gb RAM for the ARC Cache.
You’d think i’m not trying to make this setup fast and resilient at all, nahhhhh lol
l2arc is just a read cache. by definition if it fails data is not lost.
in contrast i’d discourage from making a metadata device unless you’re mirroring it - one slice of failure, metadata is not a nice-to-have. Seemingly it can be worth it, but unless there’s a serious workload, that may still not be enough reason.
One thing to consider, is how large do you consider the zpool to ever grow to. if that’s 8 bays, you might not want a pool so wide as to be only a z1. If so you might want to make it a z2 now (meaning 4th disk). That won’t be the most efficient or performant configuration now, but zfs expansion can grow a raidz pool width but not change the parity.
if you’re DIY’ing your server rather than through something like TrueNAS, careful to initialise the pool at 4k blocks (ashift=12) (also often advised is to apply atime=off, that’s at least configurable later, like sync, but ashift is not).
The nice thing is you can make your own choices for the characteristics you want. best of luck
Hi Exovert,
Well I was looking to initially go with RaidZ1 for just the NC array. The seperate 2x1Tb Drives are just for other apps, VM’s, etc etc etc.
It’s seperating the drive allocations. NC on it’s own array with necessary support, general data on it’s own array, and my mailserver is on a seperate Fusion Iodrive 2 768Gb
In terms of growing it? I’ve survived for this long on the 4x250Gb SSD’s, so i don’t really see myself expanding drives internally. Maybe external, but that’s for a down the track option if needed
To go for some form of parity, do you think RaidZ1 is the go or is there another option?
I’m planning the filesystem options here, before the server arrives, so that i have a clear gameplan. Basically spend the time to get the best i can options wise for parity & storage capacity. I know i’ll lose some, i’m prepared for that. Initially was only going to get 2 of the Seagates, but got offered a 3rd, so i took it.
And considering i’built my current server by myself over 10years ago, I like to be able to know that once i’ve got the work done, and i can be happy once it’s live, that it’s going to last for a while
for 3 drives z1 is fine. it’s only less so if the pool might grow to 5-6 wide. z1/z2 can’t be changed without a rebuild of the pool. if that’s not something you’re thinking of in ~10 yr don’t worry about that.
Right, so the 2x512Gb NVME drives, i can basically just set them as a mirror for L2ARC and leave it at that you think? As long as i remember to set the ashift to 12, then i’ll be fine
You can’t mirror anyway. L2ARC devices act on a round robin principle, so >1 device just spreads all data from the feeder threads (more or less equally) across the devices.
There is no need for redundancy because it’s a cache. And which is a big benefit over using special which requires redundancy because data actually lives there.
Hi ThisMightBeAFish,
I’ve already bought the 3 Seagate EXOS drives tho haha.
At least that helps me work out the finer details of the filesystem. Less worry now, i just ordered the Server this morning so it should be here in a week or so
Considering what I’m coming from, and going to, it’s pretty exciting
The next bit of fun is just working out the finer details of which OS setup I’m going to run, which for what it is I’m going to be doing here, I’m probably looking to UNraid > Proxmox. I was running ClearOS on my old server for the last 10+years, and lets not get into why that wasn’t the best choice lol.
In any case, I think now i’ve gotten a better idea of what i need to do now for the NC ZFS Array, cheers guys
It isn’t to late to by another 5 of them and build a glorious 8 wide RAIDZ2 for your data
One huge advantage of UNRAID is their fake RAID approach. Because of that, you will have a way better time putting drives to sleep and save on energy. And chances of you loosing data because of user error is also lower.
But you loose out on ZFS, which is pretty awesome if done right. And their license stuff would also put me off.
Oh I wish i could’ve bought a few more. But i got them with a chance find from another geek locally to me that had less than 15 cycle counts and perfect drive health, so i’m happy with 3
UnRAID, I’m mainly looking towards, not due to their licensing side, but the apps they have. Their app store for me will give me a great degree of flexibility. I could’ve gone for the default XFS/BTRFS route for UnRAID, but from what I’ve read, BTRFS can be a bit hit or miss. The server is now on it’s way for me today, so within a week i start building.
I came from a ClearOS license which cost me about $110AUD per year, especially for the featureset i was running, that was also forming part of what Server software setup i ran. I wanted to go for a like for like setup, anc considering Proxmox charges for Per Socket, means that if i wanted to go for an equivalent, it would cost me a LOT more.
Sure the upfront cost for UnRAID is a decent amount, but that’s a once off, and i’m fine with that. Heck, I paid that $110AUD p/yr for ClearOS for 10+ years, so you can understand why I was quite happy to have a one off fee
Going to be exciting now, to have a Mailserver that’s going to be a hell of a lot faster than my entire old setup. I know it’s going to be a bit of work transferring the emails from the old setup across to the new one, but hey, i like pain hahaha
Well after a long time waiting for parts to finally arrive, the server is finally in final phase configuration. Funnily enough i didn’t end up going for UnRAID as it only supported 1 Array Max, which was the buzzkill for me
Currently running on Proxmox now, and the arrays are as follows:
3xSeagate EXOS 6Tb Enterprise (RaidZ1)
1xSilicon Power 512Gb SSD (Caching)
4xSeagate 530 Series 1Tb NVME SSD (Quad Port PCIE Card - RaidZ1)
1xTimetec 512Gb NVME (PCIE Card - ZFS)
The whole system is working well, thanks to the advice from everyone here. NOw it’s just the finalising configuration of the other software i want to run on Proxmox lol
