Help finding a guide for serving data and data protection

[djotter]

Hi

I am renovating my house, so taking the opportunity to make a more robust home media and backup server. I would appreciate advice from the community on recommendations regarding hardware and software to achieve a robust system.

Current setup
An under-the-tv HTPC for holding and playing all my media (TV, movies, music, home video, photos, documents), casual couch gaming (Human fall flat, Jackbox, Trine 2, Unravel 2), and web browsing/web based apps.
Windows 10
Plex Media Server
AMD 2400G
Samsung 840 EVO 500GB (OS)
WD Red (all CMR) 4TB (personal), 4TB (TV), 8TB (movies)

Other devices in the home, 1x Windows 10 laptop, 1x Linux Mint desktop, 2x Android phones. This will be expanded to other PCs (Windows or Linux) when the kids get older.

My current backup method is running Fbackup from the Personal drive to an USB external 2TB WD Green . . . whenever I remember to do this. i.e. I just did it this week, but it was 3 years (!) since the last backup run. I have a WD 2TB Red and 3TB Green currently unused and am considering one of these as a second USB external backup to store offsite.

Future setup

1. I would like to separate the media and backup server from the HTPC as I believe best practise is not to use a server for general purpose tasks (or have have the kids mashing the keyboard when i’m not looking).
2. I will move the server, HTPC, and audio reciever to the bottom of a linen cupboard.
   2a. Cupboard will be 5m and 3 walls away from TV, what is the best way to control the HTPC and deliver video and 5.1 audio?
3. I would like advice on redundancy and backups.
   3a. Automated backup of personal files and phones. Offsite backup schedule can be a manual process if I rotate HDDs with family at birthdays etc.
   3b. TV and Movies, I would like protection from mechanical failure, but I don’t think a backup is required.
   3c. What file system? RAID levels?
4. If I keep the AMD 2400G on Windows 10 for HTPC/couch gaming tasks, what additional hardware and software is required? OS, server, HDDs, HBA or onboard?

Stretch goals

1. Remote hosting of family member backups.
2. Quiet enough to be in the cupboard across the hall from my bedroom.

Thanks!

[SgtAwesomesauce]

@djotter

would like to separate the media and backup server from the HTPC as I believe best practise is not to use a server for general purpose tasks

You definitely can. I would recommend it if you’ve got kids. Mostly to keep the expensive storage away from the little gremlins.

@djotter

I will move the server, HTPC, and audio reciever to the bottom of a linen cupboard.
2a. Cupboard will be 5m and 3 walls away from TV, what is the best way to control the HTPC and deliver video and 5.1 audio?

Disregard that.

Uh, 5m =~ 16ft (just doing conversion for muricans like myself) That’s within HDMI range, which is officially 50ft (15.2m), but I’ve found best experiences are had within 20ft (6m). You’re going to need to run cables through the wall or ceiling, I’m assuming you’re okay with this. An HDMI cable can carry PCM 5.1 audio, so you’re in luck there if you’ve got a receiver that’ll handle it properly. If not, I’m not the best person to ask about audio.

@djotter

Automated backup of personal files and phones. Offsite backup schedule can be a manual process if I rotate HDDs with family at birthdays etc.

I’m not sure about phones, I usually just nextcloud sync my photos onto my fileserver and they get backed up that way. Nothing else on mine is important. I personally like Restic, but it’s more Linux oriented and while it works on Windows, it’s command line only, so you’re going to need to know a little CLI-fu to get it working.

Offsite in the fashion you mentioned is hard with Restic, it’s better in my findings to go with BackBlaze (their B2 solution) I’m going to be doing a little guide on this, from a Linux perspective in the next week or so.

@djotter

TV and Movies, I would like protection from mechanical failure, but I don’t think a backup is required.

RAID is not a backup.

That said, if you’re looking for reducing downtime due to hardware failures, ZFS would be my recommendation. ZFS has first class support on most distros now, so don’t worry about being stuck with some boutique distro to get support, If you’re happy with Mint, I’d recommend Ubuntu server.

@djotter

What file system? RAID levels?

Kinda already touched on this. ZFS is the go to around here. As far as raid levels, are you planning on expanding your disks?

ZFS likes to have all identically sized disks to be happy and you need at least 3 disks to do a raidz. (ZFS equivalent of raid5) Given your current disk setup, I can’t really recommend raid. I’m assuming you’ve got a total of 3 spinning disks in that list.

Now, I can recommend the following: Don’t use green drives. Try to stay away from SMR disks if you’re looking for write performance. If you don’t care about that, have at them.

I can better help you come to a solution if you specify a budget.

@djotter

If I keep the AMD 2400G on Windows 10 for HTPC/couch gaming tasks, what additional hardware and software is required? OS, server, HDDs, HBA or onboard?

So it really depends on your total storage capacity needs here. I’d recommend something from the ryzen 3000 lineup, and maybe a 1050 ti for plex transcoding, if needed. (the geforce cards only support 2 simultaneous streams, but you can patch the drivers to remove that limitation)

Are you looking for 16TB? 64TB? (usable) If you’re not looking for exotic amounts of storage, you should be fine with no HBA.

I’m starting to recommend against WD HDDs because they’re not being clear about which disks are SMR and which aren’t.

Since you didn’t specify a budget and I’ve been out of the server (hardware) game for a while, I’m going to have a difficult time giving you a recommendation at the moment.

@djotter

1. Quiet enough to be in the cupboard across the hall from my bedroom.

This won’t be a problem, however, make sure that cupboard is ventilated!

[dersteffeneilers]

I of course don’t really know what the OP intended for cost and management, but HP has their micro server gen10 plus, which has a lot of expandability and a BMC if something does go wrong (https://www.servethehome.com/hpe-proliant-microserver-gen10-plus-review-this-is-super/) There are two AsRock rack x470 mobos as well with a BMC if you want to build something yourself, which would be much cheaper

[SgtAwesomesauce]

@dersteffeneilers

AsRock rack x470

The issue is that those boards are twice the cost of most CPUs that fit his use case.

[djotter]

@SgtAwesomesauce

Uh, 5m =~ 16ft (just doing conversion for muricans like myself) That’s within HDMI range, which is officially 50ft (15.2m), but I’ve found best experiences are had within 20ft (6m). You’re going to need to run cables through the wall or ceiling, I’m assuming you’re okay with this. An HDMI cable can carry PCM 5.1 audio, so you’re in luck there if you’ve got a receiver that’ll handle it properly. If not, I’m not the best person to ask about audio.

Ok. Yes running HDMI will be ok. What about keyboard/mouse and receiver remote control?

@SgtAwesomesauce

I’m not sure about phones, I usually just nextcloud sync my photos onto my fileserver and they get backed up that way. Nothing else on mine is important. I personally like Restic, but it’s more Linux oriented and while it works on Windows, it’s command line only, so you’re going to need to know a little CLI-fu to get it working.

I assume (and a quick search shows) I can run Nextcloud from Ubuntu Server, would this also fulfill the stretch goal of remote hosting of family member backups?

@SgtAwesomesauce

Kinda already touched on this. ZFS is the go to around here. As far as raid levels, are you planning on expanding your disks?

ZFS likes to have all identically sized disks to be happy and you need at least 3 disks to do a raidz. (ZFS equivalent of raid5) Given your current disk setup, I can’t really recommend raid. I’m assuming you’ve got a total of 3 spinning disks in that list.

Ok, I can build something with ZFS. At the moment I have about 8TB of data to store. So I would need 2 more 4TB drives (in addition to the two I already have) to make a 12TB RAIDZ, and sell/re-purpose the 8TB drive.

@SgtAwesomesauce

I can better help you come to a solution if you specify a budget.

Budget is ‘minimum to meet needs’. I am happy to build the system myself, but keep it simple. I live in New Zealand, so pricing will be a bit more than what is available in North America, and not all parts will be available or fall at the same price points.

So let’s try to build this:

OS - Ubuntu Server 20.04 LTS
ZFS File System
CPU - AMD Ryzen 5 3400G
Motherboard - Asus ROG Strix B450-I Gaming
RAM - Crucial Ballistix Sport LT DDR4 3200MHz 2x8GB
Boot Drive - Adata XPG SX8200 Pro 256GB
Storage - 4x WD Red WD40EFRX 64MB 4TB in RAIDZ (I already own 2, so are not included in the total cost)
PSU - SilverStone SFX SX500-LG 500W
Case - Fractal Design Node 304

Total NZD$1557 / USD$945

Options:
Move to mATX and use Fractal Design Core 1000 and a Asus Prime B450M-A, saves about NZD$200 / USD$120, but gives up an Intel NIC.

[SgtAwesomesauce]

@djotter

What about keyboard/mouse and receiver remote control?

I’d just run a USB 3.0 cable over with the HDMI and set up a small hub in the cabinet.

@djotter

I can run Nextcloud from Ubuntu Server, would this also fulfill the stretch goal of remote hosting of family member backups?

You can.

And possibly. Depending on scale. Nextcloud tends to fall over in huge deployments, but if you keep it under a couple hundred gigs, you should be fine. Anything over that, you’d be better off using SFTP or something like that anyways.

@djotter

At the moment I have about 8TB of data to store. So I would need 2 more 4TB drives (in addition to the two I already have) to make a 12TB RAIDZ, and sell/re-purpose the 8TB drive.

Yeah, that sounds solid.

So it’s important to note that you can’t just add another disk to your raidz array and go from 4 in a raidz to 5 in a raidz without destroying and recreating the pool. So if you think that a year down the line, you might want 16TB, build for 16TB today.

With 4TB disks, a raidz1 (raid5) is kind of on the edge of what I’d recommend. You might want to consider raidz2 (raid6) if you want to have a bit of additional security. When you get up to these larger disk sizes, you can easily run into issues where an array rebuild causes a second disk failure.

@djotter

So let’s try to build this:

OS - Ubuntu Server 20.04 LTS
ZFS File System
CPU - AMD Ryzen 5 3400G
Motherboard - Asus ROG Strix B450-I Gaming
RAM - Crucial Ballistix Sport LT DDR4 3200MHz 2x8GB
Boot Drive - Adata XPG SX8200 Pro 256GB
Storage - 4x WD Red WD40EFRX 64MB 4TB in RAIDZ (I already own 2, so are not included in the total cost)
PSU - SilverStone SFX SX500-LG 500W
Case - Fractal Design Node 304

Total NZD$1557 / USD$945

That sounds good to me.

If you wanted to switch to the Asus Prime, that doesn’t ring any alarm bells to me, but I don’t know the status of that realtek NIC with Linux right now.

I’ll use this opportunity to reiterate that I think there’s a benefit to be had by switching from an igpu to a 1050ti for plex transcoding. It might cost you a bit in the long run, but I don’t think the 3400g will have the grunt to do more than one stream transcode, and that may not even be realtime.

Of course, if you’re not planning on watching outside the house and the HTPC supports the codecs you’re using, you should be just fine.

[djotter]

@SgtAwesomesauce

I’d just run a USB 3.0 cable over with the HDMI and set up a small hub in the cabinet.

I thought USB3 has a maximum length of 3m? Oh, just had a look, there are active cables for 10m lengths. I’ll just have to find an IR repeater for the audio receiver remote.

@SgtAwesomesauce

Nextcloud tends to fall over in huge deployments, but if you keep it under a couple hundred gigs, you should be fine.

Should be less than 100GB total.

@SgtAwesomesauce

So it’s important to note that you can’t just add another disk to your raidz array and go from 4 in a raidz to 5 in a raidz without destroying and recreating the pool. So if you think that a year down the line, you might want 16TB, build for 16TB today.

With 4TB disks, a raidz1 (raid5) is kind of on the edge of what I’d recommend. You might want to consider raidz2 (raid6) if you want to have a bit of additional security. When you get up to these larger disk sizes, you can easily run into issues where an array rebuild causes a second disk failure.

I think 12TB is an ok limit. If I start running out of space I am sure I can find some stuff I don’t really need to keep. Can I build a RAIDZ and add another drive to make a RAIDZ2 later? Or will I need to do the same pool recreation?

@SgtAwesomesauce

I’ll use this opportunity to reiterate that I think there’s a benefit to be had by switching from an igpu to a 1050ti for plex transcoding. It might cost you a bit in the long run, but I don’t think the 3400g will have the grunt to do more than one stream transcode, and that may not even be realtime.

Having a quick Google, there isn’t an official stance on HW accelerated transcodes for AMD under Linux. Hmmm . . .

That said I don’t expect any more than 2 transcodes at the moment with most clients being direct stream.

Thank you for your advice

[SgtAwesomesauce]

@djotter

I thought USB3 has a maximum length of 3m? Oh, just had a look, there are active cables for 10m lengths. I’ll just have to find an IR repeater for the audio receiver remote.

Yes, technically. It’s best to get an active repeater cable when going over 3m.

@djotter

Should be less than 100GB total.

I think nextcloud will do just fine then.

@djotter

I think 12TB is an ok limit. If I start running out of space I am sure I can find some stuff I don’t really need to keep. Can I build a RAIDZ and add another drive to make a RAIDZ2 later? Or will I need to do the same pool recreation?

So ZFS has a heirarchy. The following illustration, if each disk was a 4TB disk would have a usable capacity of 12TiB.

pool
  raidz1-0
    sda1
    sdb1
    sdc1
    sdd1

If you run out of space, you can do the following to get 24TiB of capacity.:

pool
  raidz1-0
    sda1
    sdb1
    sdc1
    sdd1
  raidz1-1
    sde1
    sdf1
    sdg1
    sdh1

Basically, if you want to add more space, you must add another vdev (which is what the organizational units with raidz1-0 and raidz1-1 are called)

This doesn’t include things like cache and slog, but those are really only needed for high-performance pools.

@djotter

Having a quick Google, there isn’t an official stance on HW accelerated transcodes for AMD under Linux. Hmmm . . .

I don’t think AMD supports it. At least not for Plex, unfortunately. I really wish that were the case. If so, the CPU you picked would be the perfect chip.

@djotter

Thank you for your advice

Always happy to help.