Video Editing Server/Data Storage Assistance | Noobie

Hardware Hub Build a PC
, ,
1

Hello, first time poster, read the rules so I think I am doing this right. I literally signed up for the forums because I saw videos of Wendell talking about the topics I was researching and had questions.

I am a total noob when it comes to setting servers up, the extent of my knowledge is setting up Synology and AsusTor equipment but I really want to give it a go.

Here is the info I need help with:

I run a small video production agency and am a data hoarder. I have data from 11 years ago we still sometimes reference on a rare blue moon. With that said, I need to backup our 88TB of data and have expansion for the future. Additionally, having myself and my editors able to optimally edit off it would be great. Usually we have 1-5 editors at a time editing. Total of 12 on the team. We film in H.264 1080p/24fps for internal projects and ProRes 422 4k 30-120fps in LOG and non-LOG formats (depending on project scope). Very rare to have much more except maybe some motion graphics files that are ProRes 4444. We film using Panasonic, BlackMagic, and Sony cameras (again depends on the customer and scope, different packages for different types of shoots).

I was looking at the Synology servers (SA and XS rackmount line) but the price point was too high for what I would aim for, although is doable. So I went researching and found the videos Wendell did for GamersNexus a few years back and again last year. Would a used NetApp disk shelf such as that one in the video be a good investment? What would good host PC hardware be for our needs? What matters in a build like that vs what doesn’t (I built our Threadripper workstations so I understand the impact components have on a local scale and type of media but not a networked scale for a server, ex: learning the Asustor Flashtor RJ45 ports would be limited because of the CPU was a learning moment)? Also, would I be able to run the disk shelf in RAID 5 and add drives (WD Red Pro 14TB 7200rpm each) as I go or is that going cause me to format the drives every time? What are the possible limitations, pros/cons, and does that even make sense for us?

Same with software, is TruNAS Scale the way to go for my needs? Something else? Any resources to learn the setups and ease of functions?

I plan to connect whatever Data we have via iSCSI on each workstation as well to avoid any Premiere or After Effects issues. Basically having one, large single volume setup as one large “hard drive” (with SSD Caching) everyone has access to locally. Does that make the most sense as well?

We do have 10Gb networking. I have two ports aggregated on my workstation to our switch (all Unifi equipment) for a total of 20Gb, the same with the server. All other workstations have one 10Gb connection to the 10Gb Unifi switch.

More or less looking for any and all advice, thoughts on my idea for a setup, etc. Trying to beat $10,000 (drives included) price point, which would be easy if a used NetApp Disk Shelf, >$2000 host PC, and I can add more drives to the volume as time goes on (instead of needing all the drives at once like the Synology needs) should do the trick, right?

TLDR: I am brand new here and wanting to expand my horizons and understanding. Does a similar setup to the one Wendall did with GamersNexus make sense for our media editing and storage needs? What host PC hardware matters most for a server PC? Can I set a DiskShelf to be RAID 5 or is it locked in as a JBOD (am I hardware locked for some reason)? Can I also add more drives as I go to the same volume or is that an issue?

Please be gentle with the noob (me), I want to learn how to make this happen and do it. Any advice or redirection, criticisms and improvements are happily welcome! :slight_smile:

2

I would advise a hard pass there, you need a full enterprise solution such as TrueNAS running on something resembling enterprise grade hardware (ECC RAM and some redundancy).

As my username suggests, I dabble in videography but professionally I solve big problems for companies.

A Ryzen 7000 series with ECC support is your baseline and 1GB of RAM per installed TB of storage.

Having built up to 320TB usable servers for enterprise videography workloads, I strongly recommend you lurk my posts and you’ll be able to derive everything you need. (sorry, not reposting just for this until you have more specific questions)

Would provide the raw storage, but not the bandwidth you require.

No sir, you want RAIDZ2 per 6 drives (RAID 6) else you’ll end up like the SmallTree client I partially recovered who lost a 128 TB array after 2 drives died and a 3rd, then 4th bit the dust during rebuild. (this was a $200k+ deployment when new 8 years ago)

The 320 TB usable array I have in prod has 8 drives of redundancy using all 20TB WD (formerly HGST) drives per BackBlaze most reliable stats.

And you will want as many direct connections to your server as possible. Port aggregation, or individually assigned IP addresses per NIC.

I highly recommend NVME devices for your working storage.

Is what you’ll be runnin as they are EOL’ing core.

Doable, just how fast do you wanna spend?

Welcome, I tend to be a dick at times. Sorry in advance.

you need JBOD / HBA access for ZFS to function properly.
I have seen proprietary deployments with ZFS on RAID and do NOT recommend it.

Hardware RAID is dead
long live software RAID

2 Likes
3

Was more or less setting the stage. I want to try and build a custom server for less than the cost of Synology’s SA or XS line with drives included. I can easily build a Ryzen 7000 host PC, and remember to include 1GB of RAM per TB. But like you also mention, the Disk Shelf:

So, if I wanted to ensure all our footage and project files our laid out like the attached screencapture, how could I ensure an optimal editing experience (for 1-5 editors) while maintaining our current file organization (the Client Asset Library is where we keep all footage in subfolders. Basically Library/Client Name/1.Project Name/Video/files.MOV) while also ensuring we can have enough storage space?
Screenshot 2024-08-05 144741

Again, my entire XP is based on using Synology and AsusTor NAS and rackmount servers. Any resources you could point me to so I can read up/watch how/why I want to do it this way? I understand the redundancy part.

24 port 10Gb switch. 12 workstations and 1 server, 1 uplink to the rest of the network. 16 ports being used total. Not sure I can direct attach that many into a single server right?

Would NVME for a SSD Cache be enough or having a separate volume of only NVME drives for editing only be the way to go?

I should mention that each of our workstations have 3 NVME drives:
1 for the OS
1 for Media Cache
1 for Temp Proxies (auto wipes after 30 days)

I mean more or less it comes down to can I beat the synology prices, and have my cake too. 1. Expandability. 2. Efficient editing speeds 3. Storage space we can use ASAP and continue to again, expand. 4. Redundant systems a plus. 5. Modern-ish power efficiency. I can’t be pulling 5000+VA from the wall, lol. Money is not really an issue. I could just spend the $10k for the Synology systems, but if I can save a buck or two, learn something new, AND have a better, more user-fixable (when issues arise) solutions, why the heck not, right? More or less my mentality. I want to learn how to do this and will try to find your posts to learn more as well.

Same sometimes but I don’t find you being a jerk here. Merely direct and I appreciate it.

Maybe I have my terminology incorrect then? I thought JBOD would be independent from RAID (choose one or the other).

Based on what you’ve said, an idea I’m having is:

Build a Host Server PC using a Ryzen 7000 CPU, have an array of NVME drives as one volume as a place to edit off of. Only Premiere Pro Project files and After Effects Files live here. Attach it to a Disk Shelf as the Mass Storage Device, and setup an auto-backup to send all the files to back up to the proper folder from the NVME drives to the disk shelf at the EOW. Thoughts? Or is a different setup better? Not locked into the disk shelf, just trying to see what may work and may not.

Can I also add more drives as I go to the same volume or is that an issue?

4

Welcome to Level1Techs!

You obviously have some existing experience hosting and accessing that 88TB of data.

What are you using currently and what aspects would you like to improve?

5

Pretty simple recipe:
Ryzen 7700 or 9700 (for ECC support)
256 GB ECC RAM
12 20TB drives for 240TB installed, 160TB usable storage in one giga-vdev ← $3600

The RAID6 configuration is through experience performing data recoveries.
I have performed miracles to recover RAID5 arrays, and been unable to recover several.
So it’s all about your acceptable margin for data loss.

There are some proponents on this forum of RAID5, but they will explain that you need a pair of servers hosting the same data as RAID5 is the minimum. I understand it is rare for a small enterprise to have a pair of servers able to mirror all data.

My drive recommendations are sourced from:

Depends on the user’s utilization and network resource access. Dedicated ingestion machines and high res workstations typically have direct uplinks to the NAS. This optimizes work flows for the heaviest users.

both

a single 4x m.2 add in card can reasonably host 12 TB of usable NVME storage (4TB drives are the sweet spot in the market right now)
If you’re gonna cheap out on the drives, put them in RAIDZ2 and you’ll still have 8TB usable as scratch/working space.

Boot off a SATA SSD and use the onboard NVME slots for cache and acceleration devices.

Buy a 3U server with a 24 drive SAS/SATA backplane (480TB raw expandability using 20TB CMR drives) and skip the disk shelf for this config since you need throughput.

Throughput, Throughput, Throughput

Dedicated NVME VDEV on a single add-in card so you can add more at a later date

Integrating as much as we can here and now with the drive RAIDZ configuration.

Big drives and modern processors, easy peasy

EXACTLY where you are now. For half the price, you can have a superior equivalence. 3/4 and it’ll be twice as good for twice as long and expandable

Most of my posts are about similar deployments / applications

It’s a binary choice, yes. You are looking for a HBA card (skip flashing into IT mode) for a mission critical installation. Broadcom is the defacto standard and what you’ll have plugged directly into the backplane.

Yep

Skip it and go integrated for the size you are looking

ALWAYS a good idea since the NVME drives will be tiny in comparison

Not ideal, ideally you add a new VDEV with the new drives but here’s some real food for thought:

SOP is replace the main NAS every 3 years along with all the drives to head off drive failure rates at 3.5 years. We then relegate the old NAS to backup server duty with fresh drives for the next 3 years.

You don’t need to plan storage requirements for 6 years out, just 3.

The NVME working VDEV is the easiest to replace/upgrade.

6

Hi Jode,

Currently using Synology servers. Issue is we’re approaching our maximum storage with expansion units (100TB). So, I was looking at newer models to replace our older ones and just want to explore the idea of instead of buying a 1st party solution, building one.

No real problems with out current solution, just can’t expand it further and we cannot currently halt operations to rebuild everything from our offsite backup. Plan is to copy our data to a new local system while still operating on the old one, then transition.

Basically, just want to replace the old system with a larger, new one. Just retarget the LUNS and iSCSI to each workstation, etc. I just really am in the dark for anything outside of the 1st party solutions (Synology, QNAP, Terramaster, AsusTor, etc.)

2 Likes
7

Yeah, this could make for a fine backup to your NAS, but obviously it will only be able to hold so much in it’s current config.

In an ideal world: upgrade to mirror your new TrueNAS in whatever config is possible.
Reality: Mirror critical data and rely on your new server for daily operations.

1 Like
8

So what I am gathering is:

Buy a 3U server case with 24 drive bays. Use 20TB WD Red Pros (more questions on this later), ensure I have a Broadcom HBA, on the motherboard have a PCIe M.2 Card for three 4TB NvMe drives (FYI: I usually buy WD for HDD and Samsung for SSD, thoughts?)

Config it to have a separate volume and backups to the greater 24 bay volume.

Not sure I’ll replace the NAS every three years but good to be aware of since my SOP is replace our workstations every five (they are overclocked as well).

Now about the drives, 20TB HDD from WD are going for $420/piece now. If you’re saying it is better to have all the drives in an array setup to be a singular volume all at once, I’d have to buy 24 20TB drives, and build the system (that is WAY more than the $3,600 you listed). So I am back at square one again.

So I’ll lose a total of 8 drives for redundancy, that correct?

Yeah, not sure how our synology config could migrate but I’ll look into it. This is mostly about our onsite, operations unit. Not sure if this is necessary to mention but we also have 1 offsite backup about 20 miles away and a Cloud backup as well.

1 Like
9

Thank you so much so far!!! this is incredibly insightful!

1 Like
10

You’re actually replacing every 6 years.
Years 1-3: Primary NAS
Years 4-6: Backup NAS

Ideally 4 to maximize your redundancy or throughput (RAIDZ1 vs RAIDZ2) and use that empty slot, unless you want it open to rebuilds though I have had greater success rebuilding NVME arrays with the problematic drive disconnected since controllers die 3:1 vs NAND

love it

love it, but read up on the BackBlaze drive stats for the WHY
Never Seagate BTW

24x 20TB is a metric shit ton of raw storage. Half a petabyte is no joke.

I’d seriously consider the 12x drive config and see if 240TB raw, 160TB usable will serve your purposes (and for how long), then consider the 18x drive config of 360TB raw 240TB usable storage (quarter of a petabyte here).

The WD drives being made at the old HGST plant are $330-$400 brand new OEM.

not quite, I recommend 1 drive of redundancy per 2 usable drives. This config is 4 drives of redundancy for 12 drives installed. Almost everytime I have to recover an array, another drive fails during the rebuild. This safeguards against that and good SOP of rotating/replacing drives before the AFR gets too high will protect your data from someone like me performing a $10,000 recovery.

You don’t have to fully populate that yet, just have the expansion options for the future.

11

I have an unused NVME expander card that came with each of our workstations mother boards (MSI Creation X399) so I’d throw four 4TB Samsungs in there most likely.

Like I said, I only buy WD for HDD. I have bought Seagates many years ago and ALWAYS had issues with them. Over the last 11 years, I have only had 1 WD fail on me and had issues with 2 (they were external drives I phsyically dropped. The head would click after that). I’ve lost at least a dozen Seagates in the same period of time and learned my lesson.

Indeed. So 24 Bay, 3U case, etc. etc. but only populate with 12 drives for now and you are saying I CAN expand a singular volume (remember I’ll be using iSCSI to each workstation so having access to everything in one “drive” is ideal) and then expand upon that?

Also, between 2TB and 10TB a year. The last three years we’ve accumulated 30TB of data.

Where the HECK do I find 20TB HDD for $330 t0 $400? The most affordable I saw was $420.

So I can expand within the same unit?

12

I’d assume to go maybe Threadripper 2000, 3000, or 5000 used as both those CPUs, according to AMD’s website, only support 128GB and 192GB respectively?

13

E.g. here

14

PERFECT solution

CAN? yes sir
that will constitute a complete rebuild of the array so be aware it will take the NAS down for worst case excluding failure:
20TB * (8 Seconds/GB) …160000 seconds / 1660 seconds in an hour and you’re lookin at 4 days

I understand there are some optimizations and such in the works to get this down, but always plan on worst case. Realistically, you’re lookin at total storage used in percentage as a multiplier to the 4 days worst case (70% utilization) → .7 multiplier, but always best to finish projects early

Apply Moore’s law here and plan for 60TB additional storage needed.

yep

Are you a cop?
Cause you’re askin alot of questions for a solid dude…
But really, there’s a TON of vendors selling around that price point.

dammit @jode …

Yeah, you’ll be pushin a 7000 series ryzen to the brink if you go over 12 drives and need high throughput. The 1Gb / 1 TB is a good rule of thumb for max performance, which you will need in this specific installation as video editing takes all the data.

The threadripper 5000 or 7000 would handilfy fill the need without goin crazy, though you can also begin considering EPYC.

15

So when I expand the volume, is this also formatting the drives? Or more similar to refering to the parity data and rebuilding everything but I haven’t lost data yet?

16

Hahahaha, no. Just trying to learn as much as I can. Like I said in the OP, this is all mostly new to me.

I’ll probably go to the used market here for TR 2000 or 3000 depending on the 5000 prices.

Thank you @jode So go to ebay for that! :slight_smile:

17

It’s a rebuild, nothing is lost.

If you’re gonna live on the wild side, used epyc is the way to go.
The threadripper offers higher clockspeed, but that isn’t necessarily where you need to be as your random IO is low and sequential is ridiculously high (comparatively speaking versus database workloads). Just beware vendor locks on processors, or don’t and embrace it for even bigger bang for the buck on the second hand market.

Yeah, you’re takin your data into your own hands here so far as RMA, but the cost savings typically outweighs the RMA peace of mind. When we have a drive fail we rush to replace and RMA later, so it’s SOP for us.

18

Feel free to go to Amazon.

Next, I pull out LMGTFY :slight_smile:

1 Like
19

No reformatting or data loss during that time? Perfect!

So what youre saying is go 5000 series TR no earlier. Epyc would do me better anyway? I’ll admit, I never considered Epyc cause it seems overkill so anything ideal I should look at?

I try to always have two spare drives to replace any dead ones in a machine as a good measure. Peace of mind outweighs cost for me. I want things to work, work for as long as it can.

I think at this point, I’ll part something out, post it here, and get any last thoughts if that makes sense.

1 Like
20

No earlier because it’s used silicon that’s EOL’d already.

I have an EPYC 9334ES running the 320TB usable NAS in prod. It’s unlocked for overclocking, but tamed down. Be warned: I received 2 vendor locked CPU’s before finally getting the one in use now and did a TON of stability testing prior to deployment.
We actually have several ES skus in prod because they’re cheap enough to inventory, but mission critical workloads run production skus.

That entire build was around $12k and I don’t recommend it unless you need it. There is no upgrade path in that config, but all we could do before deploying a full SAN. In 3 years it’ll be the gnarliest backup server in the midwest.

12 drive 3 U enclosures are a dime a dozen, 16 drive 4u can be had pretty easily but more is far more difficult to source at reasonable prices.

The 24 drive 4u enclosure for the 320TB build was a one off fleabay find.

EPYC’s require 1 DIMM per memory channel to function optimally. This balloons cost over a TR build when you are dealing with multiples of 192GB across 12 DIMMs.