Advice needed on how to setup my drives

Hello community!

I recently received all the parts to setup my computer. I will be building it over the weekend

Amongst them is my MoBo, which accepts 4 4.0 NVMe and 4 Sata drives. I also have 4 Solidigm p44 2TB drives.

This is going to be my secondary pc. I will use it as a dedicated streaming pc eventually, but it will also be my editing rig, I do photography and lightweight video. I would also like to take advantage of the 16gb GPU I’m putting in, to do my data and, hopefully soon, my Ai courses in Linux environment. 2 of the Sata will have WD Reds I have for long term storage.

My dilemma is how to set them up, as I never used NVMe before, so I don’t know where their speed is really leveraged. I definitely want to setup at least a Raid 1 with 2 NVMes for speed and reliability for the Raw stuff I’m working on.

Basically I’m torn between setting my OSs in 2 NVMes, or getting 2 SSDs and setting the OSs and software installs there, and leaving the other 2 NVMes free for stuff that requires higher speeds.

Any suggestion and experiences are welcome. Not sure how much noticeable is NVMe vs SSD for OSs, or if splitting OS and software is really necessary. I tried that once, and the fact that some soft requires for some of its dependencies to be installed in C:\ and leaves me with everything split between 2 drives really gives me headaches.

Thanks!

Congrats on the new system!

Context: My core box is a TrueNAS system running a VM with passthrough 3080 Ti for gamestreaming and video workloads, and it’s amazing - never having a desktop ever again.

NVMe is a must for any serious workload. Professionally, everything from AutoCAD, MatLAB, PyTorch, code development, Adobe Premiere, and even file sharing benefits from the low latency and high IOPS. Once you use it, you will never use spinning rust again, except for bulk storage.

OS-wise, my recommendation would be TrueNAS Scale since it sounds like this box will be the centre of the world with regards to your data. With Scale, you’ll get the ability to do config and data, automated backups, ability to slice and dice your storage as needed without carring about hard space quotas, great WebUI (it’s the same UI they use on their storage appliances), and virtualisation and GPU passthrough support.

Mobo-wise double check the block diagram in the manual, 4x4.0 M.2 (which should be 4x lanes each) sounds extremely unusual and most likely one or more of those slots are handicapped in some way. This will affect how you pair the drives when RAIDing.

OS drive-wise, grab a small, cheap SATA SSDs for your OS (or pair of). Crucial, Micron, Samsung and Intel are all great choices. I personally have been using a pair of used Intel S3510 480GB for 3 years which had 20k Power-on hours but only 1TB of writes (they were decommissioned hot-spares), each were like $15. Since no data is actually stored on the drives other than the boot-time OS, replacing them is trivial (I have 2 extras strapped to the inside of the case, not plugged in, just in case one or both dies).

NVMe Storage-wise, put them in a striped mirror (stripe on top of a pair of mirrors). The TrueNAS GUI will walk you through the setup. That will give you a rock solid zpool to build on. After that you can create different ‘datasets’, which are a hybrid between a partition and a folder. They don’t have to have hard space quotas and they can have individual settings for on-the-fly compression, encryption, permissioning, etc… They can even act as a vdisk for VMs or as an iSCSI target (remote disk) for remote boxes. ZFS (and hence TrueNAS) has native remote differencial backups built into core architecture, so whenever you do end up setting up a remote backup at friend’s or family memeber’s house, sending scheduled encrypted differential backups off-site is trivial (literally set-it and forget-it). TrueNAS also has a UI page for cloud syncing to backup individual folders any cloud storage, use it - I use Backblaze and S3, but it works with everything.

GPU-wise - TrueNAS allows you to passthrough a GPU to a VM. In the UI just tell it you want to reserve it for passthrough, then when you create a VM, one of the options will ask you if you want to attach a GPU to the VM. If you want to be pro, you can even attach it to multiple VMs, though only one can use it at a time (at least until we get VFIO working for Intel Arc). On the VMs that have a GPU, you can install Sunshine, which will give you GPU-accelerated remote desktop access (~2ms latency locally). This is how I play games now, and it’s amazing (like 4k Starfield on my MacBook connected to an external display!)

Bulk storage-wise - Use the other two drives for now via onboard SATA in a mirrored pool. Whenever you want to expand storage, I recommend investing in a LSI/Adaptec HBA. They’re super cheap second hand and are rock solid - my go to is the 8-port AOC-S3008L-L8i (~$44USD).

On a side note, the P44s are fine drives, but whenever you do another upgrade, checkout the enterprise U.2 stuff like P4510/P5510 and CD6 - I prefer them because they don’t have the same performance drop-offs, proper PLP, super-high write endurance vs consumer drives, and aren’t that much more expensive. Just make sure your mobo can handle PCIe bifurcation, as the best way to use them is via a cheap 2xU.2 to x8 PCIe adapter card. My preference is always fewer and large enterprise NVMe drives vs many smaller consumer drives (i.e 2x4TB Intel P4510 vs 4x2TB Samsung 980Pro) as in sustained transfers they will wash out to the same.

Also if you go the TrueNAS route, checkout the Portainer setup guide to get docker running as well. There are times when a lightweight container makes more sense than a full blown VM.

2 Likes

Hey there!

Thanks for taking the time to answer.

Truth is I will build myself a TrueNAS in the future to hoard my data, I even have my Meshify 2 xl with all the trays waiting for me to fill it up.

This is a secondary PC I want to use as a dedicated streaming PC, I want it to boot into Windows and not have to deal with any passthrough for any of the tasks, I’ll have to consider GPU, capture card, cameras, etc. I’m building it to have less to worry about for those things. Dual booting into Linux is a bonus to leverage the GPU to play around during my data studies, I could live without it.

The data storage I plan to use it for is mostly my photos to have them in yet another place for backup purposes.

I don’t think I will be investing in enterprise level hardware for this pc, unless I find an interesting deal.

The motherboard is a LiveMixer z790, seems to have 4x Gen4x4 according to the diagram. 1 to the CPU, 3 to the chipset. That’s why I’m sticking to with a mirror and 2 separate drives, and not a striped mirror. Might even get a 4060 TI, if I find a good deal in BF, so 8 lanes of gen 5 is enough and I can use a gen5 NVMe attached to the CPU.

As I have the NVMes now and no SSDs, and I don’t want to do all over again in the future, I will probably go with my first option and just put an OS in each solo NVMe, use the mirrored NVMes for my ongoing work and the HDD for long term storage. Might get a couple of SSDs and add an extra mirrored storage at some point.

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.