Bioinformatics workstation build - 4x m.2s on X570 with ZFS?

Hi! I’m new and I’m configuring a bioinformatics workstation.

I’ll be doing quite a bit of data analysis in R so plenty of ram and good single-threaded performance is desireable. I’ll also be doing some devops stuff testing an deploying R shiny apps and tools for a bioinformatic workflow manager called galaxy. Also I’d like to have a system with a good upgrade path.

I’ll probably be running plasma KDE neon and now it’s available in 20.04 zfs on root, so i’d like to have ECC memory.

In addition, this is leading me to some questions about how to layout my storage. I was thinking it would be nice to create a Zpool consisting of 4 m.2 NVMe SSDs and 2 HDDs. I’d mirror 2 of the ssds for my log/write cache, 2 for my l2arc/read cache and mirror the 2 HDDs for some redundancy. Feel free to tell me this is a crazy way to layout my zfs setup - I’m new to it.

Now i’m having trouble figuring out if this is an achievable config with a 3800X and an 570 chipset - can I get 4 PCIe lanes to each of my 4 M.2s and have enough left over for my GPU?

My current thinking is this:

  • Mother board: gigabyte x570 aorus master
  • CPU: Ryzen 3800X
  • GPU: RX 580
  • Storage:
    • 2x 10Tb seagate ironwolf pros
    • 4x 1Tb WD blue NVMe SSD m.2 (PCIe gen 3)
    • M.2 card?
  • RAM 4x 32Gb DIMMs (Crucial CT32G4RFD4266 32 GB DDR4 2666 MT/s)

My understanding is that there are 24 lanes on the CPU, 4 are reserved for the chipset but that the 570 chipset can somehow wangle the equivalent of 16 lanes out of these 4 so I should be able to run 2 of the m.2 on the built-in slots that go via the chipset, with some chipset lanes for my SATA drives. Then I can use 8 of the remaining 20 lanes on to run the other 2 m.2s on an expansion card (I could do with a recommendation of a suitable one) with my GPU at x8 instead of x16.

I’m not entirely clear if the gigabyte x570 aorus master supports ECC or not, I’m also unsure of whether or not it matters for this support if I get registered or unregistered DIMMS - I can’t seem to find any unregistered 32Gb DIMMS anywhere. Should the ones I found from Crucial (above) work?
I definitely want 128Gb of ram as on top of using R, I have ram hungry browsing habits and ZFS L1arc will want some too!

Thanks in advance for any advice.

1 Like

I’m afraid you’ll be defeated by main board limitations, as it’s likely that using one or more SATA connectors will prohibit the use of one or more NVMe slots. Check the manual to be sure. Also, as you specified single-thread performance as a requirement, the AMD platform might not be your best choice. Intel still holds the crown here (but only just!) so you may want to investigate that route as well.

IIRC the X570 chipset allows for 4x4x4x4 bifurcation of certain PCIe slots, so you may want to use a 4x NVMe to PCIe adapter card to connect the NVMe drives to your proc. I don’t know if Intel offers the same functionality on any of their chipsets, although it’s fairly unlikely they don’t.

As for the ECC RAM, I don’t think the DDR4/AM4 platform offers memory in suitable sizes for your use case. Consider going Threadripper or, depending on how strong your requirements are, the Epyc platform. In either case, expect at least another digit to your total build sum :stuck_out_tongue:

This seems more like a job for Threadripper system.

1 of the nvme slots on the board is direct x4 lanes to the cpu.
But the 2 other will run though the chipset and fight over the bandwidth.

And 128 GB on x570 is gonna be a stretch, just finding a kit is gonna be hard.

Threadripper 3000 supports fast memory in the 128 GB capacity.

Biforcate the 4 m.2s on the gpu slot and run the gpu through the chipset

2 Likes

Infact, it might make sense to invest in an AIC ssd storage device like this https://www.gigabyte.com/us/Solid-State-Drive/AORUS-Gen4-AIC-Adaptor#kf

While Zen 2 IPC is technically higher, since the Intel counterparts can clock higher, you might benefit from team Intel this time around. Although Zen 3 might make up for it with higher IPC and higher single core clocks, but only with higher end parts.

How do you run the GPU on the Chipset - would that just be a bios setting?

You plug it into a slot that is connected to the chipset

No, you just choose the pcie slot that is known to use the chipset- refer to the manual.

Actually something like the 3300X or 3600XT with super good memory might make sense for your work (while you wait for Zen 3).

I’m a little too budget constrained for a thread-ripper system so I may have to lower my aspirations for the build - just trying to see how far I can push things in my limits.

ak I can’t post links yet - I found these 32Gb ECC dims on amazon: Crucial CT32G4RFD4266 32 GB DDR4 2666 MT/s any reason why 4 of these wouldn’t work?

That is registered memory. Only unregistered unbuffered memory is supported on X570.

What is your overall budget for the system?

~3000 Euros

Thanks - that was unclear to me. I can’t find and unregistered 32Gb DIMMs, I think I may have to lower my sights to 64Gb of ram this should be adaquate for most of what I do and I can tell ZFS to use less. I’ll see if I can find any 16Gb unregistered ECC DIMMs.

So if I get a 4x m.2 card which I put it in a 16x slot and put my GPU in the slot connected to the chipset I can probably manage my PCIe lanes issue.

If I lower my sights to 64Gb of RAM and get some 16Gb unbuffered ECC DIMMs possibly: “Crucial CT16G4WFD8266 16GB DDR4 2666MHz ECC Memory Module CT16G4WFD8266” ?

I might be able to build something like this?

Yeah, TRX40 is damn expensive.
The Gigabyte x570 aorus master you have listed seems to support Bifurcation of the X16 slot into x4x4x4x4, atleast from reading the manual.
So if you get the adaptor @CybeastRaystriker linked, or one from Asus LINK, you can populate the first pci-e slot with your 4 nvme drives and get full bandwidth for them.

And then run the gpu through the the chipset. The lowest PCI-E slot runs through the chipset at X4 speed.

There aren’t any 32 GB dimm ECC modules on the motherboards QVL. So you might have to try out your luck there.
But they do exist:
https://memory.net/product/m391a4g43mb1-ctd-samsung-1x-32gb-ddr4-2666-ecc-udimm-pc4-21300v-e-dual-rank-x8-module/

But it’s not certain if it’ll work at the rated speeds or in 4x32 GB on your motherboard. That’s a gamble.

The CT16G4WFD8266 module is on the motherboards QVL, so it should work.

hmm so I’d be running my GPU with only a 4x connection - I imagine that’s OK if i’m not using it for much more than driving 2-3 1440p monitors but might be an issue if I ever want to do anything else with it - I’ll have to give that some thought. Thanks for the help!

Just running monitors won’t be an issue.
But yeah, anything GPU intensive/gaming will likely result in some bottle necking.

1 Like

Are you planning on doing any sort of machine learning/CUDA accelerated workflows? It could potentially be a factor then, although depending specifically on the type of bioinformatics you’re doing, that may not be necessary.

If it’s just R data analysis, I wouldn’t worry about any GPU impact.

Would it be possible to put the GPU in and M.2 card in 2 top slots and give them 8 lanes each, then put the other 2 m.2s on the chipset like I originally thought? I’d sacrifice some speed on the storage but If I have any more GPU intensive workloads It’d at least have have 8 lanes to the GPU. I could use the chipset drives for my L2arc then I’d still have pretty fast access to key stuff because of L1arc.