Threadripper Build for Relion/Cryo-EM Protein Resolving

Hardware Build a PC
1

Hi guys,

I’m a grad student in a lab that is using Cryo-EM to essentially take pictures of large proteins and then using those pictures, piece together a 3D structure. The software we are using to do that is Relion, and it will be running on CentOS. The way it works is we basically flash freeze purified proteins onto a laser grid, and then film them with an electron microscope. Then we have to piece together the 2D averages that we took pictures of into our 3D model, and that part is very very resource intensive. Our budget is up to $6000 or so.

Looking at the Relion benchmarks, it looks like a Threadripper 2950X should be enough with its 16 cores and 32 threads, to feed the GPUs, since there’s a benchmark of a 1950X feeding two Voltas, and benching 2:43, which is the fastest time on the benchmarks page. So it seems like a 2950X should prevent any kind of CPU bottlenecking.

Similarly, that system is running 128GB RAM, so I think we will be fine with that. How much of an impact will the RAM speed have on the CPU performance? Also, if the motherboard supports quad channel, will any RAM work in quad channel, or does it have to be quad channel specific memory?

We would like to go the 4 GPU route, so I need to find a motherboard that supports 4 GPUs, preferably at full speeds. Does such a thing exist? Everything I’m finding says 2xPCI-E 16 and 2xPCI-E 8, when running 4. Is there enough bandwidth to run 4x16?

As far as the GPUs go, I was thinking 2080s. They don’t support more than 2-Way SLI, but I think I can run them independently for Relion, they will be doing compute, not video rendering, so I want 4 of them. I was thinking MSI-VENTUS, if anyone has any experience with those.

Additionally, I’ll be running probably a 1TB SSD and a pair of 4 TB HDDs. I don’t know who makes a decent hard drive nowadays. WD Red Pro’s?

As for power, it will go through spurts of heavy use most likely, a day or two under constant load, followed by weeks of nothing, etc. I’m thinking 1500W, and so the EVGA SuperNOVA 1600 caught my eye. I’m just trying to figure out the power connectors. The 2080s need an 8+6 (at least teh ones I was looking at do) and so I was looking at the SuperNOVA. It has 9 8-pin VGA plugs, which I assume should be enough. Does each plug go to an 8+8 configuration, or would I need two plugs per 2080?

Finally, for a case and cooling, well, it’s going to be in a separate room so I’m not worried about noise. I was thinking something like a Define XL R2, but the truth is I’m not sure about how to manage the heat from those 4 cards. Does anyone have any case recommendations for keeping them cool? And is the 2950X difficult to keep cool? Would a simple fan be good enough for it, since it won’t be overclocked?

Here is my PC Part Picker list, if anyone wanted to take a quick gander and let me know if something is really off:
https://pcpartpicker.com/user/ccgauvin94/saved/#view=ZW6nHx

And if anyone (@wendell maybe?) is feeling ambitious and wanted to try booting up a TR machine with a couple of video cards and seeing how it handled the Relion benchmark, that would be a huge, huge help!

1 Like
2

For home use and semi professional, it looks good. As you got essentially a 1300W heater there, I would go for airflow focused cases.

The setting at a uni, I would go with brands known for professional equipment or directly buy a server from Supermicro, Dell or HPE. The later has the benefit of OEM support.

This gives you a $2800 platform with parts you can justify should anyone ask questions:

Part Price
(in $)		 Note
Epyc 7351P 800 The “P” at the end of the name indicates only one CPU per machine. The 7301 (200MHz slower, $100 more expensive) supports 2 CPUs
SuperMicro MBD-H11SSL-NC-O 460 Server Motherboard
Micron MTA9ASF1G72PZ-2G6D1 100/piece 8GB ECC RDIMM, as per supportlist, 8x
WD Ultrastar 8TB 277/piece Server HDD, 2x
Intel D3-S4510 480GB 140 Is on support list, not sure why you need an SSD

While it would be nice to house this in a server case, something like the Supermicro SC747TQ-R1620B would set you back another $1200 it brings redundant 1620W PSUs to the table.

Should you happen to get more funding:
Supermicro Sys-1029CQ-TRT can be configured to house 4 really beefy GPUs.

1 Like
3

Thank you for the reply! I’m torn because we’re finishing up a grant and there’s only a certain amount left in the bank account that must be spent by June 30th without guarantee of what future funding will look like. So then we’re stuck with - do we build the best complete system we can now, or do we build a half of a system and hope to finish/upgrade it in the future?

I do see the argument for the enterprise gear, and I think it may be worthwhile putting together a list for it and offering the PI both possibilities. Buying a dual-socket board with 4 PCI-E slots that we could add to in the future could be a really good middle ground. However, my concerns with going that route are:

  1. I don’t know how the lower clock speeds and memory speeds of the comparable EPYC line will handle the performance needed by the sofftware, and I don’t currently possess a computer capable of running the software to get an idea of its performance. All I can go off of is the benchmarks, and systems build by other suppliers, none of which are using EPYC.

  2. I don’t have much knowledge of web servers, compatibility, brands, etc, in the server space. The only server I’ve ever owned was an original Apple XServe with PowerPC processors…

The two problems that jump out at me with the H11SSL is that:

  1. 3 PCI-E 3.0x16 slots, not 4
  2. Only 64GB RAM in that configuration, which is bandied about as the bare minimum for working with Relion.

Would a normal PSU and case work with that motherboard?

ASrock does make this board, which supports 4x PCI-E 16, although I have no idea if it’s decent hardware or not, and again, defeats the purpose of going enterprise grade.

For reference, SBGrid is very well known and trusted in the structural biology community, and here is their Cryo-EM config page:

https://www.thinkmate.com/system/sbgrid-cryoem-workstation

They are using a SuperMicro dual socket board with 4 PCI-E, but the added cost of going that directions is just too high, I’d like to stick to AMD if possible and put more money into the video cards instead, because that’s where the real performance gains are.

4

Back when dinosaurs walked the earth and I worked in academic computing getting projects like this sponsored by vendors or local companies was the big thing.

Your time frame sounds short, but worth a shot. Aim much higher on the machine, in the tens of thousands range, and see if you can talk HPE or Dell or Sun into donating it or selling for 90% discount. All three will have quad GPU configurations more or less off the shelf.

1 Like
5

Reminder that Epyc has twice the memory controller count compared to Threadripper.

From what I saw, Relion is super slow on CPU-only. If you really need more CPU performance to keep the GPUs fed, the Epyc 7401P offers 24 cores @ 3GHz for $1150.

  1. That is indeed a problem
  2. Would 8x 32GB of these fix that? Link

Yes, a normal ATX PSU would work.

It would fit your use case of 4 GPUs and is an Epyc board. I would recommend to stay with parts listed as supported (especially memory).

1 Like
6

OK great, thanks so much for the help. I think you’ve sold me on EPYC. Looking at the Ward lab at Scripps research institute’s benchmarks, they have 256GB RAM in their 4x2080. Also the higher availability of ECC memory for Epyc (on QVLs anyways) would be nice.

So with the 8 channel memory, I’m wondering if we could run 4x32 and then add another 4x32 at a later point in time, or does the memory have to run in 8 channel?

This is on the Asrock QVL:

It would enable us to start at 128GB and if we want to move to higher resolution images, double the memory to 256 by just filling the other 4 DIMMS.

My thought process now is:

  • 7351P
  • Asrock EPYCD8-2T
  • 4x32GB Crucial ECC (from QVL)
  • 4xRTX 2080 Ventus
  • 2x4TB WD drives of some sort
  • 500GB SSD (we’d like the SSD for working in PyMOL and Chimera)
  • Some kind of ATX or E-ATX case
  • ~1500W PSU (I don’t think redundant is super high on our priority list so we’ll stick with one for now).
  • Some kind of Noctua or whatever cooler
  • Bunch of case fans

Going as budget as I can on all the other parts, that puts us at $5830.95. I’m going to do a bit more research but I think this is probably going to be what I put forward to the PI. Thanks for the help, again!

This thought did cross my mind, although I’m not really sure who I would contact about it. The other problem is we don’t have any projects right now, our Cryo-EM is getting in early next year but we want to have a workflow/process down so that we can write grants right for when the microscope is installed. So for now we’re just doing some virus proteins we have purified and going to other labs in Utah and Oregon to actually get the work done. Since we don’t have some really flashy project at the moment, I’m not sure any companies would be willing to sponsor us at the moment, but it’s something I will probably look into in the future.

7

More channels occupied, the more throughput.
But running 4x 32 should be totaly possible (would have to consult the manual).

Looks good to me.