AI Disclosure:
In life and work I broadly don’t use Gen AI, the one exception I allow myself is using it as a rubber duck when I’m stuck, or to show me where I could improve code after I wrote it. I write code myself to learn programming, but also because I need to ensure scientific accuracy of what my code does. Though I guess now the Google search engine is AI now so I also allow myself to google stuff.
TOC
1- Background you can skip
2- Mission statement
3- Budget
4- Where I am
5- Where I need guidance
Some background
Hello! I’m a chemistry student. Let me explain what I’m doing here real quick.
Ramblings about the path which brought me here
I made My first post here now 5 years ago (keelah…) when I was a penniless Master’s student doing Density Functional Theory (DFT) molecular modelling in the basement of a university with 4 recycled first-gen core i7’s. I eventually learned The Linux and using SLURM to use DRAC clusters which was an absolute boon. Then I converted the i7’s into a small SLURM-directed cluster because I was allowed to go to the deserted Uni basement during the worst of COVID lockdown. Still, I had stars in my eyes at the concept of benchmarking EPYC CPUs for the kind of tasks I did back then.
Academic stuff happened. I didn’t get the time or energy to define a repeatable workload and write a guide/walkthrough for this place’s great patron to follow. (I’m sorry and maybe we can still get something going at some point)
Now I’m a slightly less penniless PhD student at a different university who tries to keep nanoparticles stable in the lab and then get data from them. I will eventually need to train ML models (something like QSARs probably) on those nanos when I have enough/good quality data, early next year. To do this, my supervisor bought a 4070Ti for me to put in a research group-shared desktop, and renewed my free access to DRAC. Shared computers though have the characteristic of being shared, and in this case, with people who don’t know better and break stuff sometimes. DRAC also have limited numbers of GPU-endowed nodes and can’t fully keep up with national demand. My personal gaming 3070 may or may not be powerful enough for these jobs, but I’d like to keep it available for the occasional gaming session.
More importantly, I’m starting to see the eventual end of my PhD at the far-away sunlit horizon of hope.
I have good ideas for a research project I’m starting in my sparse spare time that will mostly happen after my graduation, as a so-called independent investigator. That entails running DFT molecular calculations, and then if needed, a touch of light machine learning.
Since I have agreed to DRAC’s terms of service for free academic use, I’m definitely not supposed to use their weakest clusters during low use months for personal compute every now and then ( ͡° ͜ʖ ͡°). It remains that when I’m no longer a student my access will be terminated and I’ll still need to run these calculations.
Of importance: Density Functional Theory (DFT) relies on FP64.
Even just approximating Schrodinger’s equation requires a lot of sig figs - who’d’a thunk. Of note is that most DFT packages (including my go-to ORCA) aren’t GPU-accelerated.
Mission statement
-
As the title indicates, obtain some HPC or at least “MPC” (Medium-Performance Computing
) capacity of my own starting from scratch, by investing what I can and setting up a compute environment that’s comfortable to do science on, and learn fun stuff along the way; -
Define and test an open repeatable DFT benchmark to compare hardware (and ideally software*) fairly and share my findings on performance scaling for every CPU I can get my hands on;
* It is known that comparing DFT packages is inherently unfair (so much so that Gaussian is unfamous for banning people who compare it to other software), as level of theory implementations, and most importantly, parameters like integration grid, differ.
-
Use the HPC capacity to slowly make a brick of my own to add to the wall of knowledge of humans. If my idea works I’ll publish with open access or post the results somewhere
-
Critically, stay sane while I make a big push in the wet lab 'till next year
Budget
I started with $500 CAD - that is noooot a lot, but I’m a student. Fortunately in this economy, the levels of theory I usually use don’t need loads of memory! Usually a few GBs per core; more would allow me to use even more accurate methods though.
I purchased the following:
- 1x 16-core Xeon E7-4850v4
- 3x 14-core Xeon E5-2680v4
For a total of $59 CAD.
About 440$ remains. Maybe I’ll expand depending on what I think I can get (considering a V100, see the very end of this post).
Where I am
In terms of skill I’m closer to mortals than to the greybeards sysadmins whom I respect. I know intermediate Python and basic Bash. I also know basic networking. But I crave looking at a box that uses electrons to calculate electrons, and thinking "look at you, working so hard
".
In terms of hardware:
- I already have a gigabit switch, an old 3com 3812
- If needed, I can have the old first-gen core i7’s for free from the now-dismantled cluster I build at my former Uni:
- Will receive the 4 Xeons listed above soon
- There’s a good chance I can have 2x Dell Precision Tower 5810 with one 8-core E5-1660v3 each, and 32GB ECC DDR4 each, all for free!

I’m thinking of getting a dual socket system to put two 2680v4’s for 28 cores and quad channel DDR4 (is hexachannel a thing?), and put the remaining 2 Xeons (one 2680 and one 4850) in the Dell workstations. No idea what to do with the 1660v3’s. Only 8 cores but higher clocks.
Guidance needed
I have options for what to buy: cheap old Supermicro server (is 1U good for HPC? what about the weird vertical slotted nodes that you put in a bigger 3U case thing?) but that will be noisy for an apartment. Other options are to get a brand new Huananzhi board and find a cheap case, or maybe find a dual socket workstation (like a Z800 or similar) that would at least already have a power supply included. If I go the noisy server route I could maybe ask my old Uni if I can have that room in their basement again and put the machines there.
Any opinions on hardware? How are thermals on these options?
. . .
Finally, software. Aaaah, software. I’m still haunted by the memory (see ramblings in the Background section) of wasting 6 days going around in circles wondering why something didn’t work (don’t remember if it was network provisioning or SLURM itself), only to discover I needed some sort of time synchronizing thing. Don’t want that experience again. Is that the old fashion way of doing things? Any modern piece of software I should be aware of? IIRC the OpenHPC stack I used 5 years ago was; CentOS 8 (I finished the cluster 3 months before IBM said they’d pull the plug, again I do not want that experience again bc it contributed to the dismantling of that system by old Uni) and then [checks notes] nagios(???) + warewulf + SLURM. I’m open to any comments or suggestions. Hopefully doing it a second time I’ll suffer slightly less but, oh dear, I have good and bad memories of the setup process from 5 years ago. Thank god I was wise enough not to connect that thing to the internet ![]()
. . .
Well I just checked the multithread bench scores and oops
| CPU | Nb cores | Score |
|---|---|---|
| R7 5800X Personal gaming PC | 8 | 27654 |
| Xeon E5-2680 v4 | 14 | 17075 |
| Xeon E5-1660 v3 from workstations | 8 | 12575 |
| Dual 2680 v4 (hypothetical) | 28 | 35150 |
The argument for a dual socket system makes sense, but it might be wise to buy a fourth 2680v4 make 2 dual socket systems instead? hmmmm. Well, in any case I’ll probably buy more memory down the road to quad channel all the things, and make sure my dual socket system has at least 64GB of memory.
. . .
Some thinking out loud
When I have things set up, how insane would it be to email NextSilicon and ask them to borrow a Maverick2 acceleration card that supposedly adapts to the workload in real time to see if that can magically improve DFT calculations performance. I have no idea what that entails; probably modifying scientific code that’s obscure to me in ways I’ll never understand.
There’s also the MI300A which as an APU sounds like it would have massive potential for DFT (all the FP64 FLOPS!); some of the code would run on the CPU part, and what can be parallelized goes to the GPU part of the chip (I guess at Infinity Fabric speeds???). Again, I do not have the level of competence to understand what would need to be done with the packages I’m a user of, to make it work. AMD says with ROCm you “just have to tag a loop!” but all DFT code I’ve ever used is closed source anyway.
Any GPU (better than my 3070) would mean I could train ML models myself, which is enticing. Though for my PhD I can use DRAC for free and just get in the queue along with all of Canada’s scientists using and training AI models. For DFT however, maybe I could get a copy of Gaussian from the Seven Seas, which I believe can use up to a 16GB V100, or GPU4PySCF. Might be out of my budget range though, they’re going up in price as of writing these lines.



