Poor MATLAB Performance on Threadripper 5995WX System (Win10)

[ProAtOverthinking]

Just to quickly answer some questions that came up while I was away:

@alpha754293:
The Threadripper Workstation has Windows 10 installed. Changing operating systems isn’t really an option.
Yes, my account shows up as an Administrator on that system.

I am currently checking with my IT department about the Parallel server license. I followed the instructions from your powerpoint (thanks for sharing that!) and I am facing some errors when trying to spin up the job scheduler. Probably its related to some kind of licensing issue.

@twin_savage I did read up about the new AOCL libraries supposedly improving performance on AMD systems, and Mathworks has instructions on switching over from Intel MKL to AMD AOCL libraries. I have AOCL 4.1.0 setup and running on my MATLAB install, and haven’t noticed any performance uplifts from that unfortunately. Do report back if your benchmarks show a significant performance difference.

[twin_savage]

It would appear the new AOCL 4.1 libraries are inferior to MKL 2022 libraries even when running on one of the brand new threadripper machines:

Dual socket EPYC 9654 + Windows Workstation benchmarks on COMSOL, etc

7960X is up and running, with first pass tune up, prices in China were crazy so I ordered on new egg and got a friend to carry in form Hong Kong… DDR 6400 4x32G DR CL32-38-32 ADIA Bandwidth 183G Read, 69ns Latency (not sure why so high…) Manual OC 5250, 5100, 5150, 5200 for each CCD respectively 1.225V, passes prime95 320k FFT, which is my perso…

AOCL is behind MKL in performance by about 10% in this workload. It is wild that Intel can write demonstrably more efficient libraries for AMD hardware than AMD can; surely AMD has more insight into their own hardware than Intel does.

[alpha754293]

@ProAtOverthinking

I followed the instructions from your powerpoint (thanks for sharing that!) and I am facing some errors when trying to spin up the job scheduler. Probably its related to some kind of licensing issue.

No problem. You’re welcome.

It will depend on the nature of the error.

Sometimes, if you reach out to Mathworks, they’ll assign a support engineer to your issue and they might be able to help you decipher the error message and try and help you figure out whether that’s a licensing issue or if it is something else.

[ProAtOverthinking]

@alpha754293 Seems like MATLAB Parallel Server was never installed on my computer in the first place. Getting my organization’s IT to fix that.

@twin_savage Absolutely unbelievable. AMD is on the 5th iteration of Threadripper now with the 7000 series and having their in-house libraries STILL be slower than Intel’s is wild. Not to mention all the weirdness with NUMA settings & Coreprio’s DLM & NUMA dissociater going on.

[ProAtOverthinking]

@alpha754293 I got Parallel Server installed on the workstation. Running some initial tests on it right now, so far the CPU usage reported by Windows Task Manager seems to be a lot more consistent. Its mostly at 100% with drops down to ~80% once every few seconds (which corresponds to a new iteration in my code, so its expected behavior). I will try to run some more variations of tests on it, such as playing around with NUMA configurations, AOCL vs MKL, CorePrio settings etc. Will keep everyone posted.

[alpha754293]

Yeah – let us know how your testing goes.

It would appear that at least with MATLAB R2023a, it says in the Parallel Settings (IIRC) that you can actually select the default parallel execution environment to be process vs. threads (vs. GPU if you have a support CUDA GPU), and as far as I can tell, it defaults to processes.

Not sure if it makes a difference as I haven’t tested this out yet (vs. the MATLAB Parallel Server, as I used to use the MATLAB parallel server on my cluster). The “Process” option, appears to be new (don’t know if it is new as of R2023a or if they brought this in in an earlier version, and I’ve still been using an older version.)

Either way, you can run tests on your code to see which option works best for you.

With the Threadripper, there is a potential that each CCX may be treated as its own “cluster node” of sorts, so dividing the work up in accordance to that may yield benefits.

(I don’t have any experience with Threadripper neither, so you’ll have to run the tests to find out what would be the optimal configuration for your code.)

Thank you.

[ProAtOverthinking]

Reporting some initial tests:

The MJS profile enabled via the Admin center was worth an improvement of anywhere from 25-35% depending on the run. Then I enabled CorePrio and its NUMA Dissociater, and that got me another 20% improvement. So far so good, but still nowhere close to where this system should be performing. The benchmarks I’m running should ideally take around 2 hours, but are taking nearly 10-12 hours even after the improvements I outlined. Sorry about being vague, but the code I am running is proprietary and I can’t share much detail into what it actually does or share any files from it. @alpha754293, my code uses some functions that are not compatible with the “Threads” environment, so I am on the “Processes” one currently.

One interesting thing that I noted is that this performance degradation becomes worse with longer tests. I can see some pretty good performance at the beginning of the optimization run, but performance starts degrading severely around an hour into the test. HWinfo64 reports CPU temperatures are fine (maxing out at around 71-72C), but I read this thread ([SOLVED] 5995wx on ASUS WRX80E-SAGE with 1TB memory installed goes into an infinite Q-Code loop - #73 by wendell) and I’m kind of getting suspicious that it could be the memory starting to get too hot and degrading performance. Unfortunately, I don’t think the Precision 7865 is setup to sense DIMM temperatures as I am not seeing any of that information on HWinfo64 or Hardware Monitor. I am not sure how well the info from the other thread would correlate since I am running just 8x32 GB and can’t see the DIMM temperatures via software.

Basically, the performance breakdown across different runs of my optimization code is kind of looking like this:

1000 generation optimization - 400 - 450 seconds
5000 generation optimization - 2k - 3k seconds
10000 generation optimization - 40k - 60k seconds (!!)

If there was no throttling anywhere, the time taken for completion should be linearly proportional to the number of generations. The “ideal” number to hit for the 10000 generation optimization should be around 4500 seconds or less. Funny thing is, with the fixes via the Admin Center and CorePrio, the CPU usage does consistently hit 100% and stay there as expected, which is why I am suspecting the memory throttling now.

Any thoughts?

[alpha754293]

Thank you for the update. I appreciate it. And I totally understand about running proprietary code/software on proprietary data that you can’t really describe in any sort of a public forum.

Not a problem.

(I’m used to that myself.)

At least that the MJS IS yielding a benefit for you, so that’s encouraging.

As for the hypothesis re: memory temps without any direct or indirect insight to said memory temps – the simple, jerry-rig test is to lay the system down on its side and then put two 40 mm or 60 mm fans on it (or whatever you have available) and see if that helps improve the performance.

If it does – then that’s at least a PART of the answer.

That’s the fastest way to test it if you don’t have like a thermal camera available/accessible if the software doesn’t/can’t report the memory temps.

The other thing that you might also test (just for an A-to-B comparison) is try running the system with the side panel off (assuming that the Dell’s chassis intrusion sensor does not prohibit you from starting the system when the side cover is off. I vaguely remember some older systems will NOT let you power on the system if the cover is off, so I am not sure if the Dell would be like that or not).

But again – a relatively “short and simple” test that you can run.

And if a 40 mm or 60 mm fan on the memory doesn’t work, then you can also find a USB powered desk fan and just point it at the memory and see if that helps.

It is entirely possible that it can be a part of the issue.

I don’t have personal experience with Threadrippers so unfortunately, I can’t provide any more direct nor specific guidance beyond this.

But give that a shot and see what works and what doesn’t for you.

P.S.
I forget if there are like memory management tools that you can try and see if you can monitor what’s going on with the RAM (i.e. how much data is being written to it vs. how much data is being read from it and at what rate).

If there are tools like that available, maybe that’s another option as well.

(similar to iotop, but that’s for hard drives/SSDs in Linux – but like iotop for RAM.)

I don’t know of what it might be called for Windows, but if you can get some information like that – that might also help as well.

The other thing too is given what appears to be exponential growth in time where time(O(n^alpha)) – I don’t know if you’re able to run multiple tests, plot it, and then run a polynominal, exponential, or logarithmic regression through it to calculate alpha.

I know, for example, that for the Gauss linear algebra solver with partial pivoting (e.g. MATLAB’s left divide), the time that it takes IIRC, is something like O(n^3).

So…without knowing your code nor your data set – just looking at time vs. # of generations, and doing a super quick plot of that – it looks like that it has an exponential curve to it.

So whether the E(X) = O(n^~1) (linear or nearly/quasi-linear) – again – without knowing your code nor data – I would inquire about what are the conditions where E(X) = O(n^~1) is true.