If you’re gonna have CCDs and an IO die something’s got to connect them. Zen’s easily beating Arrow despite Intel’s massive process advantage on the links. ¯\_(ツ)_/¯
Other option’s to distribute the IO die across compute chiplets, which presumably would be lower latency and power for data on a chiplet’s DDR channels and PCIe lanes. But the die area costs more and it’s messier for off chiplet data. I’m 100% confident both AMD and Intel simmed out the tradeoffs and, at least for now, they’ve both decided on centralized IO.
I’m not in a position to second guess this but, from loosely similar problems I’ve worked on, it doesn’t surprise me.
A draft of the previous post distinguished bottom side vcache for that reason. It’s not a complete fix but the clock penalties are substantially smaller and the thermal disadvantage’s removed.
Yup, a bigger cache is slower in all the cases I can think of (though some processors use split L3 and L3.5 as a mitigation). With vcache the latency’s offset by bandwidth increase. I’ve some workloads that should be prone to the increased latency zone but occupation of the higher bandwidth zone looks likely to be common.
Typical workload YMMV. I think the odds are good enough to be pretty interested in a 9900X3D or 9950X3D as an evaluation vehicle despite the asymmetry but will have to see what the launch data looks like, along with pricing and availability.
Needs to be runnable by non-admin Windows users. Also there’s quite a strong preference for code that just works on whatever machine’s available instead of needing a bunch of fiddling.
Do you know why AMD still boast benefits to fluent and openfoam with 3d vcache epycs?
Also, I am wondering, would 8 channel DDR4 stlll beat out dual channel and quad channel DDR5? I am wondering if outside of the Price to performance argument for used DDR4 Eypc CPU, can the CPU keep up with large 30+ Million cell cases?
Also would love to try nvidia ominverse and modulus with CFD data.
@twin_savage, how many NUMA nodes do you have on the Xeon Max system (run lscpu to figure out)? Can you try setting -numasets to the number of NUMA nodes? Does it improve performance? This gives a good performance boost on AMD EPYC servers.
The Xeon Max system is exposed as 8 NUMA nodes to the OS since SNC4 is forced on by default on this motherboard.
The -numasets argument is more of a “userspace NUMA” (I might be making up this term) construct and kind of layers onto the existing system NUMA architecture. I’ve been meaning to write a summary of how all the different “layers” of NUMA interact but I don’t think I have a good enough grasp on it to explain it in a way that will make sense to everyone yet.
I actually had to use numactl to wrangle all the processes onto 1 processor (4 NUMA nodes as exposed by uefi) to get the best performance; the CFD-EM problem just isn’t big enough to benefit from parallelism vs extra latency dichotomy a second socket brings.
The -numasets argument wouldn’t be able to force all the processes onto a single CPU, it would just break them up “evenly” to all existing NUMA nodes if that makes sense.
I’m trying to run the 10GB and maybe 60GB sims on linux and have some problems. The runtime installed correctly, but when I open the simulations, they load and open, but after about half a second they close again. Have I installed something wrong here, or is it an error when trying to run the simulations?
I’m running openSUSE Leap 15.6, KDE on a framework 16 for reference. I installed both simulations using the konsole command and there were no issues.
It’s typically only the relatively simple or small problems that benefit from the 3d v-cache EPYCs, problems that have less interconnected-ness among the different mesh elements. Since openfoam is limited to only the finite volume method it’s problems aren’t densely connected. I’d imagine there are quite a few problems in ansys they would not run well on
It would depend on the problem being solved, the smaller problems would likely benefit more from the generally lower latency the 2-4 channel DDR5 system would have over the DDR4 system… not sure where exactly the line between “small” and “not small” is though.
I’d argue that the DDR4 EPYCs would not have any advantage in price to performance compared to newer systems due to the cost of electricity they’d use over a new system. I’m of course assuming that the system would get a lot of run time at full tilt solving problems.
EDIT: whoops, this reply had been sitting in my web browser and never actually posted.
Do the simulations close when you click the “Compute” button or are they closing all by themselves right after launching?
To rule out graphics problems you could run the simulation with the -3drend sw argument right after .sh of the file to force the software graphics renderer to run which typically has better compatibility with all the different Linux desktop environments than the default OpenGL renderer.
Although this doesn’t seem to be right for some reason. AMD 7840HS, 64GB ddr5 5600mhz, Opensuse leap 15.6 all stock. During the run it seemed like the cpu was limited to 3.8GHz, which is odd because on W11 it was going up to 5GHz. CPU usage was steady at 50% throughout because I didn’t switch off SMT, so I don’t know why the performance could have been as bad as it was
This performance makes sense for the most part since you are effectively using only one Zen 4 chiplet.
The only strange thing is the 3.8GHz clock speed, typically Linux is much better than Windows about scheduling and boosting. Maybe setting the CPU governor to performance in Opensuse could fix this?
@dahlia123’s tested a Ryzen AI Max+ 395 on the CFD-only benchmark, however this particular benchmark favors “raw compute” over memory bandwidth so it doesn’t highlight the 395’s strength (memory bandwidth). Despite this the 395’s efficiency per compute rivals Apple’s M series in this benchmark.
The 395 would do even better in the other two benchmark flavors because they are more memory bandwidth dependent but nobodies run them yet.
The last one is a 200+ GB ? So it wont run.
I may try the 60Gb if I can shedul it, but the OS/driver stack is not good on the unite I should have acces to (IT bloatware). I found some lacking performance compared to other. So results may not be usefull.
But it is still quite impressive, the 395 handle the comparison with the Xeon 8280 I used before, in small problems tested internaly.
Better than 13gen as well (i7 was better than Xeon scallable on small problems up to 8 thread).
And you can do all of that on battery !
EDIT: here we go : 07h54m54s for 60 Gb variant with powercord (of course) and performance power plan on W1124H2, on PARADISIO solver.
It is the pro variant with only AMD driver and chipset driver.
The CFD only was arround the 40 min on top of my head, will check again.
I’m wondering about strange CPU usage on my double AMD EPYC 9355 system (all memory slots filled).
When running a CFD models with nonisothermal flows (Default SAAMG solver), the CPU usage in the task manager only shows roughly 50% total usage. Is assume this is because of hyperthreading (i.e., is the actual usage ~ 100%).
Now, if I open a second COMSOL instance of the same model and run it simultaneously, in the task manager the CPU usage is around 100%.
Strangely, if I run batch sweep with 4 simultaneous jobs, for example, the CPU usage isn’t even 50%. I don’t really understand what’s the problem here.
Does anyone have experience on getting better performance from the batch sweeps?
The behavior you described is “normal” if you run comsol without any extra arguments during launch; the 50% usage is because using hyperthreaded cores is usually detrimental to performance so the application avoids it, but when two comsol instances are launched they don’t know eachother exist and break this rule.
WRT the batch’s 4 simultaneous jobs not running well, it sounds like memory thrashing might be occurring, what is your NUMA configuration?
Switching from Windows to Linux should get you ~20% performance bump right away, and let you tune system behavior better via numactl which opens the doors for massive performance increases if you have a particular complex system topology that Windows handles particularly bad.
Ah, that explains a lot about the two COMSOL instances - thanks for the explanation! Strange that the batch sweep doesn’t do the same, as it starts separate instances, but I suppose COMSOL batch prevents it from happening.
On my workstation I have 2 CPU sockets (32 cores each), each socket has 12 RAM sticks running on 6400 MT/s speed. (Not sure if that fully answers the question about NUMA configuration, apologies if vague!).
The specific model I am running takes about ~ 80 Gb Memory (yay for multigrid), and the system capacity is over 1000 Gb, hence Batch sweep would be very nice to get working efficiently.
Forgot to mention, but I also don’t start my COMSOL with any extra settings (AOCL, np, etc.). Do you think those could help?
Well that sounds like a very good sales speech for switching to Linux Unfortunately not sure if that is a possibility.
The way I probably should have asked that question is: how many sockets does Comsol say it’s running on if you look at the Log’s output (socket count should be near the top of the log).
My suspicion is if Comsol thinks the computer only has only 1 or 2 sockets (NUMA nodes) then running 4 jobs simultaneously might cause some memory thrashing and not utilize all the cores properly.
You can force Comsol to use more NUMA nodes which I think might help you. you can use the -numasets 4 launch argument to force 4 NUMA nodes; it might be worth it to try 8 and 16 as well.
…well it sort of is, Linux is still a pain when coming from the Windows world because it is so obtuse in many respects.
Unfortunately not sure if that is a possibility.