Thank you for the review, it is great.
Any chance of a comparison with a higher core count version(s)?
My use case is hybrid inference, I’m trying to find out what is the optimal core count to maximize both prompt processing and token generation performance.
When are retail boxes shipping in the U.S.? All I can see are backorders from a few different channels.
I am looking as to when there maybe a release date in Canada? no resellers that I can find as of now.
I’ve been testing the code-aster docker container from simvia, on github simvia-tech/cave, and find that mpi ranks accelerate integrations, but open mp threads accelerate factorizations.. I think that the true bench, using the default MUMPS solver, will be case dependent. But there are interlocking considerations, comparing the granite rapids to the zen architecture. With large simulations that are bandwidth limited, on the two architectures, how do you divide the simulation across ranks, and on Granite Rapids is it easier to run a large simulation without managing locking ranks to chiplets, or constraining ranks to chiplets ? For small simulations, how do the different architectures enable the strategy of running from cache, and how should the problem be divided to achieve that, and does the in cache simulation performance support running lots of small test simulations very quickly on either architecture ? how small is small on each architecture ? Code-aster gives detailed logs of memory and cpu usage for each step of a simulation.
I have spent more time with the Xeon 658X + ASUS W890 SAGE SE motherboard, and I am still unable to ever see boost beyond 4.3GHz with the “Optimized Default” BIOS settings. If I check /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq I do see the majority of cores list 4700000 but a few do hit 4900000. Nonetheless, I never see either of these clock rates reached in usage, even if I pin stress-ng to a single core with an otherwise idle system.
I did try enabling the water-cooled overclock preset, and I then saw boost up to the advertised 4.9GHz. However, my system did crash while this was enabled while effectively idling and all CPU and DIMM temperatures were ~50C.
Here are some graphs during the time I had the water-cooled OC preset enabled, the last datapoints on the graph are when it crashed:
Currently, I am suspecting it may be due to using 4x DDR5-5600 DIMMs and maybe the overclocking preset is tuned for higher-spec’d RAM. Nothing looks particularly suspicious at the time it crashed (although something triggered some CPU utilization right before the crash).
That said, I did recently notice that I am receiving AER errors infrequently from my GPUs - which is more concerning than the CPU freq scaling. I did spend a little bit of time testing these GPUs (H200 NVLs) in a WRX90 machine before putting them in this W890 board, and they reported no errors there. The GPUs are plugged directly into slots 1 and 3.
No Memory errors.
PCIe AER events:
1 2026-04-30 13:58:16 +0900 0000:92:02.0 Corrected error: Replay Timer Timeout
2 2026-04-30 13:58:16 +0900 0000:93:00.0 Corrected error: Receiver Error, Bad TLP
3 2026-04-30 23:03:16 +0900 0000:69:00.0 Corrected error: Receiver Error
4 2026-04-30 23:03:32 +0900 0000:69:00.0 Corrected error: Receiver Error
5 2026-04-30 23:09:54 +0900 0000:69:00.0 Corrected error: Receiver Error
6 2026-05-04 14:56:07 +0900 0000:8d:02.0 Corrected error: Replay Timer Timeout
7 2026-05-04 14:56:07 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP
8 2026-05-06 11:10:33 +0900 0000:8d:02.0 Corrected error: Replay Timer Timeout
9 2026-05-06 11:10:33 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP
10 2026-05-06 15:00:05 +0900 0000:61:02.0 Corrected error: Replay Timer Timeout
11 2026-05-06 15:00:05 +0900 0000:62:00.0 Corrected error: Receiver Error, Bad TLP
12 2026-05-06 15:34:26 +0900 0000:8d:02.0 Corrected error: Replay Timer Timeout
13 2026-05-06 15:34:26 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP
14 2026-05-06 15:46:41 +0900 0000:61:02.0 Corrected error: Replay Timer Timeout
15 2026-05-06 15:46:41 +0900 0000:62:00.0 Corrected error: Receiver Error, Bad TLP, Advisory Non-Fatal
16 2026-05-06 16:17:55 +0900 0000:61:02.0 Corrected error: Replay Timer Timeout
17 2026-05-06 16:17:55 +0900 0000:62:00.0 Corrected error: Receiver Error, Bad TLP, Advisory Non-Fatal
18 2026-05-06 16:20:14 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP, Advisory Non-Fatal
19 2026-05-06 19:10:55 +0900 0000:61:02.0 Corrected error: Replay Timer Timeout
20 2026-05-06 19:10:55 +0900 0000:62:00.0 Corrected error: Receiver Error, Bad TLP, Advisory Non-Fatal
21 2026-05-07 06:31:23 +0900 0000:8d:02.0 Corrected error: Replay Timer Timeout
22 2026-05-07 06:31:23 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP
23 2026-05-07 18:19:54 +0900 0000:8d:02.0 Corrected error: Replay Timer Timeout
24 2026-05-07 18:19:54 +0900 0000:8e:00.0 Corrected error: Receiver Error, Bad TLP
No Extlog errors.
No MCE errors.
I did install the most recent BIOS on 2026-05-01, which is why the PCIe identifiers shifted around, but I saw some errors before that BIOS update and many afterwards. I haven’t noticed any actual problems other than these AER error logs, and NVIDIA’s DCGM stat publisher has not reported any replays under DCGM_FI_DEV_PCIE_REPLAY_COUNTER.
Not sure if this is just teething issues with a new platform that will be fixed with firmware updates, or if maybe I have a defective board or CPU.
whats the copy-paste I can do on my board to see if I get similar behavior? I have NOT udpated my bios so I could try before/after to see if we’re dealing with a bios regression.
and what’s doing the graphing for you
Would the CPU clock be a first order factor for the p2p? I’d imagine the bottlenecks here would be in the uncore, are there settings that affect the power states / frequencies of the uncore blocks?
pstates is what I think is messed up, probably. core clock is probably second order on intel whereas its first-order-ish on amd. AMD really seems to have nailed their fabric frequency scaling when busy/not busy to handle that largely transparently.
the cpus weren’t pegged at 100% for some of the membench stuff I was doing, too, which was surprising
When you tried the HBM equipped Xeons was this a model where it fit in the HBM? I guess for that matter I don’t know that much about the way the HBM is presented to the user for those models. Is it an L4 cache or a separate memory region?
Thanks. As far as the open source alternative, I settled on palabos, which seems to be in a similar space if not the same. IIUC it’s structured in a way that’s going to work for my demo. I have the alterations I need functional, now to get them working fast.
I’m currently running this demo an old Cascade Lake platform as a POC. If it works I should be able to get a speed bump on this benchmark by ~2x. Would like to partner up to try to get this working on the 6 series. Caveat is to show the benefit will require… one of several interesting hardware configs. More details to come.
I am looking for a core/thread with a 4900000 max freq reported by the OS:
~ grep . /sys/devices/system/cpu/cpu*/cpufreq/scaling_max_freq
...
/sys/devices/system/cpu/cpu11/cpufreq/scaling_max_freq:4700000
/sys/devices/system/cpu/cpu12/cpufreq/scaling_max_freq:4800000
/sys/devices/system/cpu/cpu13/cpufreq/scaling_max_freq:4900000
/sys/devices/system/cpu/cpu14/cpufreq/scaling_max_freq:4700000
/sys/devices/system/cpu/cpu15/cpufreq/scaling_max_freq:4700000
...
On my machine, cpu13 is reporting a scaling_max_freq of 4900000 so that will be my target.
Then, in a second terminal, I run sudo turbostat --interval 1 --cpu 13 --show CPU,Busy%,Bzy_MHz and get output like:
...
CPU Busy% Bzy_MHz
- 0.54 784
13 0.54 784
CPU Busy% Bzy_MHz
- 0.20 778
13 0.20 778
CPU Busy% Bzy_MHz
- 0.09 790
13 0.09 790
CPU Busy% Bzy_MHz
- 0.11 775
13 0.11 775
...
Then in the first terminal, I run taskset -c 13 bash -c 'while :; do :; done' to put the targeted core into a busy loop and trigger 100% CPU usage (I have also tried stress-ng as well, but this is simpler and also effective).
I then see the clock rate ramp up in the terminal running turbostat and a max Bzy_MHz of ~4300:
CPU Busy% Bzy_MHz
- 0.11 775
13 0.11 775
CPU Busy% Bzy_MHz
- 54.41 4244
13 54.41 4244
CPU Busy% Bzy_MHz
- 99.76 4300
13 99.76 4300
CPU Busy% Bzy_MHz
- 99.76 4305
13 99.76 4305
CPU Busy% Bzy_MHz
- 99.76 4315
13 99.76 4315
For graphing, I am using Grafana + Prometheus, with the data source being the standard node_exporter (temperatures from IPMI and hwmon sources, CPU utilization) and GitHub - BlackDark/prometheus_turbostat_exporter: An prometheus exporter for turbotstat (for monitoring different C-states and pkg-states) · GitHub to track the Busy_MHz stats reported by turbostat.
I did a tiny bit more experimenting, setting the By-Core ratio limits to what I think are the factory defaults, and saw slightly different behavior (although still not what I was expecting).
BIOS settings I tried:
Previously they were all set to Auto. With this setting in the screenshot, I saw the a speed of 4.4GHz (instead of the prior 4.3GHz) running the same steps as above. Also, all cores reported a scaling_max_freq of 4900000 instead of only selected cores.
I had a load average <2 during the test, so I would expect the “Turbo Ratio Limit 1” to be in effect (or Limit 2 at the very least). I don’t have experience with Xeon and its turbo boost settings, so it is very likely I am misunderstanding how boost rates are determined and with research it seems like there are many factors that determine it. I do not particularly care about the boost rate hitting the 4.9GHz other than concern that this may be a symptom of a hardware problem.
Regarding the AER errors, I do see a FAQ on ASUS site about other boards having AER Rx issues with H100s - so maybe this an expected issue with Hopper GPUs 
.
I did file a support ticket with ASUS regarding the AER errors and the CPU boost, but they pointed me to the local distributor… (Although, maybe that is not a bad thing as the Japanese distributor may be more diligent in following up on the issue than ASUS themselves)
If my experience with W790 shares any relevance whatsoever then:
I wouldn’t suspect a hardware fault (yet) this boost behavior seems very similar to that often observed on W790 2000/3000 - removing default “Auto” values or using a Asus OC profile is necessary on some bios revisions, to actually get the chip to freely scale per-core instead of only boosting CPPC preferenced cores or cores with a VIF curve it considers “sustainable”.
This is a likely culprit, but do not rule out that the default PCIE Redriver profile for W890 may simply be a “manufacturer recommended default” rather than a “optimal setting for this board” as it was commonly on both the W790 ACE, SAGE and WRX80E.
Nefastor ‘s excellent post on this was instrumental to me resolving issues with PCIE device drop out in the last slot on my W790-ACE Help with WRX80E-Sage SE Render server - #6 by Nefastor
are you only using turbostat for max boost reporting? When I worked on it I had run into a bug where the “Second stage” boost was not reported correctly in turbostat. This was the “+100mhz” boost-beyond-boost that the 14900k could do, iirc. I am about to sit down with this system and work on it, if nothing else interrupts
I was relying solely on turboboost for max boost reporting. I also have /sys/devices/system/cpu/cpu*/cpufreq/ stats recorded as well and they mostly match up to what I had been monitoring. Maybe turbostat is reporting 4.37GHz in a spot cpufreq reports 4.4GHz - but it not showing different boost-bins in-use between the two.
Later today I will try to record some more metrics related to how turbo boost is selecting its boost bins, and maybe a root cause will become more obvious. Having read a bit into the topic, there are a lot of factors taken into account when selecting a turbo boost bin so it might take a bit of work to nail down.
Regarding the PCIe P2P transfers, I do see settings around IOD Mesh Ratios in the BIOS. I am making an assumption that IOD is what is responsible for PCIe RDMA work, but I am wondering if under the default ASUS “Auto” settings the IOD will run underclocked some of the time leading to the lackluster PCIe P2P bandwidth numbers I saw earlier.
Can you test this CPU with MRDIMMs as the 676X-698X Granite-Rapids WS Xeons have official support for 8000MT/s MRDIMMs?
OK, I have found the “magical” BIOS setting that resolved my issue with my Xeon 658x not boosting to the full 4.9GHz.
In the BIOS, under Socket Config > CPU C State Control, I set C6x to C1 Demotion to Disabled.
With that in place, the idle cores enter the C6 state. With the default settings, I was finding that many of my cores would stay in the C1 state rather than enter C6. It seems that if the core is not in C6, then it is being counted as an active core for TurboBoost and I was not seeing boosts to the top bins (for 2 or 4 active cores).
I have encountered the issue on both Ubuntu 24.06 and Ubuntu 26.04. I am not sure if changing this BIOS setting is necessary because of some kernel configuration that Ubuntu includes and I am not sure what drawbacks may be encountered because of this BIOS setting. I am working through support channels and will hopefully get some feedback from ASUS regarding this, but I have only just made this change and have not heard back from them yet.
I am still encountering PCIe AER errors from each of my H200s, which ASUS says they are unable to reproduce. I only know the errors occur from the logs, there is no noticeable impact otherwise, but hopefully this will be eventually resolved.
How is granite rapids (especially the 658X) holding up compared to sapphire rapids? I’ve had enormous scheduling & context switching, game streaming etc / random latency issues on SPR even with the known energy bias / os & bios level power fixes applied. I hope its fixed with GNR ? let me know!
