Background: I purchased a RAM upgrade for my server and when trying to install, found that the motherboard is damaged and needs to be replaced. Since the replacement motherboard is an unexpected expense and I just bought more RAM, I can’t afford to buy more at this time and will need to make do with the best configuration possible.
New Board: Supermicro X10DRI-T4+ with dual E5-2680v3, 4 channel RAM per CPU, 3 slots per channel.
Available RAM (all ECC): 4x16GB 2Rx4 2133MHz, 4x8GB 1Rx4 2133MHz
I know from previous experience with the C612 chipset / Xeon v3 processors that running 2 DIMMs per channel in only 2 channels (4 DIMMs total) will boot and function properly, but the memory clocks down to 1866 MHz.
This would give me all 96GB in the new build, but running slower and only in dual channel mode instead of quad.
(It’s unclear from the manual if the following would boot, but if it does: ) Would it be more performant to put 2x16GB and 2x8GB in all 4 channels (1 DIMM per channel) where 2 channels would have more memory than the other 2 channels? Assuming it still runs at 2133MHz. What if it runs at 1866MHz, still better?
Also note: the board can technically boot with only one CPU but only one of the PCIe slots works without CPU2 installed. Even though I don’t need the cores, I do need the expansion slots, and it’s a really good deal that’s cheaper and better than replacing my single CPU board like for like.
Thanks in advance for any help you can offer. And lastly note: if you’re a PC DIY’er and think that qualifies you to build your own server, learn from my mistakes and think again. I could have a way better server for cheaper if I could start over and just buy a used R730 or a Storinator or something like that.
Total system memory bandwidth, sum of both CPUs, rounded
Speed/Channels
2133MHz
1866MHz
4 channel
546Gbps
477GBps
8 channel
1092Gbps
955Gbps
Now, obviously, there are some latency concerns which may impact effective bandwidth and application performance when making memory read requests from one CPU to another, but it’s incredibly clear to me that you’re much better off with one DIMM in 8 channels than the other way around.
If you need latency, you might need to swap this, but I hope this helps.
Thanks! Does this all hold true even when some partial amount of the RAM is in quad channel (in this case 2/3rds of it) and the rest is in dual channel?
Ohh, that, I don’t know. I don’t know if the dual channel space will be load balanced with the quad channel space, or if it will be sort of “tacked on” when the quad channel gets filled.
Given that this is server memory (RDIMMs), which I failed to account for in my bandwidth calculations above, the timings are looser. The memory prioritizes data integrity over speed. This means there’s a bit less bandwidth. First: server motherboards, at least the overwhelming majority, do not allow for manual fiddling with the timings and frequency. You get what you get. Second: 2133 will not “do with good timings”. It’ll have what it has. It’s still a porsche vs a bicycle when comparing to desktop memory performance.
OP, I’m not sure what you intend to use this for, but unless you’re doing very specific, high performance workloads, you’re going to be fine. For simple homelab work, this thing’s going to be an absolute beast.
If you are, you’d have probably known to just buy something new, specced the way you need it from the manufacturer.
as long as your ram MT/s is equal to or less than your cpu’s native ram controllers MT/s you shouldnt have any issues, as its a dual cpu system…
the 1st cpu will get 64 gigs the second 32. if you mix and match giving each cpu 48 gigs each you will have issues as the single rank chips will likely have a different bit dept than the dual rank dims along with slightly different timings.
so yeah the best option is 64+32 and just light load cpu 2 till you get more ram and a replacement motherboard.
Thanks! It’s definitely way more horsepower than I ‘need’ and would still work fine if I only got half the available performance out of it. But, given that I’ve ended up way overpaying, it feels silly not to get the best possible performance out of it. Plus, number go up makes me feel all warm and fuzzy, you know?
Can you do that? I thought that dual CPU boards required both to have the exact same capacity (and configuration?) of RAM.
yes the cpu’s have there own memory controller so as long as you place the ram in the correct slot, the system wont care about the amount of ram per cpu. its more about timings and transfers for ram compatibility.
I could have a way better server for cheaper if I could start over and just buy a used R730 or a Storinator or something like that.