I apologise in advance if this is a stupid question, I am still very much learning server and enterprise grade equipment.
I have built my own white-box server using a mix of server and consumer parts.
But I will admit that I am a little bit confused by the types of server memory configurations.
I know that you have ECC and non ECC, and servers are best used with ECC.
Then you have Registered (buffered), and non Registered (buffered).
Then you have Rank type, and I am totally lost at this., eg, “1Rx4”, “2Rx4”, “4Rx4”, etc.
Doing some searching it seems that 1R = Single Rank, 2R = Dual Rank, 4R = Quad Rank.
But I do not understand the “x4” or “x8” part.
My white-box server is built with a Supermicro X8DTL-3F and installed 24GB (6x4GB) PC3-10600 DDR3-1333 ECC Memory “Samsung M393B5170GB0-CH9Q8”.
Let’s just say that if I wanted to max out the server with a total of 96GB (6x16GB) as per the manual, what would be the best / cheapest configuration of memory that I can use to upgrade my server.
Quite frankly Most modern servers will take pretty much any kind of DDR4 or DDR3 RAM, unlike older servers which took only registered ECC RAM. I do not know of a comprehensive guide to server RAM that is not specific to a certain server or a sever CPU so I will give you a brief overview of what we typically see in the server space
There are four main types of DDR server RAM:
Unbuffered non-ECC RAM. This is the same kind of stuff as you use in a desktop. Maximum sizes at the moment are 8 GB/DIMM (will be up to 16 GB/DIMM with the final 8 Gb DDR3 ICs) and up to two DIMMs/channel. The advantage of this stuff is that it is inexpensive and if you are enterprising, it can have better performance than any other kind of DDR3 especially if you are handy with an EEPROM flasher and use low-latency 1.50 V desktop DDR3. The disadvantage is that there is no error correction and that can lead to shorter uptimes and possibly data corruption although unlikely HOWEVER if ZFS is being used YOU MUST USE ECC
Unbuffered ECC RAM. This has the same module sizes and number of DIMMs as unbuffered non-ECC but adds a ninth memory IC per rank and has the same error correction and stability as "real server (registered) memory) but with slightly higher performance and lower cost. Unbuffered ECC memory was historically used as workstation memory where you needed more reliability than a typical desktop but were not using server hardware which required much more expensive and sometimes slower registered memory
Registered ECC RAM. This is not compatible with any of the unbuffered types due to the presence of a memory buffer chip which holds all memory requests for a clock cycle. The next type ofa ram is an extension on this too. (You can use either type of unbuffered memory or registered ECC but not both at the same time.) The buffer allows for much larger quantities of memory per module (currently 32 GB/module) due to the ability to run 4-bit wide ICs rather than 8-16 bit wide ICs as required by unbuffered RAM and also the ability to run four ranks of memory vs. one or two. Registered RAM can also support up to three DIMMs per channel as opposed to two with unbuffered. It is somewhat slower than unbuffered RAM and usually quite a bit more expensive. This is “server RAM” as the old grayhairs know it and what all servers used to require until the DDR3 generation. This may have changed with DDR4 but the basic principles are the same in the sense of you can only homogeneously load the ram (Same type of ram no mixing)
Load-Reduced DIMMs (LR-DIMMs.) An offshoot of registered ECC, this allows for even larger memory quantities per DIMM than registered ECC. Only supported by the later generations of DDR3-using servers (IIRC Opteron 4200/6200 and later, Intel Sandy Bridge LGA2011 and later.)… it contains a memory buffer (MB) chip, as opposed to a register, in order to reduce and minimize the load on the server memory bus. The memory buffer reduces each clock, command and address and data input to a single load, which increases the memory speed. LRDIMMs are useful in memory-intensive applications in data centers, cloud computing and in high-performance computing (HPC) environments. LRDIMM is pin-compatible with existing DDR3 DIMM sockets and is JEDEC standard.
To answer your question honestly Ebay kits of the ECC stuff would be the cheapest route. Also MAX the speed its not worth saving pennies for slower ram… RAM in server applications needs to be quick and in large quantities
Server boards will generally require ECC for multi-socket setups and may require it for even single socket - check your board’s specs.
Check your board’s support for registered requirements. In generally if you want more than one dimm per channel, you are very likely going to need registered and may need it to function at all.
x4 and x8 refers to the width of the chip’s outputs - 4 or 8 bits. The 8 bits being newer and cheaper chips to produce. The only thing of concern here is you generally can’t mix types within a channel. The Right answer is go grab the QVL for that MB and figure out what types they claimed to support and buy something very similar.
Dual rank on ddr3 and ddr4 setups is generally superior to single and quad rank. It offers the best balance of interleaving, load on the memory controller and latency.
Quad rank memory is generally going to reduce the total number of dimms you can put in the system and/or the speed at which they run. Load Reduced dimms are usually the most expensive by a long shot but offer the highest density.
So, shoot for dual rank registered ECC x4 or x8 as your best bet for working, but best to check your board’s compatibility list.
Be aware that ddr3 and ddr4 xeons generally have limits as to the dimm speed hardware enforced by how many dimms per channel are present.
1 or 2 dims per channel can run at 1333 with 10600 ram, but add a third dimm per channel (9 total for a single processer) and that drops to 800.
Whether 2 limits you to 1066 vs 1333 or whether a single channel can run at 1600 depends on your board, bios and chip, but most will do 2DPC (dimms per channel) at 1333 with 10600 registered ECC.
The drop from 1333 to 800 is not as drastic as it sounds since the CL (CAS Latency) also drops for the lower speed which improves random access times.