Running a 3.5 TB Ethereum-Node in Optane PMEM

[Mr_Moonsilver]

I know this is a place for crazy ideas, so let me share one.

The prices for 100 series Optane PMem 512GB sticks have come down quite a lot. It made me thinking, would it be possible to populate 8 slots of the ASUS “Pro WS C621-64L SAGE” with those Optane dimms in order to run an ethereum archive node?

The idea is to run a node and query it for blockchain data, and lots of it. I was wondering, if it would make a difference compared to running it on SSDs (or even Optane SSDs).

Appreciate any critical thoughts or additional considerations. If it turns out to be feasible I might as well do it and make a post about the build and the performance results.

One question in particular would be the RAM requirements. How intensive on RAM are queries on a Node and database operations to store transaction data?

Also, is it possible to create two different arrays with specific optane DIMMs? Let’s say, one array with 6 devices and another with only 2?

[Mr_Moonsilver]

Ok, I guess this means I have to test it out so I ordered 5TB worth of optane DIMMs and a Xeon Gold 6240 to go with it as well as 128Gb of ram. Will update the post in a couple of weeks when I got the hardware and did the testing.

[TryTwiceMedia]

@Mr_Moonsilver sorry no one responded, we’re typically pretty good at spotting this flavor of shenanigans…and upon further review, I understand why. This is outside the wheelhouse of most members here, lucky for you I’ve got experience with this one.

@Mr_Moonsilver

Optane PMem

VERY niche hardware. We utilize it in 2 specific environments:

1. mission critical: where every second of downtime is lost revenue (pmem DIMM’s allow crazy fast boots since the working set is already in working memory, which doesn’t seem like much to most but loading a few terabytes from even the fastest SSD array takes serious time).
2. computationally expensive: workloads where tons of compute resources are dedicated to a single set and rebuilding exceeds the ludicrous hardware costs

@Mr_Moonsilver

The idea is to run a node and query it for blockchain data, and lots of it. I was wondering, if it would make a difference compared to running it on SSDs

Optane really only helps with random IOps and write endurance in typical applications, pmem is a cornerstone of any serious AI venture however. If you’ve worked with neural networks and dynamic learning sets, you’ll quickly learn results are not duplicable or documented. A network trained on the exact same set will offer “identical” results, but no one uses static sets for anything but POC.

@Mr_Moonsilver

or even Optane SSDs

SSD’s comprised entirely of Optane are fantastic. Optane accelerated SSD’s are junk. They use sub par NAND combined with a tiny Optane cache to offer the worst of both worlds, it’s honestly quite impressive.

@Mr_Moonsilver

How intensive on RAM are queries on a Node and database operations to store transaction data

For database queries, they really aren’t…until you get into crazy quantities as RAM will only be held to verify entries. If you are serving giga quantites of queries, it’ll matter more as your bandwidth becomes the bottleneck.

@Mr_Moonsilver

I ordered 5TB worth of optane DIMMs and a Xeon Gold 6240 to go with it as well as 128Gb of ram

Build it out so you can either:

1. direct attach GPU’s
2. install ridiculously fast interface cards to a GPU node

You’re building a machine that most guys goin into AI don’t even know they need.

P.S. I’ve been active in the AI space since 2008, llm’s are currently being used out of context. the next leap will be combining llm’s with neural networks which requires crazy dev rigs (like you’ve built) to develop the algo’s that can run on smart watches.

[rrubberr]

@Mr_Moonsilver

5TB worth of optane DIMMs and a Xeon Gold 6240

The 6240 has a 1TB memory cap (sum Optane+DRAM). You’ll want an “L” suffix SKU. A 6268CL or 8275CL will perform much better, and support 4.5TB of memory per socket.

Optane requires BIOS support, in addition to CPU support. It is not “plug-and-play” as such, and must be configured in the BIOS in one of the available addressing modes (memory mode, app direct, etc.).

I’d recommend checking your motherboard’s manual to make sure it has a DCPMM configuration utility built in. Nobody seems to know if the SAGE boards had DCPMM support, and ASUS doesn’t list it anywhere on their support page.

I ran 2TB of Optane DCPMM in my workstation for about a year before shelving the sticks and replacing it with DRAM.

In memory mode, hitting Optane (that is to say, when the dataset would not fit in the DRAM cache) increased my LLC miss latency from 150ns with DRAM only, to well over 3000ns. Whether that’s performant in your use case will obviously vary.

Official documentation recommends a 1:8 configuration of DRAM to Optane. I had much better performance running 1:4. In any case, it’s best used in a 2DPC configuration with 1 DRAM and 1 DCPMM per channel.

@Mr_Moonsilver

Also, is it possible to create two different arrays with specific optane DIMMs? Let’s say, one array with 6 devices and another with only 2?

No, but you wouldn’t want to. DCPMM has to be interleaved by the IMC to be performant. Sequential read bandwidth is less than 8GB/s per stick. The more channels (up to 6) being interleaved, the better.

With current Fedora-based Linux distributions, you can partition your DCPMMs in app direct mode like any other disk. But the DIMMs are interleaved by the BIOS, and aren’t partitioned along hardware boundaries.

Debian-based distros, in my experience, lack the required NVDIMM driver OOTB.

[Mr_Moonsilver]

Hey @TryTwiceMedia thank you so much for your great reply. I only saw it a month later when I came back to report my findings so far. But first, I’d like to address some of your comments. They were all very useful, but the AI part really got me interested.

I have been experimenting with models, but haven’t gone full deep dive yet. I was trying to understand what you meant with:

llm’s are currently being used out of context. the next leap will be combining llm’s with neural networks which requires crazy dev rigs

I guess you are not referring to just combine two LLMs, but I couldn’t grasp what you meant by that. Could you elaborate what you meant by that? Also, do you see a place for PMEM as a high performance Vector DB?

In terms of the Rig, @rrubberr really helped me to understand the limitations of my initial hardware choices and I have been researching. It seems the dell Power Edge R640 and R740 are able to handle that amount of Optane Memory.

Also @rrubberr mentioned

Official documentation recommends a 1:8 configuration of DRAM to Optane. I had much better performance running 1:4. In any case, it’s best used in a 2DPC configuration with 1 DRAM and 1 DCPMM per channel.

Does that apply to using PMEM in app direct mode or is the ration important only when using it in Memory Mode?

I will be looking into getting a Dell R740 with the appropriate CPUs and come back to let you know how it went.

[rrubberr]

Correct, those suggested ratios are for memory mode.

In app direct mode, you only need as much DRAM as your OS and applications require. With a minimum of two DRAM sticks per socket.