Threadripper 2 Update! ECC Support, Linux, and More! | Level One Techs

Grim_Reaper · September 3, 2018, 8:58pm

what workloads?

mutation666 · September 3, 2018, 9:00pm

database work, vms, 4k editing, and may more

nx2l · September 3, 2018, 9:11pm

Is there a amazon link for the non ECC flareX kit you showed?

wendell · September 3, 2018, 9:15pm

Not out yet. Next week I hope.

jordif · September 3, 2018, 9:21pm

I agree that for most people having so much RAM (128GB) is difficult to justify. But there are several situations worth considering.

(Sample 1): Workstation users.
For instance in engineering simulation. Llike program Autocad, but for real pros (military level). I remember 8-10 years ago a comment (from a friend) linked to an AMD Opteron machine. The software recommending to get 64GB RAM to load all program libraries in memory. To ensure maximum speed during use. (HDD spinning drives were not so speedy).

(Sample 2): Software development.
I also considered, recently, the concept of enteprise development. Having multiple virtual machines. To simulate a real scenario “in house”, in a local area network with one physical computer. Thinking in a virtual machine having 4 cores and 16GB RAM. See restrictions at 3 virtual machines. Because 4 virtual machines means 64GB RAM. The whole memory in the system (Socket AM4). Imagine a server cluster (master slave with 3 nodes). And having “no space” to create more virtual machines to simulate 2-3 desktop users accessing the cluster (for unit testing). It is not something reserved to space scientists. Any good developer could do that easily (I think in elasticsearch + client-side programs in a desktop). Even allowing combining different desktop types: linux, windows, mac. So for instance in the this development described, a motherboard with 32 cores and 128GB RAM would mean up to 7 virtual machines (each with 4 cores and 16GB RAM).

(Sample 3): virtualization
See also the benefit of one computer being used to host multiple server side environments. Maybe supporting development environments for multiple teams or projects. One virtual machine for each one.

Conclusion:
So in essence more than 64GB RAM, might we agree is for professional users. But it really makes a big difference for them. To avoid enterprise hardware.

In my humble opinion.

wendell · September 3, 2018, 9:31pm

on fedora, the fix was:

install this:
dnf install openmpi-devel blas blas64 blas64-devel cmake hpl-openmpi

run this because the autodetect sucks

declare -x MPI_CC=/usr/lib64/mpich/bin/mpicc
declare -x MPI_INCLUDE=/usr/include/openmpi-x86_64/
declare -x MPI_LIBS=/usr/lib64/openmpi/lib/libmpi.so
declare -x MPI_PATH=/usr/lib64/openmpi/
declare -x MPI_LD=/usr/lib64/mpich/bin/mpicc

then your phoronix command… and all is well with hpcc. your fix for jtr works fine.

sceps · September 3, 2018, 9:54pm

do you mean hpl-openmpi? I don’t find anything for hpl-openmpi-dev.

Also, I’m confused why you include some declarations from mpich and some from openmpi

wendell · September 3, 2018, 9:57pm

Yes, I think that’s right. I was just sort of hamfisting it to get it to go. There is not a devel version on fedora. Juggling Ubuntu and fedora is frying my short term memory.

I thought that was odd too about mpicc but that’s the only one on my system. Everything seems to work…

sceps · September 3, 2018, 9:59pm

And I think there are several LA options too. I didn’t have blas64 installed on my system.

I used openblas, all installed with: env MPI_CC=/usr/lib64/openmpi/bin/mpicc MPI_INCLUDE=/usr/include/openmpi-x86_64/ MPI_LIBS=/usr/lib64/openmpi/lib/libmpi.so MPI_PATH=/usr/lib64/openmpi LA_PATH=/usr/lib64 LA_INCLUDE=/usr/include/openblas LA_LIBS=/usr/lib64/libopenblas.so.0 phoronix-test-suite install-test hpcc

I imagine there will be performance differences with different libraries…

RBE · September 3, 2018, 10:00pm

As has been mentioned by others, there are plenty of engineering applications which will use 128GB of memory. Ansys (a finite element analysis package) is an example of such a program, as are Fluent (computational fluid dynamics) and Comsol (multiphysics), etc etc. Not all simulations require that amount of memory of course, but when they do, boy howdy.

wendell · September 3, 2018, 10:34pm

Any idea what was used in the photo is article? I can try to match it closer

sceps · September 3, 2018, 11:32pm

No, sorry. I was trying to figure that out myself. Please post what you find or eventually go with.

Also note that just because it installs without errors it doesn’t mean it was successful. If LA variables weren’t defined, hpcc still compiled for me, but running the test took an order of magnitude longer.

CrayZeApe · September 4, 2018, 7:07am

LA variables to test with BLAS from the ‘Dark Side’, the Intel MKL.
https://software.intel.com/en-us/mkl/

LA_PATH=/opt/intel/mkl/10.0.5.025/lib/em64t/
LA_INCLUDE=/opt/intel/mkl/10.0.5.025/include
LA_LIBS=/opt/intel/mkl/10.0.5.025/lib/em64t/libmkl_intel_lp64.so /opt/intel/mkl/10.0.5.025/lib/em64t/libmkl_intel_thread.so /opt/intel/mkl/10.0.5.025/lib/em64t/libmkl_core.so /opt/intel/mkl/10.0.5.025/lib/em64t/libiomp5.so -lpthread

HPC Challenge 1.5.0:
pts/hpcc-1.2.4 [Test / Class: G-HPL]
Test 1 of 1
Estimated Trial Run Count: 3
Estimated Time To Completion: 1 Hour, 31 Minutes [12:28 AEST]
Started Run 1 @ 10:58:02
Started Run 2 @ 12:11:36
Started Run 3 @ 13:26:20

Test / Class: G-HPL:
    249.032
    246.051
    244.241

Average: 246.44133 GFLOPS
Deviation: 0.98%

sceps · September 4, 2018, 4:01pm

Could you confirm your CPU? Thanks for posting the variables.

It would be important to know what variables the phoronix article and benchmark stats used. With the openmpi + openblas config I posted above, my results (“GCC Ryzen 7 2700X”) are at the 2950X level. Either my system is “messed up” or the libraries really make that huge a difference.

HPC Challenge 1.5.0:
pts/hpcc-1.2.4 [Test / Class: G-HPL]
Test 1 of 25
Estimated Trial Run Count:    3
Estimated Test Run-Time:      1 Hour, 31 Minutes
Estimated Time To Completion: 3 Hours, 29 Minutes [05:20 EDT]
    Started Run 1 @ 01:52:05
    Started Run 2 @ 02:11:42
    Started Run 3 @ 02:31:22

Test / Class: G-HPL:
    194.758
    193.79
    194.14

Average: 194.22933 GFLOPS
Deviation: 0.25%

GFLOPS > Higher Is Better
Core i7 5960X ....... 90.02280  |====================
Core i7 8700K ....... 90.79620  |====================
Core i9 7900X ....... 116.44200 |==========================
Ryzen 7 2700X ....... 119.79800 |==========================
Threadripper 1950X .. 185.84300 |=========================================
GCC Ryzen 7 2700X ... 194.22933 |===========================================
Threadripper 2950X .. 196.18200 |===========================================

Here are my full results: https://openbenchmarking.org/result/1809046-FO-1808102RA12&obr_hgv=GCC+Ryzen+7+2700X&obr_sor=y&obr_hgv=GCC+Ryzen+7+2700X “201808262317” is also my result on the same system, but I’m not sure if GCC was used since it isn’t displayed in the system info.

Differences between the article’s and my results include: OS, kernel version, GCC version, and the article’s flags “CXXFLAGS=-O3-march=native LIBGL_AWLAYS_SOFTWARE=true CFLAGS=-O3-march=native”

@wendell The first chart shows the article used:

(CC) gcc options: -lm -pthread -lmpi -fomit-frame-pointer -O3 -march=native -funroll-loops

ATLAS + Open MPI 2.1.1

CrayZeApe · September 4, 2018, 4:31pm

CPU is Threadripper 2990WX.

I plan on testing multiple blas implementations to see how they compare, I noticed at least three distinct brackets in the scores appearing on OpenBenchmarking.org, on identical hardware. It looks like the libraries may all be vastly different in their performance.

If we’re talking about the same Phoronix article, the benchmark results page says a pairing of ATLAS + Open MPI 2.1.1 were used.
https://openbenchmarking.org/result/1808115-RA-THREADRIP08

Hujkis · September 4, 2018, 5:22pm

I’ve dealt with 4TB ram dual-socket systems

Grim_Reaper · September 4, 2018, 7:15pm

But those are not for people who make Youtube videos.

anon79053375 · September 4, 2018, 7:31pm

Wendell and team do more than make YouTube videos.

I eat up 32 GB on my development workstation and I’m conservative with how I allocate resources. I’d love to have 128GB for a full local development environment.

wendell · September 5, 2018, 12:56am

Same dependency issues with docker “You’re on your own…”

Linear Algebra Pack
Possible Package Names: liblapack-dev, lapack, lapack-devel, liblapack-devel

OpenCL
Possible Package Names: NVIDIA OpenCL, ATI Stream SDK, OpenCL SDK, opencl-headers

TIFF Image Development Files
Possible Package Names: libtiff-dev, libtiff, libtiff-devel

OpenGL Utility Kit / GLUT
Possible Package Names: freeglut3-dev, freeglut-devel

The above dependencies should be installed before proceeding. Press any key when you’re ready to continue.

It’s clear linux… so going to see what options I have for installing it inside the container…

sceps · September 5, 2018, 1:20am

Oh right. The tests without dependencies worked great.

I haven’t used Clear linux but the available “bundles” aren’t numerous: https://clearlinux.org/documentation/clear-linux/reference/bundles/available-bundles

And the bundle tool swupd doesn’t seem to be available in the pts docker image…