AMD Epyc Milan Workstation Questions

[sherma.52]

Looks like a few of the 7003 CPUs on ShopBLT just got pushed back from 7/1/21 to 9/21/21.

In terms of distribution, AMD has several partners that they ship EPYCs to:

• Arrow
• Ingram Micro
• SYNNEX
• Tech Data

1. You can buy from Arrow directly but they don’t have any 7003s to preorder or buy.
2. If you buy from ShopBLT they are a reseller for Ingram so that covers that.
3. Does anyone know resellers for SYNNEX or Tech Data?

I’m trying to make sure I have a CPU ordered from each distribution point so that I can increase my chance of getting the CPU.

[Edin_Gacic]

Thanks for your input here - I have decided to go with the same motherboard form AsRock - I see you are in France - where did you order it if you can please let me know I can’t find it in Austria where I am and also in Germany is out of stock - I have RTX 3090 and I run DaVinci Resolve and 3D apps and I am glad you did not have any issues under windows 10 is it pro or enterprise? Can you do a Vray 5 benchmark for me? Thanks a lot!

[oegat]

@oegat

After more than a week of daily operation, a few days ago: I found it off when I had left it on. Pressed power: nothing happened. Logged into BMC, got in, tried starting from there - got an uninformative error message. Soft reset BMC - no change. Unplugged power for a while and re-plugged it - it started as normal. No traces in the logs.

Ok, this problem hasn’t gone away, I had two more sudden poweroffs in the last two weeks (H12SSL). So I can’t ignore it. The symptoms:

• Machine is found powered off (likely ungracefully, I wasn’t near)
• BMC remains running, nothing unexpected in BMC event log
• Machine cannot be started, not through power button nor from within BMC
• Removing power (switch on PSU), allows normal power-on afterwards (BMC will restart)

Here is my suspicion: when seating the CPU I did not turn the screws until the torque screwdriver clicked, since I started feeling so much resistance that I stopped trusting the screwdriver. So maybe I did not tighten it enough. So maybe the first step is thighenting the screws a little more and see if it helps?

@Nefastor, you seem to have a good grasp of the electrical/physical side of things - do you agree with my diagnosis?

Btw @jtredux, have you had any progress with your RMA, or other news about your BMC issue?

[jtredux]

@oegat Do you have your machine on a UPS? If not, it’s probably a wise investment given the value of H/W you’ll have in that machine once your Milan goes in. Which reminds me, I need a bigger UPS if/when my new rig is up an running again.

As for the CPU torque, I was surprised just how stiff 1.6Nm was on my torque screwdriver, but as you’re only 15-20mm from the axis, it will feel a lot more significant than 1.6Nm would on a torque-wrench. I went back and re-torqued my other EPYC that I had done by feel, and that was way too loose too. Functional for 3+ years, but a few ECC errors in the BIOS logs that I hadn’t noticed. Not sure if that’s alpha-particles or loose pins though!

I’m still waiting for my RMA request to be actioned. I chased my MB supplier and they got back to me straight away to say they’re waiting on SM to approve it

Anyway, because I had a feeling this might happen, and because I don’t like having all this H/W sitting idle for a part that’s ~10% of the build, I’ve ordered a ‘spare’ MB. It’s a bit of a left-field choice but if/when it arrives and if it works, I’ll share some info.

[oegat]

No UPS currently - I have one but it serves my NAS, and I plan to get a new one for the EPYC (I actually had two but one died).

I suspect the torque more than the power delivery at this point. Your experience corroborates my suspicion, I probably did not come near the required torque. So I will start by tightening it.

A spare MB makes sense imo, especially as you have two Rome rigs already. I expected SM to be faster with RMA issues though

[jtredux]

What a difference a few hours can make - I got my RMA approved this morning - sent off my H12 at lunchtime. Got home, got an email with a UPS tracking # for my plan-B motherboard. It is supposedly out for delivery, but I suspect will get bumped to tomorrow.

@oegat - I don’t actually have 2 Rome rigs already! I’ve got a Naples box, and was using a cheap (£170) 8-core Rome CPU to bring-up my 2nd EPYC system before the Milans hit the shelves.

I suspect I’ll just swap out the Naples 7281 CPU and H11-DSi MB for the Rome + new-MB once I get my hands on a Milan chip.

[oegat]

Sorry when I wrote Rome I meant EPYC - but I realize now that your new MB will not fit your Naples anyway.

So that’s another reason have one extra MB (if RMA goes well), since you already will have an extra CPU in the end.

I have a similar plan, when I get a Milan chip I’ll also get a new MB for the Rome one. I already have a place for it in my home ecosystem, replacing my aging NAS/Server (which will also be subject to an RMA adventure, since its Atom 2550 MB broke a while ago - running a temporary Celeron in the meanwhile).

[Nefastor]

You’re welcome ! I ordered my board from Amazon.fr , IIRC it cost me 660 €. It’s a bit more expensive now but they still have a few left :

Asrock ROMED8-2T Carte mère LGA 4094 ATX: Amazon.fr: Informatique

@Edin_Gacic

I run DaVinci Resolve and 3D apps

So do I. Specifically, Resolve 17 and SolidWorks 2020. GPU acceleration works flawlessly in both. I also use a 3090, by the way.

I run Windows 10 Pro, latest version.

I cannot run benchmarks as I’ve now turned that machine into its original form : a 120-bay TrueNAS server. More on that soon

[jtredux]

Well, I’ve got my new motherboard - this one’s a Tyan. I managed first boot with a CPU, but I can’t connect the front panel as the case has too few pins/the motherboard has too many, so for now the only way to turn it on is via the IPMI!

[Edin_Gacic]

So I am safe then with this board I just got a quote for 463 EUR from ASRock distributor from Germany but they can’t get it before August while the CPU Epyc Milan 7443P I can get in 2-3 days. Do you know if Resolve 17 would run under Windows Server 19? How long did you test Windows 10 Pro with DaVinci 17? I also have DeckLink 4K cards and other RAID cards for legacy enerprise drives that I plan to use more for backup after I migrate everything to these crazy fast SSD drives

[Nefastor]

Hey guys. I hope you’re all doing well. I’ve been off the forum for a few days. Here’s what happened : I ordered a Core i7-6850K on eBay to replace the 5820K on my aging workstation. I say “aging” but it’s still plenty capable… it had one issue though : you may remember that while socket 2011-3 processors normally have 40 PCIe lanes, the very first and cheapest one, the 5820K, only has 28.

Back when I bought it, that wasn’t an issue : NVMe was still unobtanium and the only PCIe card in the system was a GPU. Years later I added an NVMe drive and everything was still OK.

And then I added a 10 Gb NIC, and promptly ran out of PCIe lanes. Basically, I could not get more than 350 MB/s over Ethernet when it should have been 1 GB/s. That’s really the main reason I eventually embarked on building an EPYC workstation. But I knew I could buy a second-hand Core i7 and bring my old machine up to my new standards. So I did. It cost me only 134 € and I even get a nice 300 MHz clock bump in the process.

With that out of the way, I’ve decided to use my EPYC for what it’s meant to do. I present to you my latest baby :

That’s a 25U rack with, top to bottom :

• A 1.55 KVA Eaton UPS (just 1U, a bit pricey but it’s really nice)
• Empty space for a 4U rack (future backup server)
• The EPYC 7282 / ROMED8-2T in a 24-drive 4U chassis
• Four NetApp DS4246 disk shelves (thank you eBay)

It’s very sparsely populated for now, but the whole idea was to build “the only file server I would ever need”. I don’t like wasting time every few years building entirely new infrastructure. Now I won’t have to. This thing has 108 TB of storage in the “main” array (8 EXOS X18) and two of the disk shelves are filled with almost every 3.5’’ HDD I’ve bought for the last 10 years, as one giant backup pool. So that’s 56 drives with room for 120 drives total.

I changed the jumper settings on the motherboard to use both M.2 sockets, and used them for the system pool (a pair of cheap 128 GB sticks in mirror). The drives are controlled by two 16-port SAS 6 Gb HBA’s (LSI 9201-16i, again, a nice eBay find) :

• 24 ports are directly connected to the server chassis’ backplanes
• 8 ports go to bulkhead passthroughs and from there to the disk shelves

Here’s what that looks like :

Of course I’ve had to replace the huge Noctua heatsink by something less quiet but 4U-compatible. It turned out to be a very nicely built unit. I could replace the fan, it’s a standard 92 cm but it’s less noisy than the chassis fans already.

There’s one caveat I’d like to point : due to all its PCIe slots, Asrock has seen fit to use angled pin headers for the front panel connections. I knew that would be a problem in any 19’’ chassis. I’ve had to get creative, again :

The case’s front panel connectors use shells that are too long and simply can’t fit. However, the connectors are what the Chinese like to call “Dupont crimp connectors”. They are very popular with the maker crowd, and I have a metric ton of those. It’s the same connectors used on those popular jumper wires :

It turns out that on some versions of those wires, the single-pin shells are shorter than regular shells by a few millimeters. So I “modded” the case’s power switch connector and gained just enough length for the tight fit in my photo.

Not sure if this is a design flaw on Asrock’s part or if they expect anyone will just use IPMI to control the motherboard, but hey, if you ever face this sort of problem now you know a cheap solution that doesn’t involve modifying your expensive motherboard

As for my workstation… I still want a new machine. The 6850K upgrade is not a long-term solution. But I’m hoping it’ll take me until a Zen 3 ThreadRipper is announced, or, who knows, until Intel finally figures out how to 7 nm

I know this is a thread about workstations, but since many of us like to repurpose their old workstations into servers maybe I’m not off-topic here.

The NAS has only just been freshly setup. I intend to try all sorts of VM f*uckery in the very near future. I’ll keep you posted ! It might be in a NAS-specific thread, though, what do you think ?

[Edin_Gacic]

man thats the same thing I am doing But my setup looks so ghetto maybe I will make a photo - i use intel backplanes to run cables from inside the case but it works I want my Epyc to be the workstation I don’t care about hybernation or sleep like you do and also noise is not a concern for me because I blast music or use headphones in my studio. But maybe I will go quieter who knows - i am just lazy to make it all perfect just care about speed.

[Nefastor]

I’m fairly tolerant to noise, however there is no way to keep a stack of hard drives cool without making a ton of it. I’ve learned that the hard way. My previous NAS (and current backup NAS) is due for a rebuild because of that.

I used a standard ATX tower case, from the time when dinosaurs ruled the Earth and every case had 5.25’’ bays from top to bottom. I installed several “ICY-BOX” SATA backplanes in one such case :

And that resulted in a Micro-ATX NAS with 12 drive bays (4 blocks of 3 drives).

Now, those puppies ship with what I mistakenly believed to be a “crappy” fan. Because I didn’t know about static pressure back then. So I did the dumb thing and replaced each backplane’s fan with a Noctua silent fan.

Well it did make the NAS quieter. But now the drives would cook themselves to death. Eventually I had to resort to using only 6 drives, with an empty bay between each drive. Not exactly a “win”

When the 24-bay chassis arrived, I knew the drive fans would be problematic : 24 drives cooled by just three 12 cm fans ? Those had to be “jet engines” loud. They were. They also drew 1.5 A each.

So I replaced then with industrial Noctua fans. They do warn you that those aren’t nearly as quiet as your regular Noctua, but the static pressure is excellent. Perfect for my purposes. Problem is, they really are too loud. And it’s not just the volume : the pitch is higher too, which makes them a bit annoying. Even with loud music, those are not office-compatible.

Bottom line : if you want to do serious storage, you have a choice between two scenarios :

• Huge PC cases with lots of space between HDD’s
• Or a datacenter chassis that will need to live in a separate room

But hey, that super-stupid Chia Coin craze has just started… it may be that a month from now we’ll start to see all sorts of new mining-oriented cases that might be perfect for NAS duty. They did make special multi-GPU chassis for the other cryptos, after all.

[Edin_Gacic]

yes I know this from my z800 workstation. It is running on the side with the lid off and I have 15K drives in the 4 bays inside and also run long SAS cables for additional 2 x 15K drives to a IcyBox similar to the one you just posted but I have to run 2 drives because as you say they cook eachother. The 4 x 15K drives that are inside the HPz800 are cool because the case is built for that and the fans are right on them. Also I have LSI Raid card inside the z800 and I run cables to 2 Intel backplanes and they are open on the desk outside the case I need to show you these pictures I run one slow spinning fan in front and they are quite cool and don’t make noise - Since I added RTX 3090 inside the acase and have another RAID card that powers another 8 drives that I use for backup the my raid controller sometimes complain about heat but I have been dismissing it for years and it works - i run one small fan on it most of the time its fine. But the workstation looks like monster that keeps growing outside the case its funny to see

I have a question now about the Epyc and AsRock board and RAM. Do I need to go and populate all 8 slots to gain all the lanes? Can i start with just 2 stick of 128 ? I am trying to keep it cheap and grow slow on the ram?

Keep the post going I love this and your input = thanks!

[Nefastor]

@Edin_Gacic

I need to show you these pictures

Please do ! It sounds like you’re living dangerously

[Edin_Gacic]

yes I would love to know this too with Milan

[Nefastor]

@Edin_Gacic

Do I need to go and populate all 8 slots to gain all the lanes? Can i start with just 2 stick of 128 ? I am trying to keep it cheap and grow slow on the ram?

Indeed you can use as little as one stick, although it will bring the wrath of @wendell upon you (just kidding). Here’s the info from the manual :

I’m sure you already know, but there are some EPYC’s optimized for quad-channel. With those, you get max RAM throughput with 4 sticks but can still use 8 to get higher capacity.

[Edin_Gacic]

oh that is interesting I did not know that you can go with one stick only! Are there any differences in performance? I have to read this - and I don’t know about the thing about some EPYC being optimized for 4 - can you share more? I am planning on running AMD EPYC 7443P 2.85 GHz (24C/48T)

[Nefastor]

AFAIK the 7443P is a “true 8-channel” chip. The chips I’m talking about have few cores (8, 12, 16) and that means few chiplets, which has complex effects on how the RAM controllers talk to all the cores. Because of that, those small EPYC have “full performance” on 4 channels, with higher latency on the other 4 channels (but those still work). The alpha and omega on this whole topic is the work of Patrick at STH :

AMD EPYC 7002 Rome CPUs with Half Memory Bandwidth

[oegat]

I’m running one of the 4-channel optimized parts currently: EPYC 7252. I got it as a placeholder while waiting to order what will probably be a 7313p, which is a full-bandwidth part.

The 7252 has 2 CCDs, CCD3 and CCD5, both on the same half of the MCM (same side of the IO die), but on opposite sides of that half. So they should preferably use the 4 memory channels located on the same side as the CCDs. As a consequence of the layout, these CCDs cannot be configured as separate NUMA nodes (not that it matters for most use cases).

(@Log this is also the reason I haven’t got back about the PCIe slot <-> chip section mapping. I could not configure NPS ≥ 1 on my current hardware, so I don’t know how to expose the topology).