YATPRO : Yet Another Threadripper Pro Build!

[Nefastor]

@gysi

I could test this also tomorrow on my GTX 970

That would be much appreciated ! Meanwhile, I’ve kept investigating, and it is clear that the “trigger” for the odd behavior is having a monitor connected to the 970 when I start the computer. Here’s the sequence I’ve just tried. “VGA” means on-board graphics, “HDMI” means GTX 970 :

• Turn the PC off and on at the wall
• VGA and HDMI both connected : PC starts on VGA only, 970 is “Code 43”.
• Shutdown, unplug HDMI, turn back on : PC starts on VGA, but 970 is fine. Plug the HDMI and both monitors work.
• Disconnect HDMI, reboot PC, reconnect HDMI afterwards : the 970 works.
• Keep HDMI connected, hibernate PC, restart PC : everything works, but it feels like it took longer to come out of hibernation than to boot.
• Keep HDMI connected, hibernate again, turn off power at the wall, restart PC : everything works, but it seems to take longer than booting the PC again. It’s only seconds, so I could be imagining it.
• Keep HDMI connected, reboot the PC : the PC boots on VGA only, the 970 is “Code 43”.
• Booting on HDMI alone does not work : the PC boots on VGA and the 970 goes “Code 43”.

So it’s very clear that there’s something this configuration doesn’t like about a GTX 970 being plugged into an HDMI monitor during boot. I’ve tried half a dozen times, same results I’ve just listed.

Note that when I wake up from hibernation, which is the only case where the PC boots Windows with the 970 connected, I do not see the Asus splash-screen and the invite to enter the BIOS.

When hibernating, a PC goes into ACPI state S4. For some reason, waking up from S4 makes the board “tolerant” to the GTX 970 feeding a monitor, whereas during boot (sort of waking up from state S5) this is a deal-breaker.

Quick note : my 970 is a Gigabyte Windforce, I never updated its BIOS. It works very well on X58 and X99 motherboards, as well as an Asrock EPYC server motherboard. I’m hoping its BIOS is somehow causing this issue, because it’s clearly a software problem.

This is the weirdest boot behavior I’ve ever seen from a PC, going back to the late 80’s.

It’s getting really late here. Tomorrow I’ll experiment with an RTX 3090, see if that works better.

[Nefastor]

@wendell

I bet that 970 is missing a firmware update

As a matter of fact, I never did update it. I’ll try an RTX 3090 tomorrow, if it works better then it might be the reason. But I used that 970, as is, on an Asrock Rack ROMED8-2T motherboard last year and had no problem whatsoever. It was even in the same case, with the same PCIe riser. I didn’t have to force the PCIe slot speed.

I’m a bit annoyed that I don’t know exactly how the Asus board is architected. Every other WRX80 board manual has a schematic telling how everything is connected to the processor and chipset. I wish Asus did the same.

@wendell

When you change bmc vga mode you must kill power to the psu for 1 min then plug it in and wait 1 min before powering on the system.

I’m afraid I haven’t been that patient. I only waited until the LED’s on the board went dark, usually a sign that all capacitors have discharged below VDDmin. But I’ll keep it in mind.

[Nefastor]

It’s a brand new day and I have news, some good, some dodgy.

Using a 3090 instead of a 970, I get better results. I can finally boot reliably (so far) using only the discrete graphics. I’ve disabled both the onboard VGA and the BMC. However I have to force the PCI slot to PCIe 3.0 : if I leave it on “auto”, the strangest things happened.

For example, I managed to boot to Windows only for the device manager to give me the same “Code 43” error as before. According to Windows it had to stop the 3090… and it was displaying that message on the 3090 itself. Kind of a “mission failed successfully” moment

Clearly, there’s some signal integrity issue on this board, running a PCIe riser on the slot farthest from the CPU. You would think the board’s redrivers exist specifically to compensate for that, but maybe Asus doesn’t know how to use them.

For a comparison, the Asrock ROMED8-2T had no trouble in this same configuration. Same riser, same 3090, using the last slot, it was happily running in PCIe 4.0 all day long… and it has no redrivers on the PCIe slots. The only redrivers on the Asrock are where you’d expect : near every connector that can bring PCIe lanes to cables, for example the Oculink and U.2 connectors.

Speaking as an electrical engineer and board designer, it’s painfully obvious that Asus is inferior to Asrock, at least on this type of high-end hardware. For example, the steel plate “reinforcing” the motherboard has everything to do with Asus using a cheaper PCB than Asrock. Asrock’s looked like a 14-layer, very thick and rigid. Asus’ looks like an 8-layer. I didn’t break out the caliper but it really looks like 1.6 mm thickness. It also explains why Asrock can make this kind of hardware fit on an ATX-size board while Asus needs to go 5 mm bigger than full E-ATX.

The BIOS contains interesting options for manually configuring the redrivers for slots 5, 6 and 7. I’m going to put my engineering degrees to use, find the datasheet for those redrivers and see if I can go all Thanos and do it myself. I paid for PCIe 4.0, not PCIe 3.0.

This being Europe, I have a couple of weeks to decide whether I want to keep this board or return it for a refund. Not gonna lie, if I can get my hands on an Asrock Creator (rev 1) then I’m sending the Asus back.

[nnunn]

@Nefastor

I’m a bit annoyed that I don’t know exactly how the Asus board is architected. Every other WRX80 board manual has a schematic telling how everything is connected to the processor and chipset. I wish Asus did the same.

This schematic was posted here some time back. Might help with your investigation?

[Nefastor]

Thanks, nnunn ! However, this isn’t as detailed as one would hope. For example, this is what Asrock gives you for their latest WRX80 board :

Anyway, I’m just rambling. I wouldn’t be happy unless Asus gave me the CAD files for their motherboards.

[Nefastor]

Earlier today I received an NVMe carrier. This led to further investigation of PCIe on the Asus WRX80 :

Note that this came with a heatsink covering all four positions, but for quick testing it was simpler to leave it off. You might be wondering why I got this puppy when the Asus motherboard ships with a similar gadget. Two reasons : the Asus twice as large for no valid reason, and I had originally ordered the Asrock motherboard, which doesn’t come with one.

The two drives I’ve tested with are PCIe 4.0 (Sabrent) and PCIe 3.0 (Samsung). I wanted to test if, after bifurcating an x16 slot, you could mix devices with different bus speeds.

Short answer ? Nyes.

It works, in that the drives are recognized by the BIOS and Windows. And then you start transfering some files and one (or maybe both) drives hang, forcing you to hard-reset the PC.

Conclusion : it doesn’t seem like a good idea to mix M.2 drives of different PCIe generations on the same x16 slot.

So I removed the Samsung and continued with just the PCIe 4.0 drive.

I also wanted to test if an NVMe drive would struggle to run in PCIe 4.0 if it was plugged in one of the last slots of the board. I saw no such problem there : the Sabrent behaves the same whether it’s slot 2 or slot 6 :

Incidentally : I’m not an expert in SSD testing… can someone tell me why the CrystalDiskMark speeds (which I also checked in the task manager) are so much faster than what I see when I copy large ISO files ? The best I can get out of the file explorer is 2.5 GB/s, transferring four files of 5-6 GB each. My system drive is an even faster SN850X.

[oegat]

Great to see you have progress, despite the setbacks!

@Nefastor

Incidentally : I’m not an expert in SSD testing… can someone tell me why the CrystalDiskMark speeds (which I also checked in the task manager) are so much faster than what I see when I copy large ISO files ? The best I can get out of the file explorer is 2.5 GB/s, transferring four files of 5-6 GB each. My system drive is an even faster SN850X.

It’s likely related to queue depth (number of IO commands lined up at a time) and thread count (number of parallel operations). I believe copying an ISO file to or from the drive should be the equivalent of what CrystalDiskMark calls SEQ1M Q1T1. I see you have it there, does it give comparable enough numbers?

It’s possible that Windows Explorer somehow splits the file in smaller chunks than 1M when copying sequentially, though.

[Nefastor]

@oegat

I believe copying an ISO file to or from the drive should be the equivalent of what CrystalDiskMark calls SEQ1M Q1T1. I see you have it there, does it give comparable enough numbers?

Hello there ! Yes, the numbers I see when I copy files are close to SEQ1M Q1T1. Your explanation makes sense.

@oegat

Great to see you have progress, despite the setbacks!

Let’s call that “growing pains”. As usual with this type of motherboard, there’s a learning curve and some effort required to get the most out of it. I’m not giving up. I’m digging deep into the BIOS and looking at PCIe redriver settings. I’m not the only one having issues with PCIe 4.0 on this board, using risers, in fact someone just created an account and started a thread with that specific problem !

Help with WRX80E-Sage SE Render server

I’m going to post my research on this topic in their thread, I feel it’s something a few Asus users will be interested in.

EDIT : it’s been a few evenings of research and experimentation and the result is that my 3090 now works in PCIe 4.0 all the way to the end of the motherboard and riser. A lot went into getting this result, you may want to check it out as I’ve added a lot of background info that could help you no matter what platform you’re having PCIe trouble with. END OF EDIT

That being said, even in its incomplete state (still missing its actual RAM) this machine is very promising.

[Nefastor]

F Yeah ! The first 256 GB of RAM have arrived, just in time for a three-day weekend !

Samsung even added a touch of humor by printing a UKCA logo on their sticks

EDIT : it’s been a few days. My silence can be taken as an indication that this new machine is working really well. Nothing quite like the feeling of a brand new, much more powerful workstation ! And it has a lot of room for growth.

Now I need to spend days, maybe weeks, migrating my many different tools, projects and workloads to this new monster. I’ll be running the old and new machines side by side for a while.

I’m still waiting on fun components, notably a 6-port Gigabit Ethernet NIC based on two Intel i350 and a PCI x8 bridge. Why I’d need so much Ethernet is a long story.

[Nefastor]

The build continues. I’m holding off a little on buying the final 256 GB of RAM as the price of 64 GB sticks is currently dropping a little. This is a 10 K€ machine but hey, if I can save 100 € I’m not going to say no.

Speaking of savings, Black Friday on Amazon was good. I managed to score two 4 TB SN850X SSD’s for 365 € each, which is half-price. So now all three M.2 slots on the motherboard carry the same drive, 12 TB total. I’m not going to run out of space for a little while.

The 6-port NIC just arrived today. It’s a funny thing :

Cost me 100 € on eBay. The “catch” is that it has this weird non-standard PCI bracket. It does fit in a normal PCIe slot but you can’t screw it in place. I’ve zip-tied it for now while I brainstorm how to fabricate a suitable bracket.

This card words right away, no drivers needed, it is seen as 6 Intel i350 gigabit Ethernet ports and has SR-IOV support. Here’s the product page if you’re interested :

Silicom Ltd. | 1 Gigabit Ethernet Card

There is no kill like overkill :

This Asus motherboard wasn’t my first choice but it has grown on me. Three M.2 drives, a 3090, 8 Ethernet ports in total, and I still have 5 spare PCIe x16 slots. It’s also pretty quiet.