10G Ethernet via u.2>pci-e slot riser on Asus WS x570 ACE

Did you suceed in getting the u.2>pcie riser to work on the Asus ws pro x570? @aBav.Normie-Pleb ?

Any success by anyone?

@igoodman

Unfortunately nothing positive:

ASUS Level 2 support just shrugged the question off ā€œwe donā€™t support the use of PCIe adapters on the U.2 portā€.

I have not been able to find out on myself what difference there is between an NVMe PCIe x4 SSD and a ā€œgeneral PCIe deviceā€ that also uses 4 lanes.

Itā€™s not the X570 chipset PCIe in general: M.2_2 (PCIe 4.0 x2) comes from the chipset and there you can adapt a GPU to, for axample (my configuration doesnā€™t need GPU performance but one to boot up so I connected a Radeon Pro WX 4100 there to have the large PCIe slots free for two HBAs and a dual 40 GbE AIC).

Signalling @wendell

Do you know a technological reason why an NVMe SSD works normally with an X570 U.2 port but not other devices?

well, without redrivers, pcie4.0 would be unsupported on any series of adapters. pcie3 might be ok, but maybe not.

Sometimes you see bios not enumerate all the pcie busses for certain kinds of devices. Itā€™s rare, but it happensā€¦ but even pcie 4.0 root port in pcie3.0 mode doesnā€™t have the same signal integrity as native pcie3ā€¦

1 Like

Also tried connecting a PCIe 2.0 x4 device, no luck (Sonnet USB 3.0 controller).

Signal integrity loss caused by the adapters should not be an issue, for an extreme POC Iā€™ve successfully tried going PCIe slot ā†’ M.2 ā†’ U.2 ā†’ PCIe slot ā†’ normal PCIe AIC and that worked fine with PCIe 3.0 devices.

1 Like

Probably the bios not enumerating things other than mass storage on that root port then

Might be able to echo 1 into something invlinux to rescan he bus tho and ā€œhotpugā€ it

2 Likes

Thanks for your answer @wendell :slightly_smiling_face:

This seems like light at the end of the tunnel!

(maybe incomming train, but letā€™s give it a try)

I am not to versed with PCI-e rescans.

Where to echo to?

/sys/bus/pci_express/???whatā€“device???

/sys/bus/pci/rescan

Do you still have the adapter @aBav.Normie-Pleb ?

Thanks a lot!

Yes, I still have all (and other Gen4) adapters for PCIe/M.2/U.2 bifurcation.

For Gen4 testing I only have a 3090 and Samsung PM1733 U.2 SSDs. I donā€™t do anything risky with the 3090 since as you probably know itā€™s irreplaceable at the moment but the Samsung SSDs can go beyond 7GB/s almost maxing out PCIe 4.0 x4.

Seems like I should look into how to do a livestream playing with the various adapter configurations and go through all questions that might come up for users that are new to PCIe adapters - I feel sooo in demand :stuck_out_tongue:

2 Likes

Those numbers are amazing. :heart_eyes:

Are those results from the PM1733 attached to the U.2 Port on the Asus WS ACE x570???

(via the adapter or directly?)

If YES, that would mean that the u.2 port via the x570 chipset supports pci-e 4.0!

A livestream adapter session would be great! Would join in definitely! :slight_smile:

@igoodman

Sorry, totally forgot about the U.2 port ASUS claims to be Gen3-only on their spec page after I have finally gotten some Gen4 U.2 SSDs.

As suspected, it in fact supports Gen4 and not just Gen3 and ASUS was just not thorough enough to test it properly.

The 7 GB/s test screenshot previously posted shows ideal performance with a 5950X and directly connected to CPU PCIe lanes.

Here are the results of a PM1733 connected to the X570 U.2 port with a more ā€œnormalā€ CPU (3900 non-X), the X570 chipset detour is eating some performance but itā€™s still quite usable:

Configuration:

  • R9 PRO 3900 (SC up to 4.4 GHz)

  • 128 GB DDR4-2666 ECC UDIMM

  • PCIe x16 #1 (PCIe lanes directly from CPU) running at x8 Gen3: Broadcom HBA 9400-8i8e #1

  • PCIe x16 #2 (PCIe lanes directly from CPU) running at x8 Gen3: Broadcom HBA 9400-8i8e #2

  • PCIe x16 #3 (PCIe lanes via X570 chipset) running at Gen3 x8: Intel XL710 QDA2

  • U.2 (PCIe lanes via X570 chipset) running at Gen4 x4: Samsung PM1733 SSD

  • PCIe x1 from X570 chipset: ASRock Rack PAUL for stabilty testing with a passive x1 PCIe riser to be able to mount the low-profile AIC in a regular case

  • Win 10 20H2 with all Windows Updates installed, running on an USB 3 SSD

  • All AIC firmwares updated to latest versions, as well as drivers

  • ASUS Pro WS X570-ACE, UEFI 3302, AGESA 1201

  • All security/virtualization features enabled

  • Removed transceivers from the Intel ethernet adapter in the x8 chipset slot to have the best conditions possible for the PM1733 on the U.2 port that has to share the bandwidth of the CPU-X570 chipset interface (Gen4 x4) with PCIe x16 #3.

She ainā€™t pretty but 0 crashes in about a week of testing:

3 Likes

Would you please be so kind to post good pictures of all sides of the pm1733?

Would like to see what those openings are for and what the debug port between them looks like, and other such details. There are no good pictures on the Web so far i think.

Hope the quality is good enough.

Before seeing one in person I had first thought that the four air vents were some kind of weird new connector.

Nothing at all on any other side of the SSD (except of course the standard SFF-8639 connector) - but its ā€œfaceā€ looks like it has an existential crisis

:expressionless:

They probably only support gen 3 if you run it outside of their spec youā€™re on your own

Well, then you cannot get any PCIe Gen4 SSD for the U.2 port since you cannot set the U.2 port to Gen3 manually - ASUS even removed the PCIe Gen selection options from the AMD PBS UEFI menuā€¦

1 Like

air-vents, sweet. The thought about a new connector is exactly what i feared as well. Same debug-port apparently.
Thank you very much

Do you by any chance know if you can get firmware updates for Samsung enterprise SSDs as a mere end customer?

i do not know, i have tried for PM1725Bs though and failed.

The only people that have anything available for those drives are DELL and HPE.
I looked into both and it seems like one could maybe extract actual firmware from those packages, but still nothing i can tell you for sure.

Flashing the assumed FW is also an issue, or a non issue, it depends since the NVME standard defines two commands for such purposes.

One downloads the ā€œFWā€ to the drive and the other, verifys and puts a previously downloaded FW into a FW-Slot.
FW-Slots can then be ā€œactivatedā€ to be actually in use.
There is still a ā€œBoot-Slot?ā€ Nothing i could really find any good documentation on, or that i could say is like safe to experiment with and not brick a drive.

Thanks for your in depth answer @aBav.Normie-Pleb :+1:

2 follow up questions:

Do you use a special U.2 cable specifically certified for pci-e 4.0 or just a regular pci-e 3.0 cable?

Do you see any caveats against using a riser of the m.2 slot directly connected to the CPU for attaching any non storage related hardware?

Yes, cable/adapter quality is something that is very, very important, here!

Delock has basically become my go-to manufacturer for these kinds of adapters and cables.

When testing such a configuration I enable all error reporting features in AMD CBS ā†’ NBIO options (AER, ACS, these sre disabled with UEFI default settings), stress the PCIe device in question as hard as it can be and check if WHEA errors start to show up in Windows Event Viewer.

On a different system (ASRock X570 Taichi Razer Edition) where Iā€™d like to use an Icy Dock U.2 NVMe backplane Iā€™m currently trying to fix the occurence of such errors when using the CPUā€™s M.2 port for an NVMe slot in the backplane, the slot that is connected via an M.2 from the X570 chipset is completely fine.

Iā€™ve never had issues with PCIe riser builds that were confirmed to be error-free at the time of their first use.

Regarding the Pro WS X570-ACEā€™s first M.2 slot that is handled by the CPU: Didnā€™t have any issues with non-storage PCIe devices, tested 10 and 40 Gb ethernet adapters and a Radeon Pro WX4100 with it, these could be used just as if they had been installed in regular PCIe slots.

3 Likes

I also have this Moboard and same cpu.
(for a Pro branded product I must say the pro feature list is kinda disapointing, still looking for a place to plug in a TPM module and get bitlocker working)

From what I understand of the contradicting documentation on how they do their pcie magic on the botom slot, the M.2, U.2 and bottom Pcie slot all share bandwith, and had the idea that the use of the M.2 and U.2 ports was and OR situation and not an AND one, though I guess that if you try and limit the bandwith to pcie 3.0 speeds it might work, but that could be the source of the issue, having the chipset not recognize one of the devices pluged into these two ports at the same time.
Also, thanks for the info on the compatibility with 4.0 speeds on the U.2, ASUS really should have better suport for these kind of product when they charge a premium for a Pro/ WS badge :smiley:

PS: Iā€™m kinda jelous of your 3090, if you think they are hard to get in the US try getting one in south americaā€¦ the retail pricing is about 3.5k usd (scalpers charge about 5k) Iā€™m using a borrowed gts 250 lol, its not much but gives me a display output in these tought gpu times :slight_smile:

Donā€™t know if it helps but you can change BitLockerā€™s settings in gpedit.msc so that you can use it for the operating system drive with a password only, not requiring additional TPM hardware.

1 Like