ASUS X570 WS ACE with Intel Microchip PCIE Switch/Hub no BAR space, HELP pls

Hardware Hub Motherboards
1

So I’m pretty sure I know the anwser, ask asus for a special Bios or it won’t work…
But maybe there is another option.

Long story short I got two

IMG_20230112_181045961
IMG_20230112_1810459611920×2560 625 KB

IMG_20230119_221236844
IMG_20230119_2212368441920×2560 345 KB

And I’d like to use them in my x570 ws ace server. However dmesg shows:

What I already tried:
a bunsh of livecd some arch some debian same issue
CPU: r5 5500 and r9 5950x same issue

Above 4G decoding is enabled

pci=realloc
MMIO no space - shows quickly
this just craches the during boot. I just get the busybox but since the doesnt seem to be neither more or less included I cant scroll through the dmesg…

chipset lanes and cpu it makes no difference

a couple is the recent bios’s no difference

In freebsd they maybe working nothing seems to stand out in the dmesg however I’m not a bsd pro

I ordered the oculink to sff-8639 cables just today so I cant test with ssds

If someone has any insides :slight_smile:

2

Unfortunately I don’t have any experience with Intel PCIe Switch AICs for NVMe SSDs but I’ve tested Broadcom x16-to-x32 PCIe Gen3 and Gen4 switches on the very same ASUS Pro WS X570-Ace without such issues.

When looking at your photos my DANGER sense is tingling: Did the cards actually come without any heatsink?! You see the intended mounting holes in the PCB.

ANY active PCIe switch I encountered so far required a heatsink and additionally active cooling - don’t know about the Intel one specifically, couldn’t see a TDP or typical power draw value on their website.

I hope the silicon didn’t fry itself overheating.

3

Relax they have heatsinks, I just wanted to know the switchchip model. Also in the 2nd pic the card is in the top slot with the heatsink

4

Another gut rumbling I got is from Intel’s website mentioning the AICs needing to be connected with a special cable to their server motherboards to function.

Proprietary DRM stuff going on?

1 Like
5

Maybe they dont have a public datasheet they have some headers for bmc

IMG_20230119_224654524
IMG_20230119_2246545241920×2560 300 KB

but I doubt thats is the problem.
I also can test it with other boards… z690 tachyon or x10qbi but I was hoping to avoid that…
May I ask what bios you are using?

6

Have you checked running Windows (see Windows drivers on Intel’s website) with the card and check if the latest public firmware version is installed?

On my Broadcom PCIe Switches that came new from a big destributor horrible non-public pre-production firmwares were installed with which they didn’t even function properly.

1 Like
7

I tried however Its shown with the yellow triangle and I was missing 1 driver however the driver is for winows server 16/19 and i test with win10

8

I just tried it on the z690 tachyon and the dmesg is pretty identical

9

Interesting little cards.

I see the nouveau module crashing. Maybe install the nvidia driver?

I don’t see anything out of the ordinary in the log. What does lspci -v say?

Also, it looks like Intel issued a firmware upgrade:

Not to give you the total runaround, but maybe try with one of the cutting-edge distros (Fedora 37, Ubuntu 22.04 or later)

Finally, I’d try with bare minimum hardware first (iGPU if possible, single boot drive, this card) reset UEFI to default. Go from there.

10

Grasping at straws…

BIOS options to try toggling:

  • kvm: disabled by bios SVM = Enable
  • IOMMU = Enable
  • Above 4G Encoding = Enable

Kernel 5.19 is available in hwe-edge, you could try that.

Kernel command line options to try:

  • pci=assign-busses,realloc
11

lspci -v:

I tried with nvidia drivers on manjaro

“I don’t see anything out of the ordinary in the log.”

[    1.482672] Call Trace:
[    1.482673]  <TASK>
[    1.482675]  amd_iommu_enable_interrupts+0x183/0x2a0
[    1.482678]  ? e820__memblock_setup+0x85/0x85
[    1.482681]  state_next+0x1ed/0x300
[    1.482684]  ? e820__memblock_setup+0x85/0x85
[    1.482685]  iommu_go_to_state+0x28/0x2d
[    1.482687]  amd_iommu_init+0x15/0x4b
[    1.482688]  ? e820__memblock_setup+0x85/0x85
[    1.482690]  pci_iommu_init+0x1a/0x44
[    1.482691]  do_one_initcall+0x48/0x1d0
[    1.482695]  do_initcalls+0x12f/0x155
[    1.482698]  kernel_init_freeable+0x162/0x1b1
[    1.482699]  ? rest_init+0x100/0x100
[    1.482702]  kernel_init+0x1b/0x150
[    1.482703]  ? rest_init+0x100/0x100
[    1.482705]  ret_from_fork+0x22/0x30
[    1.482707]  </TASK>
[    1.482708] ---[ end trace 5c00049607b1db6f ]---

This doesnt seem right

12

Hold the presses pci=assign-busses,realloc looks better

  1. amd iommu doesnt crash and 2. BAR’s assign on 3rd try
13

It is not “right” but unrelated to your cards. This is the nouveau driver crashing.

Your two cards (PMC-Sierra Inc. PM8533 PFX 48xG3 PCIe Fanout Switch) are detected and drivers are loaded (pcieport).

The cards don’t seem to be identical. One is reporting Memory behind bridge ... [size=8M] and a single memory controller as subdevice; the other one reports it as disabled (Memory behind bridge: [disabled]) but lists 7 subdevices. All in their own IOMMU groups - nice!

I’d say you have two (differently) working PCIe cards. Report what you find when the cables arrive.

My guess is that a valid install of the nvidia driver will fix the issue of the nouveau driver crashing.

14

the latest dmesg and lspci are just with one card.

With both cards and Kernel command line options pci=reealloc or pci=assign-busses,realloc the kernel crashes during boot.
With one card it seems to be working however BAR’s init needs a couple of tries. But adding PCI Devices also causes the kernel to crash during boot
But I cant figure out how to check the dmesg in busybox…

15

Just to be sure: You changed IOMMU from “Auto” to “Enabled” in the ASUS BIOS?

CSM disabled?

16

CSM is disabled however I just found out that the cmos battery died… So some test may have been with 4g Decode off, SVM disabled more testing needed
Also I dont have an IOMMU option in the bios just AMD SVM.

17

It’s there. I’m not in front of my Asus X570 system at the moment but you have to dig a bit into AMD CBS I believe.

1 Like
18

@Blissed

Yes, its there, explore the AMD CBS menus.

Enable:

IOMMU
ACS
DMA Protection (not DMAr, that can prevent posting until you reseat a different CPU)
Ten-bit Tag Support

The only thing ASUS hides in their AM4 BIOS is the option for PCIe Advanced Error Reporting for some unkown reason.

1 Like
19

Not ARI?

20

Yeah, ARI also, didn’t have the complete option set memorized.

Should be able to reboot that system soon, will make some screenshots of the settings.

1 Like