ASUS B650 Creator, to many PCIe lanes :)

Hardware Hub Motherboards
1

Hi,
I have a luxury problem here, my B650 Creator seems to have a PCIe switch, but it is not specified in the ASUS documentation for the board.
I trust lspci, I don’t think the output is wrong, lspci is used daily by thousands of Engineers around the world, what’s going on here?

To test it, I replaced my Intel P4618, which has a PEX 8718 chip, with a second Mellanox CX-3 and lspci shows 16-8-4 PCIe lanes for the three PCIe slots

Slot 1 - 6800XT
LnkSta: Speed 16GT/s, Width x16

Slot 2 - Mellanox CX-3
LnkSta: Speed 8GT/s, Width x8

Slot 3 - Mellanox CX-3 pro
LnkSta: Speed 8GT/s, Width x4 (downgraded)

----------------
[manja-02 ~]# lspci -vv | grep -i -A 60 Radeon
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c1) (prog-if 00 [VGA controller])
Subsystem: Sapphire Technology Limited NITRO+ Radeon RX 6800 XT
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 255
IOMMU group: 16
Region 0: Memory at fce0000000 (64-bit, prefetchable) [size=256M]
Region 2: Memory at fcf0000000 (64-bit, prefetchable) [size=2M]
Region 4: I/O ports at f000 [disabled]
Region 5: Memory at fbc00000 (32-bit, non-prefetchable) [size=1M]
Expansion ROM at fbd00000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
Status: D3 NoSoftRst+ PME-Enable+ DSel=0 DScale=0 PME-
Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 16GT/s, Width x16

----------------

[manja-02 ~]# lspci -vv | grep -i -A 60 Mellanox
05:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3]
Subsystem: Hewlett-Packard Company InfiniBand FDR/EN 10/40Gb Dual Port 544QSFP Adapter
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 131
IOMMU group: 21
Region 0: Memory at fa100000 (64-bit, non-prefetchable) [size=1M]
Region 2: Memory at f8000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at fa000000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Vital Product Data
Product Name: HP ConnectX-3 QSFP
Read-only fields:
[PN] Part number: 649281-B21
[EC] Engineering changes: A5
[SN] Serial number: IL242902TG
[V0] Vendor specific: HP 2P 4X FDR VPI/2P 40GbE CX-3 HCA
[RV] Reserved: checksum good, 0 byte(s) reserved
Read/write fields:
[V1] Vendor specific: N/A
[YA] Asset tag: N/A
[RW] Read-write area: 107 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 252 byte(s) free
End
Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
Vector table: BAR=0 offset=0007c000
PBA: BAR=0 offset=0007d000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 116W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
MaxPayload 512 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x8

------------

08:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
Subsystem: Mellanox Technologies Mellanox Technologies ConnectX-3 Pro Stand-up dual-port 40GbE MCX314A-BCCT
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 25
IOMMU group: 23
Region 0: Memory at fb100000 (64-bit, non-prefetchable) [size=1M]
Region 2: Memory at fa800000 (64-bit, prefetchable) [size=8M]
Expansion ROM at fb000000 [disabled] [size=1M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] Vital Product Data
Product Name: CX314A - ConnectX-3 Pro QSFP
Read-only fields:
[PN] Part number: MCX314A-BCCT
[EC] Engineering changes: A6
[SN] Serial number: MT1528K01653
[V0] Vendor specific: PCIe Gen3 x8
[RV] Reserved: checksum good, 0 byte(s) reserved
Read/write fields:
[V1] Vendor specific: N/A
[YA] Asset tag: N/A
[RW] Read-write area: 101 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 253 byte(s) free
[RW] Read-write area: 252 byte(s) free
End
Capabilities: [9c] MSI-X: Enable+ Count=128 Masked-
Vector table: BAR=0 offset=0007c000
PBA: BAR=0 offset=0007d000
Capabilities: [60] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
pcilib: sysfs_read_vpd: read failed: No such device
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 116W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 8GT/s, Width x4 (downgraded)
2

I find something like this more useful probing lanes, for each slot you are interested in:

lspci -s 03:00.0 -nnvv | egrep '^[0-9a-f]|Lnk(Sta|Cap):'

or you could strip -s 03:00.0 and get all slots

lspci -t is useful too if you haven’t found it yet

3

Yes, much more elegant, but you have to be able to remember that and the result doesn’t change :slight_smile:

[manja-02 ~]# lspci -t
-[0000:00]-+-00.0
           +-00.2
           +-01.0
           +-01.1-[01-03]----00.0-[02-03]----00.0-[03]--+-00.0
           |                                            +-00.1
           |                                            +-00.2
           |                                            \-00.3
           +-01.2-[04]----00.0
           +-01.3-[05-08]----00.0-[06-08]--+-01.0-[07]----00.0
           |                               \-02.0-[08]----00.0
           +-02.0
           +-02.1-[09-0f]----00.0-[0a-0f]--+-00.0-[0b]----00.0
           |                               +-08.0-[0c]--
           |                               +-0a.0-[0d]----00.0
           |                               +-0c.0-[0e]----00.0
           |                               \-0d.0-[0f]----00.0
           +-02.2-[10]----00.0
           +-03.0
           +-04.0
           +-08.0
           +-08.1-[11]--+-00.0
           |            +-00.2
           |            +-00.3
           |            \-00.4
           +-08.3-[12]----00.0
           +-14.0
           +-14.3
           +-18.0
           +-18.1
           +-18.2
           +-18.3
           +-18.4
           +-18.5
           +-18.6
           \-18.7
4

I knew there was a reason I bought this board, that and the pretty great iommu grouping.

1 Like
5

yes it is the go to AM5 VFIO board, the IOMMU groups couldn’t be better and also inexpensive

I have to thank @sotiris_bos that he drew my attention to the board, I wanted to buy the MSI 670E ACE first

1 Like
6

Much easier to read than wading thru vv. I just remember LnkCap and LnkSta, the rest is just normal lspci and grep, which I can do while asleep, almost. Regex is with learning, still useful 30+ years later.

7

Wow. I thought I was the only one who had this motherboard. Thanks for the finding this!

8

the days of repairing other people’s systems are over for me. But yes, it’s always important to learn something new

9

yes hardly anyone has the board, that made me hesitate a bit, but luckily the other options were more than twice as expensive, that was the decisive argument

1 Like
10

The PCIe switch is called B650 chipset. From the tech specs:

Screenshot from 2023-07-14 13-16-33
Screenshot from 2023-07-14 13-16-331652×307 76.3 KB

Also, look out for slot 3 turning off if you connect a m.2 into the third m.2 slot.

11

not really, a ZEN 4 CPU has 28 PCIe Lanes.
Four of them are for the chipset, leaving 24 for PCIe slots and M.2 connectors.
That means for this Board 8-8-4 and one M.2 slot.
I have two more M.2 NVMEs, one directly on the CPU and one via chipset.
The third M.2 is free because otherwise I would use PCIe lanes from the third PCIe slot.

I use 32 PCIe lanes directly from the CPU In total.

CPU Lanes 16-8-4-4
Chipsatz Lanes 4

Edit: The Intel P4618 has two controller and is plugged into the second PCIe slot

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
07:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54] (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s <64ns
                LnkSta: Speed 8GT/s, Width x4
08:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54] (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s <64ns
                LnkSta: Speed 8GT/s, Width x4
0b:00.0 Ethernet controller [0200]: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:1007]
                LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
                LnkSta: Speed 8GT/s, Width x4 (downgraded)
04:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkSta: Speed 16GT/s, Width x4
10:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive [1cc1:8201] (rev 03) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
                LnkSta: Speed 8GT/s, Width x4
1 Like
12

So which is it, 28 or 32? :joy:

The tech specs clearly state that you have x16/x0 or x8/x8 expansion slots from the CPU and two x4 M.2 slots from the CPU, plus four lanes connecting the chipset to the CPU, for a total of 28. That’s the correct number. The remaining expansion slots and third M.2 slot are hanging off the chipset.

2 Likes
13

if you can explain the output of lspci correctly, you have a reason to laugh, otherwise it just shows that you don’t understand it. :wink:

It should be 24 but 16-4-4-4-4 are 32, how can that be?

[manja-02 ~]# ./iommu.sh 
IOMMU Group 0:
        00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 1:
        00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 2:
        00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 3:
        00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 4:
        00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 5:
        00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 6:
        00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 7:
        00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 8:
        00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 9:
        00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 10:
        00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 11:
        00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 12:
        00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
        00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
        00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
        00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
        00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
        00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
        00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
        00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
        00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
        00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
IOMMU Group 14:
        01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
IOMMU Group 15:
        02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]

--------------
PCIe Slot 1 (Radeon 6800XT) = x16 PCIe4 CPU Lanes
--------------
IOMMU Group 16:
        03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
IOMMU Group 17:
        03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
IOMMU Group 18:
        03:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
IOMMU Group 19:
        03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]
#####

--------------        
M.2 Slot 1 (Seagate FireCuda 530) = x4 PCIe4 CPU Lanes
--------------
IOMMU Group 20:
        04:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01)
#####

-------------
PCIe Slot 2 (Intel P4618) = x8 PCIe3 CPU Lanes (2x4 because the P4618 hast two controller and one 8718 switch)
-------------
IOMMU Group 21:
        05:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8718] (rev ab)
IOMMU Group 22:
        06:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8718] (rev ab)
IOMMU Group 23:
        06:02.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8718] (rev ab)
IOMMU Group 24:
        07:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]
IOMMU Group 25:
        08:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]
#####


IOMMU Group 26:
        09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)

-----------
PCIe Slot 3 (ConnectX-3 Pro) = x4 PCIe3 Chipset Lanes
-----------
IOMMU Group 27:
        0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0b:00.0 Ethernet controller [0200]: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:1007]
#####

IOMMU Group 28:
        0a:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 29:
        0a:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0d:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 30:
        0a:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0e:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
IOMMU Group 31:
        0a:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0f:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
IOMMU Group 32:
        0a:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        10:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
        
---------------
M.2 Slot 2 (ADATA XPG SX8200) = x4 PCIe4 CPU Lanes
---------------
IOMMU Group 33:
        11:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive [1cc1:8201] (rev 03)
#####
        
IOMMU Group 34:
        12:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c1)
IOMMU Group 35:
        12:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
IOMMU Group 36:
        12:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
IOMMU Group 37:
        12:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
IOMMU Group 38:
        12:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
IOMMU Group 39:
        13:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8]

PCIe Connectivity all devices

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14d8]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:14d9]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x8, ASPM not supported
                LnkSta: Speed 16GT/s, Width x8
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM not supported
                LnkSta: Speed 16GT/s, Width x4
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db] (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM not supported
                LnkSta: Speed 8GT/s, Width x8
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db] (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 16GT/s, Width x4, ASPM not supported
                LnkSta: Speed 16GT/s, Width x4
00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db] (prog-if 00 [Normal decode])
                LnkCap: Port #2, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkSta: Speed 8GT/s, Width x4
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L1, Exit Latency L1 <64us
                LnkSta: Speed 16GT/s, Width x8 (downgraded)
02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479] (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
03:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6] (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
04:00.0 Non-Volatile memory controller [0108]: Seagate Technology PLC FireCuda 530 SSD [1bb1:5018] (rev 01) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkSta: Speed 16GT/s, Width x4
05:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8718] (rev ab) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
                LnkSta: Speed 8GT/s, Width x8
06:01.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8718] (rev ab) (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <4us
                LnkSta: Speed 8GT/s, Width x4
06:02.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8718 16-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8718] (rev ab) (prog-if 00 [Normal decode])
                LnkCap: Port #2, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <4us
                LnkSta: Speed 8GT/s, Width x4
07:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54] (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s <64ns
                LnkSta: Speed 8GT/s, Width x4
08:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54] (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s <64ns
                LnkSta: Speed 8GT/s, Width x4
09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <32us
                LnkSta: Speed 16GT/s, Width x4
0a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <32us
                LnkSta: Speed 8GT/s, Width x4
0a:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x1, ASPM L1, Exit Latency L1 <32us
                LnkSta: Speed 2.5GT/s, Width x1
0a:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 16GT/s, Width x1, ASPM L1, Exit Latency L1 <32us
                LnkSta: Speed 2.5GT/s, Width x1
0a:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #2, Speed 16GT/s, Width x1, ASPM L1, Exit Latency L1 <32us
                LnkSta: Speed 5GT/s, Width x1
0a:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #12, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 2.5GT/s, Width x1
0a:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #13, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 2.5GT/s, Width x1
0b:00.0 Ethernet controller [0200]: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] [15b3:1007]
                LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s unlimited
                LnkSta: Speed 8GT/s, Width x4 (downgraded)
0d:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                LnkSta: Speed 2.5GT/s, Width x1
0e:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                LnkSta: Speed 5GT/s, Width x1
0f:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01) (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                LnkSta: Speed 2.5GT/s, Width x1
10:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01) (prog-if 01 [AHCI 1.0])
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                LnkSta: Speed 2.5GT/s, Width x1
11:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive [1cc1:8201] (rev 03) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
                LnkSta: Speed 8GT/s, Width x4
12:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c1) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
12:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
12:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6] (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
12:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7] (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
12:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16
13:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8] (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkSta: Speed 16GT/s, Width x16

where are the pros here, @wendell, @gnif, please take a look at it!

edit: after checking the Board manual again and lstopo, I got wrong that the 3 PCIe Slot comes from the chipset and not from the CPU, but that still doesn’t explain 16-8 PCie Lanes for the first two PCIe Slots.

AMD Ryzen™ 7000 Series Desktop Processors*
2 x PCIe 4.0 x16 slots (support x16 or x8/x8 modes)
AMD B650 Chipset
1 x PCIe 4.0 x16 slot (supports x4 mode)**
1 x PCIe 4.0 x1 slot

So the CPU lanes are distributed like this, which still means 32 CPU lanes :slight_smile:
PCIe Slot 1 = 16
PCIe Slot 2= 8
M2. 1 = 4
M.2 2 = 4

Which, according to AMD, can’t be true either

lstopo-all
lstopo-all878×830 56.3 KB

1 Like
14

First slot isn’t x16 but x8 in your case. That explains the remaining 8 lanes. Once you plug in a card in the second slot, the first slot becomes x8. That’s normal for boards with x8/x8 switch.

It’s 8+8+4+4 = 24. Everything according to specs

edit: this x8/x8 is not bifurcation. This is a physical hardware switch. Usually seen on the block diagram. So don’t get confused with this.

Thanks for raising attention to this board. Looks like the cheapest AM5 board with x8/x8 that also lists ECC compatibility. I guess my search has ended, good package.

15

You need to look at the lspci -t output and then map the lane widths thru that.

If you have x4 going to chipset, then you can only ever get x4 bandwidth thru that connection even if you have multiple x4 and/or x1 devices hanging off that chipset, they have to share the bandwidth with the chipset.

16

It is technically bifurcation because it’s a single x16 IOU port being bifurcated to x8/x8. If it were a true PCIe switch it would show up in lspci/lstopo as a separate device with the full e.g. Gen5 x16 lane allocation at 512 GT/sec, with the devices in slots one and/or two downstream of it. That doesn’t appear to be the case here.

However, there is a relatively simple circuit to detect slot presence and route the lanes electrically between slots, and—as you said—some board makers will show this circuit on the block diagram as a switch.

17

it has to be 8 Lanes, but I’ve never seen it that way, GPU-Z also shows 16 lanes for the GPU, a PCIe throughput test would be interesting

edit: it is x8 PCIe4, so no switch functionality, just a strange peculiarity of the board

14.326557820216237 GB/s

launch_binary_x64
dist/bin/x64/3DMarkPCIExpress.exe


launch_binary

C:\ProgramData\UL\3DMark\chops\dlc\pci-express-test\dist\bin\x64\3DMarkPCIExpress.exe





PciExpressBandwidthP
14.326557820216237
OK


PciExpressFpsP
15.030041509644963
OK

18

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.