A far easier and cheaper way to move PCIe lanes for GPUs

After watching the AI council video, I figured some folks would like the cheaper, simpler, still perfectly working version.

A SlimSAS card is $20 and 2 SlimSAS cables is $40. Voila you just moved 16 PCIe lanes 1M for $60USD. If you want to split it up into 8 and 8, you do it in the BIOS as normal.

1 Like

It’s likely >Gen3 speeds won’t work reliably with 1M cables, without additional redrivers or retimers.

1 Like

I can do full PCIe 4.0 over 1M cables so :person_shrugging:
Personally, I’ve seen it happen a lot of times, I think they saw a potential problem and engineered around it without determining if its real.

And for clarity, many of those times I’ve seen it, I did it :slight_smile:

Not with a $20 SlimSAS card or even a $60 budget.

There is a never-ending thread about why not. This is just the tl;dr:

If you made it work with PCIe AER (advanced error reporting) turned on and have no errors, then you did it and you should share. :slightly_smiling_face:

See this is whats blowing my mind here, I literally just did it, and it didn’t take anything special. I was waiting on a PSU so I was running my GPU outside the case with another PSU. And it was absolutely no problem at all. I verified it was working to speed and went on about my business, unaware I’d just don’t something hard.

And I’d love to share the specific hardware, but this forum keeps blocking my messages with links in them.

2 Likes

Hi, links and pictures, are automatically restricted for new users, to slow bots and spammers.

You’ve posted a few times now, maybe try again?

I would beninterrested in a card that is affordable, and allows PCI movement

I have a bunch of SAS cards, but they don’t expose PCI lanes (none of them are even Tri- mode cards, as they are still not quite affordable enough for me)

If links still blocked, try the old (dot) com trick?

If you’re seeing PCIe 4.0 work over 1M SlimSAS cables, that’s impressive for the price. Most guides warn about Gen3+ signals failing without redrivers or retimers, so your setup could be a useful reference. Sharing exact hardware and BIOS settings (even without links) would help others replicate it safely.

2 Likes

Ok, first, there is a piece missing from my list, a slimsas x16 slot, thats another $30-$40. So, around $100. Still less than a sas card with repeaters Second, I can’t find one of my 1M cables, but I did find 2 .5m so I have LSPCI from both configs.


Thats the slimsas card.


With the x16 adapter.

This time around, I also ran clpeak, to get an actual bandwidth number, and I don’t believe I am getting all 16 lanes, just 8 of them.

2 .5m:

LSPCI:
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 44 [Radeon RX 9060 XT] (rev c0) (prog-if 00 [VGA controller])
	Subsystem: Tul Corporation / PowerColor Device 2437
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 86
	IOMMU group: 16
	Region 0: Memory at f800000000 (64-bit, prefetchable) [size=16G]
	Region 2: Memory at fc00000000 (64-bit, prefetchable) [size=256M]
	Region 4: I/O ports at f000 [size=256]
	Region 5: Memory at f6b00000 (32-bit, non-prefetchable) [size=512K]
	Expansion ROM at f6b80000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [64] Express (v2) Legacy Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
		LnkSta:	Speed 32GT/s, Width x16
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: ReqEn+
			 IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
			 10BitTagReq+ OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 32GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00000  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [200 v1] Physical Resizable BAR
		BAR 0: current size: 16GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB
		BAR 2: current size: 256MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB
	Capabilities: [240 v1] Power Budgeting <?>
	Capabilities: [270 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-
		LaneErrStat: 0
	Capabilities: [2a0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [2d0 v1] Process Address Space ID (PASID)
		PASIDCap: Exec+ Priv+, Max PASID Width: 10
		PASIDCtl: Enable+ Exec- Priv-
	Capabilities: [320 v1] Latency Tolerance Reporting
		Max snoop latency: 1048576ns
		Max no snoop latency: 1048576ns
	Capabilities: [410 v1] Physical Layer 16.0 GT/s
		Phy16Sta: EquComplete- EquPhase1- EquPhase2- EquPhase3- LinkEquRequest-
	Capabilities: [450 v1] Lane Margining at the Receiver
		PortCap: Uses Driver-
		PortSta: MargReady+ MargSoftReady-
	Capabilities: [500 v1] Physical Layer 32.0 GT/s
		Phy32Cap: EqualizationBypass+ NoEqualizationNeeded-
			  ModTsMode0+ ModTsMode1- ModTsMode2-
		Phy32Ctl: EqualizationBypassDis- NoEqualizationNeededDis-
			  Modified TS Usage Mode: PCI Express
		Phy32Sta: EquComplete+ EquPhase1+ EquPhase2+ EquPhase3+ LinkEquRequest-
			  Received Enhanced Link Behavior Control: Equalization bypass to highest rate support
			  ModTsRecv- TxPrecodeOn- TxPrecodeReq- NoEqualizationNeededRecv-
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu

CLPEAK:

Platform: AMD Accelerated Parallel Processing
  Device: gfx1200
    Driver version  : 3649.0 (HSA1.1,LC) (Linux x64)
    Compute units   : 16
    Clock frequency : 2620 MHz

    Global memory bandwidth (GBPS)
      float   : 270.41
      float2  : 281.76
      float4  : 286.50
      float8  : 300.49
      float16 : 298.55

    Single-precision compute (GFLOPS)
      float   : 7913.89
      float2  : 8824.82
      float4  : 9219.55
      float8  : 9591.53
      float16 : 9715.56

    Half-precision compute (GFLOPS)
      half   : 10399.02
      half2  : 19554.80
      half4  : 18973.15
      half8  : 20401.22
      half16 : 19495.62

    Double-precision compute (GFLOPS)
      double   : 412.51
      double2  : 421.26
      double4  : 427.41
      double8  : 428.57
      double16 : 423.81

    Integer compute (GIOPS)
      int   : 2577.07
      int2  : 2597.87
      int4  : 2633.82
      int8  : 2607.76
      int16 : 2572.70

    Integer compute Fast 24bit (GIOPS)
      int   : 11524.70
      int2  : 11738.36
      int4  : 11693.35
      int8  : 11519.03
      int16 : 9256.39

    Integer char (8bit) compute (GIOPS)
      char   : 12468.76
      char2  : 6940.71
      char4  : 6741.26
      char8  : 6063.06
      char16 : 5577.70

    Integer short (16bit) compute (GIOPS)
      short   : 12205.95
      short2  : 15161.38
      short4  : 14763.57
      short8  : 14509.21
      short16 : 14901.87

    Transfer bandwidth (GBPS)
      enqueueWriteBuffer              : 21.81
      enqueueReadBuffer               : 7.95
      enqueueWriteBuffer non-blocking : 21.74
      enqueueReadBuffer non-blocking  : 7.96
      enqueueMapBuffer(for read)      : 596523.25
        memcpy from mapped ptr        : 7.98
      enqueueUnmap(after write)       : 842150.44
        memcpy to mapped ptr          : 21.85

    Kernel launch latency : 27.61 us

1 1m:

LSPCI:
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 44 [Radeon RX 9060 XT] (rev c0) (prog-if 00 [VGA controller])
	Subsystem: Tul Corporation / PowerColor Device 2437
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 86
	IOMMU group: 16
	Region 0: Memory at f800000000 (64-bit, prefetchable) [size=16G]
	Region 2: Memory at fc00000000 (64-bit, prefetchable) [size=256M]
	Region 4: I/O ports at f000 [size=256]
	Region 5: Memory at f6b00000 (32-bit, non-prefetchable) [size=512K]
	Expansion ROM at f6b80000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [64] Express (v2) Legacy Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- TEE-IO-
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- FltModeDis-
		LnkSta:	Speed 32GT/s, Width x16
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
			 10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
			 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
			 FRS-
			 AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
			 AtomicOpsCtl: ReqEn+
			 IDOReq- IDOCompl- LTR+ EmergencyPowerReductionReq-
			 10BitTagReq+ OBFF Disabled, EETLPPrefixBlk-
		LnkCap2: Supported Link Speeds: 2.5-32GT/s, Crosslink- Retimer+ 2Retimers+ DRS-
		LnkCtl2: Target Link Speed: 32GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
			 EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
			 Retimer- 2Retimers- CrosslinkRes: unsupported, FltMode-
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00000  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP-
			ECRC- UnsupReq- ACSViol- UncorrIntErr- BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+
			ECRC- UnsupReq- ACSViol- UncorrIntErr+ BlockedTLP- AtomicOpBlocked- TLPBlockedErr-
			PoisonTLPBlocked- DMWrReqBlocked- IDECheck- MisIDETLP- PCRC_CHECK- TLPXlatBlocked-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+ CorrIntErr- HeaderOF-
		AERCap:	First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [200 v1] Physical Resizable BAR
		BAR 0: current size: 16GB, supported: 256MB 512MB 1GB 2GB 4GB 8GB 16GB
		BAR 2: current size: 256MB, supported: 2MB 4MB 8MB 16MB 32MB 64MB 128MB 256MB
	Capabilities: [240 v1] Power Budgeting <?>
	Capabilities: [270 v1] Secondary PCI Express
		LnkCtl3: LnkEquIntrruptEn- PerformEqu-0.
		LaneErrStat: 0
	Capabilities: [2a0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [2d0 v1] Process Address Space ID (PASID)
		PASIDCap: Exec+ Priv+, Max PASID Width: 10
		PASIDCtl: Enable+ Exec- Priv-
	Capabilities: [320 v1] Latency Tolerance Reporting
		Max snoop latency: 1048576ns
		Max no snoop latency: 1048576ns
	Capabilities: [410 v1] Physical Layer 16.0 GT/s
		Phy16Sta: EquComplete- EquPhase1- EquPhase2- EquPhase3- LinkEquRequest-
	Capabilities: [450 v1] Lane Margining at the Receiver
		PortCap: Uses Driver-
		PortSta: MargReady+ MargSoftReady-
	Capabilities: [500 v1] Physical Layer 32.0 GT/s
		Phy32Cap: EqualizationBypass+ NoEqualizationNeeded-
			  ModTsMode0+ ModTsMode1- ModTsMode2-
		Phy32Ctl: EqualizationBypassDis- NoEqualizationNeededDis-
			  Modified TS Usage Mode: PCI Express
		Phy32Sta: EquComplete+ EquPhase1+ EquPhase2+ EquPhase3+ LinkEquRequest-
			  Received Enhanced Link Behavior Control: Equalization bypass to highest rate support
			  ModTsRecv- TxPrecodeOn- TxPrecodeReq- NoEqualizationNeededRecv-
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu

CLPEAK:
Platform: AMD Accelerated Parallel Processing
  Device: gfx1200
    Driver version  : 3649.0 (HSA1.1,LC) (Linux x64)
    Compute units   : 16
    Clock frequency : 2620 MHz

    Global memory bandwidth (GBPS)
      float   : 269.90
      float2  : 278.95
      float4  : 287.72
      float8  : 291.13
      float16 : 295.78

    Single-precision compute (GFLOPS)
      float   : 7886.64
      float2  : 8495.42
      float4  : 9035.06
      float8  : 9380.99
      float16 : 10107.19

    Half-precision compute (GFLOPS)
      half   : 10849.87
      half2  : 19641.99
      half4  : 22137.11
      half8  : 21278.89
      half16 : 19187.52

    Double-precision compute (GFLOPS)
      double   : 416.26
      double2  : 424.76
      double4  : 430.64
      double8  : 430.42
      double16 : 426.93

    Integer compute (GIOPS)
      int   : 2581.67
      int2  : 2615.94
      int4  : 2653.49
      int8  : 2622.71
      int16 : 2612.27

    Integer compute Fast 24bit (GIOPS)
      int   : 11560.11
      int2  : 10926.58
      int4  : 11814.78
      int8  : 11913.64
      int16 : 9769.06

    Integer char (8bit) compute (GIOPS)
      char   : 12414.99
      char2  : 6479.79
      char4  : 6265.91
      char8  : 5600.18
      char16 : 5194.58

    Integer short (16bit) compute (GIOPS)
      short   : 12302.38
      short2  : 15925.47
      short4  : 15078.00
      short8  : 15484.33
      short16 : 15052.46

    Transfer bandwidth (GBPS)
      enqueueWriteBuffer              : 21.98
      enqueueReadBuffer               : 8.02
      enqueueWriteBuffer non-blocking : 21.93
      enqueueReadBuffer non-blocking  : 8.03
      enqueueMapBuffer(for read)      : 447392.41
        memcpy from mapped ptr        : 8.01
      enqueueUnmap(after write)       : 604925.00
        memcpy to mapped ptr          : 21.88

    Kernel launch latency : 27.00 us

(continuing from above)

I have an actual benchmark thats made to measure pcie bandwidth, I made it for TB but it’ll do fine for this) . Vector copies and sensor data. But on this machine running bazzite I can see opencl devices, but my benchmark using ILGPU can’t see any.

This machine is:
HP OmniDesk
AMD Ryzen 7 8700G
32GB DDR5 5200 MT/s
Micron SSD (1TB)
AMD WIFI (mini-card)

I’m not 100% sure the 8700g has enough lanes for all 16 on the slot to work at once, so this may not be the machine to confirm all 16 lanes work. But lspci shows physical 16 GT/s, and clpeak shows just under 22GB/s. Someone tell me if I am wrong, but that sure looks like its full speed to me and the lspci shows zero errors.

It occurs to me that even if you can’t get all 16 lanes to one normal x16 card, you can still get 8 lanes to 2 cards (with another slot) or 16 to a dual accelerator card like a dual Tesla or one of those upcoming dual B60’s.

One way to check for PCIE errors on that cable, would be to install AMD SMI tool, and check the recovery-count values.

Someone did something similar for an RTX 6000 Pro via nvidia-smi.

1 Like

The count in the lspci output above is zero, but more ways to confirm that is always better.

But I think I’m going to have to ditch bazzite on this machine to poke at it any further. I’m missing metrics, including that one. I did play many games at 32x9 1440p for hours at a rock solid 144fps vsync’d. Not that that means anything at all.

Let me put something else on here and get out an Nvidia card and maybe the B580…

I’m pretty sure that the G-series Ryzen CPUs only expose 8 lanes for PCIE graphics.

My curiosity’s piqued, because I would dearly love to have my two R9700s running on PCIE 5.0 x8 CPU connections for some balanced bandwidth, but I’m stuck with a consumer motherboard that limits one of them to x4 through the chipset. It doesn’t so much matter at the moment, but when llama.cpp’s tensor parallelism PR goes live…I’m really going to want to use it.

Ya, that appears to be the case. (From Ubuntu, same hardware)

amd-smi metrics (0 errors, width 8, speed 16 GT/s)

GPU: 0
    USAGE:
        GFX_ACTIVITY: 100 %
        UMC_ACTIVITY: 1 %
        MM_ACTIVITY: 0 %
        VCN_ACTIVITY: [0 %, N/A, N/A, N/A]
        JPEG_ACTIVITY: [N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A, N/A]
        GFX_BUSY_INST: N/A
        JPEG_BUSY: N/A
        VCN_BUSY: N/A
    POWER:
        SOCKET_POWER: 24 W
        GFX_VOLTAGE: 1102 mV
        SOC_VOLTAGE: 810 mV
        MEM_VOLTAGE: 1350 mV
        THROTTLE_STATUS: THROTTLED
        POWER_MANAGEMENT: ENABLED
    CLOCK:
        GFX_0:
            CLK: 3395 MHz
            MIN_CLK: 500 MHz
            MAX_CLK: 3395 MHz
            DEEP_SLEEP: DISABLED
        MEM_0:
            CLK: 1258 MHz
            MIN_CLK: 96 MHz
            MAX_CLK: 1258 MHz
            DEEP_SLEEP: DISABLED
        VCLK_0:
            CLK: 25 MHz
        DCLK_0:
            CLK: 25 MHz
        FCLK_0:
            CLK: 2016 MHz
            MIN_CLK: 307 MHz
            MAX_CLK: 2400 MHz
            DEEP_SLEEP: DISABLED
        SOCCLK_0:
            CLK: 1371 MHz
            MIN_CLK: 417 MHz
            MAX_CLK: 1476 MHz
            DEEP_SLEEP: DISABLED
    TEMPERATURE:
        EDGE: 55 °C
        HOTSPOT: 83 °C
        MEM: 50 °C
    PCIE:
        WIDTH: 8
        SPEED: 16 GT/s
        BANDWIDTH: N/A
        REPLAY_COUNT: N/A
        L0_TO_RECOVERY_COUNT: N/A
        REPLAY_ROLL_OVER_COUNT: N/A
        NAK_SENT_COUNT: N/A
        NAK_RECEIVED_COUNT: N/A
        CURRENT_BANDWIDTH_SENT: N/A
        CURRENT_BANDWIDTH_RECEIVED: N/A
        MAX_PACKET_SIZE: N/A
        LC_PERF_OTHER_END_RECOVERY: N/A
    ECC:
        TOTAL_CORRECTABLE_COUNT: 0
        TOTAL_UNCORRECTABLE_COUNT: 0
        TOTAL_DEFERRED_COUNT: 0
        CACHE_CORRECTABLE_COUNT: N/A
        CACHE_UNCORRECTABLE_COUNT: N/A
    ECC_BLOCKS: N/A
    FAN:
        SPEED: 66
        MAX: 255
        RPM: 592
        USAGE: 25.88 %
    VOLTAGE_CURVE:
        POINT_0_FREQUENCY: N/A
        POINT_0_VOLTAGE: N/A
        POINT_1_FREQUENCY: N/A
        POINT_1_VOLTAGE: N/A
        POINT_2_FREQUENCY: N/A
        POINT_2_VOLTAGE: N/A
    OVERDRIVE: N/A
    MEM_OVERDRIVE: N/A
    PERF_LEVEL: AMDSMI_DEV_PERF_LEVEL_AUTO
    XGMI_ERR: N/A
    VOLTAGE:
        VDDBOARD: N/A
    ENERGY:
        TOTAL_ENERGY_CONSUMPTION: 0.0 J
    MEM_USAGE:
        TOTAL_VRAM: 16304 MB
        USED_VRAM: 897 MB
        FREE_VRAM: 15407 MB
        TOTAL_VISIBLE_VRAM: 16304 MB
        USED_VISIBLE_VRAM: 897 MB
        FREE_VISIBLE_VRAM: 15407 MB
        TOTAL_GTT: 15590 MB
        USED_GTT: 70 MB
        FREE_GTT: 15520 MB
    THROTTLE:
        ACCUMULATION_COUNTER: N/A
        PROCHOT_ACCUMULATED: N/A
        PPT_ACCUMULATED: N/A
        SOCKET_THERMAL_ACCUMULATED: N/A
        VR_THERMAL_ACCUMULATED: N/A
        HBM_THERMAL_ACCUMULATED: N/A
        GFX_CLK_BELOW_HOST_LIMIT_ACCUMULATED: N/A
        GFX_CLK_BELOW_HOST_LIMIT_POWER_ACCUMULATED: N/A
        GFX_CLK_BELOW_HOST_LIMIT_THERMAL_ACCUMULATED: N/A
        TOTAL_GFX_CLK_BELOW_HOST_LIMIT_ACCUMULATED: N/A
        LOW_UTILIZATION_ACCUMULATED: N/A
        PROCHOT_VIOLATION_STATUS: N/A
        PPT_VIOLATION_STATUS: N/A
        SOCKET_THERMAL_VIOLATION_STATUS: N/A
        VR_THERMAL_VIOLATION_STATUS: N/A
        HBM_THERMAL_VIOLATION_STATUS: N/A
        GFX_CLK_BELOW_HOST_LIMIT_VIOLATION_STATUS: N/A
        GFX_CLK_BELOW_HOST_LIMIT_POWER_VIOLATION_STATUS: N/A
        GFX_CLK_BELOW_HOST_LIMIT_THERMAL_VIOLATION_STATUS: N/A
        TOTAL_GFX_CLK_BELOW_HOST_LIMIT_VIOLATION_STATUS: N/A
        LOW_UTILIZATION_VIOLATION_STATUS: N/A
        PROCHOT_VIOLATION_ACTIVITY: N/A
        PPT_VIOLATION_ACTIVITY: N/A
        SOCKET_THERMAL_VIOLATION_ACTIVITY: N/A
        VR_THERMAL_VIOLATION_ACTIVITY: N/A
        HBM_THERMAL_VIOLATION_ACTIVITY: N/A
        GFX_CLK_BELOW_HOST_LIMIT_VIOLATION_ACTIVITY: N/A
        GFX_CLK_BELOW_HOST_LIMIT_POWER_VIOLATION_ACTIVITY: N/A
        GFX_CLK_BELOW_HOST_LIMIT_THERMAL_VIOLATION_ACTIVITY: N/A
        TOTAL_GFX_CLK_BELOW_HOST_LIMIT_VIOLATION_ACTIVITY: N/A
        LOW_UTILIZATION_VIOLATION_ACTIVITY: N/A

amd-smi static

GPU: 0
    ASIC:
        MARKET_NAME: AMD Radeon RX 9060 XT
        VENDOR_ID: 0x1002
        VENDOR_NAME: Advanced Micro Devices Inc. [AMD/ATI]
        SUBVENDOR_ID: 0x148c
        DEVICE_ID: 0x7590
        SUBSYSTEM_ID: 0x2437
        REV_ID: 0xc0
        ASIC_SERIAL: 0x8FFF4657ACF8550F
        OAM_ID: N/A
        NUM_COMPUTE_UNITS: 32
        TARGET_GRAPHICS_VERSION: gfx1200
    BUS:
        BDF: 0000:03:00.0
        MAX_PCIE_WIDTH: 16
        MAX_PCIE_SPEED: 32 GT/s
        PCIE_LEVELS:
            0: (2.5 GT/s, 8)
            1: (16.0 GT/s, 8)
            2: (16.0 GT/s, 8)
        PCIE_INTERFACE_VERSION: Gen 5
        SLOT_TYPE: CEM
    IFWI:
        NAME: Navi44 XT G26201 bring up 5Phases
        BUILD_DATE: 2025/06/24 04:37
        PART_NUMBER: 113-EXT118967-100
        VERSION: 00155333
    LIMIT:
        PPT0:
            MAX_POWER_LIMIT: 182 W
            MIN_POWER_LIMIT: 112 W
            SOCKET_POWER_LIMIT: 160 W
        PPT1:
            MAX_POWER_LIMIT: N/A
            MIN_POWER_LIMIT: N/A
            SOCKET_POWER_LIMIT: N/A
        SLOWDOWN_EDGE_TEMPERATURE: 100 °C
        SLOWDOWN_HOTSPOT_TEMPERATURE: 110 °C
        SLOWDOWN_VRAM_TEMPERATURE: 105 °C
        SHUTDOWN_EDGE_TEMPERATURE: 105 °C
        SHUTDOWN_HOTSPOT_TEMPERATURE: 115 °C
        SHUTDOWN_VRAM_TEMPERATURE: 110 °C
        PTL_STATE: N/A
        PTL_FORMAT: N/A
    DRIVER:
        NAME: amdgpu
        VERSION: 6.16.13
    BOARD:
        MODEL_NUMBER: N/A
        PRODUCT_SERIAL: N/A
        FRU_ID: N/A
        PRODUCT_NAME: N/A
        MANUFACTURER_NAME: Advanced Micro Devices, Inc. [AMD/ATI]
    RAS:
        EEPROM_VERSION: N/A
        BAD_PAGE_THRESHOLD: N/A
        BAD_PAGE_THRESHOLD_EXCEEDED: N/A
        PARITY_SCHEMA: N/A
        SINGLE_BIT_SCHEMA: N/A
        DOUBLE_BIT_SCHEMA: N/A
        POISON_SCHEMA: N/A
        ECC_BLOCK_STATE: N/A
    SOC_PSTATE: N/A
    XGMI_PLPD: N/A
    PROCESS_ISOLATION: Disabled
    NUMA:
        NODE: 0
        AFFINITY: NONE
        CPU_AFFINITY: N/A
        SOCKET_AFFINITY: N/A
    VRAM:
        TYPE: GDDR6
        VENDOR: SAMSUNG
        SIZE: 16304 MB
        BIT_WIDTH: 128
        MAX_BANDWIDTH: N/A 
    CACHE_INFO:
        CACHE_0:
            CACHE_PROPERTIES: DATA_CACHE, SIMD_CACHE
            CACHE_SIZE: 32 KB
            CACHE_LEVEL: 1
            MAX_NUM_CU_SHARED: 1
            NUM_CACHE_INSTANCE: 32
        CACHE_1:
            CACHE_PROPERTIES: INST_CACHE, SIMD_CACHE
            CACHE_SIZE: 32 KB
            CACHE_LEVEL: 1
            MAX_NUM_CU_SHARED: 2
            NUM_CACHE_INSTANCE: 16
        CACHE_2:
            CACHE_PROPERTIES: DATA_CACHE, SIMD_CACHE
            CACHE_SIZE: 16 KB
            CACHE_LEVEL: 1
            MAX_NUM_CU_SHARED: 2
            NUM_CACHE_INSTANCE: 16
        CACHE_3:
            CACHE_PROPERTIES: DATA_CACHE, SIMD_CACHE
            CACHE_SIZE: 256 KB
            CACHE_LEVEL: 1
            MAX_NUM_CU_SHARED: 8
            NUM_CACHE_INSTANCE: 4
        CACHE_4:
            CACHE_PROPERTIES: DATA_CACHE, SIMD_CACHE
            CACHE_SIZE: 4096 KB
            CACHE_LEVEL: 2
            MAX_NUM_CU_SHARED: 32
            NUM_CACHE_INSTANCE: 1
        CACHE_5:
            CACHE_PROPERTIES: DATA_CACHE, SIMD_CACHE
            CACHE_SIZE: 32768 KB
            CACHE_LEVEL: 3
            MAX_NUM_CU_SHARED: 32
            NUM_CACHE_INSTANCE: 1
    CLOCK:
        SYS:
            CURRENT_LEVEL: 1
            CURRENT_FREQUENCY: 1490MHz
            FREQUENCY_LEVELS:
                LEVEL 0: 500 MHz
                LEVEL 1: 1490 MHz
                LEVEL 2: 2620 MHz
        MEM:
            CURRENT_LEVEL: 1
            CURRENT_FREQUENCY: 456MHz
            FREQUENCY_LEVELS:
                LEVEL 0: 96 MHz
                LEVEL 1: 456 MHz
                LEVEL 2: 772 MHz
                LEVEL 3: 875 MHz
                LEVEL 4: 1124 MHz
                LEVEL 5: 1258 MHz
        DF:
            CURRENT_LEVEL: 1
            CURRENT_FREQUENCY: 1841MHz
            FREQUENCY_LEVELS:
                LEVEL 0: 307 MHz
                LEVEL 1: 1841 MHz
                LEVEL 2: 2400 MHz
        SOC:
            CURRENT_LEVEL: 1
            CURRENT_FREQUENCY: 1200MHz
            FREQUENCY_LEVELS:
                LEVEL 0: 417 MHz
                LEVEL 1: 1200 MHz
                LEVEL 2: 1476 MHz
        DCEF:
            CURRENT_LEVEL: 1
            CURRENT_FREQUENCY: 236MHz
            FREQUENCY_LEVELS:
                LEVEL 0: 147 MHz
                LEVEL 1: 236 MHz
                LEVEL 2: 1800 MHz
        VCLK0:
            CURRENT_LEVEL: 0
            CURRENT_FREQUENCY: 25MHz
            FREQUENCY_LEVELS:
                LEVEL 0: 25 MHz
        VCLK1:
            CURRENT_LEVEL: 0
            CURRENT_FREQUENCY: 25MHz
            FREQUENCY_LEVELS:
                LEVEL 0: 25 MHz
        DCLK0:
            CURRENT_LEVEL: 0
            CURRENT_FREQUENCY: 25MHz
            FREQUENCY_LEVELS:
                LEVEL 0: 25 MHz
        DCLK1:
            CURRENT_LEVEL: 0
            CURRENT_FREQUENCY: 25MHz
            FREQUENCY_LEVELS:
                LEVEL 0: 25 MHz

And now Windows 11.
I’m still not get squat from the card on PCIe transfer rates, but neither is anything else, so its not just me XD
image

I’ve tested 2 more cards, PCIe Gen 4x8 confirmed.

After running the 5050 all morning, with the primary display connected to it, nvidia-smi says:

        Replays Since Reset                            : 0
        Replay Number Rollovers                        : 0

Bandwidth Benchmark:

CLPeak bandwidth:

    Transfer bandwidth (GBPS)
      enqueueWriteBuffer              : 14.25
      enqueueReadBuffer               : 14.50
      enqueueWriteBuffer non-blocking : 14.15
      enqueueReadBuffer non-blocking  : 14.23
      enqueueMapBuffer(for read)      : 11.55
        memcpy from mapped ptr        : 6.80
      enqueueUnmap(after write)       : 14.49
        memcpy to mapped ptr          : 21.65

And for the Arc B580 (on the slower side, still in spec):

    Transfer bandwidth (GBPS)
      enqueueWriteBuffer              : 12.08
      enqueueReadBuffer               : 11.97
      enqueueWriteBuffer non-blocking : 14.08
      enqueueReadBuffer non-blocking  : 13.92
      enqueueMapBuffer(for read)      : 12.79
        memcpy from mapped ptr        : 5.75
      enqueueUnmap(after write)       : 14.42
        memcpy to mapped ptr          : 24.16

The 9060xt is lopsided, its dedicated more bandwidth to sending data to the GPU and less to receiving it, but they all tell the same story:

I can go test my synology NAS too, its also using 8 lanes of slimSAS (the slot is physically x8) to a 2070 Super, and it encodes video happily all the time without issue. Completely forgot, that’s where my other 1M slimSAS cable is. :grin:

Its PCIe gen 4 speeds, with zero errors, and its relatively cheap.

1 Like

You should post this in Wendell’s thread about relocating pcie lanes. If he’s actually totally wrong and retimers/switches are not needed at all, I am sure he would be thrilled to hear about it.

its impressive! I’ll have to get one of these and experiments. for the boards that have built in redrivers/retimers… it can work out? but I think with your particular setup this is a special case that happens to work.

I have had long-running jobs on gen4/gen5 nvidia hardware and the gpus barf in weird ways after many hours of fine-tuning even in the most ideal “retimerless” scenarios. theres some long threads here about suffering.

physically tiny motherboards with more layers are better than uatx/atx boarsd with just 8 layers too, fwiw. signal integrity…

I find it being an edge case that happens to work far more likely than I fumbled into the answer, always lol.

But it remains that I have yet to find a way it doesn’t work flawlessly. My NAS been up for months, and there are zero errors or retransmissions there either. The 2070 Super re-encodes video happily every day, so it’s finding where it falls down that is becoming elusive.

1 Like