Chelsio T520-CR does not recognize sfp module

Hardware Hub Networking
1

I have a Dell Chelsio T520-CR installed on a Dell PowerEdge R220 the system I use is FreeBSD. The SFP module that I use is the CPGOS03-0490 v.2.0

The controller used for the T520-CR is the cxgbe(4)

The SFP module automatically listens at the address 192.168.1.1, So I configure the interface of the T520-CR:

ifconfig cxl1 inet 192.168.1.2 netmask 255.255.255.0 192.168.1.255

Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
127.0.0.1          link#3             UH          lo0
192.168.1.0/24     link#5             U          cxl1
192.168.1.2        link#5             UHS         lo0

cxl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP>
    ether 00:07:43:53:a2:06
    inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255 
    media: Ethernet none
    status: no carrier
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

But I can’t connect to the SFP module, there is no ping or any kind of response.

I don’t know if it is a compatibility problem between the module and the card, could someone help me?

Thanks.

2

Id be really interested to see if you get this working. I have wanted to use a GPON type SFP directly in my PC as well to take fiber right into it instead of going through an ONT but didn’t have enough expertise to do all the proper configuration of the SFP module GPON interface. For myself, I actually want to use an XGS-PON because that is the type Frontier uses in their 2gb+ fiber deployments, but it is the same situation as any PON SFP as far as configuration of the gateway goes.

I guess for your situation the first thing to check would be if the SFP module powers up and will “talk” to the Chelsio card? I know some brands of network gear require the SFP modules to be coded for the manufacturer. I do not know if this is the case with Chelsio or not. Do you know if another brand SFP for a typical Ethernet fiber network (with LC plugs for example) functions with the NIC? I don’t really know of a way to test this without having a standard SFP around to try and connect it to a switch or something.

3

No problem, I’ll keep the thread updated for any news.

If the problem seems to be that the card and the module are not “talking” I wrote to the seller of the module but I did not get a response from this almost 1 week ago.

If you want you can read the thread I have open in the FreeBSD forum. Hay could follow the thread from the beginning.

SFP FreeBSDForum topic

Maybe you are more interested in page 2 of the thread.

No, unfortunately I don’t have any other LC connector SFP to perform another test, but these are some threads I’ve seen from people with the same module, but they connect the module to equipment like mikrotik etc… not my idea.

Unity

Mikrotik

If you have any other questions tell me, I have everything ready, the problem is this incompatibility problem. Which I didn’t expect.

Regards.

4

I found this thread here where the specific SFP GPON module is being talked about:
https://community.ui.com/questions/UDM-PRO-GPON-SFP-CarlitoxxPro-CPGOS03-0490-v2-0/362172d4-8a5f-4bda-a841-4a34c075c57f#answer/a27a5556-a4b9-4b3a-8b20-b2b7368bf7a3

It sounds like the GPON module only supports single byte reads. The thread mentions that the linux kernel does support auto detection of this when it attempts a multi-byte read and only receives a single byte reply, but just wanted to bring it up as I dont know what kernel you are on with your system or when this specific change was implemented in the linux kernel or when FreeBSD may have picked it up.

5

Interesting, I have looked at the possible configurations that can be done to the cxgbe driver.

hw.cxgbe.autoneg

     Link autonegotiation settings.  This tunable establishes the de-
     fault autonegotiation settings for	all ports.  Settings can be
     displayed and controlled on a per-port basis via the
     dev.<port>.X.autoneg sysctl.  0 disables autonegotiation.	1 en-
     ables autonegotiation.  The default is -1 which lets the driver
     pick a value.  dev.<port>.X.autoneg is -1 for port	and module
     combinations that do not support autonegotiation.

It gives me to understand that if I get a -1 it is that autonegotiation is not possible and the module will not be able to work

And it is effectively the result I get, this is an output of the kernel state:

$ sysctl -a |grep autoneg
hw.cxgbe.autoneg: 1
dev.cxl.1.autoneg: -1
dev.cxl.0.autoneg: 1

Being port 1 where the SFP is connected, if I exchange the port SFP the same thing happens, port zero becomes -1 and port one becomes 1

That can make us think that we are having the problem that is shown in the link that you have shared with me, maybe not? I’ll ask the FreeBSD people maybe they are more familiar with the driver.

Edit:

I forgot to say that I set these values in loader.conf

if_cxgbe_load="YES"
hw.cxgbe.autoneg=1
dev.cxl.0.autoneg=1
dev.cxl.1.autoneg=1

To imply that it loads the driver at kernel startup, and configures autonegotiation.

6

Auto negotiation uses hex values in specific spots of the EEPROM to understand what link rates are valid, and some of these modules simply have 0’s for all values. You could try specifying the link speed within your configuration of the interface.

7

Ok, these are the speeds supported by my interface.

$ ifconfig -m cxl1

cxl1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP>
	capabilities=6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP>
	ether 00:07:43:33:a2:08
	media: Ethernet 1000baseSX <full-duplex,rxpause,txpause>
	status: no carrier
	supported media:
		media 1000baseSX mediaopt full-duplex,rxpause,txpause
		media 1000baseSX mediaopt full-duplex,rxpause
		media 1000baseSX mediaopt full-duplex,txpause
		media 1000baseSX mediaopt full-duplex
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

According to chelsio’s paper the T520-CR supports the IEEE802.3z coding.

T520-CR Paper

But according to ifconfig it doesn’t support the 1000baseLX standard which is what the module is supposed to use.

I can’t configure the interface with said base speed. So here we have a new problem?

Edit:

Hex values, from the link of unify:

0x03: Transceiver Compliance Code = "\x00\x00\x00\x02\x00\x00\x00\x00" (1000Base-L (1000BASE-LX) - good)
8

Base SX is multi-mode fiber and LX is single mode fiber. GPON is a single mode system so it uses 1000baseLX as you said, but the driver config would most likely be the same for either standard. The electrical portion shouldn’t need to change, and the SFP transceiver should be handling the optical portion itself so I doubt it will be a problem.

9

But for example the standard 1000baseSX It does not fit, for example, the laser specifications of my module, it supports laser 850 nm.

The SFP module that I am using conforms to the laser 1310 nm, SFP module that I am using conforms to the 1310 nm what is the standard 1000baseLX

That couldn’t be a problem?

I will write back to technical support.

10

Apparently the card does recognize the module, it just wasn’t executing the correct options:

$ ifconfig -vvvm cxl0

cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP>
	capabilities=6ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6,HWRXTSTMP,NOMAP>
	ether 00:07:43:33:a2:00
	inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
	media: Ethernet 1000baseSX <full-duplex,rxpause,txpause>
	status: active
	supported media:
		media 1000baseSX mediaopt full-duplex,rxpause,txpause
		media 1000baseSX mediaopt full-duplex,rxpause
		media 1000baseSX mediaopt full-duplex,txpause
		media 1000baseSX mediaopt full-duplex
	nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
	plugged: SFP/SFP+/SFP28 1000BASE-LX (SC)
	vendor: C PN: V SN: C DATE: 202
	module temperature: 52.00 C voltage: 2.82 Volts
	lane 1: RX power: 0.00 mW (-inf dBm) TX bias: 0.00 mA

	SFF8472 DUMP (0xA0 0..127 range):
	 03 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	 00 00 00 00 00 00 00 00 69 00 00 00 00 00 00 00
	 20 00 00 00 00 00 00 00 56 00 00 00 00 00 00 00
	 20 00 00 00 00 00 00 00 56 00 00 00 00 00 00 00
	 00 00 00 00 00 00 00 00 32 00 00 00 00 00 00 00
	 31 00 00 00 00 00 00 00 30 00 00 00 00 00 00 00
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
	 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

We can see some values ​​such as the speed of the module, the temperature and the voltage, apparently the vendor and the SN do not appear correctly.

The most interesting thing is the DUMP I think that in the link you shared on the second page they talk about it.

I still can’t put the card at the same speed as the sfp module.

But I think the problem is in the dump, can you think of something?

11

I’m not sure, this is well beyond my knowledge level now.
I do find it interesting that it is not reporting that it cannot auto-negotiate any speeds though, as well as the vendor and serial being messed up.

12

I have written again to the support of the module, I do not think they will answer me, they still did not in the previous one that I sent.

I don’t know what else I can do, I think the problem is in the autoneg, which is what the link you shared refers to, and the driver gives an error in the autoneg.

The only thing that occurs to me right now is to use another platform to check if it is a problem in the driver that does not support reading a single byte.

I think I’ll try with pfsense, or by default some linux, I guess the linux distribution doesn’t matter as long as it has the driver.

EDIT:

And that you can’t adjust the speed of a card that is supposed to support 1000baseLX, that’s also another point.

13

I have tested ubuntu server and fedora server, without result.

Same with pfsense, but that wasn’t a very good idea since it’s based on FreeBSD.

I will try to use another card, the problem could be there, since I cannot adjust the speed of this one.

I’ll let you know when I have something.

14

Apparently there is a tool to view and debug cxbge. But I still can’t get the module to work, but with this tool I was able to identify it and see, for example, that the ifconfig dump does not correspond to the i2c port:

cxgbetool t5nex0 modinfo 0

ID: SFP/SFP+/SFP28
Vendor CarlitoxxPro
SN CP202003180114
PN V2801F
Rev V2.
Temp: +44C
Vcc 2.867200V
TX Bias 0.000000uA
TX Power 0.000000mW
RX Power 0.000000mW

And this is when I read the i2c that does not correspond to the ifconfig dump the first 3 bytes:

cxgbetool t5nex0 i2c 0 0xa0 0....3

0x3 [3]
0x4 [4]
0x0 [0]

This tool also has filtering, but I don’t know how it works, I go back to the start according to the controller, I have synchronization problems, I don’t know what else to do. If it is a definite compatibility problem or a bad adjustment on my part, but according to the manufacturer you only have to assign it an IP address and it will work.

15

I returned to resumethe thread, I found this information apparently it could work but with another sfp G-010S-A GPON SFP.

In this github thread, there is information about firmwares and patches for the FreeBSD kernel, I’ll have to read it calmly as soon as I can, and see if it’s the correct solution.

https://github.com/hwti/G-010S-A

But hey, as I said at the beginning of the thread, it would only be a part, if I make it work, then I will have to see what I can associate with the OLT of my ISP, that is another part of the problem

Thanks.