So I build a NAS server recently, and I am thinking of getting a 10GIG card for it. Was wondering if anyone had experience with, preference for or suggestion for a better alternative of any of the following cards: ( I don’t know any of these. they just have the right number and type of ports that I want ):
Thanks,
Whats on the other side? Switch or Point to point? I am a fan of the intel/aquantia but I use copper for my 10gbe not SFPs. Plenty of people have used the Chelsio stuff here, so any of them would probably work well.
What other hardware is this connected on to? What OS?
FreeBSD, Linux and WIndows drivers as well as heat are the main differences between NIC’s.
Hey everyone,
Thanks for the replies.
It would be connected to an spf+ switch via DAC ( most probably a mikrotik one, I am working on it ) . The host OS would be Linux. BSD driver support would also be important ( for passthrough situations )
That is what I am asking here for. If you are familiar with any of them , do they over heat? ( like I had to get a fan for my LSI card since they do have that issue outside of a server chassis) and how good are the drivers?
I have 2 aquantia 10GbE nics…
Hey Thanks, but I need a few more ports
I have a T422-cr in a FreeNAS system and no complains, although it is a server, so lots of airflow. I do really like the combination of ports.
Look at mcx354a-fcbt … or perhaps a mcx353a-fcbt
These are dual port qsfp+ (or single port), and if you want only 10G port you can use a breakout cable.
They’re between 30 and 120 bucks for a dual port, depending on country etc…
wow, Those are interesting. I don’t have enough pcie lane for a mcx354a-fcbt, but mcx353a-fcbt could work.
Thanks for the suggestion.
$19 ?
how about this one $40?
just bought the Chelsio T422. Has anyone used them in a normal case? How much airflow do they need?
Hi,
I am having problems debugging my issues with a T422. I was wondering if it is not too much trouble, and if you still have your T422, could you post the output of the following commands please:
ip link
lspci -vnn | grep -A 20 ‘Chelsio’
Thanks,
It’s in a FreeNAS box but:
% lspci -vnn | grep -A 20 ‘Chelsio’
04:00.0 Ethernet controller [0200]: Chelsio Communications Inc T422-CR Unified Wire Ethernet Controller [1425:4002]
Subsystem: Chelsio Communications Inc Device [1425:0000]
Flags: bus master, fast devsel, latency 0, IRQ 30
Memory at fae40000 (64-bit, non-prefetchable)
Memory at fae3e000 (64-bit, non-prefetchable)
Expansion ROM at fae80000 [disabled]
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
Capabilities: [60] MSI-X: Enable- Count=8 Masked-
Capabilities: [6c] Express Endpoint, MSI 00
Capabilities: [a8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Multi-Function Virtual Channel <?>
Capabilities: [1b8] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [1c4] Single Root I/O Virtualization (SR-IOV)
Capabilities: [204] Alternative Routing-ID Interpretation (ARI)
04:00.1 Ethernet controller [0200]: Chelsio Communications Inc T422-CR Unified Wire Ethernet Controller [1425:4002]
Subsystem: Chelsio Communications Inc Device [1425:0000]
Flags: bus master, fast devsel, latency 0, IRQ 37
Memory at fadc0000 (64-bit, non-prefetchable)
Memory at fadbe000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
Capabilities: [60] MSI-X: Enable- Count=8 Masked-
Capabilities: [6c] Express Endpoint, MSI 00
Capabilities: [a8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [144] Single Root I/O Virtualization (SR-IOV)
Capabilities: [184] Alternative Routing-ID Interpretation (ARI)
04:00.2 Ethernet controller [0200]: Chelsio Communications Inc T422-CR Unified Wire Ethernet Controller [1425:4002]
Subsystem: Chelsio Communications Inc Device [1425:0000]
Flags: bus master, fast devsel, latency 0, IRQ 39
Memory at fad40000 (64-bit, non-prefetchable)
Memory at fad3e000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
Capabilities: [60] MSI-X: Enable- Count=8 Masked-
Capabilities: [6c] Express Endpoint, MSI 00
Capabilities: [a8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [144] Single Root I/O Virtualization (SR-IOV)
Capabilities: [184] Alternative Routing-ID Interpretation (ARI)
04:00.3 Ethernet controller [0200]: Chelsio Communications Inc T422-CR Unified Wire Ethernet Controller [1425:4002]
Subsystem: Chelsio Communications Inc Device [1425:0000]
Flags: bus master, fast devsel, latency 0, IRQ 38
Memory at facc0000 (64-bit, non-prefetchable)
Memory at facbe000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/8 Maskable+ 64bit+
Capabilities: [60] MSI-X: Enable- Count=8 Masked-
Capabilities: [6c] Express Endpoint, MSI 00
Capabilities: [a8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [144] Single Root I/O Virtualization (SR-IOV)
Capabilities: [184] Alternative Routing-ID Interpretation (ARI)
04:00.4 Ethernet controller [0200]: Chelsio Communications Inc T422-CR Unified Wire Ethernet Controller [1425:4402]
Subsystem: Chelsio Communications Inc Device [1425:0000]
Flags: bus master, fast devsel, latency 0, IRQ 30
Memory at f9fc0000 (64-bit, non-prefetchable)
Memory at fa000000 (64-bit, non-prefetchable)
Memory at f9fbe000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/32 Maskable+ 64bit+
Capabilities: [60] MSI-X: Enable+ Count=128 Masked-
Capabilities: [6c] Express Endpoint, MSI 00
Capabilities: [a8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [144] Alternative Routing-ID Interpretation (ARI)
04:00.5 SCSI storage controller [0100]: Chelsio Communications Inc T422-CR Unified Wire Storage Controller [1425:4502]
Subsystem: Chelsio Communications Inc Device [1425:0000]
Flags: bus master, fast devsel, latency 0, IRQ 37
Memory at f9f40000 (64-bit, non-prefetchable)
Memory at f9f3e000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/32 Maskable+ 64bit+
Capabilities: [60] MSI-X: Enable- Count=32 Masked-
Capabilities: [6c] Express Endpoint, MSI 00
Capabilities: [a8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [144] Alternative Routing-ID Interpretation (ARI)
04:00.6 Fibre Channel [0c04]: Chelsio Communications Inc T422-CR Unified Wire Storage Controller [1425:4602]
Subsystem: Chelsio Communications Inc Device [1425:0000]
Flags: bus master, fast devsel, latency 0, IRQ 39
Memory at f9ec0000 (64-bit, non-prefetchable)
Memory at f9ebe000 (64-bit, non-prefetchable)
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/32 Maskable+ 64bit+
Capabilities: [60] MSI-X: Enable- Count=32 Masked-
Capabilities: [6c] Express Endpoint, MSI 00
Capabilities: [a8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [144] Alternative Routing-ID Interpretation (ARI)
04:00.7 Ethernet controller [0200]: Chelsio Communications Inc Device [1425:0000]
Subsystem: Chelsio Communications Inc Device [1425:0000]
Flags: bus master, fast devsel, latency 0, IRQ 255
Capabilities: [40] Power Management version 3
Capabilities: [48] MSI: Enable- Count=1/32 Maskable+ 64bit+
Capabilities: [60] MSI-X: Enable- Count=32 Masked-
Capabilities: [6c] Express Endpoint, MSI 00
Capabilities: [a8] Vital Product Data
Capabilities: [100] Advanced Error Reporting
Capabilities: [138] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [144] Alternative Routing-ID Interpretation (ARI)
05:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2116 PCI-Express Fusion-MPT SAS-2 [Meteor] [1000:0064] (rev 02)
Subsystem: Broadcom / LSI SAS 9201-16i [1000:30c0]
Flags: bus master, fast devsel, latency 0, IRQ 32
I/O ports at e000
Memory at fbd3c000 (64-bit, non-prefetchable)
Memory at fbd40000 (64-bit, non-prefetchable)
Expansion ROM at fbd80000 [disabled]
Capabilities: [50] Power Management version 3
Capabilities: [68] Express Endpoint, MSI 00
Capabilities: [d0] Vital Product Data
and relevant part of ifconfig:
cxgbe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: member of lagg1
options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause>
status: active
cxgbe1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
description: member of lagg1
options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-Twinax <full-duplex,rxpause,txpause>
status: active
cxgbe2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: member of lagg2
options=ac00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
cxgbe3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: member of lagg2
options=ac00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
Thanks. I think mine is broken:
So I worked on that server today and the 422 was acting very erratic. Would only negotiate links with certain devices. In some cases, the other end would show a link while the 422 stayed dark. Very weird. Going to retire it.
After my last post, I ordered another one from eBay, and that one has been working fine so far. Easily established a 10G with my mikrotik router over a DAC. I was mostly worried about my router (rb4011) than the card as apparently this model wasn’t able to establish a connection over a passive cable with older version of the RouterOS. But everything works now, so I am happy
Also, FWIW, On lots of posts that I have read, a lot of people also mentioned having success with turning auto negotiation off and forcing a 10G connection.