Mellanox works in windows not esxi

I’ve been installing a number of the 100Gbit Mellanox Cards, note these cards come
Up fine in windows, they’re on the supported list, proper firmware, I’ve done about 60 in the last two weeks but I’ve been running into the below situation:

Ports are up, lights showing however no “vmnics” shown

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 15492: vmnic1: clear flags 0x51822 DEVICE_OPEN_FAILED

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 15493: vmnic1: set flags 0x51826 DEVICE_OPENED

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 12471: vmnic1: set flags 0x51a26 DEVICE_SCHED_CONNECTED

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 13003: The default queue id for vmnic1 is 0x3c4000.

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)VMKAPIMOD: 110: failed to query port alias for Not found:

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 13015: enabled port 0x82000005 with mac b8:ce:f4:21:22:33

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 3122: vmnic1, fast switch on uplink(0x82000005) connected to PPS is not supported

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 15683: vmnic1: clear flags 0x51a0e DEVICE_OPENING

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_UplinkStateSet - (nmlx5_core_en_uplink.c:4806) change state to <ENABLED,PROMISC,BROADCAST,MULTICAST>

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_UplinkStateSet - (nmlx5_core_en_uplink.c:4857) current state is <ENABLED,PROMISC,BROADCAST,MULTICAST>

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_UplinkStartIOLocked - (nmlx5_core_en_main.c:2487) called

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_SetDevPortMtu - (nmlx5_core_en_main.c:2018) Port MTU (-22) is different than configured MTU (1500)

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_OpenChannel - (nmlx5_core_en_channel.c:1977) called for channel idx 0

2024-04-17T20:00:46.370Z Wa(180) vmkwarning: cpu0:2097533)WARNING: <NMLX_WRN> nmlx5_CoreCreateCq:110 command failed: IO was aborted

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_ERR> nmlx5_core: vmnic1: nmlx5_en_OpenCq - (nmlx5_core_en_channel.c:1903) nmlx5_en_EnableCq failed: IO was aborted

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_ERR> nmlx5_core: vmnic1: nmlx5_en_OpenChannel - (nmlx5_core_en_channel.c:2052) done, status: IO was aborted

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_ERR> nmlx5_core: vmnic1: nmlx5_en_UplinkStartIOLocked - (nmlx5_core_en_main.c:2512) nmlx5_en_UplinkOpenChannels failed: IO was aborted

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_ERR> nmlx5_core: vmnic1: nmlx5_en_UplinkStartIOLocked - (nmlx5_core_en_main.c:2620) done, status: IO was aborted

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_ERR> nmlx5_core: vmnic1: nmlx5_en_UplinkStartIO - (nmlx5_core_en_main.c:2639) done, status: IO was aborted

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 14858: vmnic1: set flags 0x51a4e DEVICE_CLOSING

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)NetPort: 1879: disabled port 0x82000005

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 12382: vmnic1: clear flags 0x5184e DEVICE_SCHED_CONNECTED

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu67:2098689)NetSched: 724: vmnic1-0-tx: worldID = 2098689 exits

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 14717: vmnic1: clear flags 0x51842 DEVICE_OPENED

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu0:2097533)<NMLX_INF> nmlx5_core: vmnic1: nmlx5_en_UplinkQuiesceIOLocked - (nmlx5_core_en_main.c:2653) called

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 14862: vmnic1: clear flags 0x51802 DEVICE_CLOSING

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Uplink: 14874: vmnic1: clear flags 0x51800 DEVICE_PRESENT

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Device: 637: uplink_drv:driver->ops.startDevice:0 ms

2024-04-17T20:00:46.370Z In(182) vmkernel: cpu9:2098682)Device: 642: Device 0x4b9a430be3e257d3 start from driver uplink_drv failed with bad00e0

Give TrueNAS Scale a whirl, chances are the drivers are not available in ESXI but they might be in Linux.

Can’t have to use esxi, I have this same card albeit a different country of build Israel vs India, none of the ones made in India work