I currently have an issue where some NVME drives are not showing back up after reboot. I’ve ruled out bad drives (they are new WD SN850x Blacks). I’ve tried various OSes: Windows Server, Debian, Ubuntu Server, CachyOS, ProxMox, and a few more. They all seem to have the same issue.
As soon as I remove and reinsert the drives, they all show back up immediately after reinserting and they work at full speed. The data link is PCIE4.0 x4 (expected).
I have them inside of a HighPoint Technologies Rocket switching adapter (https://www.highpoint-tech.com/nvme-switch-adapter/gen4/rocket-1528d)
Status on the Rocket is all green leds (OK). No firmware updates. This adapter requires bifurcation to be off. I tested turning it to x4x4x4x4, and that just causes the Highpoint to start giving RED leds and drives not appearing regardless if I reseat them.
My current guess is the IcyDock MB118VP-B is not doing something correctly on reboot. The fans on the icydock are screaming (working as intended). The LEDs on all drives are green indicating they are on. However, I have to reseat the drives every time I do a reboot. No software seems to make them show up. (lsblk, ls -lF /dev/disk/by-id shows they simply aren’t connected. Windows Disk mgmt does not see them either). Sometimes there’s 1-2 drives that are still available after reboot, but rarely ever more than that until I reseat.
I tried formatting all drives and assigning them a drive letter to see if maybe some sort of power savings was maybe turning them off to conserve power. No dice.
Anyone know if there’s anything in the BIOS that I could try or have any ideas what could cause the issue? I have a WRX90 WS Evo with Threadripper Pro 7965W.
Drives are lost on reboot regardless if it’s cold or warm. Have to reseat each time. I do have Fast Boot disabled. Have the memory persistence enabled so the memory doesn’t retrain every reboot. I haven’t seen anything about PCIE persistence?
Just bought some shorter cables to see if that will help. Going to explore turning on PCIE debugging. Also looking at possibly purchasing a Kingwin NVME enclosure to see if it uses a different backplane than the Icy Dock,unless anyone else knows for sure it does(nt)?
I found two GTEK NVME SFF5684 cards I’m going to replace for the Highpoint adapter to see if they fair any better.
No, they don’t. I’m aware of difference between cold and warm.
Just set up the GTEK cards. 5 drives are showing after reboot now. One drive is still not showing. SInce the GTEK cards aren’t switching adapters, they aren’t hot swapable.
In quite the predicament now. If I use the Highpoint card, I can get all drives to work by reseating after booting to the OS. However, I lose most of them on reboot until physically reseating. If I use the GTEK cards, I’m permanently without 1 drive.
Tried changing drives in different slots in the Icy Dock and no difference.
I’ve tested all my cables and they seem to all work with 1 drive. While testing to see if they worked with 2 drives, I noticed that every other reboot that 1 drive would disappear. Tested multiple cables and the same thing happened. However, seems to happen only when using the Highpoint card.
Just rebooted about 20 times while plugged into the GTEK cards, and they seem to be working fine with the exception of the 1 drive. I’m fairly certain the reason that 1 drive doesn’t work on the GTEK because of a bad Icy Dock NVME m.2->U2 adapter. Seems to always have trouble with this one bay regardless of the slot it’s in or the brand of NVME drive inside of it.
I just got the Highpoint card and I was hoping I could consolidate to using that single card for 8 drives instead of 2 GTEK cards, but doesn’t look like it’s going to be possible. To me, it doesn’t really make sense getting these easily removable nvme bays from Icy Dock if you don’t have hot swap capability. May try RMAing it.