I’ve got an Asus K14PA-U12 motherboard with an Epyc 9684X CPU and lots of RAM. I want to connect 6 SSDs, 2 on each MCIO connector using this MCIO x8 to U.3 cable. I see the drives in BIOS and all their info is there, but when I try to boot from USB (or iKVM) I just get a black screen with a non-blinking cursor in the upper left.
This happens if I connect even just 1 or 2 SSDs. If I connect no SSDs, then I can boot from USB just fine (it took an immense amount of pain to figure that out). The SSDs are Micro 7500 MAX.
Is there something with that cable? Is the motherboard or firmware screwy? Is what I’m trying to do wrong in some other way? Maybe some BIOS setting is needed? There’s nothing in the BIOS about bifurcation, so I assume it’s automatic.
I’ve opened a case with Asus, but I don’t have high hopes.
Some bases I’ve covered: Secure boot is disabled. The BIOS has no legacy/CSM options. BIOS and BMC firmware are the latest. I tried both with my desired BIOS settings and factory defaults. I tried rolling back the BIOS to old versions, but doing so failed – seems it’s a one way ticket. I connected a GPU and monitor and got the same black screen and cursor. I tried other grub-based USB boot drives (Proxmox, Ventoy) and even a Windows rescue USB behaved the same way.
I assume your system by default tries to boot from directly connected disk and in case none is connected tries to boot from USB and other devices. That’s normal behavior for any machine.
As soon as you connect a disk (any one of your Micron 7500 MAX) it tries to boot off of these devices, but none of those have a OS, so it fails with a black screen with a non-blinking cursor in the upper left.
Connect your SSDs and change the boot preferences to USB (or whatever your needs are).
There is another interesting failure mode where the bios scans the contents of the SSD and crashes because of unexpected data (presumably looking for EFI partition/etc).
While I strongly suspect its just bios boot settings on your setup (take some pics for us?) it could also be this weird edge case. If you can boot with them in another system and use linux to add an efi partition and set the disk label, that could also resolve the problem.
The SSDs don’t show up in the list of bootable devices. The USB does and I choose to boot from it explicitly (boot override or F8 popup), but the screen just goes black.
That would be quite nasty! The drives are brand new from DigiKey, FWIW.
Unfortunately I don’t have another system with U.3 or MCIO.
I’m happy to take pics! I reset the BIOS to defaults and took these screenshots, with comments:
Maybe your UEFI/BIOS allows some modification of drive? E.g. secure erasing / formatting?
The mobo manual refers to a “ASUS Storage Viewer” in the Tools menu. It doesn’t state anything beyond that.
Maybe drive format is an IPMI function (no experience)?
If none of this works, another (more esoteric) step may be to find a UEFI app on the interwebs that allows secure erasing the SSDs and kick that off from the BIOS.
Is hot-plugging the SSDs a thing in your environment? Meaning, boot into a comfortable live linux (or other) environment, hot-plug a SSD, format, create gpt partition table, etc.
DON’T try hotplugging if it’s not a documented feature! I don’t want to be responsible for any damage caused.
You’re right! I found Media sanitization under Security that can clear a drive. I did the option Purge/Format/UserData on 2 drives (the only connected). Rebooted, F8, boot from USB… black screen. Damn, it was a promising try.
I don’t know if hotplugging should work. There are a few BIOS options mentioning it, so maybe! Proxmox has no way of refreshing its drive list, so I need to hotplug after boot but before Proxmox does it’s thing. So, with no SSDs attached I booted Ventoy off USB, then plugged in 2 SSDs, then booted the Proxmox ISO. Wow it worked!
What are you trying to boot?
Do us a favor and try booting a Server 2022 iso, Ventoy, etc.
I have seen this behavior from a few servers where the iGPU is missing a driver or parameter from various distros.
I tried the first Linux Boot Manager and UEFI OS options, but they both give a black screen like before. Why can’t this motherboard boot with the NVMe drives connected!?
Is there anything else I can try?
Are we ready to accuse the BIOS of being a buggy piece of junk?
haven’t looked up the board, did you set the MCIO dip switches to NVME?
I am guessing so based on:
Which brings us to the boot configuration. Enable secure boot and disable all but your USB drive from the boot order. Further, delete all UEFI entries.
You want to boot from a ZFS pool on U.3 NVME drives?
Have you tried chainloading from your USB bootloader to the installation on your ZFS pool on the U.3 NVME yet?
There is one jumper that changes 2 MCIO from NVMe to SATA. The other MCIO (which have my SSDs) are always NVMe.
Why would I do this? The Proxmox USB can’t boot when secure boot enabled. In the past USB was the only bootable device, this doesn’t seem like it would test anything new.
I’ve already installed Proxmox, I don’t need to boot from USB anymore. Now I need to boot from the SSDs. I choose the boot device each boot, I’m not (yet) relying on the BIOS to choose automatically. It’s trying to boot what I choose (USB or SSD) but it fails if the SSDs are plugged in.
What do you mean? I can disconnect the SSDs and boot anything off USB, like Ventoy. If I then hotplug the SSDs, Ventoy doesn’t see them, as it doesn’t have any way to refresh the drives it sees.
Famously, signalling can cause all kinds of problems. Find the PCIe/MCIO/… section in the BIOS and force the drives to PCIe Gen 4 or lower signalling. See if that makes a difference.
which MCIO ports did you try plugging into? based on the block diagram I’d try 1,2,3 or 4, and stay away from the others because they look like they might have a weird pinout.
I gave that a try: PCIe Link Speed Capability = Gen2 PCIE Speed PMM Control = Static Target Link Speed (GEN4) (the lowest available)
Unfortunately, no improvement.
If this motherboard ever works it’s good to know I should fill 1-4 first. I’ve tried all the MCIO, using 1 drive. I tried 6 SSDs on 2, 3, and 4, no luck.
I went through the jumpers and disabled VGA (I’m using iKVM) and SmaRT. I tried with MCIO 7 and 8 jumper set to auto (NVMe) and SATA. No change.
I’ve escalated with Asus B2B and pointed them to this thread. What are the chances that helps eh? If it doesn’t, I think soon I’ll return this board to the vendor, as it seems most likely the BIOS is faulty. The motherboard box says “ASUS: Reliable Server Building Block”.
Asrock GENOAD24QM3-2L2T/BCM: 7 MCIO x8 for NVMe gen5, 2 for SATA (good), 2 RJ45 10GbE, 2 x16 slots (need 1).
Gigabyte MZ33-AR0: 4 MCIO x8 for NVMe gen5, 1 gen4, 2 for SATA (ok), 2 RJ45 10GbE, 4 x16 slots (need 1).
Gigabyte MZ33-CP0: 4 MCIO x8 for NVMe gen5, 2 for SATA (ok-ish), 2 RJ45 10GbE, 4 x16 slots (need 1).
SPF+ would be better than RJ45. Only the Asus has SPF+ for some reason.
I had a bad Asrock experience once and swore never again. I like the MCIOs though.
The Gigabyte MZ33-AR0 looks OK. There’s enough MCIOs for now, but in the future I may be annoyed. With a brief look I don’t find a PCIe x16 card that can do 4+ U.3 x4 gen4 NVME drives.
I’ve never had a Tyan board, but they seem to have a great reputation, slightly better than the others. The Tyan Tomcat CX S8056 seems to be best match for my needs, even if I don’t need the OCP slot, or the crazy x24 and x28 slots (what are those even for?). Do the weird PCIe slots line up to the slots in a standard case? I only need to mount one GPU, but if the slot doesn’t line up that would be bad.
One more question:
does the post code readout on the motherboard itself change when the SSDs are plugged into the MCIO ports?
<-this would help rule out PCIe bus problems from bad signal integrity which is always a possibility with MCIO.
With 6 SSDs connected, after boot while it waits for DEL/F8/etc, the post code is 0x00. After I press F8 for boot device selection it changes to 0xA9 “BIOS Setup Utility Start”, which seems normal. After I press enter to choose a boot device, it changes back to 0x00 and hangs at the black screen with the non-blinking cursor in the upper left.
Yeah, I’m also at a loss. Hair loss. The hotswap shenanigans prove the drives work 100%. The Proxmox install is really on there, I verified it. Whatever goes wrong at boot really looks like a BIOS bug. Asus takes days to get back to me. My guess is they offer to RMA the board, but I doubt that’s a fix (given their bad PR lately, they’ll probably charge me). Even if they are willing to make a BIOS fix, how long does it take to publish a BIOS update? Plus if I can’t trust Asus for such basic functionality, I don’t want to use Asus server gear. At least their gaming stuff is popular enough that they seem to fix the issues.
U.3 is supposed to be backward compatible. It’s probably a stretch but maybe something about U.3 is causing this issue? Maybe Asus only tested U.2. They did test this absolutely cursed board, right? Right?
I have experience with both boards in prod.
SuperMicro is…SuperMicro and a main player in the enterprise space for a reason. Their flashing process is more verbose, but a bigger pain in the ass to actually take upgrades. This board has more internal space for a GPU.
Gigabyte has more DIMM slots if you need it and is receiving much faster updates. I have a thread about the TPM modules for it.
Neither will boot from an iKVM uploaded Server 2022 iso.
Remember EPYC requires 1 DIMM per memory channel for optimized performance.
Cool, thanks for your input! Not booting certain ISOs via iKVM is pretty lame. The SM has so few MCIOs. The Gigabyte or Tyan are a better fit for me. I lean toward the Tyan S8056, but might pick based on lead time – waiting for quotes to come back.
The Tyan PCIe slots are weird and it has a switch sticking out near them. Are either of the two slots aligned with an ATX case slot? Isn’t the CPU in the way of a GPU? Eg an A100 or H100. My 4U case can fit a GPU without shenanigans, if the board has a standard slot.
The Gigabyte has 2 slots blocked by the GPU, and the other are blocked by the RAM (right?). Using a riser/cable then has the problem of how to mount the GPU. I’d much rather a motherboard when I can just slot in a GPU, like the damned Asus that doesn’t boot! Damn!
The Gigabyte MZ73-LM1 has reasonable PCIe slots but it’s dual CPUs. I’d need to buy a second 9684X and 12 more DIMMs. I wasn’t really planning on that, and powering that beast plus a GPU is harder. I’ve got a 1600W PSU, but 2x 400W CPUs and 1x 700W GPU is 1500W before any drives (8 SSDs, 4 HDDs so far).
Any other options?
Aye, thanks. I’ve got 12x96GB.
Asus got back to me after ~2 days, but only to say they are working on it. They promise another update in 24-48 hours.
Previously I tried to flash older BIOS versions via iKVM, but it gave an error. I found today that I can flash them via USB. I tried all 3 (0302, 0401, 0601) without success. It’s mildly interesting that the BIOS flash utility can browse all the SSDs just fine. Why can’t this board boot them?! whyyyy
The hotswap shenanigans prove the drives work 100%. The Proxmox install is really on there, I verified it. Whatever goes wrong at boot really looks like a BIOS bug. Asus takes days to get back to me. My guess is they offer to RMA the board, but I doubt that’s a fix (given their bad PR lately, they’ll probably charge me). Even if they are willing to make a BIOS fix, how long does it take to publish a BIOS update? Plus if I can’t trust Asus for such basic functionality, I don’t want to use Asus server gear. At least their gaming stuff is popular enough that they seem to fix the issues.
