[SOLVED] Kdump cannot use NVME on Debian

Are you using virtio-pci and binding a device that is sharing your rootfs NVMe or SATA controller IOMMU group?

1 Like

i use vfio-pci but not on the ssd in question nor anything within it’s IOMMU group.

UPDATE: i have been told that there is a known bug in all kernels 5.5 and newer that can cause this behavior. but reverting my kernel to 5.4 made no difference in kdump’s behavior. specifically, i tried 5.4.0-4
this is the bug that it was suspected i am encountering. https://bugzilla.kernel.org/show_bug.cgi?id=209351

2 Likes

Thanks for the update. Try going back to 5.3 or older?

1 Like

as of 5.14. this is still happening.

according to ppc64el/Recommendations - Ubuntu Wiki

NVMe is simply not supported. but this is ubuntu, the Debian documentation says nothing about this. is there any workaround to force NVMe support?

It may be a kernel command line argument maybe.

Unfortunately, I do not have a PPCle system to use. While I am a Debian and ArchLinux shill, you may want to take a look at Void Linux PPC edition and see if it has support for NMVe.

Also, have you tried one of the newer kernels like 5.13 or newer?

i am not on PPCIe, i am AMD64, which is why im not sure if that lack of NVMe support applies to me here.
i am on Linux 5.16 and still have this issue.

Oh, PPCle is PowerPC (POWER) little endien. Completely different CPU architecture. It has nothing to due with your x86_64 CPU.

If you are on the new hot kernel and still having issues, then more than likely, it is your Mobo or Mobo chipset and driver/kernel config combo.

Have you reached out to your distro maintainers. You are probably an extremely edge case.

yes. i have reached out to debian.
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=990464

Wow. I wonder if the kernel is suffering from bit rot. René Rebe has been complaining about this for a while as he maintains old architecture support in the Linux Kernel.

updated to linux 5.17.5. no change in this behavior.

Thanks for the bump.

i believe the problem can be solved by changing the “Nvme firmware source” option in a motherboard’s firmware. however, my motherboard lacks this option. is there any know software/kernel way to override the nvme firmware source?

or maybe is there a way to idenitfy and flash a different firmware to my mobo’s NVMe controller?

side note: i have discovered that without ACPI, NVMe just doesnt work a all. even on the normal kernel. not sure if its at all related.


Updated to Linux 6.0
different output now.

Upgraded to Debian Bookworm. No change in behavior.

1 Like

Does anybody know if it’s at all possible to use kdump without a root fs? I do have a network drive I could upload the dump to.

I’ve never done this at home, but your replacement/kdump kernel is supposed to boot all the way to init (could be initramfs/intrd) where e.g. a busybox script can do whatever with /proc/vmcore ; such as e.g. save it on local disk.

e.g. you can “crash your way into dropbear”, so that external network machine can slurp your ram over ssh or do whatever on the machine (poke stuff around various devices). What functionality is initialized or isn’t depends entirely on the kernel options for the replacement/kdump kernel and on the userland that ends up running.

… but as I’ve said, I’ve never done this at home, I don’t what various distro packages/maintainers do for this kind of debugging. (all I know is that it sucks, … my Debian router kexecs new kernel versions after upgrades, it was a pita to set up).


It’s hard to tell from the glare in your screen, do I see init starting happily there?

1 Like

Yes, it seems initramfs loads fine, just panics when it can’t find the root fs.

It sounds like you need a different initramfs image, or perhaps you could add a hook to the existing one, that would detect the kdump context and do something else?

Using an initramfs hook is probably the way to go (because the initramfs image will get rebuilt all the time whenever there’s a new kernel, etc).

1 Like