How to diagnose kernel crashes without display, serial, or IPMI?

Operating Systems & Open Source Linux
1

my server is mostly stable. however sometimes the system hangs for random and unknown reasons. i have configured my kernel to panic on lockups and to use kdump to save kernel crashes and then reboot.
this however, never happens. when my kernel panics, the entire system becomes 100% hung until i physically go to my server room and power cycle the machine. this is something i hate doing.

connecting a display device directly to my server is not an option. my server has no serial connections at all (cheap motherboard). and my server has no IPMI.

i do have an old laptop that is just collecting dust, and my server has an unused USB connection. is there an easy way to have this laptop constantly read dmesg (and/or other logs) over USB from my server such that i can use the laptop to find the cause of these random crashes? and maybe find out why kdump isnt working as well?
or is there a much better approach to this sort of issue?

2

You can attach a USB to serial cable or one of those cheap FTDI dongles (just be aware of the difference in the RS232 voltage levels) to that USB port of your server. When connected a /dev/ttyUSB0 file should be created (or something similar). That /dev/ttyUSB0 is your “serial interface”.

Run this command:

stty -F /dev/ttyUSB0 speed 115200

A baudrate of 115200 is usually what a console serial interface uses, but you can try lower ones.

You should now have a terminal through that serial interface.

To enable this terminal at boot, you need to edit the /etc/sysconfig/grub or /etc/default/grub or wherever that file is in your server. Just be careful to not be locked out of your system.

1 Like