Remote Server Locked Up (Room Locked)

I have a remote server that has locked up on me. It’s an consumer X570 Motherboard, so there’s no IPMI or out of band management.

I still have access to a VPN to the router. The server has reboot on AC power-loss enabled along with WoL.

Is there anything I can do to get it to reboot or prevent this in future. I saw that with the Pi KVM there is an attachment that splits the front panel buttons to an ethernet PCI slot. Would it be possible to use something like that for a simple remote on/off switch.

1 Like

It specifically mentions “Onboard ATX power control” so I think that would be your best bet. If it locks up you could just remote in and reboot it

Of course fixing the locking up issue would be priority number 1!

Is this part sold/work separately from the Pi KVM? If so where can I get it.

It looks like its all included, I’m going off of this listing

image

That’s a shame, It’s a great product but that price + a Pi is a bit much for me. I wouldn’t mind it as much if I my experience with trying to run RPis as 24/7 tools was better

On closer inspection there’s no logic on that PCI board just traces. Of all the crap-tastic IoT devices I would have thought this would be a thing.

1 Like

Agreed, remote KVM/IPMI has always been a sore spot

What are the other specs of the system? Is it possible to grab a used Supermicro board that has IPMI built in and swap it?

Another idea could be a Z-Wave/WiFi/Zigbee smart switch for the PSU, and then just have it turn off and on when it locks up. Would cost like $20

I’ve had very good luck with these. I like that they are WiFi so I can actually tell they are connected etc - https://www.amazon.com/TP-Link-Kasa-Smart-Wifi-Plug/dp/B07RCNB2L3

Or, a controllable UPS or PDU, and then control that

1 Like

If you had something connected to the UPS serial port you could possibly reboot. If you had a staff member with access to the power you could do it that way. Failing that crash a drone into the power pylons and hope it puts the power out to the town for a while.

1 Like
System
Component Information
Brand Custom
CPU AMD Ryzen 5 3600 @4.20GHz
RAM DDR4 16GB 3600MHz @3600MHz (2x)
Microarchitecture Zen 2
Motherboard ASUS x570-F

HBA for 13 HDDs, Nvidia GT 1030, 750 W Corsair PSU

That Z-wave idea isn’t bad. I was thinking of putting a SonOff switch inline with the IEC C-13 to the PSU. Had a look for network controlled switched PDUs too turns out go for $400+.

That reminds me, is there some way to control the power ports of an Eaton MGE UPS with the USB port?

Drones are out of the question, we’re smart enough not to have power cables above ground here haha.

There is a Linux based Power Management Suite for Eaton MGE UPS. I’ve not used it myself.

The Crypto miners have a watchdog that plugs into the reset connector on the PC motherboard. Software has to keep restarting the watchdog timer or it will time out and hit the reset. That might do you.

@wayland That’s a really good idea, I’ll have a look at that.

1 Like

could you get a relatively inexpensive “smart” power plug/power switch?

Doesn’t help you get into a locked closet though.
and might only be accessible via wifi, and not sure if the router you can connect to has a wifi modem.

I mean, if the item is in a co-lo, then the hands-on charge for a guy to power cycle at $50, will still be cheaper, depending on rates.

But if it secured elsewhere, then I guess that’s a no-go

Yeah looks like there’s no in-between. The comprehensive options are $300 - $500 and the ones that perform only the act of power cycling are about $20.

  • I can’t use the all warts because it’s connected to a UPS.
  • I’ve decided I’m going to build myself a wireless in line IEC C13 switch.

That’s the good news, but I can’t for the life of me find anyone that makes one that has three conductors (the ground pin is missing on most). Seems they’re all targeted at lighting.

1 Like

Cycle power to the building

Maybe something like a remote finger?

A similar concept as demonstrator… As a solution for the future.

Or one of the sonoff switches… you’ll just have to prepare yourself two cables without plugs on one end.

2 Likes

That genius, made me chuckle a little that this Adaprox Fingerbot Plus exists.

The issue that I have with the Sonoff devices and others in the same vein is that they don’t have ground lines, just live and neutral.

  • Buy consumer motherboards with Intel vPro or AMD PRO.
  • Buy server boards with IPMI.
  • Buy one of the (few and expensive) IPMI cards that be plugged into just about any system.
  • Connect an 8-port USB relay to each power switch, and plug it in to a reliable system you can access remotely. Will allow you to hold the power switch down on each system as needed.

Less reliable:

  • Connect your system to a PDU (or a UPS) with individually switchable outlets. (becomes rather reliable in combination with WoL, which helps in the event of accidental clean shutdowns)
  • Enable the watchdog timer in the BIOS. The system should automatically reboot when the OS locks-up hard.
  • Enable Magic SysRq so that you can always reboot the system over your IP-KVM, in the event of anything but a kernel panic (which the above watchdog timer will handle).
  • Schedule regular nightly/weekly/monthly reboots, so the system isn’t left running for long periods between returning to a known clean state.

You can do this from the os/Linux.

Pikvm software works on other / cheaper things too, and there’s several options on getting things working if you’re willing to diy - the hardest part is USB device mode. In theory , a pi zero + $15 HDMI → USB 2.0 adapter + $10 esphome compatible socket might be a working option.

It’s not nice of sonoff that they don’t have. :wink:

I thought about the purchase of Fingerbot myself, but in the end the project went in a different direction. :wink:

Then maybe instead of sonoff, buy a regular on / off switch and mount it between the psu-ups and the fingerbot will do the remote on / off. Or plug a long cable with the new power / reset button into the server and lead it out of the back of the rack and let the fingerbot touch it there. :slight_smile:



or