I have a remote server that has locked up on me. It’s an consumer X570 Motherboard, so there’s no IPMI or out of band management.
I still have access to a VPN to the router. The server has reboot on AC power-loss enabled along with WoL.
Is there anything I can do to get it to reboot or prevent this in future. I saw that with the Pi KVM there is an attachment that splits the front panel buttons to an ethernet PCI slot. Would it be possible to use something like that for a simple remote on/off switch.
That’s a shame, It’s a great product but that price + a Pi is a bit much for me. I wouldn’t mind it as much if I my experience with trying to run RPis as 24/7 tools was better
On closer inspection there’s no logic on that PCI board just traces. Of all the crap-tastic IoT devices I would have thought this would be a thing.
If you had something connected to the UPS serial port you could possibly reboot. If you had a staff member with access to the power you could do it that way. Failing that crash a drone into the power pylons and hope it puts the power out to the town for a while.
HBA for 13 HDDs, Nvidia GT 1030, 750 W Corsair PSU
That Z-wave idea isn’t bad. I was thinking of putting a SonOff switch inline with the IEC C-13 to the PSU. Had a look for network controlled switched PDUs too turns out go for $400+.
There is a Linux based Power Management Suite for Eaton MGE UPS. I’ve not used it myself.
The Crypto miners have a watchdog that plugs into the reset connector on the PC motherboard. Software has to keep restarting the watchdog timer or it will time out and hit the reset. That might do you.
Doesn’t help you get into a locked closet though.
and might only be accessible via wifi, and not sure if the router you can connect to has a wifi modem.
I mean, if the item is in a co-lo, then the hands-on charge for a guy to power cycle at $50, will still be cheaper, depending on rates.
But if it secured elsewhere, then I guess that’s a no-go
Yeah looks like there’s no in-between. The comprehensive options are $300 - $500 and the ones that perform only the act of power cycling are about $20.
I can’t use the all warts because it’s connected to a UPS.
I’ve decided I’m going to build myself a wireless in line IEC C13 switch.
That’s the good news, but I can’t for the life of me find anyone that makes one that has three conductors (the ground pin is missing on most). Seems they’re all targeted at lighting.
Buy consumer motherboards with Intel vPro or AMD PRO.
Buy server boards with IPMI.
Buy one of the (few and expensive) IPMI cards that be plugged into just about any system.
Connect an 8-port USB relay to each power switch, and plug it in to a reliable system you can access remotely. Will allow you to hold the power switch down on each system as needed.
Less reliable:
Connect your system to a PDU (or a UPS) with individually switchable outlets. (becomes rather reliable in combination with WoL, which helps in the event of accidental clean shutdowns)
Enable the watchdog timer in the BIOS. The system should automatically reboot when the OS locks-up hard.
Enable Magic SysRq so that you can always reboot the system over your IP-KVM, in the event of anything but a kernel panic (which the above watchdog timer will handle).
Schedule regular nightly/weekly/monthly reboots, so the system isn’t left running for long periods between returning to a known clean state.
Pikvm software works on other / cheaper things too, and there’s several options on getting things working if you’re willing to diy - the hardest part is USB device mode. In theory , a pi zero + $15 HDMI → USB 2.0 adapter + $10 esphome compatible socket might be a working option.
I thought about the purchase of Fingerbot myself, but in the end the project went in a different direction.
Then maybe instead of sonoff, buy a regular on / off switch and mount it between the psu-ups and the fingerbot will do the remote on / off. Or plug a long cable with the new power / reset button into the server and lead it out of the back of the rack and let the fingerbot touch it there.
