Another tip on these boards to pass through the primary GPU. By setting the RTL8117 KVM display mode to either Local Multi-Display or Remote Multi-display it appears to release the GPUs and allow them to be bound with VFIO. I am not sure exactly what is going on but I suspect the internal RTL graphics is booting the GPUs instead of using the normal POST process which allows the VFIO stub driver to take over even for the primary GPU.
I have a windows VM on my server running for this purpose specifically.
You can power the machines on, off, force off, reboot etc from the ACCE software. It also has the ‘Remote Desktop’ component which lets you connect to the IPMI video device. I can setup the system remotely from start to finish by mounting an ISO from ACCE, accessing the BIOS and booting to the mounted image and then installing an OS and going from there all without having physical access.
So Windows VM is on some other machine right? Because later you wrote …
Without the RTL8117 set to remote single-display then when WIndows boots you have to use RDP to access the machine since the display from the onboard video is disabled
… so this refers to windows on Asus? You have windows VM on some other machine to access the IPMI and Windows machine as host on ASUS motherboard. But host doesnt have to be windows right?
One more question. Can you choose boot device in UEFI/IPMI? In my current setup i have proxmox on one disk and windows 10 (for games) on other disk. My plan is to set proxmox as default (to run it when im remotely) and Windows on second drive to use it when im directly use computer. Im not going to do gpu passthrough so I would be able to leave IPMI on main card (default).
So can you guys recommend this motherboard as kind of alternative to IPMI? I havent seen any AM4/TR4 motherboards with remote control beside of mentioned Asrocks which sadly are not available in my country.
and
I am experiencing exactly the same two issues as well. However, It might not be able to boot/post until I power off PC and unplug from the power for several seconds.
My specs:
CPU: Ryzen 9350X
RAM: 128 ECC DDR4
GPU: RX580 8GB
OS: ESXi 7.0
Hello @droric,
Could you explain in a bit more details on how you have updated ACCE firmware? Like the params you used in *.bat file and why those values. I have not found any manual for that. When I try to update the firmware it hangs/does not proceed at step when it checks for a connectivity to the RTL8117.
I also want to chime in here. I recently bought this board.
@Rusman I can safely say that it’s not possible to update the firmware through the ACCE_Firmware_Update_0111.zip archive. There are no instructions available and the bat-file failed everytime. The only way to update the firmware was through the RTL_8117_Firmware_0111.zip archive and the provided 0111_20200318_website_k_sign.img image. You will have to activate the IPMI in the BIOS under Advanced > RTL8117 setting > RTL8117 Manager controller and set it to Enabled. For me it was crucial to set it to a static ip under Configuration Address source and plug in another Ethernet cable to the Intel NIC. After that I had to cut the power to the machine completely and bring it back online. You can see that the RTL8117 NIC on the backside of the mainboard starts flickering and picking up a signal. Furthermore I had to boot into Windows 10 as a host. Finaly after that I was able to update the firmware via the ASUS Control Center Express v01.04.24 Software from a second Windows machine pluged in to same network. Under Management Controller > Scan > Select the device through clicking on the UUID > in Management Control Information windows I had to select Firmware Update > than I had to click Import File and choose the aforementioned 0111_20200318_website_k_sign.img then click Firmware Update > Voila!
1 Like
Some quirks I discovered: In order to use the IPMI features through the ACCE I will have to power down the machine completely and wait till the RTL8117 NIC turns off. After that I will have to turn the power back on and wait till the RTL8117 NIC picks up a signal which is visible when the lights on the NIC start to flicker again. I then have to connect through ACCE and power on the machine through Management Controller > Scan > click on the found UUID > Control > Power On.
To my knowledge there is no way to save the found machine and monitor it. I deployed the Setup.msi found in Agent Management manually but I’m still unable to add the machine in ACCE.
If I power down the machine either through IPMI or on the machine itself I’m not able to control the IPMI afterwards anymore. I can confirm this because the RTL8117 NIC turns off. I then have to cut the power again, wait, and power on back to be able to control the IPMI again.
Has anybody successfully be able to control a linux based machine (like proxmox or unraid) after enabling Remote multi-display or Remote single-display through the inbuild IPMI graphics with ACCE? I will just get a black screen after linux boots. Windows works fine.
EDIT: Got it working. I had to add a static ip configuration in proxmox as well.
$ cat /etc/network/interfaces
auto enp8s0f1
iface enp8s0f1 inet static
address 192.168.1.250/24
Now for some reason I’m unable to send any keyboard input to the tty console. My guess is that there are no drivers available for it. This is how the NIC is listed in proxmox:
Group: 30 0000:08:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. Device [10ec:816e] (rev 1a)
Group: 30 0000:08:00.1 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 1a) Driver: r8169
Group: 30 0000:08:00.2 Serial controller [0700]: Realtek Semiconductor Co., Ltd. Device [10ec:816a] (rev 1a) Driver: serial
Group: 30 0000:08:00.3 VGA compatible controller [0300]: Racore Computer Products, Inc. Device [10ef:816f] (rev 1a)
Group: 30 0000:08:00.4 USB controller [0c03]: Realtek Semiconductor Co., Ltd. Device [10ec:816d] (rev 1a) Driver: ehci-pci
Group: 30 0000:08:00.7 IPMI SMIC interface [0c07]: Realtek Semiconductor Co., Ltd. Device [10ec:816c] (rev 1a)
I dont know which device is responsible for the remote control.
The next issue is that I have to bind the tty console output to the Racore Computer Products VGA [10ef:816f]. I guess something like video=astdrmfb , but I dont know which driver is used.
root=ZFS=rpool/ROOT/pve-1 boot=zfs amd_iommu=on iommu=pt vfio-pci.disable_idle_d3=1 disable_vga=1 nofb nomodeset video=vesafb:off video=efifb:off textonly video=XXX
$ lspci -vvv -s 0000:08:00.3
08:00.3 VGA compatible controller: Racore Computer Products, Inc. Device 816f (rev 1a) (prog-if 00 [VGA controller])
Subsystem: Realtek Semiconductor Co., Ltd. Device 8168
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin C routed to IRQ 255
Region 0: I/O ports at b000 [size=256]
Region 2: Memory at 7ff0800000 (64-bit, prefetchable) [size=8M]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 01
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Via message/WAKE#
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [b0] MSI-X: Enable- Count=1 Masked-
Vector table: BAR=0 offset=00000000
PBA: BAR=0 offset=00000000
Capabilities: [100 v2] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [168 v1] Device Serial Number 00-00-00-00-00-00-00-00
Capabilities: [178 v1] Transaction Processing Hints
No steering table availableSo I just picked up one of these boards to replace an old asus sabertooth something or other as a home server. Im running Windows 10 Pro as the host os and have installed ACCE on another windows 10 client on the same lan/subnet (all windows firewalls are off), yet as mentioned above I can not for the life of me add this thing to ACCE. Always get “Remote Procedure failed to connect [SYSTEM_ERROR:5,Access is denied.]” when I try to deploy/add to ACCE. the account type/user/pass are correct and local admins on everything but still no worky. 
Has anyone had any luck with getting this board to work with ACCE? Eventually I will end up putting Ubuntu or another good stable *nix distro on it. But too much data to migrate atm.
Thanks in advance!
~SOL
@SomeoneOnLine Did you try to install ACCE Agent manually on the host and then connect through ACCE on another machine?
If you go down the *nix route I would be interessted to know if you can get the remote control feature fully work. I’m still unable to send any keyboards input.
Thanks for the reply!
I did try to manually install the agent (setup.msi generated from ACCE) on the host os but somewhere in the middle is fails saying that the client may already be registered to the ACCE server, yet when I look in the ACCE interface I dont see any clients/agents listed. I even uninstalled/reinstalled ACCE thinking it may clear/clean whatever DB it thinks has the agent in it but same results.
I know the OP was more or less referring to linux but, oddly enough, shy of contacting ASUS support directly I cant really fund much of any forum based folks that either have this mobo or have played with ACCE.
~SOL
Have had this board a while. Here is the way I got the remote management to function:
Im thinking I need to update the bios, which is what im stuck at atm.
question, when I go into acce (from another machine on same lan) and go into controller and do a scan (to update the firmware), it sees the nic/server via ip but, the uuid does not show in the uuid field. tbh, where is it looking for the uuid from? bios? I looked all around the bios and dont see a uuid. realtek firmware is 2019 something. The board is literally brand new, was really hoping to get the kvm function working on this thing.
Iv seen the raspberry pi kvm units. I have thought about building/buying one, but figured this board is supposed to have that function.
Well, I’ll just say this: messed with this feature of the board (albeit in 2019, scroll up to find the posts) and never could get it running (and there was no hope of any linux back then, only windows). Combine the time spent tinkering with no success with what I perceive as a lack of inteterest/support from ASUS on this feature (they have such few boards that use this inhouse specialty build vs the likes of ipmi of Asrock Rack and Supermicro) and my cost/benefit analysis says shell out for the pi. Obviously your analysis will be different, but I can say the pi route works, without flaw or exception.
I got one of these boards and running Linux on it and couldn’t resist investigating the RTL8117 stuff a bit today to see if I could use it without a Windows client and ACCE.
I really don’t have time to investigate this further (I already spent too much time on this), but below is some ramblings in case someone is interested or wants to pick it up, so they don’t need to necessarily start from scratch.
tl;dr Basic commands mostly work, KVM/VNC does not (but is probably solvable with some effort).
I mostly discovered the below using wireshark and mitmproxy.
First, there is an unauthenticated REST GET endpoint https://IP/cgi-bin/luci/apiasus/descriptor that returns JSON with some basic info and a list of functions.
Then GET https://IP/cgi-bin/luci/?luci_username=root&luci_password=PASSWORD where PASSWORD is the last 15 characters from the UUID contained in the JSON in the last request. You will get a Location: redirect to /cgi-bin/luci/;stok=STOK where STOK is a new dynamic token. Next register the token using GET https://IP/cgi-bin/luci/;stok=STOK/apiasus/reg_stok?appuid=consoleTest (replace STOK) - any appuid seems to work, but ACCE uses “consoleTest”.
Now you can call the rest of the functions with the stok, e.g.:
-
https://IP/cgi-bin/luci/;stok=STOK/apiasus/dmesg(dmesg of the Linux system running on RTL8117) -
https://IP/cgi-bin/luci/;stok=STOK/apiasus/get_dxe_info(sensor data etc., but there is noise in the output, maybe there is some encapsulation on top of JSON…) -
https://IP/cgi-bin/luci/;stok=STOK/apiasus/reboot_pc(reboot) - etc, see the descriptor JSON for all functions.
There is a WS-Management API on port 623, HTTP Digest auth, root:PASSWORD with PASSWORD from before.
One can use wsmancli, or AMD DMTF DASH tools, or just manually POST XML into the http://IP:623/wsman endpoint.
A lot of my wsmancli commands caused the remote wsman server to crash (?) and not respond to commands anymore, I used REST “stop_service” and “restart_service” functions to bring it back (I now see there is “restart_wsmand” too, maybe it would be enough).
The KVM uses a VNC-like protocol but only seems to allow a single connection each time so you need to disable KVM and enable KVM via WS-Man each time before connecting.
The raw XML command to enable/disable KVM is as follows. Replace IPADDRESS with the RTL8117 address, and the MessageID UUID should be unique (but I doubt the server cares). Replace XXX with 3 for disable, 2 for enable. I used curl -i --data-binary '@input.xml' --digest --user 'root:PASSWORD' http://IP:623/wsman.
<?xml version="1.0" encoding="UTF-8"?>
<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" xmlns:wsa="http://schemas.xmlsoap.org/ws/2004/08/addressing" xmlns:wsman="http://schemas.dmtf.org/wbem/wsman/1/wsman.xsd"
xmlns:n1="http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_KVMRedirectionSAP">
<s:Header>
<wsa:Action s:mustUnderstand="true">http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_KVMRedirectionSAP/RequestStateChange</wsa:Action>
<wsa:To s:mustUnderstand="true">http://IPADDRESS:623/wsman</wsa:To>
<wsman:ResourceURI s:mustUnderstand="true">http://schemas.dmtf.org/wbem/wscim/1/cim-schema/2/CIM_KVMRedirectionSAP</wsman:ResourceURI>
<wsa:MessageID s:mustUnderstand="true">uuid:9bf323ca-016e-4bb5-aca3-84acda80088e</wsa:MessageID>
<wsa:ReplyTo>
<wsa:Address>http://schemas.xmlsoap.org/ws/2004/08/addressing/role/anonymous</wsa:Address>
</wsa:ReplyTo>
<wsman:SelectorSet>
<wsman:Selector Name="__cimnamespace">root/interop</wsman:Selector>
<wsman:Selector Name="Name">KVMSAP:2149449728</wsman:Selector>
<wsman:Selector Name="SystemName">IPADDRESS</wsman:Selector>
<wsman:Selector Name="CreationClassName">CIM_KVMRedirectionSAP</wsman:Selector>
<wsman:Selector Name="SystemCreationClassName">CIM_ComputerSystem</wsman:Selector>
</wsman:SelectorSet>
</s:Header>
<s:Body>
<n1:RequestStateChange_INPUT>
<n1:RequestedState>XXX</n1:RequestedState>
<n1:TimeoutPeriod>00000000000000.000000:000</n1:TimeoutPeriod>
</n1:RequestStateChange_INPUT>
</s:Body>
</s:Envelope>
The VNC is running in standard port 5900, no authentication (just disable/enable KVM as above each time).
The VNC protocol is unfortunately not standard VNC. The server identifies the protocol as RFB 003.00R\n, i.e. version 3.R (Realtek?), which is not understood by e.g. tigervnc. After faking support for 003.00R in TigerVNC (just consider 3.R as 3.8 and inform server 3.R is supported) I did get something to display, e.g. the AMIBIOS logo was recognizable but was somehow messed up, and occasionally seemingly the VNC client kept losing the synchronization and tried to interpret pixel data as commands (“Unknown message type 85”), especially when rebooting the server system.
I think the VNC issues should be solvable with some effort, the protocol is simple enough that one should be able to figure out what is going wrong. Could just be some disagreement between client and server on the exact pixel encoding or screen size.
AMD DMTF DASH tools produced errors when trying to connect to the KVM VNC.
2 Likes
huh seems like this would be a CVE security issue if you can just unauth get the password and there isn’t a clean way to set it.
This is way more functional than I thought. Nice post!
That was my first thought, but I guess no security was implied in the first place since you can already execute commands and use KVM with ACCE without authentication anyway.
There is a “set_psw” method, though, I believe it could be used to change the password.
The full list of REST functions is below. There are also function descriptions below (under “Function_list”), but some functions have no descriptions so there is no 1:1 mapping between the function names and descriptions.
There is no description of function parameters but luckily the functions seem to give out descriptive errors if mandatory parameters are missing.
"Asus_api" : [
"init_check",
"get_stok",
"reg_stok",
"set_psw",
"get_info",
"get_dxe_info",
"download_dxe",
"upload",
"upload.htm",
"upload_file",
"upload.web",
"download",
"wd_enable",
"wd_set_timer",
"wd_set_interval",
"gpio_op",
"function_status",
"test",
"get_pcstate",
"power_on_pc",
"power_off_pc",
"clear_cmos",
"reboot_pc",
"switch_spi_to_pc",
"switch_spi_to_8117",
"probe_bios_flash",
"remove_bios_flash",
"descriptor",
"get_device_info",
"clean_ring_buffer",
"stop_service",
"restart_service",
"restart_wsmand",
"upgrade_fw",
"upgrade_safemode",
"check_mode",
"uart_module",
"set_ip",
"factory_upload",
"factory_setenv",
"push_rma",
"dump_rma",
"clear_backup",
"clear_stok",
"get_kvm_usbr",
"set_kvm_usbr",
"get_gop_status",
"recovery_backup",
"get_kvm_display",
"set_kvm_display",
"dmesg",
"get_firewall_mode",
"set_firewall_mode",
"get_firewall_ip",
"set_firewall_ip",
"get_machine_name",
"set_machine_name",
"get_misc",
"set_misc"
],
"Function_list" : [
"Registered/Get stock",
"Set password",
"Get info",
"Get/Download dxe info",
"Upload/Download file",
"Watchdog",
"Gpio control",
"Function status",
"Test",
"Get PC status",
"Power on/off PC",
"Force power off PC",
"Clear cmos",
"Reboot PC",
"Switch spi to PC/8117",
"Probe/Remove bios flash",
"Clean ring buffer",
"Descriptor",
"Get device information",
"Stop/Restart Service",
"Upgrade FW/Safemode",
"Check kernel mode",
"Uart module",
"Set IP",
"Upgrade FW for factory",
"KVM",
"USB-R",
"RMA record",
"Clear backup/stok",
"Get/Set kvm and usbr status",
"Get GOP Status",
"Recovery Backup",
"KVM Display Mode",
"dmesg",
"Trust_Zone",
"Get/Set machine name",
"Get/Set misc"
],
The WS-Man API looks like it is there mostly for KVM, or at least I didn’t notice anything too interesting there. AMD Management Console GUI does show e.g. TCR over Telnet and TCR over SSH support (text console I believe), but those didn’t seem to work either (no response from port). There is also e.g. IP setup and “boot to boot option X” stuff, but I didn’t try them.
Hello, general linux and open source tinkerer here. This board caught my eye as being a modern ryzen motherboard with pcie 4.0 and no rgb, but after I realized it had this management console/bmc type thing my interest has increased. I’ve done a bit of tinkering and poking around in the firmware and source code and am probably going to end up buying it. I was hoping you could provide some information on the hw in question.
Just some standard linux hw probe stuff (sensors-detect, dmesg, lspci -nn, lsusb, dmidecode and whatever else you could think to provide), as well as using the rtl8117’s REST function to dump whatever info you could out of that (the dmesg is of particular interest). Obviously, sanitize your mac addr/ip/other information out of it, but I think with some work the rtl8117 could be made a bit more useful, especially since it has its own flash (under the nvme heatsink) on a standard chip clippable soic-8 spi flash.
Hey there, I’ve bought one of these beauties, too, after @hanetzer had told me about this neat board. I am going to use it for developing open source firmware, i.e., oreboot (downstream fork of coreboot written in Rust). I already got some initial basics to work on the ASRock DeskMini A300, which had a similar super IO, so I’ll see if I can get some bytes out of the WS X570-ACE as well.
The core features will be remote flashing and power cycling, which is key to steady development.
@wendell is right in that the BMCesque firmware is basically a pile of CVEs. I have tweeted some of the things I just did, though I did see a slightly different API. I should probably upgrade to gain more features. 
Lest I say that you can just register arbitrary tokens.
For the original images, check for OrangeCMS on Twitter.
The forum limits me to two images and forbids links, unfortunately.
1 Like
hey i have a info for you maybe, you know you can just download ACCE_Firmware_Update_0111.zip and inside of it extract ACCE_Firmware_Update_0111.exe with 7zip to get the source code of the webapi?
should make it a lot easier to reverse
1 Like
If anyone is still experiencing RAM instability and random power cycling on this board, you might have to tweak VDDG. I have VDDG CCD set to 1.1V and VDDG IOD set to 1.0V on the latest BIOS with AGESA 1.2.0.7.
I’m going to make another run at the RTL8117 this week. My goals are BIOS redirection via ACCE and bog-standard serial-over-LAN for the “client” OS (Linux).
EDIT: I ended up burning the midnight oil and working on it last night and… wow, what a disaster. I thought SURELY the latest firmware, 10 months later, would make it a bit more stable and open up some of the missing functionality, but it’s exactly as everyone has described above. Just enabling the KVM feature was enough to make the system un-(re)bootable. I tried setting up the serial console on ttyS4, from the RTL8117 device, but I can’t figure out anyway to get into it. In short, nothing works.
I have a lot of devices hooked up—and a lot of features enabled e.g. SR-IOV—so there could be a compatibility issue causing the booting. I think if I cared to spend any more time on this I’d first strip the system down to the bare minimum and do a full CMOS reset. Unfortunately I need this particular system to be up and running and stable so I guess that will have to wait for another day.
I thought maybe I could repurpose this board for a headless virtualization host but for the time being at least I’m going to keep shopping around…
3 Likes