It’s been a few months since I wrote about the network bonding issue.
I have a little time now to sum up my findings.
So here goes:
I started with a situation where I had a bonding enabled without a way to disable it (no option in the ipmi web UI - see post 1222 )
The main point of disabling the bonding for me was a complete separation of IPMI network. With bonding enabled it would be accessible if I used the first ‘normal’ lan port - this is a security issue in my eyes.
At first I discovered that I can force disable/enable the bonding by modifying the web UI html - the option is there - it’s just disabled. Details are also in post 1222
After force enabling/disabling/etc I somehow gotten the option in the web-ui to stay visible. I am not sure why - maybe because of fixing MAC adresses (described below)
Then I contacted Asrock support about this and got some info and a tool.
The info was that they have some issues with some boards setting incorrect MAC addresses - they expected that one of the MACs on my board was set to zeros.
The tool is here:
mac_tool_from_asrock.zip (1.1 MB)
It is supposed to be able to enable/disable bonding and change/fix MAC adresses.
I got it in a binary form so instead believing what it is supposed to do and running it, I did some reverse engineering…
I managed to extract some commands that can be used to identify/fix the MAC issues. (they are listed at the bottom of this post)
As it turned out my board had duplicate MACs on IPMI side.
To make it more clear: the ports have different MACs on the IPMI and host side. My board was i this situation:
physical port: | LAN_1 | LAN_2 | IPMI_LAN |
---------------+-------------------+-------------------+--------------------+
ipmi side MAC: | d0:50:99:e3:44:d9 | N/A | d0:50:99:e3:44:d9 |
host side MAC: | d0:50:99:d2:d0:9c | d0:50:99:d2:d0:9d | N/A |
And after manually fixing the duplicate (commands below):
physical port: | LAN_1 | LAN_2 | IPMI_LAN |
---------------+-------------------+-------------------+--------------------+
ipmi side MAC: | d0:50:99:e3:44:d8 | N/A | d0:50:99:e3:44:d9 |
host side MAC: | d0:50:99:d2:d0:9c | d0:50:99:d2:d0:9d | N/A |
The commands I extracted are as follows:
#get MAC 0
ipmitool -I lanplus -H <IP> -U <user> -P <password> raw 0x3a 0xa1 0x00
>d0 50 99 e3 44 d8
#get MAC 1
ipmitool -I lanplus -H <IP> -U <user> -P <password> raw 0x3a 0xa1 0x01
>d0 50 99 e3 44 d9
#set MAC
#ipmitool -I lanplus -H <IP> -U <user> -P <password> raw 0x3a 0xa0 0x00 <NEW MAC>
#ipmitool -I lanplus -H <IP> -U <user> -P <password> raw 0x3a 0xa0 0x00 <NEW MAC>
#for example:
#set MAC 0
ipmitool -I lanplus -H <IP> -U <user> -P <password> raw 0x3a 0xa0 0x00 0xd0 0x50 0x99 0xe3 0x44 0xd8
#set MAC 1
ipmitool -I lanplus -H <IP> -U <user> -P <password> raw 0x3a 0xa0 0x01 0xd0 0x50 0x99 0xe3 0x44 0xd9
#detect bonding
ipmitool -I lanplus -H <IP> -U <user> -P <password> raw 0x32 0x72 0x01 0x00 0x00
>00 00 01 00 00 00 01
# check first byte: (no idea what other bytes mean)
# disabled: 00 00 01 64 00 03 01
# enabled: 01 00 01 64 00 03 01
#get bmc version:
ipmitool -I lanplus -H <IP> -U <user> -P <password> raw 0x06 0x01
> 20 01 01 90 02 bf d6 c1 00 02 02 00 00 00 00
# 1.90 00 00 00 00
