Background
There was an absolute flood of -ON suffix 100gbe Dell switches in November of '21 – I got, at the time, a really good deal on this switch – less than $1200. I was really hoping you could get 25gbe switches for < $100 per port by now!
In the time since then it has been near impossible to buy this switch at this price. Part of the reason the switch was so cheap is that it was supposed to be ONIE compatible but it had weird compatibility issues – mostly down to rough edges around the Broadcom SDK. As a result it would only work with Dell’s OS10 OS (and that was/still is wildly expensive, for what it is).
There are several vendors who used to sell third party OSes for these switches (which is AWESOME).
Unfortunately, Nvidia bought Cumulus Linux, which also caused Broadcom to pull t heir licenses for the software SDK for the chipset. Cumulus had been a kind of Freemium… but the enthusiasm for it has died after the NV acquisition.
Worse, the S5212F-ON isn’t really Dell OS9 compatible (which doesn’t require a license) and, worse still, OS10 will only work on these for 3 days at a time. After which the switch reboots.
Unless you have Dell OS10 version 10.2.2E. If you can find it, has a bug (feature?) where it complains about a missing license, but doesn’t actually reboot every 72 hours.
Most of the time OS10 WILL reboot after 72 hours though.
Thanks to enterprises with this hardware experiencing this crippling 72-hour idiocy, non-Dell operating systems for this hardware have been advancing the last couple of years. (Dell realized this was dumb and for some -ON switches they do actually now give you a free OS10 license, but the ones I have were scrapped from a datacenter that didn’t have time to deal with this shenanigans and instead went with different Open Compute hardware. And won’t be buying Dell for that kind of equipment again…that all happened in 2019 though…)
Enter SONiC – an alternative OS… for your switch?! I want to show you the SONiC operating system which is easy to setup, and fully open source.
Where do I download OS10?
That’s the neat part, you don’t! Don’t need that garbage. Stop it. 
Initial Setup of Sonic - Start with Dell S5212F-ON firmware
It is helpful to read over the SONiC Quick Start before beginning.
Even before doing anything in that document, understand that (as a best-practice) you should be able to download some platform updates from Dell. These are the updates NOT stored in your “digital locker.”
I would recommend using a serial console cable. I used serial setings of 115200 with my cable.
Think of the switch as a computer with a bios and firmware. The “switch” part is basically an add-in peripheral (Broadcom) controlled by the computer. ONIE is sort of the platform firmware/bios/out of band management stuff and SONiC as the OS.
There is, unfortunately, more crippling idiocy from Dell. This time unintentional – their web downloader seems to hide the file you need behind a PDF link.
Here’s a PDF! “Other Formats” is actually the firmware…
Yes, Dell, the “other format” of the PDF I need is actually the firmware. Perfect Sense!
Once you have the zip file, unzip it and follow the instructions in the Dell PDF to update your switch. It was necessary for me to apply the update twice because the ONIE firmware was so old that it did not cleanly apply the first time. It seemed to update the BMC, BIOS and FPGA firmware in different steps. There are commands in Dell’s PDF for examining the firmware version from the “Linux” environment as well as the Evaluation version of Dell OS10 likely installed on your switch.
The default OS10 credentials if you need them is admin/admin for the switch OS and linuxadmin/linuxadmin for the Linux shell user. If you login as the Linux shell user it’ll ask you to change the password, and you must enter the linuxadmin password a second time. Otherwise you get a token manipulation error).
With Dell’s stuff fully updated, it is now possible to properly install SONiC and ditch OS10.
Download the SONiC bin for Broadcom; install on your Dell S5212F-ON
Again, I did this via the serial console and with a USB stick. It is possible to do this remotely and via tftp downlodas but… this is more reliable. Unless you have more than 10 of these to do, I’d probably not bother with anything more complex than via serial port. If these had been used previously it is possible that they are in a weird state internally and you’d want to notice that as you (effectively) reimage them anyway.
For the Dell S5212F-ON, you can download the broadcom SONiC Bin file and copy it to a USB drive. I found the USB installation the fastest/safest method. It does take a while.
GNU GRUB version 2.02~beta2+e4a1fe391
+----------------------------------------------------------------------------+
|*ONIE: Install OS |
| ONIE: Rescue |
| ONIE: Uninstall OS <---- Uninstall |
| ONIE: Update ONIE |
| ONIE: Embed ONIE |
+----------------------------------------------------------------------------+
Start by uninstalling OS10, reboot then select Install OS.
this will dump you to a linux terminal.
From here you can use the onie-os-install script –
mkdir /mnt/media
mount /dev/sdb1 /mnt/media
onie-os-instlal /mnt/media/sonic-broadcom-filename.bin
Don’t blindly copy paste! The commands will be something like that to make a directory, mount your usb and the install the sonic OS.
TODO screenshots?
That’ll take a bit, and then it will be possible to login and configure SONIC.
Setting Up Sonic
The default SONiC Password is admin/YourPaSsWoRd (yes, really).
Linux sonic 5.10.0-18-2-amd64 #1 SMP Debian 5.10.140-1 (2022-09-02) x86_64
You are on
____ ___ _ _ _ ____
/ ___| / _ \| \ | (_)/ ___|
\___ \| | | | \| | | |
___) | |_| | |\ | | |___
|____/ \___/|_| \_|_|\____|
-- Software for Open Networking in the Cloud --
Unauthorized access and/or use are prohibited.
All access and/or use are subject to monitoring.
Help: https://sonic-net.github.io/SONiC/
Last login: Tue Jun 27 09:57:07 UTC 2023 on ttyS0
NICE. hacker noises You’re in.
Sonic out of the box comes in “router mode” – so each interface gets an IP and it is setup for IP Routing. Not super handy if you want it to be a “simple” switch!
I’m not sure why but most commands require sudo, even just show commands, which is contrary to SONiC documentation.
admin@sonic:~$ show interface status
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/sonic_eeprom/eeprom_base.py", line 244, in read_eeprom_bytes
F = self.open_eeprom()
File "/usr/local/lib/python3.9/dist-packages/sonic_eeprom/eeprom_base.py", line 232, in open_eeprom
return io.open(eeprom_file, "rb")
PermissionError: [Errno 13] Permission denied: '/sys/class/i2c-adapter/i2c-0/0-0050/eeprom'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/sonic_platform/eeprom.py", line 34, in __init__
self.eeprom_data = self.read_eeprom()
File "/usr/local/lib/python3.9/dist-packages/sonic_eeprom/eeprom_tlvinfo.py", line 258, in read_eeprom
h = self.read_eeprom_bytes(self._TLV_INFO_HDR_LEN)
File "/usr/local/lib/python3.9/dist-packages/sonic_eeprom/eeprom_base.py", line 267, in read_eeprom_bytes
raise IOError("Failed to read eeprom : %s" % (str(e)))
OSError: Failed to read eeprom : [Errno 13] Permission denied: '/sys/class/i2c-adapter/i2c-0/0-0050/eeprom'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/bin/intfutil", line 836, in <module>
main()
File "/usr/local/bin/intfutil", line 819, in main
interface_stat.display_intf_status()
File "/usr/local/bin/intfutil", line 448, in display_intf_status
self.get_intf_status()
File "/usr/local/lib/python3.9/dist-packages/utilities_common/multi_asic.py", line 157, in wrapped_run_on_all_asics
func(self, *args, **kwargs)
File "/usr/local/bin/intfutil", line 535, in get_intf_status
self.table += self.generate_intf_status()
File "/usr/local/bin/intfutil", line 479, in generate_intf_status
port_oper_speed_get(self.db, key),
File "/usr/local/bin/intfutil", line 202, in port_oper_speed_get
return appl_db_port_status_get(db, intf_name, PORT_SPEED)
File "/usr/local/bin/intfutil", line 167, in appl_db_port_status_get
optics_type = port_optics_get(appl_db, intf_name, PORT_OPTICS_TYPE)
File "/usr/local/bin/intfutil", line 224, in port_optics_get
if is_rj45_port(intf_name):
File "/usr/local/lib/python3.9/dist-packages/utilities_common/platform_sfputil_helper.py", line 120, in is_rj45_port
platform_chassis = sonic_platform.platform.Platform().get_chassis()
File "/usr/local/lib/python3.9/dist-packages/sonic_platform/platform.py", line 24, in __init__
self._chassis = Chassis()
File "/usr/local/lib/python3.9/dist-packages/sonic_platform/chassis.py", line 102, in __init__
self._eeprom = Eeprom()
File "/usr/local/lib/python3.9/dist-packages/sonic_platform/eeprom.py", line 37, in __init__
raise RuntimeError("Eeprom is not Programmed")
RuntimeError: Eeprom is not Programmed
Fortunately sudo show interface status works just fine.
sudo show interface status
eeprom'onic:~$ During handling of the above exception, another exception occurred:
Interface Lanes Speed MTU FEC Alias Vlan Oper Admin Type Asym PFC
----------- ----------- ------- ----- ----- ------------------ ------ ------ ------- --------------- ----------
Ethernet0 29 25G 9100 N/A twentyfiveGigE1/1 trunk down up N/A N/A
Ethernet1 30 25G 9100 N/A twentyfiveGigE1/2 trunk down up N/A N/A
Ethernet2 31 25G 9100 N/A twentyfiveGigE1/3 trunk down up N/A N/A
Ethernet3 32 25G 9100 N/A twentyfiveGigE1/4 routed down up N/A N/A
Ethernet4 33 25G 9100 N/A twentyfiveGigE1/5 routed down up N/A N/A
Ethernet5 34 25G 9100 N/A twentyfiveGigE1/6 routed down up SFP/SFP+/SFP28 N/A
Ethernet6 35 25G 9100 N/A twentyfiveGigE1/7 routed down up N/A N/A
Ethernet7 36 25G 9100 N/A twentyfiveGigE1/8 routed down up N/A N/A
Ethernet8 37 25G 9100 N/A twentyfiveGigE1/9 routed down up N/A N/A
Ethernet9 38 25G 9100 N/A twentyfiveGigE1/10 routed down up N/A N/A
Ethernet10 39 25G 9100 N/A twentyfiveGigE1/11 routed down up N/A N/A
Ethernet11 40 25G 9100 N/A twentyfiveGigE1/12 routed down up N/A N/A
Ethernet12 41,42,43,44 100G 9100 N/A hundredGigE1/13 trunk up up QSFP28 or later N/A
Ethernet16 45,46,47,48 100G 9100 N/A hundredGigE1/14 trunk down up N/A N/A
Ethernet20 49,50,51,52 100G 9100 rs hundredGigE1/15 trunk up up N/A N/A
In this screenshot I have reconfigured some ports for trunking and left some of the 25g ports as routed. Unlike with Dell OS10 there is no monkeying with enabling unsupported transceivers.
Troubleshooting
Not everything is always roses, though. Mellanox ConnectX-4 NICs, for example, expect RS style forward error correction and those won’t link up through this switch by default. The fix is easy, however:
sudo config interface fec Ethernet20 rs
This should allow the connectx-4 clients to link up on one of the 100gbe ports.
vtysh
This command will give you a shell that is less like a linux shell and more like a traditional switch shell. Understand that the interface numbering can be configured to be more linux-like OR more traditional switch like. Much of the documentation you might encounter online can be using one or the other convention and it might seem mysterious that this thing you’ve searched for doesn’t work… if you understand the ports can be referred to by different names, depending on the mode of the switch, then it makes a little more sense.
I Just Want A Dumb Switch That Goes Fast
Okay, I gotchu fam.
# take ports out of their default routing mode
sudo config interface ip remove Ethernet0 10.0.0.0/31
sudo config interface ip remove Ethernet1 10.0.0.2/31
sudo config interface ip remove Ethernet2 10.0.0.4/31
sudo config interface ip remove Ethernet3 10.0.0.6/31
sudo config interface ip remove Ethernet4 10.0.0.8/31
sudo config interface ip remove Ethernet5 10.0.0.10/31
sudo config interface ip remove Ethernet6 10.0.0.12/31
sudo config interface ip remove Ethernet7 10.0.0.14/31
sudo config interface ip remove Ethernet8 10.0.0.16/31
sudo config interface ip remove Ethernet9 10.0.0.18/31
sudo config interface ip remove Ethernet10 10.0.0.20/31
sudo config interface ip remove Ethernet11 10.0.0.22/31
# now the 100gbe ports. note you can run these as 4x 25gbe ports instead if you want... thats why the gaps in the numbers
sudo config interface ip remove Ethernet12 10.0.0.24/31
sudo config interface ip remove Ethernet16 10.0.0.26/31
sudo config interface ip remove Ethernet20 10.0.0.28/31
# create aribtrary vlan 10, then trunk all the ports to it
sudo config vlan add 10
sudo config vlan member add -u 10 Ethernet0
sudo config vlan member add -u 10 Ethernet0
sudo config vlan member add -u 10 Ethernet1
sudo config vlan member add -u 10 Ethernet2
sudo config vlan member add -u 10 Ethernet3
sudo config vlan member add -u 10 Ethernet4
sudo config vlan member add -u 10 Ethernet5
sudo config vlan member add -u 10 Ethernet6
sudo config vlan member add -u 10 Ethernet7
sudo config vlan member add -u 10 Ethernet8
sudo config vlan member add -u 10 Ethernet9
sudo config vlan member add -u 10 Ethernet10
sudo config vlan member add -u 10 Ethernet11
sudo config vlan member add -u 10 Ethernet12
sudo config vlan member add -u 10 Ethernet16
sudo config vlan member add -u 10 Ethernet20
… and now we have a normal switch! Save that config.
sudo config save -y
Note also the default MTU here is 9100 and you may want to make other changes to this to suit your setup.
Other handy commands:
show ip interfaces
Interface Master IPv4 address/mask Admin/Oper BGP Neighbor Neighbor IP
----------- -------- ------------------- ------------ -------------- -------------
Ethernet3 10.0.0.6/31 up/down ARISTA04T2 10.0.0.7
Ethernet4 10.0.0.8/31 up/down ARISTA05T2 10.0.0.9
Ethernet5 10.0.0.10/31 up/down ARISTA06T2 10.0.0.11
Ethernet6 10.0.0.12/31 up/down ARISTA07T2 10.0.0.13
Ethernet7 10.0.0.14/31 up/down ARISTA01T0 10.0.0.15
Ethernet8 10.0.0.16/31 up/down ARISTA02T0 10.0.0.17
Ethernet9 10.0.0.18/31 up/down ARISTA03T0 10.0.0.19
Ethernet10 10.0.0.20/31 up/down ARISTA04T0 10.0.0.21
Ethernet11 10.0.0.22/31 up/down ARISTA05T0 10.0.0.23
Loopback0 10.1.0.1/32 up/up N/A N/A
docker0 240.127.1.1/24 up/down N/A N/A
lo 127.0.0.1/16 up/up N/A N/A
This thing has docker?!? How do you think it does what it does 
?
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e52bd5c08eef docker-snmp:latest "/usr/local/bin/supe…" 13 hours ago Up 13 hours snmp
831c00b6686e docker-sonic-mgmt-framework:latest "/usr/local/bin/supe…" 13 hours ago Up 13 hours mgmt-framework
6564f7dd311c docker-lldp:latest "/usr/bin/docker-lld…" 13 hours ago Up 13 hours lldp
4c880b36dcb6 docker-platform-monitor:latest "/usr/bin/docker_ini…" 13 hours ago Up 13 hours pmon
2a8ed2f4c57f docker-fpm-frr:latest "/usr/bin/docker_ini…" 13 hours ago Up 13 hours bgp
70e6a4e94141 docker-router-advertiser:latest "/usr/bin/docker-ini…" 13 hours ago Up 13 hours radv
82f2cc7c075b docker-syncd-brcm:latest "/usr/local/bin/supe…" 13 hours ago Up 13 hours syncd
4b452f7f5101 docker-teamd:latest "/usr/local/bin/supe…" 13 hours ago Up 13 hours teamd
e5a550a022e8 docker-orchagent:latest "/usr/bin/docker-ini…" 13 hours ago Up 13 hours swss
d5f32ec18136 docker-eventd:latest "/usr/local/bin/supe…" 13 hours ago Up 13 hours eventd
6fc6f47580ef docker-database:latest "/usr/local/bin/dock…" 13 hours ago Up 13 hours database
… that can also be a source of things that need troubleshooting, though.
Fin
It’s nice to enjoy 25 gigabit from a gaggle of windows computers with basically zero headache. And the linux-based file server can easily saturate 100gbe if enough clients are busy. Since it’s built with a giant mountant of optane!!!11!! Soon ™
.