High End Switches
… do not have a web gui. Period. Mostly. And, in fact, most of these are designed for ‘zero touch’ deployment. That means you’ve bought ‘something else’ that actually configures these switches. And you have a lot of them.
Most of the documentation/guides are around using this high-end gear ‘at scale’ but actually a lot can be learned working on and configuring just one.
This is the switch we’ve used with content about our Gigabyte and Tyan servers as well; vlans, software-defined storage solutions, mellanox exploration, etc.
If you watch a switch like the S5212F-ON boot, it seems like they are a computer – UEFI setup options, press X to reconfigure, etc. But in reality the “PC” is just a front-end for the control plane.
TODO
Our test switch here for the write-up is the Dell S5212F-ON – it is a 2019 model designed to fit in a 1u half/width rack shelf. (For redundancy you usually do two of these at a time in 1u).
It has 12 25gbe SFP28 ports and 3 100gbe QSFP ports.
Out of the box these switches appear to be dead because there is no OS on it. Remember, though, the OS doesn’t really do much heavy lifting – instead it interfaces the control plane and custom silicon to decide if you want this to be a layer 2 switch, layer 3 switch, packet filtering, vlans, etc. Because of scalability and edge providers and the trends of the industry, there are plenty of non-dell OSes you can run on these which gives one control way, way up the chain.
Think software defined networking, big vmware cluster, lego-brick like components. If you’re buying these by the dozen you have zero interest to sit down, log in locally, and explore the capabilities.
For our setup we are using a simple Cisco-pinout RJ45 to DB9 to USB RS232 serial adapter. We are using Linux for the configuration computer, and Minicom for the terminal program (apt install minicom or dnf install minicom). I also recommend to use a fat32-formatted USB drive to do the initial OS load (though it can be done via TFTP too… again… meant for volume/scale customers).
Installing the OS
Connect the console cable; fire up minicom and you should start to see text. Out of the box the S5212F-ON is in a kind of infinite loop looking for configuration and software. You’ll see a grub menu! We have some options – install OS, rescue, remove OS, etc.
You can use the “ONIE Install OS” option but having done a few of these now, rescue is slightly different and easier. Pick that then hit enter after a minute or two, and you should have a shell.
By the way, the OS10 for these is not freely downloadable (!!). It can be a perpetual license, or a subscription license. “OS10 Enterprise” from Dell is what I would consider the base package on these. You have to register your service tag and download the image from a protected area of the Dell website. It is based on Debian, though, which is nice.
Debian GNU/Linux 9 OS10 ttyS0
Dell EMC Networking Operating System (OS10)
OS10 login: admin
Password:
Last login: Sat Nov 13 09:40:00 UTC 2021 on ttyS0
Linux OS10 4.9.168 #1 SMP Debian 4.9.168-1+deb9u3 x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
-* Dell EMC Network Operating System (OS10) *-
-* *-
-* Copyright (c) 1999-2019 by Dell Inc. All Rights Reserved. *-
-* *-
-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-
This product is protected by U.S. and international copyright and
intellectual property laws. Dell EMC and the Dell EMC logo are
trademarks of Dell Inc. in the United States and/or other
jurisdictions. All other marks and names mentioned herein may be
trademarks of their respective companies.
OS10#
If you see t his, an OS has already been loaded and you need to reboot and either uninstall OS10 or go into rescue mode.
Aside: These also have an “A” and “B” slot for OS to make it easy to roll between versions and to have an undo path, which is nice.
TODO
Instead, you should see something like this.
Extract the PKGS_OS10_something.bin file to the USB3 fat drive and plug it in to the front of the switch.
execute
mount /dev/sdb1 /mnt/usb
and
ls /mnt/usb
and you should see your OS image. In this mode you are in a shell, basically, but understand once the OS is loaded you’ll be in a different operating environment when using minicom. The OS10 prompt is not a shell-based prompt like this one is.
TODO
Handy Commands for Common Errors and Problems
Help, my 1gbps GBIC is not working in the SFP28 ports!
If it sees the gbic, but won’t get link, there is probably something you’re unaware of.
The ‘secret’? Look at those 100gbps ports. They’re actually 4 channels. And the 12 ports on this thing that are SFP/28? In hardware, those seem to actually be 3 100gbps ports broken up into 4x25gb ports, probably for cost reasons. So it doesn’t seem to work in most scenarios to have 10gb/25gb/25gb/1gb on port 1/2/3/4. But you can change a port group to 10gb/1gb mode, and then it works.
Use the show inventory command to show what modules you have. show inventory
OS10# show inventory
Product : S5212F-ON
Description : S5212F-ON 12x25GbE SFP28, 3x100GbE QSFP28 Interface Module
Software version : 10.5.0.1P1
Product Base :
Product Serial Number :
Product Part Number :
Unit Type Part Number Rev Piece Part ID Svc Tag Exprs Svc Code
-------------------------------------------------------------------------------------------------
* 1 S5212F-ON 0GKK8W A05 TH-xxxxxx-CETxx-04x-00NR AAABBBB 123 345 456 23
1 S5212F-ON-PWR-2-AC-R NA NA NA
1 S5212F-ON-FANTRAY-1-R NA NA NA
1 S5212F-ON-FANTRAY-2-R NA NA NA
1 S5212F-ON-FANTRAY-3-R NA NA NA
1 S5212F-ON-FANTRAY-4-R NA NA NA
and show inventory media
OS10# show inventory media
-----------------------------------------------------------------------------------------------------------------
System Inventory Media
-----------------------------------------------------------------------------------------------------------------
Node/Slot/Port Form-Factor Media Serial-Number Qualified
-----------------------------------------------------------------------------------------------------------------
1/1/1 SFP28 SFP28 25GBASE-CR-3.0M CN07720681A0L5C true
1/1/2 SFP SFP 1000BASE-T P0DC7CP true
1/1/3 Not Present
1/1/4 Not Present
1/1/5 Not Present
1/1/6 Not Present
1/1/7 Not Present
1/1/8 Not Present
1/1/9 Not Present
1/1/10 Not Present
1/1/11 Not Present
1/1/12 Not Present
1/1/13 Not Present
1/1/14 Not Present
1/1/15 QSFP28 QSFP28 100GBASE-CWDM4 17190079 false
Why do we guess that internally it’s actually 3 100gbps ports broken into 12 25gbps ports?
Port-group Mode Ports FEM
port-group1/1/1 Eth 25g-4x 1 2 3 4 -
port-group1/1/2 Eth 10g-4x 5 6 7 8 -
port-group1/1/3 Eth 25g-4x 9 10 11 12 -
port-group1/1/4 Eth 100g-1x 13 -
port-group1/1/5 Eth 100g-1x 14 -
port-group1/1/6 Eth 100g-1x 15 -
the show port-group command shows us. This might not look the same as yours.
I have configured port group 1/1/2 which is labeled ports 5/6/7/8 to operate at 4x10g instead of 4x25g. In so doing now these ports will work with 10g and even 1g media converters. Otherwise they cannot establish link.
How do you reconfigure the port group for 1gbps/10gbps gbic operation?
OS10# configure terminal # enter config mode
OS10(config)# port-group 1/1/2 # select port group for the ports you want
OS10(conf-pg-1/1/2)# mode Eth 10g-4x # reconfigure ports for 10g operation
Once you do that, the 10gbps or 1gbps gbic should work and get link.
You’ll also notice that other commands under configure like interface ethernet now have some different options. e.g. here is a show command you can use to inspect port 6, now in this new mode, for example:
OS10(config)# show interface ethernet 1/1/6:1
Ethernet 1/1/6:1 is up, line protocol is up
Hardware is Eth, address is 1c:72:1d:bf:93:86
Current address is 1c:72:1d:bf:93:86
Pluggable media present, SFP+ type is SFP+ 10GBASE-SR
Wavelength is 850
Receive power reading is no power
Interface index is 28
Internet address is not set
Mode of IPv4 Address Assignment: not set
Interface IPv6 oper status: Disabled
MTU 1532 bytes, IP MTU 1500 bytes
LineSpeed 10G, Auto-Negotiation off
Flowcontrol rx off tx off
ARP type: ARPA, ARP Timeout: 60
Last clearing of "show interface" counters: 13:36:41
Queuing strategy: fifo
Input statistics:
222419 packets, 132520444 octets
11009 64-byte pkts, 87709 over 64-byte pkts, 25245 over 127-byte pkts
14531 over 255-byte pkts, 8182 over 511-byte pkts, 75743 over 1023-byte pkts
850 Multicasts, 49 Broadcasts, 221520 Unicasts
0 runts, 0 giants, 0 throttles
0 CRC, 0 overrun, 0 discarded
Notice the gbic is 10gbps so the SFP28 to gbic speed is 10gbps but then the line speed on this particular adapter is just a paltry 1gbps.
If you used the speed command before your only option was 25000 (or auto). But now new options:
S10(config)# interface ethernet 1/1/6:1
OS10(conf-if-eth1/1/6:1)# speed
1000 10000 auto
These switches work well for their relative low cost (~$1k street) for high-speed interconnects to support technologies like VMware VSAN and xcp-ng XO-SAN (v2?). Dual 25gbe connections in each node in a cluster to a pair of these S2512F-ON switches makes an incredible low-cost platform that has mind-bending performance vs other implementations around the same price point.
Videos soon, I hope.
Save Configuration
Did you mess up? Reboot, the configuration is not saved by default. If you want to save your config:
copy running-configuration startup-configuration
If you want to show your running config
show running-configuration
OS10# show port-channel summary
Flags: D - Down I - member up but inactive P - member up and active
U - Up (port-channel) F - Fallback Activated
--------------------------------------------------------------------------------
Group Port-Channel Type Protocol Member Ports
--------------------------------------------------------------------------------
OS10# show port-group
Port-group Mode Ports FEM
port-group1/1/1 Eth 25g-4x 1 2 3 4 -
port-group1/1/2 Eth 10g-4x 5 6 7 8 -
port-group1/1/3 Eth 25g-4x 9 10 11 12 -
port-group1/1/4 Eth 100g-1x 13 -
port-group1/1/5 Eth 100g-1x 14 -
port-group1/1/6 Eth 100g-1x 15 -