Mikrotik will sell you the boards directly for projects like this.
2 Likes
I mean you could get a bunch of 4 port nics, set each nic to be a plan, and have like 6-8 in a box. Ta da?
Vlan
Up to 8 10GbE and around 20 1GbE which I realize is probably unrealistic but I still want to know what is realistic and what the most performing option is.
Of course I could. The question is about the performance of doing that.
I was looking at these or just buying a whole mikrotik switch and trying to put openbsd on it, but the switches have around 32/64MB of ram. Even with openbsd, idk if that’s usable. Other than that though, switch hardware with an OS on it is basically my ideal situation.
1 Like
This sounds doable in cpu depending on packet size.
For 8x10GbE , you can get a single dual port qsfp+ pcie card and get a pair of 1-4 breakout cables.
For the 20x1GbE … I don’t know, I’d get a $150 mikrotik switch that has 24x1gbps + a pair of 10gbps uplinks, and connect the uplinks to the Linux box. Then configure mikrotik to put each port in their own VLAN.
1 Like
Have you looked at a used Mellanox switch? They have a custom linux distro that runs the switch.
1 Like
I have not, but that sounds good. I’ll take a look.
They are not as cost effective as used Arista because of the lower adoption rate, so you wont get amazing deals. BUT with a little leg work you can find one with Cumulus Linux
1 Like
Yeah, Cumulus Linux sounds good, but I cannot figure out how the pricing works.
For instance, if I’m looking at switches on Supermicro’s site, it tells me that the switch is cumulus compatible but to contact Cumulus Networks for ordering information.
https://www.supermicro.com/en/products/accessories/Networking/SSE-G3648B.php
And then on the Cumulus site, it says “try for free” but I can’t find any pricing info anywhere.
Ok, so Cumulus Linux is proprietary. A license is perpetual, but software updates require a support contract. Also, of course, owned by Nvidia… 
Perpetual Licenses: General Questions (Current Licensing Model) – Cumulus Networks® Knowledge Base?
Still couldn’t find the license cost but usually when it’s this secret, it’s not pretty… I’ll ask just for kicks though.
Lol, yikes
https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2380057.m570.l1313&_nkw=mellanox+cumulus&_sacat=0
Ok, so there you go:
Also TIL nvidia is selling switches under its own brand name now.
4 Likes
First thing they did after the Mellanox acquisition was rebrand everything to nVidia Networking. Hopefully it won’t affect things too much, Mellanox was rather friendly (or at the very least, not hostile) to homelabbers, at least in my experience.
1 Like
Apparently there are Cumulus competitors that will load on ONIE switches, but none that are actually FOSS.
PicOS
And there’s something called Switchlight, that appears to have been acquired by Arista. So not sure if that still exists or was rebranded or what.
Neither are listed in distrowatch so not really on the radar.
Oh here we go…
Well, there is this…
https://opennetlinux.org/index.html
Which does appear to be everything I’m looking for (FOSS, ONIE). Only red flag is no new release since July 2019. No activity on the site at all since 2019 from what I can tell. The github isn’t dead though…
There are a lot of the edge-core switches on ebay for cheeeeeap and they are ONL certified. I just hope it’s not abandonware…
Ehh, expired certificate, barely any commit activity, mailinglist doesn’t give off great vibes either…
Did find a mention of SONiC through the ML though, which is Linux based, pushed by Microsoft, and Open Source. So might be some more life in that one…
Didn’t dig too deep, so no idea if there’s any hardware options that don’t cost a kidney…
1 Like
Oh yeah, SONiC is also backed by OCP so maybe it’s replacing ONL? While all the info on ONL was old, info on SONiC looks like it’s too new. There are a lot of descriptions but nothing about how to actually use it. No mention of ONIE anywhere, and it runs several docker containers? Is it a controller? The other network project under OCP is SAI which looks like a set of API definitions for managing switches.
It almost looks like SONiC is a collection of containers that manages ONL switches via SAI, but then why would ONL be dead and there be mention of specific switch hardware in SONiC? It’s really difficult to get a birds eye view of these OCP projects.
Oh hey, nvm. The github wiki is actually fleshed out vs the wiki they link to in on the OCP page.
1 Like
Tbh, I might just pick up one of these and see how this stuff really works…
Edge-core lists ONL on the switch spec sheet (idk what’s up with the sketchy IP address link), but they have a whole section on SONiC which reinforces my impression that SONiC is replacing ONL.
I am going to reach out to Edge-core and confirm SONiC support on their ONIE switches (at least the current-gen ones). If they confirm, I’ll buy one. Looks like DCs are dumping a lot of these into used markets based on the eBay listings.
SONiC is mentioned as “future support” in this video from last October…
Also, lol at the “feed the hunger” translation.
Is anyone else concerned that datacenter switches are running their own docker containers?
New music genre for emo kids?
6 Likes
You’d think so right? Somehow Taiwanese tech beat them to it.
3 Likes
Changed the title of this topic to better suit the direction it’s taken.
2 Likes