Confessions of a 10 Gigabit Networking NEWB (Mikrotik CRS226 Review)

Networking Networking Hardware
1

Hello Tek Syndicate!
After Wedell's video review of the Netgear XS712T:

@wendell please bring us the next part of that series!!!!!!!

My hunger for more network bandwidth was reignited. I've been interested in deploying 10G at home for a while now, but until this point the cost was just too absurd, especially because I don't really "need" it.

Now, let me explain how I saved a boatload of money and got 2 10 Gig clients on my network.
Mikrotik, a small European company basted in Latvia, makes a line of very powerful (specifically in terms of Software) Routers and Switches running an operating system based on Linux they call RouterOS.

In the past couple years they have released several Switches in their Cloud Router Switch (CRS) series that pack two SFP+ 10 GIG cages. What does that mean? 10 GIG fiber for cheap. Currently they offer 3 versions of these switches, an 8 port, and 2 24 port models, one is a desktop model and one is a rackmount one.

  • The 8 port one, the CRS210-8G-2S+IN sells for around $180 bucks on Amazon in the US.
  • The 24 port desktop model sells for around $245.
  • The Rackmount version sells for around 20 bucks more.

For those of you looking to dig a bit deeper, theres a pretty good review of the 24 port desktop model Mikrotik CRS226-24G-2S+IN, on ServeTheHome. it includes a breakdown of the hardware, and an explaination of the nifty LCD screen.

Now; let me start with my review, I bought the rack mount version of the switch and so far I love it. My first impressions of the switch were actually pretty bad.

  • The build quality of the case is kinda crappy metal
  • It's actually very light, much lighter than my old Cisco SG300 10 port switch.
  • Heatsinking is okay, but could be better
  • Although it has a fan mount on the back of the case, it lacks both a fan and a fan header.
  • It does NOT support LACP or Spanning Tree

But there is much good!

  • With all the compaints above, temps have been fine, the CPU temp has not been reported to go over 40 degrees when testing. I don't know how it will perform when it is really loaded up, however. I've only done testing with 4 gig clients (laptops) and 2 10gig clients simultaneously.
  • The board PCB itself looks pretty good
  • The design asethetic is very nice, especially with the LCD
  • Performance is excellent.
  • RouterOS

Here is a picture. Please keep in mind I need to re-do the wiring xD.

IMG_0114.JPG2448x3264 370 KB

Now with all those caveats in mind, let me explain the reason I think that this switch is the perfect switch for the home user looking for more bandwidth. I live in a house with 5 other people, and according to some network statistics, around 95% of the network utilization (inside and out to the internet) comes from my server and my main computer. All major file transfers (like multi TB ones) are handled here as well.
So, when I was weighing the option of buying a $1,200 10G copper switch, or a >$300 24 port Gig switch with 2 10 Gig SFP+ ports, I think the answer was obvious. I would imagine that for most of you looking to roll out 10G in your home, it will be as well, and this is why I made this thread.

I managed to get 2 Mellanox ConnectX2 10Gig NICS on ebay for $35. I DONT KNOW IF THESE WILL PLAY NICE WITH FREENAS OR NOT and then got some pretty cheap Direct Attach copper SFP+ cables form Fiberstore. But for less than $400 bucks I have my server and my main workstation/gaming rig connected at 10Gig.

For your amusement I have prepared some screenshots of the performance in action of this switch. This is basically out of the box perfromance, the only settings I changed on the switch was to change its IP, give it a DNS server, and update it.

Here are my iPerf Numbers:

iPerf.png955x695 49.7 KB

And a picture of the switch's fancy LCD while running iPerf (not much else going on on the network):

IMG_0113.JPG2448x3264 241 KB

Here are some comparisons of my 3-disk RAID5 WD Red 3TB array.

Native, ran Crystal Disk Mark on my server:

Native.png675x670 496 KB

The same raid array via my main computer, mounted as a CIFS share over 1G.

RAID 1g.png803x695 399 KB

The same raid array via my main computer, mounted as a CIFS share over 10G.

RAID 10g.png803x695 440 KB

Now here are some done with a Samsung 850 PRO 256 on my server, mounted via CIFS on my main computer:
Over 1G

ssd 1g.png803x695 397 KB

Over 10G

ssd 10g.png803x695 400 KB

Now what do I plan to do with all of these new found bandwidth?
The goal is to expand my RAID array to increase performance, and to backup to an external either Thunderbolt or USB 3.1 array attached to my main computer.

7 Likes
2

Nice, I'd really like to see a 4 sfp+ version of their 24 port switch.

2 Likes
3

Mr Kane,
While I agree that would totally be awesome, having room for expansion and what not, I don't think it's really needed for most people on this forum.

Nick

4

Nice setup and a good way to getting a limited number of 10G links for relatively reasonable money.

1 Like
6

Thanks weibo! I hope this helps others who want to do the same

7

Nice setup man.. wish I could do the same.. but where I live is going to be a couple of years or less (if I am lucky) to get those Switch...
Cheers :D

8

Yeah, I just mean that I'd really like one with 4 sfp+ ports as right now I have two 10gb machines wired directly to each other so it wouldn't really change anything for me to add a two port switch. I'll be keeping am eye out though, hopefully the next generation might have more ports.

One question I have is, is it possible to have the 10gb ports on a separate VLAN using jumbo frames and have the switch route the traffic to the rest of the network using regular frames? And if so I wonder what the performance would be like.

1 Like
9

Yeah, I agree that if you already have a good switch it might not be ideal, because you can do what you've done.

To answer your question;
Yes it is possible because it has a pretty feature rich RouterOS V6 license, however because it will be done in software and not in the switch chip it will end up being much slower because the CPU can't handle much routing.
You would need one of their Cloud Core Routers (CCR) in order to do that at wire speed, but then we are talking about alot more money...

But besides, I'm getting 9 Gigs on iperf so I'm not really worried about it.

10

9 gigs is good, I was getting around 3 until I set the MTU to 4000 (the highest the card would support) and now I get between 9 and 10gbps. But that card is junk and I'm replacing it soon.

I have one desktop that I wouldn't mind having on a 10gb network but it's not really worth spending too much money on without also reconfiguring my storage for speed.

11

I am just curious; what cards are you using now and what are you planning on swapping them to?

12

I got a chelsio s310-cr to replace this one and the other is a sun dual port XFP card which seems pretty decent but I don't remember the model.

EDIT: Sorry, the one I'm replacing is a netxen NXB-10GXSR XFP card

13

Great write up! Wish I would have stumbled on this post earlier haha.

I have a similar upgrade for my small home network in the works with plans to use the MikroTik crs226 as the primary switch. Been playing around with 2 x intel x520-da2 adapters (one in my main PC and the other in my big file server) testing direct connection transfer speeds via 5m DA SFP+ twinax. Using a PCIe OCZ RevoDrive SSD on the file server as a write cache to reduce the write speed bottleneck some.

My question is how has this system (specifically the MikroTik crs226) been operating for you? Any major issues that have come up thus far? I'm close to pulling the trigger on the crs226 but am torn between it and a few other options like the Netgear s3300.

Like your setup, I only really need the 10Gbe between my main PC and the file server and the SFP+ DA twinax route is cost effective enough for me to be worth it. Would love to hear any feedback good or bad you may have after running this setup for a few months now.

14

I wonder what the easier method of running a line down the entire length of my home would be

Ideally I'd like to go from my bedroom down my bedroom wall through the wall down the dining room wall through that wall to the outside and then down the length of the house into the back house across the wall 90 degree turn across the wall down four feet and into a computer

Currently I'm making that run with regular old Ethernet cable and I can hit about 70/80 megabytes per second over gigabit

15

Nice setup that you have!! I was wondering, what kind of rack shelving are you using if you don't mind me asking?

16

`Hello Reed,
Thank you!
The CRS226 has been great for me, and since it was deployed in October it has shown zero downtime except when I either unplugged it or did a software upgrade. I did a bunch of reconfiguring on my network (moved back to FreeNAS, upgraded my router/UTM), and the switch has been great. If your usecase is like mine and you just need a fast an reliable pipe between your two systems as well as a nice switch for everything else; you cannot go wrong with the Mikrotik. I think you would be crazy to spend almost double on that Netgear. Unless your network is constantly hammered with more than 3/4 of this switches capacity (like in a data center or as a core switch in a big office building) you simply can't get a better deal right now.

17

Allent,
It is a lackrack, 3 Ikea RACK tables cobbled together.
https://wiki.eth0.nl/index.php/LackRack

1 Like
18

Thank you for the reply! and thank you for the link as well! I'm really diggin the design alot!

19

No problem.

Be sure to post back if you end up building one!!

20

Great to hear. Thanks so much for sharing your experience, this is definitely the route I'm going.

You've had no issues with your DAC cables compatibility/signal at all thus far?