Preparing for 5Gb internet

Networking Networking Hardware
1

I’m getting 5Gb internet from ATT installed today. I’ll be getting probably a BGW320 which has a 5Gb lan port and 3 1Gb lan ports.

I don’t have anything better than 1Gb on all my devices (my main desktop has 2.5 though) so I need to upgrade everything.

It seems to me that upgrading everything to 10Gb is the better move. In looking for equipment, I found the Ubiquiti Flex XG switch which has 4 10Gb ports. I’ll probably order that if theres nothing better for around the same price.

I have a server rack where my 4u nas sits running proxmox. This is in the same room as my desktop PC and work laptop, so keeping noise down is extremely important. I considered getting the Mikrotik CRS312-4C+8XG-RM but apparently that is exceptionally noisy.

I also need 10Gb nics. Im considering this Intel X540 from “10gtek” but the amazon reviews seem to say that its a hit or miss and may get a “non-genuine intel” part, so I’m very hesitant there. 10Gb rj45 nics seem to generally be impossible to find good ones for.

Any ideas here?

2 Likes
2

First of all I want to point out that I envy you. Second of all the Intel X540 does not feature NBASE-T support as far as I remember. I think it only does 1G, 10G, not the 2.5G and 5G in between. You might look into a used X550. I have one of those and they support NBASE-T speeds. If you only want to connect to a 10G switch the 540 should be OK too, but if you want to connect to the router directly you might need a X550. I am not 100% sure, though.

1 Like
3

I’m following this post because I’m also interested in recommendations others have, since I’m in the same situation as you are. I currently just have 1Gbps service from AT&T but I have both 2 and 5 available if I want to pay more per month, but like you I’d have to upgrade a bunch of gigabit equipment if I really want to realize the speed difference. For right now I figured it probably wasn’t worth it, but when the time comes in the future I’m interested in equipment recommendations too.

4

If you have all in the same room, a switch with sfp+ ports would probably be much quieter…

These cards go for 50usd

You could use direct attach cables and save the money for transceivers, you would also be using way less power…

For the switch, you could go for a mikotik CRS305 for the bare minimum, and an 10baset transceiver to connect to your ont. The crs305 is fanless
If you want room for growth and have more budget, the QSW-M1204-4C-US or one of its variants may fit your use case … It is definitely going to be quieter than the mikrotiks with fans in switch mode…

5

I’m sure the op has already picked up a router by now. What did you get?

But just for anyone who finds this post and want to consider the
MikroTik (CRS312-4C+8XG-RM) 12-Port 10G Switch with Dual Power Supplies which has 12 10G RJ45 Ethernet ports and 4 10G SFP+ ports (4 ports are combo).

You can set it up with either SwOS (switch) or RouterOS (Layer 3 features) as primary but can switch back and forth.

It’s loaded with features including 2.5G & 5G support but you need to learn to speak “Mikrotik” to make use of advanced router functionality. It’s not hard once it clicks and grows on you as it’s quite powerful.

Best of all no transceivers needed and will likely work just fine at 10Gb with your existing Cat 5 or better wiring already in place.

6

It will ‘work’ but it will struggle to route at 10Gbps, definitely will not be able to packet filter at these speeds.
Also, beware of applying processing operators to the ports that are normally handles to the dedicated bridging ASIC … they will be sent to the CPU and performance will tank …

7

I decided not to go that route as its a rather loud unit. My server rack sits directly next to me so noise is important (or lack thereof)

So, onto what I got and the issues I’m running into:

2x SFP+ cables from “10Gtek”

2x Mellanox ConnectX-3 (from ebay)

1x Ipolex RJ45 transceiver (for the 5Gb connection from the modem)

1x CRS305-1G-4S+IN Mikrotik switch

Connected all of this up and ran a speed test. I can get the full 5Gb download speed, but the upload speed is abysmal at less than 1Gb. Testing without all this, I connected that 5Gb link directly to my motherboards 2.5Gb port, I can get full 2.5Gb up and down.

I thought maybe this was the switch or maybe the cards/cables so I ran iperf between my nas and my desktop, and saw much better speeds:

Connecting to host 192.168.1.79, port 5201
[  4] local 192.168.1.216 port 56866 connected to 192.168.1.79 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.07 GBytes  9.19 Gbits/sec
[  4]   1.00-2.00   sec  1.07 GBytes  9.16 Gbits/sec
[  4]   2.00-3.00   sec  1.06 GBytes  9.12 Gbits/sec
[  4]   3.00-4.00   sec  1.06 GBytes  9.14 Gbits/sec
[  4]   4.00-5.00   sec  1.04 GBytes  8.91 Gbits/sec
[  4]   5.00-6.00   sec  1.01 GBytes  8.68 Gbits/sec
[  4]   6.00-7.00   sec  1.05 GBytes  9.00 Gbits/sec
[  4]   7.00-8.00   sec  1.07 GBytes  9.17 Gbits/sec
[  4]   8.00-9.00   sec  1.05 GBytes  8.99 Gbits/sec
[  4]   9.00-10.00  sec  1.05 GBytes  9.03 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  10.5 GBytes  9.04 Gbits/sec                  sender
[  4]   0.00-10.00  sec  10.5 GBytes  9.04 Gbits/sec                  receiver

iperf Done.

There is no ip filtering on the switch, it is literally just a dumb switch at the moment, nothing is active on it. Looking at the ATT modem’s interface, I can see that is says that 5Gb connection is connected at 5Gb. However, in the switch’s software (winbox), it says it’s connected at 10Gb. The thinking here then is that it isn’t negotiating 5Gb properly. There is no way to “hard code” the interface to 5Gb. So, maybe a problem with the transceiver? I’ve just ordered a MikroTik S+RJ10 so we’ll see how that fairs. Edit: came in, better but not great. upload speed up to ~1700Mbps. nowhere near where it should be

I suspect this kind of setup might become more common as people try to take advantage of 5Gb internet? Am I maybe missing something in my setup that might perform better? Any other thoughts people might have?

8

The transceiver interface to the switch is going to always be 10Gbit when using SFP+, it is then up to the transceiver chip to negotiate a speed on the rj45 side, so you should be good with what you see …
Another transceiver will definitely give you a better idea on where to look for the performance drop… Just make sure the speed test data is reliable …

9

To test the transceiver, I moved some cables around.

Desktop is now connected to the switch using the 2.5Gb interface on my board and going into the rj45 transceiver. Nas is still connected to the switch using 10Gb sfp+.

Running iperf as a server on the nas, I saw 2.5Gbps from my desktop. Running iperf as a server on my desktop, I saw 2.5Gbps from the nas. So clearly the transceiver is capable of 2.5Gbps at the very least without much trouble.

So then why is the internet speed so much worse? If I connect my modem directly to my 2.5Gb interface on my desktop, I can see 2.5Gbps up and down. But when the modem is connected to the rj45 transceiver, it struggles to reach gigabit. Sometimes it goes above gigabit, most times not. It should be getting 5Gb.

10

Transceivers can be fickle. Does the modem manufacture have a recommended product to use?

11

I would assume connecting their modem to an rj45 transceiver isn’t exactly a supported use case. Its a BGW320-500

12

Ok, enabling flow control on the mikrotik switch, specifically on the port going to the modem, my upload is much faster:

750×400 31.8 KB

This seems weird, right? This doesn’t seem like the right solution

1 Like
13

I am having this same issue on my Ubiquiti equipment. Many people seem to have bad upload speeds when dealing with above 1gbit speed internet. The answer to most with Ubiquiti hardware too is to enable Flow Control. Not sure why that fixes it for most people. Sadly for me it did not :confused: