Direct connection between 2 PCs (in addition to the router)

My mobo has 2 built in NICs, one of which is 2.5G.

I have a NAS I built from old parts. Today I got a cheap 2.5G nic for it, since it seemed a dead simple upgrade to speed up transfers between my PC and NAS.

Making the connection breaks my network. This is not unexpected, but I’d like help in configuring things to work.

My NAS is on linux, my computer is on Windows 10.

It would also be cool to explain what we’re doing, instead of just giving me commands, so I can learn a bit of networking in the process.

For example, I know loops are bad, but am I creating a loop in this instance?

What does your IP assignment look like on the adapters that are directly connected?

I have done something similar with a unmanaged switch where the computers all 192.168.100.X and no gateway IP, while my main PC has an additional adapter that is 192.168.0.X with gateway to the router.

Can you also explain how the network is breaking? Is your internet access gone or the NAS Drive

1 Like

I’m starting to understand things.

First I let it auto, the I set unused IPs to the same subnet as my router. That’s a mistake.
On third try, I chose a different subnet.

My router network is
192.168.0.1 / 255.255.255.0
Computer: 192.168.0.2
NAS: 192.168.0.5

I tried putting the 2.5G nics on
Computer: 192.168.0.3 / 255.255.255.0
NAS: 192.168.0.4 / 255.255.255.0

That broke my LAN. I had internet, but my shares no longer worked.

I tried setting 192.168.0.3(4) / 255.255.255.255
Still no worky.

What worked is choosing a different subnet - e.g. 192.168.10.2 / 255.255.255.0
(Which does make sense, when I think about it. The default gateway for 192.168.0.x / 255.255.255.0 is 192.168.0.1.

So 192.168.0.3(4) would be unreachable from there.

Now I’m curious if there is a way to make it work on the router subnet?
Like an explicit route on both sides? But I’m not sure what it would be.
On windows it’s:
route add destination_network MASK subnet_mask gateway_ip metric_cost


My other problem is, I’m only getting 1G speeds, even tho the Link speed is advertised as 2.5G (on both sides).

Trouble is, even if I map the drive using the 2.5G IP, I see no activity when using it!
Traffic is being set exclusively thru the 1G connection.
I also can’t get iperf to work thru that IP…

It’s not very clear what your NIC situation is, let’s clarify that first.

You mention using /32 and /24 subnets, and if I understood correctly trying to use 2 IPs in /32 (255.255.255.255) subnet which is not possible.

I’d start with configuring 2.5G NICs to a different subnet, so not the one on your router, than pulling 1G cables and testing the speed. This way your only possible link is 2.5G and you can begin to isolate the problem.

3 Likes

Maybe I didn’t present it in the best way, but I did give all relevant information above already.

Right now I am playing around with the routes.

What exactly am I isolating here? Once again, I have already confirmed traffic NOT going thru the 2.5G nic. It’s super easy to confirm because I can just open the connection status on windows, look at the packets sent/received and see them NOT moving when I copy a file.

So once again, what do you want me to isolate?

1 Like

Ah I was under the impression that the NAS was isolated and had a single 2.5G link to the PC. I now see it also has a 1G connection to the router (is this correct?)

We want to know if 2.5G works when there is no other NICs active. It is very unlikley for 2.5G NICs but if there is no Auto-MDI/X it may not work without a crossover cable. Just eliminating variables so we are sure direct connection works.

Correct.

Well, I didn’t wanna shove myself back there, but okay. It is now confirmed. Tho I’m sorta dissapointed, I’m only getting around 160 MB/s transfer. Trying to make iperf work now.

Also, it needs to work WITH the other cable plugged as well.

I have a server set up exactly how you’re describing, and I ran into same problem with traffic going over the wrong interface.

In my case, using Windows 10, I had to unmap the drive using the old ip and reboot before remapping the new one. Without the reboot, traffic would go over the wrong interface.

Back after a reboot.

Unfortunately that didn’t work for me. Traffic is once again flowing thru the 1G connection.

I think I will need to play around with the routing on both machines to fix this. But I’m starting from scratch over here. If anyone has other ideas, please share.

Here are my routes

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    102    0        0 eno1
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 enp4s0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-2bfd66249344
192.168.0.0     0.0.0.0         255.255.255.0   U     102    0        0 eno1
192.168.10.0    0.0.0.0         255.255.255.0   U     101    0        0 enp4s0

eno1 = 1G (internet)
enp4s0 = 2.5G

===========================================================================
Interface List
  8...70 85 c2 ab 9a bd ......Realtek Gaming 2.5GbE Family Controller
  5...70 85 c2 ab 9a bf ......Intel(R) Ethernet Connection (7) I219-V
  6...0a 00 27 00 00 06 ......VirtualBox Host-Only Ethernet Adapter
  1...........................Software Loopback Interface 1
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      192.168.0.1      192.168.0.2     25
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    331
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    331
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    331
      192.168.0.0    255.255.255.0         On-link       192.168.0.2    281
      192.168.0.2  255.255.255.255         On-link       192.168.0.2    281
    192.168.0.255  255.255.255.255         On-link       192.168.0.2    281
     192.168.10.0    255.255.255.0         On-link      192.168.10.2    276
     192.168.10.2  255.255.255.255         On-link      192.168.10.2    276
   192.168.10.255  255.255.255.255         On-link      192.168.10.2    276
     192.168.56.0    255.255.255.0         On-link      192.168.56.1    281
     192.168.56.1  255.255.255.255         On-link      192.168.56.1    281
   192.168.56.255  255.255.255.255         On-link      192.168.56.1    281
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    331
        224.0.0.0        240.0.0.0         On-link      192.168.56.1    281
        224.0.0.0        240.0.0.0         On-link      192.168.10.2    276
        224.0.0.0        240.0.0.0         On-link       192.168.0.2    281
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    331
  255.255.255.255  255.255.255.255         On-link      192.168.56.1    281
  255.255.255.255  255.255.255.255         On-link      192.168.10.2    276
  255.255.255.255  255.255.255.255         On-link       192.168.0.2    281
===========================================================================
Persistent Routes:
  None

The gateway field is empty on both ends? My setup didn’t work with anything populated for the default gateway on either end.

Yes, it’s empty. There is not gateway, to speak of. I understand at least that much of networking. Well… since today. Gateways route traffic outside of your subnet. There is no outside in this case.

I had this same kind of issue after setting up an old proxmox node with 10gbe on the network. When it was connected at all, it would nuke the internet for everyone. Not sure the setup or resolution exactly unfortunately but it would be nice to know what was going on in case I experience this again. Hope you get a good resolution so I can remember lol. Ill let you know if I remember anything that could potentially help.

what protocol is the NAS using? Is there any configuration into specifying adapter(s)? I am assuming SMB 2.0 since you are using windows

Multiple Network Interfaces with Samba (Could be useful)

It has been a while since I set mine up and my topology is quite different since I use a dumb switch for my devices (PiKVM, Scopes, Lab Instruments, NAS)

1 Like

SMB3, but that’s irrelevant. A protocol like this is way above the level of Internet Protocol and routing and stuff.

I’m trying to get iperf running, so we don’t rely on copying files as a benchmark.

Just a shot in the dark, but smb 3 is the one with multichannel support? Maybe try and disable that.

Now that you mention it, I had to disable multichannel in my smb share as well.

I will check out my server config. But with multichannel wouldn’t it be possible to achieve 2.5+1G ? (I’ll dive into the docs later.)

Regardless of that however, I managed to get iperf running by doing
iperf -s -B 192.168.10.5

This binds the server to that interface.
The problem now is this:

>iperf-2.2.n-win64.exe -c 192.168.10.5 -P 2
------------------------------------------------------------
Client connecting to 192.168.10.5, TCP port 5001
TCP window size: 1.00 MByte (default)
------------------------------------------------------------
[  2] local 192.168.10.2 port 50163 connected with 192.168.10.5 port 5001
[  1] local 192.168.10.2 port 50164 connected with 192.168.10.5 port 5001
[ ID] Interval       Transfer     Bandwidth
[  1] 0.00-10.06 sec   657 MBytes   548 Mbits/sec
[  2] 0.00-10.04 sec   731 MBytes   611 Mbits/sec
[SUM] 0.00-10.04 sec  1.36 GBytes  1.16 Gbits/sec

Which is pathetic.

P.S. The above is win → linux ; the opposite fails with tcp connect failed: Connection timed out

I just fired up iperf3 on my server, and the first run is barely over gigabit speeds but subsequent runs result in the full 10 gigabit.

Do you see any difference with multiple sequential tests?