Need help setting up multiple PC's connected through two different networks

I’ll try to explain my layout as best as possible.

  • PC 1: Personal Desktop using Windows 10
  • PC 2: FreeNAS Server
  • PC 3: Ubuntu Machine

Each one of these machines is using their integrated Gigabit RJ45 NIC’s to connect to a small switch I have connected to my router.

Also, each of these machines has a 10Gbps SFP+ NIC connected into a MikroTik switch (CRS305-1G-4S+IN) 4 SFP+ ports and 1 gigabit RJ45 port.

To better show it, I made a chart of how it’s currently set up:

So here’s where the problem lies:

  1. FreeNAS does not allow for multiple NIC’s to be registered as different IP’s. I read one post from the FreeNAS forums that says you can’t have multiple network interfaces connect through the same subnet/netmask. This is, I manually configured both the RJ45 and SFP+ NIC’s to be on different subnets, but I can’t connect to the FreeNAS machine from either of my other machines. Doing on a ping on either machine to the FreeNAS one results in a message: From XXX.XXX.XXX.XXX Destination Host Unreachable
    Which from what I’ve read means that my router knows that IP is registered inside my home network, but it can’t find a way to get to that IP. Even when I remove the gigabit network interface and only have the SFP+ one, my other machines report the same issue. Also just want to be clear that this issue isn’t present when doing actions to/from my desktop and the Ubuntu machine, FreeNAS is the only problem there.

  2. How would I go about binding the different connections between my machines? Say I want to copy a file from my desktop to a FreeNAS machine: I want it to only use the SFP+ connection through the SFP+ switch, but right now it defaults to using the gigabit connection through the gigabit switch. For FreeNAS, I tried binding the SMB service for network drive sharing to the one on my SFP+ NIC, but it doesn’t work and the drives get disconnected.
    I’m also assuming that whatever sort of solution for this question would also help solve issue #1, or at least I’d hope so.

What IPs/subnets are you trying to use on each interface? You just need to use a different subnet for the storage than you do for the main network. You also need to use static addresses on storage while you can use dhcp on the main network.

The issue with FreeNAS is just that it won’t let you have 2 interfaces on the same network and it won’t let you have more than one interface with a dhcp address.

The main network uses subnet 24 (255.255.255.0), while the SFP+ network is set to use subnet 29 (255.255.255.192). I checked on all three machines that they should be on the same subnet, and they are, but the connection still doesn’t work.

So, to make sure I understand this.

You have (for example) 172.16.25.0/24 and 172.16.25.192/26 as your networks?

3 Likes

Specifically, here’s what I got:

  • Desktop
    • RJ45: 192.168.1.229/24
    • SFP+: 192.168.1.97/29
  • FreeNAS:
    • RJ45: 192.168.1.98/24
    • SFP+: 192.168.1.99/29

I’m able to ping 192.168.1.99 from my desktop, but can’t access SMB or any other feature from it, even when explicitly set to use that IP / network interface.

So the problem here.

Desktop

  • RJ45: 192.168.1.229/24 <---- These are in the
  • SFP+: 192.168.1.97/29 <---- same network segment.

change the SFP+ network to

192.168.2.0/24

You will be fine.

Also, above you said your SFP+ network is 255.255.255.192 Which is /26

/29 is 255.255.255.248

Which none of that really matters because

192.168.1.97 is on the same network segment as 192.168.1.229
If you had put them in 192.168.1.0/25 and 192.168.1.128/25 it would work, as they are now in different segments.

4 Likes

For human readability, I’d just go with:

192.168.1.0/24

And

192.168.2.0/24

2 Likes

Man, networking is confusing.
The 192.168.1.229/24 address was set by DHCP, so I suppose I’ll set it manually.
If I understand this right, I can keep everything on the same subnet, but the 2nd to last number group of the IP address should be different on the SFP+ connections than on the RJ45 connections. So something like this:

  • Desktop
    • RJ45: 192.168.1.229/24
    • SFP+: 192.168.2.97/24
  • FreeNAS:
    • RJ45: 192.168.1.230/24
    • SFP+: 192.168.2.98/24

Should work fine?
Also I just realized it’s the real DeusQain, really honored to see you in the forums!

1 Like

You can let dhcp do whatever it wants to do.

Those addresses should work for the sfp+ interfaces. If you continue to have connectivity issues, we should look at your default gateways.

Yeah I can’t ping the SFP+ IP anymore when setting them to what I described above.
What should I do with the gateways?

Can you ping the two hosts from freenas?

Yes, and for some reason I checked again after more than 15 minutes and I can now also ping FreeNAS from my desktop.

Wait, holy crap it all works now! Thank you all for the help, this has been frustrating me for the past few days.

2 Likes

So, to recap…

192.168.1.0/24
192.168.2.0/24

These are different networks (subnets)

You know what. This actually calls for

I think I covered all of it, in this video.

3 Likes

That was actually a much easier explanation of the whole thing than anything I’ve seen online. Good stuff!

But for some reason, despite setting the IP addresses as mentioned above, any network drives I create always try to go through my 1g connection. So far I’ve tried:

  • Force SMB service on the Linux host to use the SFP+ NIC and IP address
  • Explicitly attach the network drive using the Linux host’s SFP+ IP address

I’m not sure how or why, but it seems that the connection is still trying to go through my 1g network instead of the 10g one. I did a tracert on my Windows machine to the Linux host, and even when specifying the source and destination IP addresses, it goes through my 1g router.

I’m so confused as to how to get this working. Should I be setting a VLAN on the SFP+ switch? I truly have no idea what I’m doing at this point

If the SFP+ Switch is physically separate from the 1G switch. That’s better than a VLAN.

I’d be interested to see how you are attempting to do this.

Making network drives over SMB by specifying the hostname would default to using the 1G network connection.
Because your computer would attempt to resolve DNS, then go out the 1G interface as it’s going to resolve the 1G address.

What happens when you disconnect the 1G interfaces all together?

Are you binding the interface in smb.conf? That should solve the problem.

It sounds like it’s trying to use the default gateway for everything. Also worth checking firewall.

If it’s trying to use the Default gateway for everything.
It would indicate that he’s got his IP scheme improperly configured somewhere.

Also, @HunterAP if you haven’t already disable IPV6. It’s terrible on windows machines. Causes stupid things to happen.

2 Likes

Yeah IPV6 is disabled on all my NIC’s, don’t think I’ve ever turned it on either (as it’s checked off on all network adapters).

@oO.o I did just that, restarted SMB and nothing changed, restarted the host and still the same thing.
I disabled the 1g network adapter and the funniest thing happened: I copied a file to the SMB share, and the Bluetooth adpater was the one reporting that is was doing the file transfer at 1.7Gbps.

I disabled the bluetooth adapter and the file transfer continued correctly through the 10g network connection.

How was your bluetooth adapter on a network to transfer files?

I truly have no idea, I should’ve taken a screenshot the network data in Task Manager when it happened.

When I re-enabled both the 1g adapter and the bluetooth adapter, the connection seems to stick with the 10g SFP+ port.
Still don’t how this happens, but I’ll take it I guess?

1 Like