New apartment, New Networking problems!

Networking
1

Hi everyone, I’m new here but have been lurking and watching for quite a long time. A few years ago I built a Pfsense router after watching a bunch of stuff about them and realizing that as time goes by I’m going to need to learn more about this stuff anyways. So I built my device and though it was a bit of a pain to get setup for the first time, it was awesome! I love it and it worked great giving me the tools to learn more over time.

So I had cable and that was great. ISP → Modem → Pfsense. Done. Simple.
Then I moved.

First Problem.

Now I have DSL and its my only option in the new place. I spent some time trying to get the gateway router modem combo thing to bridge and trying to setup my Pfsense box to talk to no avail. I have done a ton searching to see what I am doing wrong but it doesn’t seem to have worked out. I am going to try again soon but its though because I can only have the network down for a few hours in a row or the family starts to get pretty upset. I have a few of what I think are stupid noob questions but thought I would toss them out here and see if there are some ideas from you guys. Just trying to learn more.

1, Pfsense box and ISP supplied router have the same login of 192.168.1.1. If the ISP router is going to be set to bridged mode, should I still change the address of my Pfsense box so they don’t interfere?

2, Is the MTU the same for all ISP’s or do I need to call them and get that information?

3, Is there a simple walkthrough somewhere that I haven’t been able to find that may point out something that I have missed or can check against? My google-fu has failed me on this.

So that’s where I’m at for Stage 1.

Stage 2

Replacing my access point. I have an older Ubiquiti AC lite and it has been good, but I have just received my new Ubiquiti 6 lite and learned what PoE+ is first hand! So because the PoE injector for the AC lite is 24v passive wont work for powering the 6, I was thinking that it is time to get a new switch that has PoE to run it. I was thinking that I should use the old ac lite to have a different wireless network that I can put all of my IoT stuff on. So I have come up with a network diagram for what I think makes sense. Paint for the win! Please tell me if I’m being dumb in the setup of this. I am by no means a network specialist!

Network plan

So a few things I need help with in this respect are,

1, Should I get a PoE+ switch instead of an injector? The only switch I have right now is an $8 5 port.

2, Should I get a managed switch? I don’t have a guest WLAN or second network for the IoT things yet but I would like too. I have a 4 port NIC in my Pfsense box, but have used only 2 ports until now.

3, For a bit of extra money, a downgrade in specs, and it is currently out of stock, Should I wait and get a Ubiquiti Switch Lite 8 PoE so that I can manage it with Unifi?

4, If not the 8 lite, What would you recommend? TP-link, Netgear, Cisco scares me because of the $$$, something else?

5, 8 port?

I think these are what I need to really start with and then I can try and get everything setup before I start to worry about future expansions into camera network, or the old server that I got from work that I would like to load TrueNas or Unraid on and try and learn about. ( I chickened out and got my Synology but really would like to learn how to do my own NAS)

I’m sorry if these are dumb, over asked, silly questions. I’m not trying to waste anyone’s time, and I feel dumb for not being able to figure it out but I thought I would ask since you guys sure as hell know what you are doing and I’m trying to learn.

Thanks in advance!

1 Like
2

it could be that you actually have to contact your isp,
for getting your modem switched into bridge mode.

1 Like
3

When I tried to do it the last time I had them on the phone for a while. They were completely lost on what I was doing and didn’t understand it. Is it possible that they MUST do something for me to be allowed to hook up my Pfsense box? I don’t know enough to know if they have that ability. I sure hope not. That seems like a conflict to need to use their router. I wouldn’t be surprised but I would sure be disappointed.

4

If their CS is terrible you might have to pony up for a business account with them. You will get better support that way/

5

Well basically when you hook up your pfsense box,
to your modem when the modem is still in router mode.
Then i assumes that you will be getting a double NAT,
router behind router basically.

So the pf sense box will get an internal ip from the modem / routers dhcp server,
which will of course be different from the standard gateway,
of the said modem.
That’s probably why you can’t reach it.

I’m not fully a network expert either.
So maybe @PhaseLockedLoop or others could help you out with,
configuring this properly behind the modem / router.
Because i never used pf sense myself.

1 Like
6

If you use your ISP name normally you can find really good info online… to see if you need bussiness account or not for it.

7

So I haven’t tried to double NAT. I don’t know anything about double NATing but I guess I could read into it? I don’t know if that is my only option though. I only have tried to hook put the Pfsense box while the router from the ISP was in bridged mode. I thought it needed to be in bridged mode to work how I was thinking I wanted it too. Are there negetives to having its NATted? I’m such a NooB at this stuff… I could install a furnace for you or fix your car though! lol :sweat_smile:

8

I have tried looking for information on their site. Distributel doesn’t have much good information as far as I can find. They are a pretty small player in the game for sure.

9

I see, my provider does IPTV/Internet and stuff… so i had to configure some extra steps for Vlans and was really straight forward after it.

I would try to be forwarded to a higher tier of support when you call.

Just play dumb and get forwarded to tier 2 (if they have it, because is a small company most likely they have a higher tier support that does normal stuff + bussiness things)

10

That is actually a really good idea. I didn’t really think about that. I just assume that I’m a dumbass and have done something stupid… which still might be the case. I struggle because I only know so much. Networking seems so important to me to know, but I feel like it is wizardry and there is a meeting that I need to go to to actually understand any of it. I’m trying my best though!

11

I have legit 0 clue about anything network related don’t get anything i type as gospel. But i legit had the same issue on my Restaurant, at home? just worked. On my Restutaurant because of how the internet provider did TV/Phone/Internet in the same line i had to do some extra steps, that where really not that straight forward, but i did it with the technician on site while he was doing the fiber cable : )

12

Well yeah like i said sometimes your isp has to remotely switch,
your modem into permanent bridge mode.
I assume with your previous cable isp that was likely the case.

When you log into to the modem itself you should be able to see,
which internal ip it sends out to the pf sense.

I assume others with allot more networking and pf sense knowledge,
will hop into this topic soon to give you some tips and help,
with configuring this properly.

13

Very recommended yes. Having access to modem once in a while for troubleshooting is useful. Although it’s not 100% strictly required to renumber your network, you can get around having to do it by using NAT and source based routing and port forwarding etc, etc, etc… renumbering is a one time medium amount of work kind of a job, and once you’re done stuff ends up being simpler permanently / at least until you move.

No. 1500 is a widely accepted standard. Various ISPs skimp on equipment and admins. You can test and verify by trying to ping 1.1.1.1 or 8.8.8.8 with larger packets and DF (do not fragment) bit set. Ping commands differ slightly between different OS, the thing to remember is that 1500 is the Ethernet payload size – your 20byte IPv4 header and 8byte ICMP or UDP result in a 1472byte datagram. Your OS ping command might need you to pass 1472, 1480, or 1500 as the flag value… You can verify the sizing using TCP dump. Maybe this helps you as you move to separate networks and VLANs and tunneling.

I’d recommended not blackholing all ICMP or fragmented packets with pfSense, but rejecting them as properly as much as you have patience to set things up and troubleshoot stuff properly.

If you’re set on using unifi and VLANs/guest networks, get a unifi switch POE+ switch to power your AP and other stuff, e.g. an 8 port or a 16port USW lite placed strategically where you have lots of devices is a good choice.

As for POE+ specific / maybe strange gear, other than just typical wall powered POE+ switches and 24W 100mbps and 24W 1Gbps injectors, Ubiquiti sells e.g. these larger 54V/80W injectors injectors in the ISP accessories category, which you can use to power e.g. USW Flex passthrough switche, which can in turn power multiple devices like e.g. a 802.3at U6-LR and/or your e.g. a raspberry pi over a poe hat, or through a gigabit usb-c 802.3af POE splitter or various Chromecast like HDMI dongles and TV attached streaming boxes potentially external hard drives, if you get a 12V 2.1mm barrel jack splitters and/or 2.1mm barrel jack squids (it’s a commonly used plug)

I have a USW flex on a shelf under the TV, it’s nice.

On switches, unlike injectors, you get power usage monitoring and the ability to remotely power cycle the port over network if/when the attached device gets stuck or needs a reboot.

Ubiquiti also sells a Unifi Flex Mini which can be POE powered but doesn’t pass through the power.

Netgear and TP-Link also sells some POE switches for slightly cheaper, but they’re not drastically cheaper and don’t integrate with unifi controller for easily setting up LANs/VLANs and so on for guest networks and iot stuff. If you’re using Unifi APs, I wouldn’t recommend adding Netgear/TP-Link even though they’d work fine / perform typical expected basic functions ok.

Do not buy second hand corporate/enterprise network gear for an apartment (you mentioned Cisco, but also HP/dell/…), it’s too bulky and noisy for typical apartments. (If you’re homelabbing then sure, you’d don’t care if sleep next to a highway… but I wouldn’t recommend it)

14

Ok perfect. I can do this no problem I think. Makes sense.

:exploding_head: I believe that you stopped speaking english somewhere in this but I can’t figure out where… lol. Should I ask my ISP what they use?

I’m only using Unifi for my AP management so far but I would like to add VLAN’s/Guest network I think, so if you think it makes sense to stay in the same ecosystem and that its worth waiting, I think that makes sense too. I just needed a little confirmation I guess. Like I said, this stuff is all a learning experience for me for sure.

:exploding_head: Oh goodness! I don’t think I need that much. I’m just worried about how much power the Ubiquity 6 long range or future access points might take, as well as if I’m putting 5-8 PoE IP cameras on the same switch. Good to know that stuff is around though I guess! So many fun toys!

Thanks Risk, I think I will get started at least on this stuff. I will make sure I can change the IP for the Pfsense and I am on the waitlist for the Ubiquity 8 lite switch too. Thanks!

15

Heh, sorry.

Most ISP customer support usually says, turn everything off and on again, and connect directly to what they provided and if it doesn’t work they’ll send you a new one, or you should try with a different device.

TCP and path MTU discovery in a nutshell

So MTU is how many bytes you can stuff into each Ethernet packet. Normally, TCP would “discover” MTU sizing “automatically” along a path.

Why does TCP care and how does it figure out the MTU size.

Well, it cares because its job is to take an http or a Netflix or what have you stream, and split it into packets (aka. it segments the stream). And it needs to know the optimal size that TCP calls MSS (max segment size).

(Obviously, TCP also reassembles packets as they’re received back into a stream, and also deals with confirming packets are received and/or telling the other side they’re missing and maybe were dropped, so the other side can resend them… and there’s a bunch of other stuff… e.g. guessing the bandwidth for each stream… science and statistics and stuff).

So what happens if you get the MSS number wrong / you send a packet too big?

It might not be too big for your router, it might not be too big for your gateway, or your ISP first hop… somewhere along a path, you might run into a router that can take it from one of its interfaces… and can see which other interface it needs to go to, but the MTU is too small for it to go out that way.

  1. It can silently drop it (easy way out)
  2. It can return information to the sender that says “you’re holding it wrong how could you not know my MTU is 512 bytes”, for example.
  3. It can split the packet into 2, and send both to destination, the destination can reassemble it.

Turns out:

  1. Is kind of a dirtbag move.
  2. Requires sender being smart
  3. Turns out, people like firewalls and peeking inside of IP packets for filtering and stuff and this fragmentation and reassembly is hard for your corporate firewall to peek inside of.

… so 2 it is, senders became smart and TCP stacks of various operating systems now deal with these reject messages and this process is called path MTU discovery.

…

Now, you could still form a packet of any size and try to send it through and see what’s the max size that goes through, the ping command can do this for you, just set the right flags.
To avoid packet accidentally getting split as per 3. above, the ping command also supports setting the “do not fragment” bit, which routers typically respect.

why does MTU vary

I think first (or at least very old version) of ethernet had an MTU of 512 bytes… it had to do with 2Mbps data rate and something with cable length and electron/signal propagation speeds through a coax cable of a certain length and spec, and the fact it was a shared medium and multiple computers were sharing one cable and needed to detect when someone was talking and back off randomly on collisions.

And TCP is older than this I believe.

Then, things evolved, speeds went up, 2->5Mbps and you could use shorter cables with bigger MTU or longer cables with smaller or something along those lines.

At some point 1500 value stuck as a magic number for gigabit, and then someone did the math around the time 10Gbps was starting and 9000 came up as another magic number.

Nowadays you have most modern ethernet capable of 9000-ish but defaulting to 1500 ; and wifi is pretty much stuck on 1500.

Some switches sometimes may do 16k, fancy datacenters/ research labs and so on, internally may run at 16k or 64k and they’re experimenting with ethernet replacements because of massive scale and cost turns out $5000 400Gbps nics and switches are worth optimizing when you’re buying tens of of thousands of them.

Somewhere along the way people started tunneling and VPN-ing and using VLANs, first with IP over IP but soon other combinations appeared.

What became common is things like, take an IP packet, give it to wireguard. It wraps it in UDP, sends it over IP, that then happens to live on a PPPoE link that your carrier may handle on a VLAN.

So you end up with various encapsulations (think padded envelopes), adding various number of bytes of overhead, which at the end of the day still ends up using same chips and wires with same physical limitations baked into silicon.

And that’s how we ended up with 1460, and “baby jumbo” and other crap…

… oh and IPv6 has different header sizes.

… all that and folks not understanding how this works and how to test things or what to expect and/or overcorrecting for stuff is how we ended up with “random MTU” limits.

Usually this all “just works” but it’s a pain in the rear to figure out when it does not work/or when it does not work well, why that is.

@Letsgetsteve added more detail ; there’s more exhaustive reading on Wikipedia

An access point will take around 8-15W steady, might peak up to 20 or 30. Most cameras about the same.

Raspberry pi and similar would use 2-5W, might peak up to 15W, hard drives depending on size probably same.

Cables also have length and some internal resistance that’s lower in thicker cables per unit of length, and higher in slimmer cables. Depending on current which depends on how much power the end device ends up needing, you’ll also lose some power in the form of heat over the cable (POE “experts” may refer to it as “voltage drop”), not enough to feel it on your skin/fingers, but it’s technically there. Problems start to occur when you’re running close to your power budget limit, and you end up with brown outs or your end devices spontaneously rebooting. The power losses are higher when device needs more power… and it’s a PITA to troubleshoot, best to aim for at least a 20% margin, so if you need 24W for an accesspoint at peak, get 30W, if you need 60W in total, get 80W power supply.
You might also consider separate switches. If you need more power for e.g. a fancy camera in the future (powerful IR LEDs, high resolution h.265 encoding and maybe PTZ motors, that all adds up).

1 Like
16

They won’t interfere no You should set your ISP router to bridge mode

Sometimes there can be complications and I usually see this with fiber gateways where you need to clone the MAC address of the modem so that your PFsense boxes issued and Ip4 and 6

I do not run PF sense I run OPN sense. The process is different

Just so you’re aware the MTU is handled automatically by the protocols. If a packet is too big is simply will be broken up Yes this console things down at times but it doesn’t mean it’ll cause a catastrophic failure of the network however if you are insistent on setting that value which I can understand I’m going to give you the following information

I’m assuming you’re familiar with Linux or you’re at least familiar with Windows both have a ping tool I’m going to give you the introduction for finding the maximum transmission unit size via a Linux terminal. If you do not use Linux or BSD or Mac OS then I suggest you look up how to do this for Windows

ping -s $((1500 - 28)) -D 8.8.8.8 -c 1

What does this essentially doing is its defining the size of the packet that you are pinging to 8.8.8.8 which is Google DNS. We don’t need to do more than one as we shouldn’t have any timeouts. 1500 is the testing size for the MTU the subtraction of 28 is the packet header that should always remain the same but what you’re going to do is you’re going to step down the MTU until you find the size often the size is for DSL or 1460 and 1492. The size for cable is 1472. The size for a fiber network may either be 1500 or a jumbo frame which is usually 9K.

It’s simply a matter of setting the bridge mode for your modem from your ISP but the issue I think that we’re going to have here is we know nothing about the modem he has and he needs to kind of figure that out and that’s why there’s no simple guide for him to follow because we don’t know the specifics of the hardware however we do know enough about PFsense

@Novasty has more skill with PFsense
I have more skill with OPNSense

17

Yup like i said above, sometimes the user cannot switch the modem into bridge mode entirely.
In certain cases this has to be done by the isp remotely to get it switched into bridge mode permanently.
I assume that the said customer support cannot do this them selfs,
and they likely have to make a ticket to pass it through a line up in the tech support chain.

But yeah we also need to know more about the said modem / router indeed.
So that there could be guides for TS to follow.
In a worst case scenario the isp might only offer this kind of support for business clients.
But that would be kinda weird.

18

So it shouldn’t matter but potentially could? Is it possible that the ISP needs to see the Pfsense router as 192.168.1.1? Just trying to learn.

I initially wasn’t planning on touching MTU but I keep finding references to needing to change it to fix bugs with talking to the ISP. I did the ping test and found that the highest I could go was 1464. From what can read on first blush, that’s normal for PPPoE?

The modem/router combo thing is a Smart/RG SR515ac. I was able to follow the instructions that I found on a page by Sonic to bridge it, but I couldn’t get it to all work. I still think to problem is probably me and missing some sort of setup in Pfsense. I hope. I hope its not the ISP trying to lock it all down.

19

Not at all. The ISP will issue a public IPv4 address that is transparent NATd (usually or CGNATd if they are assholes) on their end. 192.168.1.1 is a private address for the internal network of that routing modem.

yes then set it to 1460 (RFC standard size)

post all the steps you dont and what its results were (maybe with screenshots). Thats how people here can best help you.

20

I followed all the steps on this page

I’m not sure which screenshots that you would like but I can anything that you would like. I just need to know which one.