PFSense - Certain Websites not working on VLAN - Working fine on LAN

Networking Networking Software
1

Hi everyone,
I’m very new to networking and I’m having an issue in PFSense with certain websites not loading on my VLANS (https://mail.yahoo.com/ https://community.spiceworks.com/), a couple of them load slowly without any .CSS styling (http://community.ubnt.com), and others work perfectly fine.

On my LAN network all these websites work fine.

I can ping these websites on VLANS fine just get an ERR_TIMED_OUT error.
I tried reducing the MTU but it makes no change.

I also have whole network PIA VPN configured but disabled, when its enabled on the VLAN these websites work fine.

Here is my Firewall rules on VLAN10

1529454906050-capture-resized
1529454906050-capture-resized.png760x66 12.1 KB

Automatic Outbound NAT rules enabled

Trace route for community.spiceworks.com

Capture

PFSense is installed Virtually on XCP-NG on a dell r710 if it makes a difference.
I have no packages snort, etc. installed
PF Sense Version 2.4.3-RELEASE-p1

Any input from experienced user would be greatly appreciated.

2

My hunch is that these sites use a CDN to for their CSS.

3

Check if it’s not an IPv4/IPv6 problem.

4

Do you have anything like mss clamping or are you filtering out packet fragments or icmp traffic?

5

I only noticed the .css on the ubnt forum when I was trying to diagnose the issues.
facebook, youtube etc. that the wife/kids browse all work fine.

IPv6 is disabled. SYSTEM / ADVANCED / NETWORKING

I don’t understand what these are so i’m going to say no but not 100% sure.
As i said I’m new to networking and just doing this as an hobbyist who is curious how these things work.

It was a clean install with a vlan added for guests that goes out through a Unifi AP.

I then came across the video about whole network VPN on this forum and when I ran into issues there, I registered so I could post and went to check my email realized I couldn’t access it (on my laptop connected to guest network.) Jumped on my desktop and it worked fine.
I added a couple more vlans with different firewall rules but same results.

6

A TCP sconnection stream need to gets segmented into packets before being sent over the network. Considering internet is a series of tubes with different and hard to predict maximum packet sizes, there’s multiple approaches to figuring out how much is safe to send, these algorithms are called "Path MTU Discovery, and they usually rely on ICMP.

Early internet/IP worked around this in a different way - send whatever, and routers will fragment/split packets and reassemble them… This was a bad idea, … simultaneously ICMP that PMTUD relies on turns your router into a public server so, so some guides recommend blocking ICMP.

The only way to figure this stuff out is to find a use case that’s broken, and stare at packet traces, generally major PITA to figure out.

7

Ok, is this likely a hardware limitation of the Broadcom NIC?
Or more likely because PFSense is virtualised?

I wouldn’t know what to look for in the packets, and even if I noticed something wouldn’t know where to start fixing it.

I might spin up a new PF Sense instance and start over to see if it may have been a configuration error.

8

Neither hardware nor a problem specific to virtualization.

VLANs traffic has a smaller MTU than non VLAN traffic - likely cause of your issues.

You could maybe start by adding a rule allowing all ICMP and ICMPv6 traffic , and checking to see if that makes any difference.

Edit: here’s a cloudflare blog entry from a few years back when they had an outage because of MTU issues: https://blog.cloudflare.com/path-mtu-discovery-in-practice/

9

I tried lowering the MTU on VLAN interface but it made no difference. Here is a test from my PC on VLAN10, full 1472 bytes plus 28 byte overhead = default 1500 byte MTU.

Capture

Should this be done on the WAN or Vlan interface?

EDIT: Just realised that from the VLAN I can ping other devices on the network even across VLANS, or out to the web will Full MTU.
However when I try ping my gateway my limit is 1468 bytes. So 1458 + 28 overhead = 1486 bytes. Should I set the MTU to 1458 or 1486?

Capture
Capture.PNG470x564 12.5 KB

And should I be setting the MTU on all interfaces or just on the VLANS?

EDIT 2: Just set all interfaces to 1486 and now I am experiencing the same behaviour on the LAN, the exact same websites have stopped loading.

10

As an easy experiment, try all of them.

On Linux at least (read: 99.9% of home routers), icmp traffic is covered as ‘related’ in iptables , and forwarded back to the original host which helps MTU discovery for hosts.

–

If you can figure out the IP that’s causing your host to fail to load the website, let us know. Or try pinging it with -l 1472 from pfSense directly and see if it works.

–

Any chance you could try adding this to your pf, assuming em0 is your wan:

scrub in on em0 proto tcp max-mss 1452
scrub out on em0 proto tcp max-mss 1452

also maybe try max-mss 1220 ?

11

Pinging from PFSense directly works fine.

Dont know where I should add this, but changing MSS on the VLAN interface to 1486 does the trick.

All offending websites are now working (so far anyway)

Thanks for all your help!

1 Like
12

https://www.netgate.com/docs/pfsense/firewall/editing-the-pf-ruleset.html