Stuck on Cisco/pfSense router on a stick setup (Inter VLAN routing)

Hi guys,

I have a cisco 48 port switch that Im trying to setup with pfsense as a router on a stick but having some issues. I believe all the configs to be correct however I cant ping from one vlan to another despite everything looking ok. I cant tell if the issue is a firewall or switch config.

Heres what I got. Any help is appreciated, Ive been looking at it too long to be any more useful to myself.

On the switch i got port 26 config’d as a Trunk with the following vlans.
The two Im working with are VLANs 19 and 20 to try and be able to ping from one to another.
image
Switch config shot 1
image
Switch config shot 2

Interfaces
image

FW rules


If you ssh into pfSense, you can run tcpdump -ni vlan19 proto icmp. Tcpdump picks up packets to show you as close to the wire as possible, incoming before firewall, outgoing after firewall.

Run ping alongside tcpdump, and you’ll know whether your switch and other interface config besides firewall is ok.

Thanks Risk, appreciate your response.

You gave me an idea and I enabled the logging for VLAN 19 and 20 (too tired ATM to setup SSH). Saw this which leads me to believe the FW rules are working correctly.

Logic would say its a possible switch issue but any ideas where/what that might be?

Also worth stating the firewall on the host systems allow ping.

Hm, maybe tcpdump or Wireshark on the hosts?

Clearly ICMP is going through the switch in at least one direction from at least one host - host1 → firewall.

tcpdump on firewill show you whether it’s exiting and tcpdump on host2 will show you what’s going on there.

Are the physical ports where hosts are connected pure access ports, or trunk or hybrid (untagged/tagged/both)?

Asking since host os/network stacks can be finicky, are they just Linux boxes with 8021q module loaded?

Provided you have the package installed:
man traceroute

HTH!

You are not showing how you configured the access ports on the switch for the clients from where you’re testing the pings … are both set as untagged VLAN and using the proper VLAN IDs, and for these have yu set up PVID properly?


Screenshot 2022-01-16 at 11.40.49

The source system is a linux box, the target system to ping is a windows box. I will remove the linux box and put in a windows but Im not expecting anything different but I appreciate the additional item to check.

The client ports are access ports that are untagged. Settings are below.

image
image

image
image

Im not as sure about the PVID if that needs to be setup/config’d elsewhere, if you can elaborate Id appreciate it. Ive read several things online about it but still a bit unclear.

It might be worth mentioning Im able to ping the V:AN 19 and 20 gateways from either VLAN to the other’s gateway, if that makes sense.

The P in the UP is what tells you that PVID is set correctly, on some other switches other than cisco/linksys it needs to be set on a separate config screen, I believe the ciscos do it when you set the port up in access mode…

ok . next checks … the dumb ones …

  • are you pinging windows/Linux , linux/windows or both? I assume all firewalls on the two boxes have been checked and allow traffic, right?
  • in pfsense are you blocking bogon networks/private ip on either interface?
  • on the clients, is the pfsense the default gateway? If not, did you set up a route to send packets to pfsense?

So ive removed the linux box from the equation and replaced with a Windows to even the playing field even tho its prob unnecessary.

I have confirmed the target system can receive ping/icmp responses when on other parts of my network but not when connected to these these two VLANS.

The source systems can for sure ping no problem.

For the clients, bogon and private networks are not being blocked and the default gw for the clients is pfsense (Ie clients on VLAN 19 default gw is 192.168.19.1) (vice versa, clients on VLAN 20 have 20.1 as their default gw)

Can you ping the clients from pfsense?

And can you post a diagnostic-> routes from pfsense?

Hmm, I am from the default source address (not sure what this constitutes)

From VLAN 19 Im not.

To reiterate Im trying to ping from VLAN 19 to VLAN 20 (Ideally both ways but for now just one way Ill be happy with)

Ok, can you re-run the ping that is not working in one tab, and run packet capture, on interface VLAN20, protocol ICMP?

If you see the packet exiting the interface the issue is either on the switch config or routing (i.e. not on pfsense), if you don’t see the outgoing packets then pfsense is filtering them and we need to understand why given the rules in place …

Test settings.

Test results. (Running a ping with -t switch to keep it going indefinitely)

image

Not sure if I have to break open this .cap file to view further…

shouldn’t the ip you’re pinging from be 192.168.19.1 (the ip of the gateway)?

I would have thought so as well but it seems, from some detective work that the .19.10 address may have come from the first avaialble address in the DHCP scope. Dont know if there is a way to specify it coming from 19.1

image

From packet cap
image

hmm, that doesn’t sound right, it must use the ip of the interface …

can you post Status->Interfaces and Diagnostic->Routes
?

Blotted out my public IP

Also dumping cap from earlier with Wireshark.

It’s not like you want to NAT between VLANs, why would IP of pfSense box show up anywhere in the packet?

We were trying a ping from the gateway, using the other VLAN ip. …