Virtualised pfSense - Seems to work in Windows, not in Ubuntu

Derkades · February 4, 2020, 9:07pm

KVM noob here, doing this all with the “virtual machine manager” GUI.

My setup

Virtual network 10.0.0.0/24 in isolated mode. DHCP disabled.
pfSense VM. WAN = macvtap bridge to host NIC. LAN = virtual network (has IP address 10.0.0.1)
Windows VM. LAN = virtual network (got IP address 10.0.0.2 from pfSense). I can access the internet and the pfSense web interface.
Xubuntu 18.04 VM. LAN = virtual network (got IP address 10.0.0.10). I can’t access the internet and when I go to 10.0.0.1 I see a webserver from the host instead of the pfSense web interface.

I can’t find any logical explanation for this, since the network settings are exactly the same. Maybe one of you can help?

SgtAwesomesauce · February 4, 2020, 9:10pm

output of arp on xubuntu VM?

output of ip route and ip addr on xubuntu VM?

What’s the mac address of all your VMs and your host?

So, just to be clear, you’re seeing the nginx instance you’re running on your physical machine (the VM host) when you hit the IP of your gateway?

Does pfsense see the xubuntu host?

Derkades · February 4, 2020, 9:13pm

robin@xubuntuvm:~$ arp
Address                  HWtype  HWaddress           Flags Mask            Iface
_gateway                 ether   52:54:00:bf:90:0f   C                     enp1s0
robin@xubuntuvm:~$ ip route
default via 10.0.0.1 dev enp1s0 proto dhcp metric 100 
10.0.0.0/24 dev enp1s0 proto kernel scope link src 10.0.0.10 metric 100 
169.254.0.0/16 dev enp1s0 scope link metric 1000

I’ve edited my OP with a screenshot that hopefully makes this clearer. When I access the gateway IP in the guest, I get a webserver the host is running which is not related to any of this.

SgtAwesomesauce · February 4, 2020, 9:14pm

Everything looks good in terms of the virtual network config.

Gotcha.

Can you ping the windows VM from xubuntu?

Derkades · February 4, 2020, 9:15pm

Good question. It does:

SgtAwesomesauce · February 4, 2020, 9:16pm

Okay, I think PFSense is doing something here. I’m not good at PFSense stuff, so I’m probably not going to be much help here.

I’ll ping a couple people that might be able to help.

Derkades · February 4, 2020, 9:17pm

Thanks! I am overwhelmed by the kindness here

SgtAwesomesauce · February 4, 2020, 9:18pm

Happy to help. I’ve sent out the call.

SgtAwesomesauce · February 4, 2020, 9:19pm

Hold on. Your firefox instance is going to https://10.0.0.1/ and it looks like IE might be going to http://10.0.0.1/

Difference being 443 vs 80.

Can you verify if you’re hitting the same port?

Derkades · February 4, 2020, 9:21pm

Yeah, I am going to port 80, but the host webserver is set up to redirect port 80 to 443 which apparently works as well. (the host is an Ubuntu machine with a lot of services, docker containers, etc)

SgtAwesomesauce · February 4, 2020, 9:22pm

Firefox likes to try 443 first.

I would specify http:// when you hit the URL.

Or, try https://10.0.0.1/ on your windows machine and see if you’re getting similar results.

Novasty · February 4, 2020, 9:23pm

It doesn’t seem like xubuntu is picking up the DNS settings. Are you able to hit https://1.1.1.1/ in the Linux system?

Derkades · February 4, 2020, 9:23pm

Even using wget:

robin@xubuntuvm:~$ wget -O- http://10.0.0.1
--2020-02-04 22:22:55--  http://10.0.0.1/
Connecting to 10.0.0.1:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://10.0.0.1/ [following]
--2020-02-04 22:22:55--  https://10.0.0.1/
Connecting to 10.0.0.1:443... connected.
    ERROR: certificate common name ‘*.derkades.xyz’ doesn't match requested host name ‘10.0.0.1’.
To connect to 10.0.0.1 insecurely, use `--no-check-certificate'.

I am not

SgtAwesomesauce · February 4, 2020, 9:25pm

Okay, that’s a standard redirect.

What IP range does your non-virtual network use?

Derkades · February 4, 2020, 9:27pm

192.168.1.0/24 (host is 192.168.1.111, router (also pfsense) is 192.168.1.1).
pfSense VM:
Screenshot%20from%202020-02-04%2022-28-05

SgtAwesomesauce · February 4, 2020, 9:27pm

Okay, so no address collisions.

SgtAwesomesauce · February 4, 2020, 9:28pm

Bumped you up, you should be able to post links now.

Novasty · February 4, 2020, 9:29pm

Wait. you said DHCP is disabled on the PFSense VM?

Derkades · February 4, 2020, 9:30pm

DHCP is disabled in QEMU/KVM, enabled in pfSense (DHCP client for WAN, hosting a DHCP server on virtual LAN)

SgtAwesomesauce · February 4, 2020, 9:30pm

something’s clearly handing out leases.