HOWTO Virtualised pfSense, with _actual_ control of physical network card + Unifi ~ Didn't think it'd be this easy

Networking
1

Introduction

In the background you’ll see a simple PC build, in this case

In the Asus UEFI make sure to enable Intel Virtualisation option.

System Config

  • Installed XCP-ng v7.6 onto single SATA SSD.
  • XCP-ng is tied to a FreeNAS backend.
  • Mainboard Intel NIC used for XCP-ng management access - connected to internal EdgeRouter-x (ER-x) at gateway 10.0.0.1/24

Appliance ports (yes, it’s a PC though in this case…)

It’s important to note that these are configured accordingly in the XCP-ng control software

  • NIC0: Mainboard Intel NIC
  • NIC1: I350 port 1 // pfSense WAN (xn0)
  • NIC2: I350 port 2 // pfSense LAN (xn1)

Ubiquity Unifi Switch 8 Managed PoE+ Gigabit Switch

  • I have Unifi controller software running in an Ubuntu VM, on my primary XCP-ng server. This handles all provisioning and control over Unifi hardware.

  • The switch used here is the Unifi Switch 8 Managed PoE+ Gigabit Switch with SFP shown below on the left in silver/grey.

  • Blue cable --> Connects to the 10.0.0.0/24 network from the ER-x router

  • Yellow cable --> pfSense LAN

  • pfSense WAN port is also tied to the 10.0.0.0/24 network from the ER-x router

  • Black cable --> Connect to my MacBook Pro.

Ubiquity Unifi Switch 8 Port Configuration

  • Port 1: Set to default ‘All’ Network, which by default is VLAN1 (trunking port).
  • Port 2: Set to a custom Profile tied to Native LAN VLAN2 // This is just a LAN profile with VLAN ID 2 and DHCP set to ‘off’. Profile is called Profile-VLAN2
  • Port 3: Black cable to the MBP - in the image below is set to Network LAN

Notice how the laptop has a 10.0.0.0/24 IP.

IMG_7989
IMG_7989.jpg4032×3024 1.77 MB

Now, if we change the switch port Profile to Profile-VLAN2 – the magic happens here – the laptop now gets a 192.168.100.0/24 IP – YAY!! This is crucial, as this is a virtual port (xn1) within pfSense.

IMG_7988
IMG_7988.jpg4032×3024 1.36 MB

Great, so why would you do this?

By simply changing the port profile, any downstream traffic can now either be routed (1) straight from the main router (ER-x) or (2) through the ER-x —> into the pFSense “Appliance” --> out the Black wire.

This setup allows you to switch the physical network tied to the Black cable :grinning:

I’ve also got IDS (Intrusion detection system) running in pFSense in the form of Suricata, which at the moment monitors the WAN interface.

I can also configure pFSense as an OpenVPN client, and all traffic through it will then be routed through a VPN – my ultimate goal.

Downsides to this approach

Right now, I don’t have any need for VLANs through pFSense, but had I wanted to tag a VLAN entering the pFSense WAN port – this is not possible as pfSense reports there are now VLAN capable interfaces attached.

Xen/XCP-ng has support for SR-IOV but I haven’t gone down that rabbit hole.

Generating VLANs

I wanted to be able to tag traffic coming out of xn1 (pfSense-LAN port) with a custom VLAN tag, but couldn’t find away to do this; (1) it seems pFSense doesn’t have the ability to do this (??) but only tag/decode incoming traffic via the WAN interface (2) it wouldn’t be possible since XCP-ng insulates it from VLAN capable hardware (which the I350 NICs are capable off, 802.1q).

If you got this far, share your thoughts – I’m keen to hear from you!!

CC: @wendell @SgtAwesomesauce

2 Likes
2

I’m actually more curious how to run pfsense in QEMU, with the pfsense VM having actual access to the NIC, and the virtual QEMU network being the only network connection exposed to a simultaneously running Windows 10 VM, so that the Windows VM network connection is sanitized before it hits the physical NIC.

1 Like
3

If I boot up Win10 (VM) and assign it the pfSense-LAN Network (as per the initial HOWTO); since that is the same LAN with access to the actual NIC, it is sanitised by pfSense.

Only difference between the MacBook Pro & the Win10 VM, is just the latter is a VM.

When you figure out how to set this up in QEMU, do please share a HOWTO!! Thanks.

4

I was more thinking the pfsense VM drives the local LAN inside QEMU rather than QEMU making it’s own DHCP server. With the physical NIC being the pfsense WAN. That means anything going out of the physical NIC is sanitary for tin foil hat networks.

5

I think it’s unusual that you have suricata running behind the edgerouter. I would put it on the wan side and then let the edgerouter handle routing, vlans, etc.

1 Like
6

Given the ER-x router is a small appliance, it has proved to be 100% reliable, and keeps the primary pieces of my infrastructure up and running on the 10.0.0.0/24 space.

For example, I have IPs mapped to the FreeNAS box and 2x Synology Diskstations, printers, and even the PS4, Xbox One X, Chromecast etc.

While it would be nice to have Suricata at the main WAN ingress point, if something goes wrong in the PC in terms of hardware, or some mis-config with the VM etc – it would effectively kill the entire network (i.e. in this scenario, there would be no need for the ER-x in my network; the pfSense/Suricata setup would replace it).

So - for critical bits of the network, that cannot go offline, I’m routing the 10.0.0.0/24 to those components – mainly the management access (at least, that’s the plan for now).

Thoughts?

1 Like
7

Sure that works. Ideally, you’d trust the pfsense machine and have a more traditional Gateway > DMZ > Router > LANs config, but I understand your reasoning.

The only other thing I thought was odd is the unifi switch. I’ve never configured one outside of a complete unifi ecosystem. I’m not sure to what degree you can configure vlans on it without a unifi gateway. I’m guessing you can manually configure tagging by port, but not sure if you’ll get anything more advanced than that.

8

Well I’ve tagged VLANs in the EdgeRouter in the past, and in the Unifi config if you create Networks with VLAN tags (they must of course correspond accordingly), then the Unifi Switches will effective untag accordingly. It’s still the same 802.1q setup – the only difference with Unifi is the ‘controller’ approach to provisioning vs. traditional manual control (similar to one of the TOUGHSwitches I had in the past).

9

Out of curiosity, with a USG, what’s “more advanced” that than? Routing (l3) /firewall features or…?

10

With the USG you can authenticate into vlans via RADIUS and I think you can do it by MAC address as well. What I had in mind though was private and/or isolated vlans which you can set up on an Edgeswitch (along with 802.1X authentication, GVRP and other fancy stuff).

I believe a full Unifi network does some of those things under the hood, but some of the features are only available if you have a USG in addition to the switch.

1 Like
11

Gotcha. I’ve got the USG Pro 4, but it was for a different network; not sure I have the time to replace the existing ER-x with this though, given all the configs it has.

1 Like