I’m at a loss here after a week of attempting this;
I’m trying to run pfsense as bare-metal as possible on my haswell i5 4670k (which doesn’t have vt-d/virtio) with my WiFi nic as the WAN, and a virtual local Ethernet network passed to the host OS as LAN using e1000 (Intel’s virtual nic protocol before virtio was a thing)
I was actually able to do this fairly easily on windows 10 hyperV by locking my WiFi nic to a bridge that is reserved/locked to the pfsense VM, and a virtual local Ethernet nic that the host OS shares with the pfsense VM. Essentially windows doesn’t have internet access until the pfsense firewall boots up and passes the internet to what it sees as a physical LAN nic. I give pfsense all 4 cores for load balancing and dynamic RAM with a generous buffer, so I get to enjoy encrypted DNS and all the other goodies without much added network latency.
My main reason for this setup is it saves me power and money since I’m powering my home on a limited 200AH lifepo4 battery I put together, so the more I can do on one PC the better.
I tried to do the same in KVM (because its apparently bare metal like hyperV) but I can’t seem to get past creating the virtual Ethernet nic for LAN on pfsense; everything I’ve tried so far (bridges, VLANs, virtual Ethernet nics with their own mac ID using nm-connection-editor in gnome) doesn’t work and I can’t connect to the pfsense configuration page from the host like I can on windows with hyperV, and there seems to be almost no documentation on KVM or Fedora on how to do this. (I’m running Fedora 33 workstation BTW, forgot to mention)
Any help would be greatly appreciated. I feel like I’m close, and this would allow me to eventually move on to my other projects like a virtual NAS and caching my video camera footage
That’s what is odd, maybe I have the virtual LAN nic in the wrong mode? Because I can get both virtual NICs to show up in the pfsense KVM as em0 and em1, and both are running their own NAT and show IP addresses and UP state. But when I go to log in to the pfsense firewall browser configuration page or just access the internet from the host, I get nothing. My host gnome Fedora OS won’t use the virtual LAN nic for some reason.
VMWare has the VSwitch to configure which is pretty easy, I don’t know about KVM. If you create two Windows VM’s can they pass traffic to each other over the network with the default KVM vm network adapter config?
I’ve heard of VMware and the vswitch it provides but I don’t believe I necessarily need to even use a vswitch if I can somehow replicate the same configuration that’s working with windows hyperV. I also believe the vswitch would add additional overhead, or at least another layer to deal with. I’m looking to basically use pfsense in a VM with a nic dedicated as WAN and then a basic virtual nic dedicated as LAN… And I kind of want to get to the bottom of why this isn’t working in KVM, since it works on hyperV
qemu/kvm can create a TAP interface that’s basically “the other end of a virtual cable” that pfsense is plugged into.
You can have a permanent (created on boot) tap, that you let qemu use. Or you can have one created dynamically by qemu when it starts up. The dynamic ones are more frequently used, they’re usually added to a bridge when they’re created at VM startup time because typically you want a VM to communicate with other VMs or you’d want to use a physical nic or you’d want to configure your host ip on a bridge that has a constant name.
When you start up the VM, you can poke around various network interfaces from your host command line to see how your bridges are put together, and you can use tcpdump or wireshark on various interfaces and check if your pings or arp requests are going through the bridge correctly and so on.
You can inspect how packets traverse Linux firewalls by adding -j LOG rules into various places with iptables (or nftables).
You can look at routing tables (yes there’s multiple in Linux for each one of the network namespaces).
Finally, you can just ditch pfsense if you’re not locked into using it for any of its various random features and replicate the network setup of what pfsense was doing on the fedora host without VMs or other strange networking contraptions.
Well I just painstakingly enabled DNS over TLS in systemd resolved.conf on Fedora 33 to cloudflare domains for now after an hour of troubleshooting (no UI, fedora just switched their whole DNS system too so it’s confusing as hell), since I have so many other important projects to get to and can’t be bothered with KVM right now. I really tried almost every KVM bridge combination without success, but I will be returning to this in the near future.
That’s why I thought that assigning the WiFi nic to a KVM virtual bridge running its own NAT would work for pfsense WAN as it did with windows hyperV. What still baffles me is why it almost works out of the box with hyperV but KVM requires command line arguments and other shenanigans
That is not related to KVM or Linux in general. I think it’s a shortcoming of WiFi, which ms windows tricks to make it work. But honestly I’m in no way expert in that nor do I know why that is really the case.
Maybe someone does and can shed some light on that matter?
