For part 3:
when you said you use partitions on the boot ssd: one for the host and one for the vm storage
Would namespaces work for that as well, or only partitions?
1 Like
Ah so what I was talking about there is the SSD can be used bare metal bypassing the hypervisor. and it keeps working as-is. Best trick ever?
2 Likes
But would namespaces on nvme work for that also?
1 Like
It can be sketchy not all bioses support efi partitions outside nvmeXn1
If you just use it for data that’s fine. If you expect to create two namespaces each with efi parts … test it ahead of time as it may be bugged if you need to swap it to a new chassis
3 Likes
I had in mind something like the The Forbidden Router to do for a while. I started to use my hypervisor(proxmox), as a desktop, windows vm for gaming and ubuntu vm for work, for a start, now I’m definitely going to put my router(pfsense) on the same box for sure. I have ordered a quad intel card. Should be intresting to see how all of it will turn out. Fun part I’m doing all of this on consumer hardware. Cheers, great project!
2 Likes
I just wanted to share my experience, while I started with my home lab server years ago I’ve always tried to keep stuff simple in case of failure.
My first server was an AMD Athlon 2200+ with 256MB of ram and a crappy ESC Elite Group MB with SIS chipset! What a pain!
During lockdown I was bored… so my plan to upgrade my infrastructure started taking place! Since then I had: My NAS and VM host (HP Microserver), Router (pfsense machine), SFP ONT, and its Media converter…
Back then my ISP started to have some issues and I was looking for an IPv6-capable one. pfSense always had issues with PPPoE even with mitigation so also priority went to an ISP with IPoE instead of PPPoE as most of the ISP over here do.
I’ve made my plan to upgrade my HW, I needed something to replace my P55+i5-750 router and virtualize it. And then use my HP Microserver as an offline backup solution.
I’ve got some drives, an HBA controller and I had an Intel NIC laying around. In that period I’ve seen some YT videos about cheap china Xeon MB and CPUs.
They had a lot of PCIe lanes! I’ve then scrapped the idea of the no-brand china MB due to ECC and power on loss factors.
I went with an Asus X99 WS IPMI. I really wanted the IPMI since the machine will be located in the mechanical room. Accessing it is pure pain! But I had my pain too with that MB!
I’ve been quite lucky I’ve got the MB from Poland, the CPU from Germany, and the RAM from who remembers? It was a bare minimum of 16GB DDR4 ECC ram!
I have recycled my PSU from the P55 build, and the chassis… oh the chassis I needed one that was capable of “EATX”! I didn’t want to buy 40USD of fans so I settled for Antec P101; It has a lot of drive cages, space for 2.5 SSDs, and space for EATX MBs!
It was also cheap back then!
I’ve patiently waited for all the parts to arrive, POST service is so slow compared to Amazon 1 day delivery! I’ve then assembled everything!
It has posted successfully but the BMC was protected by a password! That would only be my first small issue with that crappy BMC!
Once updated the BMC, and the BIOS and tested the HW I’ve configured the software. I’m using Proxmox. Yes, I know not the best solution but it is my personal choice. I had issues with XEN back on LGA771 days since then I’ve used Proxmox.
So HW my specs are:
- Intel E2620v3 (30EUR Shipped)
- Asus X99-WS IPMI (250EUR shipped)
- RAM 2x8GB 2R ECC HMA41GR7MFR8N (30EUR + Shippment)
- Noctua NH-D9L (Brand new from Amazon)
- Antec P101 Silent (100EUR shipped with 3 fans included)
- Dell PERC H200 (HBA FW, VFIO to OMV VM about 25EUR)
- 4 x WD Red 3TB
- WD ~~Black/Gold/~~Enterprise 3TB (it is failing)
- WD Purple 3TB
- Crucial P2 500GB NVME (Added later for Linux VM)
- Kingston 120GB SSD as boot drive (Coming from the P55)
- PNY 250Gb SSD for Docker (unused for now, added later to increase docker volume space)
- Nvidia GF119 GT-X20 no idea which rebraind it is (Windows VM)
- Intel PRO/1000 (To be replaced by Intel I340)
The SW requirements was:
- Router VM with VFIO of the NICs, only Virtual NICs for VMs/containers
- VoIP PBX (FreePBX)
- Docker containers (Reverse proxy, Plex, Mailu, etc)
- NAS (OpenMediaVault)
- Local/Remote Windows VM just in case I need something.
Added later: - Ubuntu/PopOS VM for development (Crucial NVME as boot drive via VFIO)
The Windows VM
I have used VFIO to pass the USB controller so the USB3.0 on the back and some on the front panel are available to the VM. Also, the HD audio and the GPU are directly connected!. Unfortunately on resume from standby the VM dies. (I’ve never discovered why, but I barely use that machine)
The network setup
3 NICs are in LACP to the managed switch and one is the uplink to the ONT.
The switch is a cheap GS1900-24, most of its ports are used by the Ethernet sockets I have around the house.
Due to the issues with my ISP I’ve migrated to the new one about 1 year ago. The new one has IPoE and IPv6 but IPv4aaS (MAP-T). So I cannot use pfSense anymore. I’m using OpenWRT in a VM and it is really fast. (OpenWRT is the only solution so far that implements MAP-T)
I’ve used OpenWRT for years on my router, AP, etc. I also ported some cheap repeaters too. It never failed me but I was using pfSense since a lot of people use it and I wanted to try.
Since its first “install” I never had any issue, but I’m still trying to figure out how to configure LACP.
On my network, I have 2 VLANs one for the public stuff (services), and the other for my LAN traffic. PUBLIC cannot connect to LAN but LAN can connect to PUBLIC. Both can connect to the WAN. Docker uses MCVLANs to connect to the internet/local networks.
I use Portainer to manage my containers but I’m thinking to migrate to k8s/k3s.
My docker host doesn’t have a lot of storage (I’m using the LVM driver for docker for volumes and using a tiny LVM group on the boot drive) and I would love to move to a GlusterFS setup using my NAS VM to store the volumes.
All the public services have data going through Traefik and to protect my services I’m using Crowdsec (with various collections); the bouncer is running in the Opewrt VM.
So far my only big issue with the setup itself is the power-on sequence for the VMs and the Docker containers. When I have to power cycle the whole machine I have to do it manually. the only VM with auto-boot is the router and the PBX.
Other issues? I could write a book about the issues I had with the BMC!
I even looked for leaked schematics of the MB online hoping to find out the pinout of the ASPEED chip to port OpenBMC to it! But no luck!
EDIT: typos, added pictures, added prices
3 Likes
Hopefully someone can help shed light on what is likely an understanding gap I have…
I have been using pfsense for a few years at a very basic level as a home user (~4 months as a VM now with a passed through quad NIC under proxmox), I used to run pihole, but ditched it in favor of pfsense’s package pfblockerNG.
I don’t presume to understand which way is better, I am just looking for a better understanding of the differences or why we wouldn’t just use pfblockerng in this situation since we are already using pfsense? Honestly - I enjoyed pihole more as it has shinier graphs which are obviously a critical part of network infrastructure (joking, but, I did like the pihole webUI a lot) but I switched as I figured it would be more seamless to just do it all in pfsense. Is this not the case?
Just to make sure my understanding is at least sort of correct; pfsense (with unbound running) + pfblocker should be effectively equivalent to pihole? As in, both of those options would provide local DNS cache for faster DNS lookup in addition to DNS black holing based on block lists?
1 Like
I cant wait for the wife to go on vacation so I can mess my system up…LOL
I have a portainer instance I can set up, So I will try to run from there… See if I have as good of a grasp as I think I do on DNS. I have Pihole with Unbound on a Pi4, and I’m using OPNsense as my firewall/ DHCP server… I think I can make this work with some IP magic, but we will see.
Its especially relevant to me now as I had accidentally deleted a 4TB Samsung SSD worth of games on my new X570 build… Boo
I hit my download cap in a day with 1/10th my library of games 
WITHOUT Mods lol
I may see if I can possibly add in the IP for Nexus Mods to see if I can cache those as well as some are huge downloads… @wendell Think its possible to add in requests from Nexus Mods without too much difficulty?
Also I saw the
## Change this to limit the maximum age of cached content (default 3650d) CACHE_MAX_AGE=3650d
Is there a way to set it for infinite?
1 Like
10 years = infinity in internet years?
2 Likes
Yeah I guess your right there lol
Anyone have any input for this? Why pihole over built in pfsense tools? @wendell I am sure there is a good reason why, I’m just trying to understand the choice to potentially help inform my own choices and recommendations to others :).
2 Likes
I think most people would be able to use pihole pretty easily.
I can’t say the same about
Also can pfblockerNG let you create DNS records for stuff on your LAN?
2 Likes
That’s Unbound’s job. Afaik pfblockerng is sort of like an “addon” for Unbound, the DNS magic still happens on Unbound itself.
2 Likes
I think for most people its the GUI quality. PIhole is much simpler to interact with.
3 Likes
That’s certainly true, pihole webUI is in a different league then pfsense’s unbound/pfblocker.
Now I am considering switching back to pihole, hmm.
2 Likes
I was already looking at doing a build like this, with Proxmox and PCIe passthrough of a 4 port NIC. Currently I have PFSense running baremetal on a Dell Micro 5060 and its good enough but after working with pfBlocker I decided Pihole probably would be simpler, problem is I dont want yet another baremetal device for it.
So VM’s for these devices makes lots of sense. I may set this all up and then just move it back to the Dell Micro but make it a VM this time so I can run a few different services on that same little very under used PC.
2 Likes
But I want to store ALL THE GAMES LOL. No but really it would be useful with three gaming pcs in the house its easy to hit a data cap.
I built a forbidden router using a Dell R210 is this week. XCP-NG running a pfSense vm and a RHEL vm with podman for container management. I might change back to portainer but I haven’t decided yet
4 Likes
but how is it though?
2 Likes
