Totally Overkill pfSense Router + Suricata + OpenVPN client

Hi all,

For the past couple weeks I’ve also been rewamping the network at home. Right now, at my WAN ingress point I have an EdgeRouter-X from Ubiquity, which connects to a TOUGHSwitch 8-port PoE managed (lvl2) router.

I’ve added a 1U 24-pro Netgear ProSafe GigE Switch, and will be moving the older 8-port version to the lab-area downstairs.

The WAN ingress right now is 100Mbps fibre. While I have the firewall configured on the ER-X, I have noticed a couple limitations already.

  • If you plan to go the EdgeRouter route (pun intended!) consider getting a beefier unit. The ER-X has very limited RAM; I’m typically limited to at most one copy of the firmware to sit in Vyatta; last time I did an upgrade, I basically had to wipe the backup to free up enough space to download the new firmware.
  • Secondly, it is alright for a 100Mbps connection, but is too slow to say run pfSense and maybe an OpenVPN client all at the same time.

Right now, I’m running my OpenVPN server inside a VM on my FreeNAS box. That same VM has the UniFi Cloud Controller as well for UAP access points around the house.

So this is the totally overkill build I’ve gone for -

Link to the pcpartpicker config above.

Couple things to note -

  • It will run an OpenVPN client (at least 1), and I plan to assign VLANs and potentially certain MAC addresses to that egress route.
  • Suricata will be run for Intrusion detection/prevention, and from what I’ve read this is processor intensive. Latest versions needs AES-NI on chip for all the crypto related stuff that it offloads to the CPU. This is why, the 7700K isn’t entirely insane of a choice here.
  • the 7700K also means I don’t need to add a GPU.
  • I ultimately went with the TUF Mark1 as it has Dual-GigE ports. Having WAN/LAN at the start is all I need to get off the ground.
  • I’ll most likely slap a Noctua in this if the Coolermaster has any trouble.
  • Chose bog-standard RAM. Nothing fancy. Crucial are a solid choice though, been choosing them for over a decade easily.

I’m running 3x Corsair AX860i PSUs here, 1x HX1000i and 1x HX750W. I believe in buying good PSUs; they’ll generally perform longer and be more efficient in the long run.

I am expecting to receive my order later this week, just hoping customs won’t hang onto it for too long though. Will post an update once I start on the build!

Thanks, do let me know your thoughts!!

2 Likes

Overkill is an understatement :open_mouth:

You could have gotten more than adequate performance with an i5. An i7 won’t even work up a sweat, but you shouldn’t loose much if any of your bandwidth.

2 Likes

Maybe money better spent on low power equipment. Build an 80watt xeon (or less) with a little ecc ram, an ssd and a quad port intel pro/1000 nic.

Also include a high quality UPS with enough capacity to handle your pfSense box, switches and modem for 30 min or so…

4 Likes

Really good point SudoWolf (epic handle BTW!) - I’m running 4x relatively new APC UPSes, each one around 1400VA each. Will be throwing a beefier one soon on the network stack.

I went the el cheapo route with used enterprise hardware from ebay.

Currently I’m $500 in but just waiting until I have enough for some more hard drives and then I’m going to finally set up a pfsense box for my network.

My main concern outside of the straight cost of the gear is the power draw\noise.

That said the hyper212 will be great at stock speeds, and an OC would be just silly when your using a modern i7 just for a home level router… PF sense would do normal duties on a 6w SOC, and adding Suricata only bumps your actual needs up to modern i3 levels as ALL skylake parts have the correct extensions (a change from the past when the Pentiums and celorns did not, just as in the past K series parts missed out of virtualization extensions).

The PSU is also an odd choice… why 850w? Best efficiency is ~50% load, and a titanium rated supply that does more than your actual wattage as 50% rating is worth less than that.

All in all I don’t really understand… a lower power use xeon with similar (or even more) ECC memory and duel NICs would cost less money, even buying new with longer warranty than the consumer stuff.

2 Likes

Why not get a CPU with an higher core count and a lower TDP instead? I think a low power 6/8 core Xeon might be a better choice if you want to go overkill and you’ll get better overall system reliability with server parts. I know even consumer parts, if used wisely, last long and don’t have any issues but why take even the slightest risk? Also, since you’re going to make such a machine you’re not going to need all the PCI lanes so having a system with expansion cards it’s not and issue in my opinion.

1 Like

Agreed - you guys are lucky, well anyone that lives in EU/USA. Unfortunately, I’m restricted by location, and for the past year and a half have been ordering my stuff from B&H USA, simply because their shipping rates are ‘reasonable’, i.e. it still costs a lot, but far less than what it could be all things considered.

Locally, we don’t have anything like FreeGeek or anything like that. Most locals are scammers, and that’s a headache I just don’t want to touch on either.

@SheepInACart - valid points there for sure. Went with the 860W cause I have a couple of them already; if any of my other systems ever need a replacement PSU for the short-term, I can cannibalise this one. I tend to do this in a hurry, since ordering a replacement at any point means there’s a 12 day wait usually once factoring customs.

No OC on this, it’s already over the top hehe.

Would have liked to have gone Xeon, but sourcing specialist stuff is very hard as I can’t order from say NewEgg/Amazon etc. It’s a total pain.

@MetalizeYourBrain - well, I’ve got the parts arriving today/tomorrow; I would have preferred WS/server grade components for sure; will see how long this lasts and if things so go sideways, I will replace with server grade stuff.

Couple questions to all -

  • Which low-power Xeon would you guys recommend?
  • If you had to attempt this project, would you be able to choose the server-grade components from https://www.bhphotovideo.com/ alone? If so, please link the CPU/Mobo/Ram combo that you would have gone for please.
1 Like

Fair enough on parts you already had.

Firstly, as a random who has never been there, what is wrong with your national online stores in Sri Lanka? Barclays, PCnet, Tera ect… Sure the range may not be cutting edge, but I can find most parts somewhere, and you don’t need to get stuff on the bleeding edge for this type of application.

I ask because not only is shipping expensive, but B&H are really a camera and film industry store, they list lots more things, but non-core market listings are pretty high price and often not in stock. For example they don’t list any SOC boards beyond the old AMD thin clients, or any LGA1151, 1150 or 1155 low power xeons. If B&H really is your only choice they do have one broadwell low power xeon (35w 8 threaded complete with ALL extensions), but its in a miniserver barebones. The small size is nice, and it gives you 6 RJ45 gigabit Ethernet ports, 2x SFP+ and a single slot half height PCIe, but the 3x 40mm fans are louder than a single 120mm would be.

You can find that xeon miniserver here:
https://www.bhphotovideo.com/c/product/1302805-REG/supermicro_sys_e300_8d_e300_8d_superserver_black.html

Before I list any other actual recommendations though, a note that Suricata only used to be super system intensive and not well threadable back before 2012 (and then only 4x more than Snope), so for an example to run home how overkill your idea is these days; The E5-2680 0 @ 2.70GHz, an 8 core hyperthreaded xeon, was demonstrated to l push 10 gigabits through Suricata in 2012, you need 1% of that traffic, yet due to processors improving in the last 5 years and the higher clockspeed of consumer parts your i7 7700k is almost the same raw work potential… scaling is not perfect but your on the order of 100x more CPU than it needs to be at minimum, and >5x as big as I’d buy.

That said unless you already have the i7 and not another use for it, I’d recommend just keeping using it, you won’t save enough power with anything that you can put in that socket to make it worth the swap. I say that because even though its a 88w part vs a 16w-35w, those are the peak operating numbers, and at idle or low loads it’ll be much more even as the i7 won’t push to its max clockspeed and thus will run at less voltage (in fact you could even underclock\undervolt in the bios if you feel like it… 3.0 ghz would be heaps for your needs and allow considerably lower voltage). Underclocking could maybe have the TDP of your i7, so ~44w to 35w, which is not so bad.

If you don’t already own you CPU or Mobo the most import question is how fast is your actual max linespeed. Since if your on a nominally 100 megabit line even if only get an average of 40 megabit, it still can on a second or so level arrive as a short burst at up to the 100 megabit rate, so the systems needs should be specified to that. Also the rough rule of thumb is you get about 200Mbps per modern Suricata and bro worker, so two modern cores at regular enterprise type frequencies (3.0-3.5ghz) would handle your traffic with a max of 50% load. Thus don’t shy away from a lower wattage xeon, it’ll get the job done.

16gb ram is perfect for your needs Suricata wise, you’d probably get away with 8gb but no reason to push it. I fully support going for 2 non-fancy DIMMs from a good company so you can upgrade RAM in future. Motherboard wise you have no special needs… so the most basic server board with duel gigabit NICs will do, and normally basic is good for reliability, power use and your wallet, especially vs a gaming mobo. I also recommend going mITX and getting a cheap tiny case that can go on a shelf with your switch and modem, you don’t need the space (no expansion cards) nor want to be running high wattages or many fans.

A 4 core atom SOC board would be perfect for your needs, >25% of the i7’s performance, 4 threads, duel NIC’s and 4 ram slots on a 16w passively cooled mITX form factor, and yes, something new like the C3558 has the AES-NI on chip (and all the virt extensions). A weird but cool feature they have is the ability to run off 12v DC only OR a normal 24 pin ATX connector, so you can avoid stepping up to mains power from your UPS to run an ATX PSU to turn it back to 12v DC… which can save money, space and power. If you need more threads, there is the 25w 8 core, but beyond that the 35w xeons make more sense.

Link to atom SOCs: http://www.supermicro.com/products/motherboard/atom/

If your doing full packed capture you’ll need big amounts of storage though, as in twice your highest average actual line traffic rate over the time you want to keep logs multiplied by the number of seconds the log is for. In other words if you pull full 100mbps all day and want 48hrs logged you’ll need a terrabyte. A week will fit on a 4tb drive, and I’d consider something basic like raid1 if this is actually critical to you (no real need for ZFS or anything though).

Finally you NEED a good UPS. Lots of these features hate unexpected power offs and will give buggy performance on a partical brown out untill next propally power cycled. It only needs to last long enough to shut the machine down (from full load and once the batteries are at your nominal end of life), but it needs to be present and work.

2 Likes

Consider something similar to my pfsense box https://imgur.com/gallery/gCzfe
I’ve since added an i350 quad port nic, but it’s still extremely beefy for my needs. My interwebs maxes out at 150Mbps anyways.
If you need more power for VM’s or the likes, just swap out the 1220L-v3 for one of the higher end 12XXL-V3 chips or even a used i5-4XXX chips(though you’ll lose some of the virtualization and management features). lots of server parts can be had for cheap off eBay and some even include free shipping. Even as a citizen of MapleSyrupLand, we can still find decent parts nearby Canada only listings often have cheap shipping.

While I haven’t actually tested out my config with full load and openVPN traffic(waiting for my Cisco sg-300 to swap out my temp 10/100 thrift store special switch), from what I’ve read you won’t really need anything more than an i3/i5 if you’re not getting anywhere near gigabit from your ISP.

4 Likes

Question for all - Right now, I have the following components

  • Asus ROG Maximus VIII Formula Z170 Mobo
  • Asus Z270 TUF Mark 1 Mobo (has dual-GigE NICs)

I plan to have the pfSense box use the TUF board due to the dual GigE ports.
The ROG board is going in my multipurpose ‘HT + Gaming PC’

I also have -

  • Intel 6700K Skylake
  • Intel 7700K Kaby lake CPU

Here’s my question - any downside to having the 7700K in the ROG board, with the latest BIOS? Just to double check, am I missing out on any “features” that the Z270 brings, well potentially apart from Optane, which I don’t care for.

I want to have the generally more overclocking friendly board with the 7700K for the gaming aspect, and throw the older 6700K chip on pfSense + Suricata + multiple OpenVPN daemon duty. Appreciate thoughts and advice…

Those in Florida - stay safe!!

1 Like

Initial stages to the build - smoke testing on the table, and the case arrives from the US