Ryzen motherboard for virtualized but dedicated pfSense build

Hardware Motherboards
1

So I’m thinking of doing a new build for pfSense (old one had a motherboard accident and was built on a circa-2010 Core i3)

I have a friend who runs pfSense on a VM setup so he can easily snapshot and restore the VM to easily recover from failed upgrades and similar. I’m thinking of doing the same thing with this new build, but I’d like to have PCIe passthrough so I can pass through my Intel 4-port NIC to the pfSense VM, and using the onboard NIC for management of the hypervisor on it’s own VLAN.

I’m thinking of just buying a Ryzen R3 1300X (I have 150/150 internet, with 250 and gigabit eventually) and using Proxmox or ESXi as the hypervisor, but I need to find a AM4 motherboard that supports IOMMU first. I would also be using host CPU type if possible (I know it is on Proxmox, not sure about ESXi) so RDRAND and AES-NI work in the pfSense VM.

Thanks

1 Like
2

Im using Proxmox VE host, OPNsense in VM, on a MSI X370 Gaming Plus mobo and Ryzen 1200.
The PCIe passthrough worked out for my 4-port Intel NIC. For the setup I followed the video instructions of the unicks.eu youtube channel.
I only have a 75 download/7,5 upload Mbit connection. Most of the time my CPU stays idle. Im definitely gonna run a few VMs more on it.

3

Cool. I’ll check out the video later.

A few questions I thought of

  • It seems this board doesn’t support ECC RAM. Unfortunate. Given how this is a 24/7 router I’d like to use ECC RAM to maximize reliability. I know on my Supermicro board it has a event log that supposedly logs ECC RAM errors. Knock on wood in the 8 months I’ve owned said server I’ve logged zero.
  • How exactly are the IOMMU groupings on the board?
  • Do you use ‘host’ CPU type with the OPNsense VM? I use it for a Debian VPN appliance/load balancer and successfully benchmarked 2.5gbps IPSec between VM’s on the same host. (Using an Xeon E3-1230 v5 with above mentioned Supermicro board)
  • If you can test the above for me with the CPU type set to host (if it isn’t already) you should be able to run dmesg -a | grep Features over ssh on the OPNsense VM to see if AES-NI and RDRAND are working. For reference running pfSense on the above mentioned server with the host CPU type, here is the output I get.
  Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c4f3a<TSCADJ,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP>
  XSAVE Features=0x7<XSAVEOPT,XSAVEC,XINUSE>
  Features=0x1f83fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,SS,HTT>
  Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x1c4f3a<TSCADJ,BMI1,HLE,AVX2,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP>
  XSAVE Features=0x7<XSAVEOPT,XSAVEC,XINUSE>

Thanks for your help.

4

Honestly unless your a business with a ton of users, or planning to run something like complete traffic analysis for malware, chances are your better off getting something cheaper\smaller\lower powered than ryzen, like a passively cooled SOC, even on gigabit. Used passive laptops can be an affordable way to get into low powered gear too, and can be surprisingly quick.

On the other hand more powerful hardware gives you more to mess around with and experiment, but if you want ECC on ryzen you currently have very few options, as to my knowledge only one or two boards have any ECC dimms on the QVL, and then only one or two options (however some people find non QVL items work fine, like Asrock AB350 Pro4 and Crucial CT9994880 32GB). Comparatively there is heaps of cheaper server boards that support ECC with a xeon on the intel side, especially if your willing to again, look at non-current gen parts (and haswell or newer consumer socket xeons should support everything you need).

5

I’m not the one paying for power currently, and regardless. It’s very inexpensive where I am.

Looking to get a reliable setup as possible. I do have a few low-power options in mind regardless.

The SoC options I looked into, have Intel quad-port NIC integrated (good), but don’t support ECC RAM. If I do end up going that route I might just save my 4-port NIC for a AMD system to get away from Realtek.

I might look into older Xeon server boards as well, like you said, you don’t need much for even a gigabit connection. The core i3 setup I was using, having two 2015 Macbook Pro’s with Ethernet, one being WAN and the other being LAN. I could do 850mbps worst case.

Thanks.

1 Like
6

Never thought about virtualizing on dedicated hardware. That actually sounds like a nice idea.

For a low power system I just build my new router on a Xeon E3 1220L v3 that I ordered on ebay for 30,- or 40,- bucks. Here is a parts list.

1 Like
7

im using a g4400t (35W version of the g4400) on a asrock server board; it supports everything pfsense needs such as AES-NI . it also supports ECC ,although im not running ECC ram atm.

8

There’s a fantastic bit of kit called Suricata (and intrusion detection/prevention system) - it can even guard you from hitting potentially malware sites with an ‘online’ DB that constantly updates it. It needs AES-NI to the point that it needs power too. Even throwing a Ryzen R3 or R5 won’t be completely nuts…

TL;DR if you want to play with this useful add-on, go for a bit more power. And RAM too.

9

To be fair, in suricata they did demo that running at 10 gigabits per second off a older xeon E5-2680 0 @ 2.70GHz (16 threads 8 cores but sandy bridge, and only 2/3rds the clockspeed of the newer higher IPC parts), but thats about equivalent performance (allcore passmark 12,700) wise to the modern consumer i7’s (7700k passmark 12,000 at stock speeds, the ryzen 1600x would also do at 12,350 and supports ecc). I get its not perfect scaling, but even the ryzen3 1200 has a passmark score of around 6,835, which is literally 5x more than you need if it where. A ultra low power xeon or multicore atom (yep, they do have AES-NI in some) would seriously still handle being a Suricata router, its demands on storeage if your doing a complete rolling traffic log are much more notable than the demands on CPU these days.

The move from snort based to dedicated ruleset was a game changer and one of the many reasons people dramatically over estimate suricata’s needs, but even under the old system typical xeon cores where doing 200-500mbps each. The one other thing that catches people out is you need to build to your incoming WAN connections theoretical speed, not your average actual WAN speed, as your average 300Mbps may include some speed drops and bursts all the way up to the max connection speed of 1000Mbps.

1 Like
10

Im sorry, I dont know that.
I built it as a home server with limited budget, and ECC RAM is expensive so ECC wasnt of that much importance to me, although I would like to have that feature too.
If you want ECC support you probably should look after a board which officially supports it.
The MSI X370 SLI Plus is almost identical to the MSI X370 Gaming Plus:
https://geizhals.at/?cmp=1616803&cmp=1591066
Wendell did a Review of the MSI X370 SLI and on 22 April 2017 it did NOT support ECC on the Bios.

Latest Bios as of today.

USER@pve:~$ lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1450
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 1451
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467
03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43b9 (rev 02)
03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b5 (rev 02)
03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b0 (rev 02)
04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
04:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
04:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
04:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
04:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
04:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
1e:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
22:00.0 USB controller: ASMedia Technology Inc. Device 2142
23:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770/8760 / R7 250X]
23:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
24:00.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
25:02.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
25:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12N3A PCI Express Switch (rev 0e)
26:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
26:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
27:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
27:00.1 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (Copper) (rev 06)
28:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
28:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
28:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] USB3 Host Controller
29:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
29:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
29:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457
USER@pve:~$

What do you mean with “‘host’ CPU type with the OPNsense VM”? Would you explain it to me, please? English is not my native language :sweat_smile:

USER@OPNsense:~ # dmesg -a | grep Features
Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
Features2=0xfff83203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV>
AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
AMD Features2=0x3f3<LAHF,CMP,CR8,ABM,SSE4A,MAS,Prefetch,OSVW>
Structured Extended Features=0x201c012a<TSCADJ,BMI1,AVX2,BMI2,RDSEED,ADX,SMAP,SHA>
XSAVE Features=0x7<XSAVEOPT,XSAVEC,XINUSE>
USER@OPNsense:~ #

11

Your software setup is an order of magnitude less reliable than ram, typically.

Also, ram that’s not used heavily tends to bit flip less.

Don’t get me wrong I like the idea of ECC, I think non-ECC ram shouldn’t exist, one extra chip is a small price to pay, for the extra peace of mind, but you’d need it more in a workstation than in a router.

12

I don’t trust Ryzen for anything mission critical.
Just stick with Intel. Here’s a cheap system ($170) that would make a great pfSense system. You may need additional network cards depending on your hardware.
13888996_vgallery01_DC_gl_5173247
http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=5038747

13

I have the CPU type set in Proxmox for my pfSense virtual machine set to “host” so AESNI and RDRAND are passed through, so that the VM can use those functions natively, similar to if opn/pfsense was installed as the native OS.

Regardless I see AESNI and RDRAND in the output of that command, so it’s working :slight_smile:

14

Your exactly right, and it would be “nice to have” really, not 100% critical. Just been bitten by strange RAM issues in the past with various systems.

Regardless of ECC RAM or not I’d run Memtest86 and Prime95 for 24 hours each so that should catch the majority of issues.

15

Now if that was a rackmount system…

Hmm. Wonder if this and an i3-6100 would have working IOMMU passthrough with ESXi/Proxmox? (so I could passthrough my NIC to the pfSense VM)

16

On the above note. I tested enabling IOMMU on my server that has a SuperMicro X11 board and a Xeon E3-1230 V5 and running the IOMMU group listing script shows the various IOMMU groups. For kicks I’ll go test it later :slight_smile:

root@angrybear:~# for d in /sys/kernel/iommu_groups/*/devices/*; do      n=${d#*/iommu_groups/*}; n=${n%%/*};     printf 'IOMMU Group %s ' "$n";     lspci -nns "${d##*/}"; done;
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Device [8086:1918] (rev 07)
IOMMU Group 10 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a149] (rev 31)
IOMMU Group 10 00:1f.2 Memory controller [0580]: Intel Corporation Device [8086:a121] (rev 31)
IOMMU Group 10 00:1f.4 SMBus [0c05]: Intel Corporation Device [8086:a123] (rev 31)
IOMMU Group 11 01:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 12 02:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 13 03:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 14 04:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 15 05:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 03)
IOMMU Group 15 06:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 30)
IOMMU Group 1 00:13.0 Non-VGA unclassified device [0000]: Intel Corporation Device [8086:a135] (rev 31)
IOMMU Group 2 00:14.0 USB controller [0c03]: Intel Corporation Device [8086:a12f] (rev 31)
IOMMU Group 2 00:14.2 Signal processing controller [1180]: Intel Corporation Device [8086:a131] (rev 31)
IOMMU Group 3 00:16.0 Communication controller [0780]: Intel Corporation Device [8086:a13a] (rev 31)
IOMMU Group 3 00:16.1 Communication controller [0780]: Intel Corporation Device [8086:a13b] (rev 31)
IOMMU Group 4 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:a102] (rev 31)
IOMMU Group 5 00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a110] (rev f1)
IOMMU Group 6 00:1c.1 PCI bridge [0604]: Intel Corporation Device [8086:a111] (rev f1)
IOMMU Group 7 00:1c.2 PCI bridge [0604]: Intel Corporation Device [8086:a112] (rev f1)
IOMMU Group 8 00:1c.3 PCI bridge [0604]: Intel Corporation Device [8086:a113] (rev f1)
IOMMU Group 9 00:1c.6 PCI bridge [0604]: Intel Corporation Device [8086:a116] (rev f1)
17

In addition to memtest86+ and prime, aida64 stresstest looks good (to verify thermals)

1 Like
18

Actually, I just remembered why I never wanted to deal with virtualizing my router again. That is why my router has a hot-swap bay so if I want to try out a different software or config, I’ll just use another drive for it.

1 Like
19

Virtualizing pfSense or really any critical networking gear can be a headache. If you want to go Ryzen for the build, I would run it bare metal especially since you said this is dedicated hardware. As @noenken said, it can cause all sorts of issues and complicates troubleshooting due to the hypervisor. With that being said, I’ve also seen it run beautifully as a VM.

20

It’s not that hard to look up:


tl;dr: yes

1 Like