Building a 10Gig pfsence Machine

Hardware Build Logs
, ,
1

So I have Xfinity because that is the only choice for internet where I live. Other than super fast DSL. :rofl: :rofl: :rofl: :rofl: :rofl: So I cut the TV cord over 2 years ago but the more we streamed and the older the kids got I needed more upload speed. Without paying $300 a month and the massive install bill to get fiber run to my house. I ended up going with there gigabit service to get about 900 down and 35-40 up. Earlier this year they increased the download speed but forgot about the upload speed as usual. So the Gigabit is now 1.2 Gig. To actually get the speed they now provisioned my modem at I needed to get the ARRIS Surfboard S33 that has a 2.5Gig port on it. But my Ubiquiti EdgeRouter 4 only had 1Gig ports so I would never get the full speed even thought the rest of my house is 10Gig. So I started looking at pfsence but no one had a write up about a 10Gig pfsence build. So I did as much research as I could to make sure I got compatible NIC’s. I finally decided to pull the cord and get everything I needed. This build I know is probably way over powered but I will never have to worry about what I want to do with it.
BUILD
pfsence 2.5.0-RELEASE
Rosewill 4U Server Chassis (RSV-L4000) with Rails
GIGABYTE B450 AORUS ELITE V2 ATX
AMD AMD RYZEN 5 3600
G.SKILL 32GB 2X16 D4 3200 RIPJAWS
CRUCIAL 250GB P5 NVME M.2
EVGA 650GT 80+G FM PSU
ZOTAC GeForce GT 710 1GB DDR3 PCIE x 1
Ebay Intel X550-T2 10Gb Dual Port 10GBase-T
Ebay Chelsio T520-CR Dual Port 10Gb SFP+
CAT8 Ethernet Cable Cord Patch Cable
10Gtek SFP+ Twinax Cable, 10GB DAC

So I put it all together and put it in my server rack. I did the initial bios setup like I normally would have turned on xmp, set the time, and turned on SVM. I then saved and installed pfsence with a flash drive to the computer. It booted up no issues I then went through the auto recognize for which port each cable was plugged into. The X550-T2 is connected to my modem cause its just a normal ethernet cable NIC on the modem. The T520-CR is connected to my Unifi 10G US-16-XG. I got that initial setup done around 1am after starting the install of it in the rack at midnight. Didn’t want to listen to the kids complain there was no tv. I then spent 30 mins trying to figure out why I couldn’t get to the internet. I could pull up the web GUI for it but there was no outside connection. Then I realized I never restarted my modem or the the pfsence box. Once I restarted the modem and did a normal restart of the pfsence router everything came up with no issues. I set the Lan connection manually to 10G and MTU to 9000. The WAN side I had to leave at Default autoselect for it to connect at the 2.5G of the modem because there is no drop down for it and I set the MTU to 4500. After that configurations saved I did a speed test and got 1.35Gig down and 41 MB up. I though sweet everything is working so I can set up the last of the stuff I need to. I then proceeded to make the port forwarding for my Plex server and set up my Vlan for my guest network. At 2AM I thought sweet I have all the internet speed and went to bed. This morning I woke up to take 2 of my kids to school while my high schooler has remote learning and there was no internet. some time between 2:30am and 7am the thing crapped out. So I thought ok maybe it just needs a restart no big deal. But I had to turn off the power supply to shut it down which was a bad sign. But I did that gave it a minute turned it back on it booted up no issues but before I could even make it to my computer to check for internet or the router it had crashed again. The wifey emailed the oldest’s school saying we were having internet problems while i ran the other 2 to school. I reconnected the old router so I could get work down and so he could get to online class. I did some searching around and found some issues with crashing with this cpu and BIOS versions so I downloaded the most recent bios for this board figuring maybe something happened to it and just needs to be written again. The newest version was F61 the version on the board was F60 I noticed that there was no version F60 in the download for this board so I’m not sure where this bios was from. So I installed the latest bios and left all the presets alone no XMP and no SVM. I know that should have been one of the first things I did last night when I started it up. But tired brain forgets things. After installed the bios it started up no crashing I left it running for about an hour without being hooked up to the network. At lunch i did the change over. Restarted the modem restarted the pfsence box everything came up and has been working correctly ever since. Were at 4 hours of uptime so far.

I really hope ASRock Rack come out with there m.2 VGA adapter so I don’t have to switch the ZOTAC GeForce GT 710 1GB DDR3 PCIE x 1 between this and my TrueNAS machine for setting up stuff. I would get 2 of them right away.

After this i will always double check the BIOS versions to make sure i am on the latest.

1 Like
2

Reading through the post, it seemed like an issue with power states? Vanilla PFsense uses very little CPU the majority of time so maybe the bios is doing something to save power PFsense doesn’t like. I hope you got it solved with the bios update

I use pfsense on a dell r330 with a X520-DA2 card. I don’t have >1Gb yet but do have the capability. It has been 100% stable even with a 10Gb and several VLANS. Unfortunately since I bought the server new, I can’t really comment much else on the hardware configuration since the only thing I did was take it out of the box, pop in a card, rack it, and install PFsense.

3

Do you plan to run PFSense as a VM or will it be baremetal on this setup? Saying its overkill is an understatement.

4

It’s a baremetal setup. Everywhere I read online says to never use it as a VM because too many things are required to be opened up for it to work as intended. I needed a board with PCIE 3.0 that had a 8X slot and a 4X slot if I could have gotten my hands on a Ryzen 5 3400G for a normal price I would have gone with that but finding those is like finding a 3080 right now.

5

crazy to use a 4U server as a router/gateway but if its working!

I was looking at doing something like this as a VM on my Cisco UCS 240 but ended up just picking up a Dell Micro Optiplex 7060 and modding a 2nd 1g nic into it. Now I have a nice 35w 1L mini PC doing all the heavy lifting.

6

Didn’t have much of a choice I needed to get a 10g nic that was capable of 2.5g that worked with pfsence and I wanted to use a SFP+ nic that I could hook to my switch and also have already installed if fiber becomes affordable where I am. Kinda wanted this to be the last upgrade to a router I ever did. Plus I share my plex with a bunch of my friends so this handles all that traffic with no issues at all.

7

It happened again last night it seams to only happen when there is nothing going on in the internet. i was good for 2 days and then today no internet until i reset the server PSU switch off give it a second then back on then power button. Think i need to look a the bios and check an few things to make sure its not trying to save power

8

Load motherboard defaults maybe? I dont think this is a PFSense issue as many run 24/7.

My gut tells me it may be a memory issue, you can also try running memtest for 24hs and see if any errors pop up.

9

It only happens at idle. So I did some looking and it sounds like Linux doesn’t like the way AMD lowers the power to just about nothing during idle cause this issue happens on a lot of motherboards. The fix seams to be to turn of the C6 power state and these crashes stop happening. I’ll have to try this once the kids are done with school today.

10

PFSense is BSD not Linux. They both use similar consuls and cmnds but they are not the same kernel.

I run 6 different linux machines on a mix of intel and AMD, and 1 PFSense on intel, none have power issues (except a very very old intel board and its a known issue Intel DP35DP w/ q6600).

I would check the RAM: idle or load, random issues often are RAM based.

11

How would i go about checking the RAM idle

12

https://www.memtest.org/#downiso

Memtest on a bootable USB it will check your ram with several different tests to see if there are any errors.

1 Like
13

Thanks i didn’t know Memtest would do idle tests too.

14

it wont specifically, but ram is never really idle, its constantly being refreshed as its a form of volatile memory. So while the value may not be updated the memory controller is constantly refreshing all the cells. If there are errors and bits are being flipped memtest will find it.

15

I performed the Memtest and came back with no errors. I was surprised. So i tried disabling all of the idle power control and C6 stuff and it has been working fine since. Up time so far is 3 days 1 hour

1 Like
16

awesome! One of those times where I am happy to be wrong.

17

Oh I know that feeling. Thanks for the ideas though. I’m just glad its all up and running and kicking some ass too.

18

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.