Combining pfSense and FreeNAS?

Jinxxter · April 8, 2014, 11:40pm

Hi there,

I got curious about FreeNAS and already experimented with it and also about pfSense which I'm currently sniffing into right now since I always liked the idea of using older hardware as well as an NAS as well as a router.

But since both are seperate FreeBSD-Distros my I'm currently a little clueless of how to combine those two to have them run on one machine.

I thought of sniffing into Linux and use virtual machines but I tried it with windows and am kind of lost of how to make the virtual pfSense-machine the dhcp-server for the FreeNAS-machine as well as the windows-host I run it on at the moment.

For the hardware I'm planning on using

- MSI 890FXA-GD70

- Phenom II X6 1100T

- 32GB OCZ Evo Leggera 2133 (4x8GB)

- 5x 1,5TB as RAIDZ (or maybe RAIDZ2 if the performance is right) for capturing WQHD with Dxtory)

For the NAS-part

- Voltaire HCA 400EX

For the Router-part

- 4-port Intel Pro/1000 (if I can get it running with Link-Aggregation then with an 8-Port-Switch)

- (Maybe) an AVM USB-WLAN-Stick or my Fritz!-Box 7390 as wireless Access-Point

To get it to work smoothly I need some advice/help on some issues I'm not able to sort out at the moment:

- Is there a way to combine FreeNAS and pfSense on ONE machine (i.e. using virtual machines)? If so how would the virtual LAN-environment have to be configured so that pf-Sense takes over the routing and DHCP instead of the host-machine?

- I've read through google but couldn't find out how to get Infiniband up and running on FreeNAS. There are a lot of threads in a lot of forums where they get the driver compiled fine but end up with a kernel panic and are stuck not finding out what's causing it

Oh and please don't hit me because up until now I have absolutely NO experience with compiling in FreeBSD.

My only expertise is when I fiddled around with Fedora Core back in 2005 but as of now I need to get my head into it.

Sould my ambitious plans of using one machine for both not be makeable I could use other older hardware and use seperate machines for each task. But I'd sure like have only one running if possible.

Thanks for your reading-patience and in advance for your help

Jinxxter

Freq_Labs · April 9, 2014, 2:30am

You maybe could run both as VMs with a FreeBSD host using bhyve I think. The only catch is that on AMD I'm not sure how well that's supported yet.

Probably a better option for now would be to use kvm or xen with Linux as the host. ESXi might be something as well, but I've never used it so have no idea whether that's a good idea.

Take a barebones install of some distro like debian or centos, install hypervisor of choice. A vm for each FreeNAS and pfSense. Configure the pci devices to pass through to their respective guests. I think this should be supported in hardware by the 890 chipset on your board.

For networking.. hmm a picture is worth a thousand words

Please try to forgive my horrible color scheme, etc.

Jinxxter · April 9, 2014, 7:07am

HI,

thanks for the quick replay and the awesome input, much appreciate it.

The scheme helps a light year.

I'm kind of ost though with what tap0 and tap1 are but I recon I'll find out when fiddling with kvm and/or xen.

Thanks alot so far I'll post here if it works out.

Greetz

Jinxxter

Freq_Labs · April 9, 2014, 8:19am

I did some testing to confirm, FreeBSD bhyve for the host is definitely a no-go at this point in time on AMD. I'll try to test Linux with both XEN and KVM tomorrow. It should work without a hitch. I think I'll also work the dual NICs on your motherboard into the network setup as well. If you have any more specifics about what else would be on the network, I can weigh where to best put them to use for your needs. Maybe freeing up that first port on the switch would be a good start.

Tap interfaces are like software loopback that can be used to attach to a virtio-net device in the guest.

Jinxxter · April 9, 2014, 10:33am

Well my idea would be

- Connecting to the Modem via onboard-LAN going directly into the virtual pfSense-environment via passthrough (Do I understand passthrough correct if I assume that the Linux-Distro I run it in from that point on has no direct use of that LAN-Port anymore and the connection to the modem is strictly and only available for the virtual machine?)

- From there routing Internet access via the 4-Port-Intel-Card and LAG to the switch and from there onwards (not sure how to handle the WLAN-Portion though at the moment but I presume that's where I'll be hooking up the Fritz-Box to the switch and configuring as Access-Point or am I able to integrate my avm WLAN-stick into that bridge?)

- Connecting pfSense and FreeNAS via the virtual network inside kvm or xen (I presume that this happens through tap1 right?)

- Connecting the Infiniband-card (Voltaire HCA 400EX) to FreeNAS (The Infiniband-controller doesn't work in FreeNAS out of the box since it is not recognized so I'll need to find out whether I'm going to be able to make it work integrating drivers which a lot of guys haven't successfully done yet or getting it to run in Linux and virtually integrating it from there into FreeNAS with only FreeNAS having access to that connection) with the other end in my gaming-rig

Here's a scheme of how I'd like to set it all up:

The virtual connection between pfSense and FreeNAS would only be needed for updates concerning FreeNAS but it isn't really necessary for I could roll updates/upgrades via externally downloading the iso and doing it manually.

I hope the scheme is understandable.

Freq_Labs · April 9, 2014, 11:23am

Ok that helps a lot. I see now you're using the infiniband card to connect the nas to the network. I'm not familiar with infiniband and I just found out it's a network device, not a storage thing (I was thinking it was like SAS or something, oops).

I'm fiddling around with XEN right now on a test rig that is somewhat similar to your config:

FX-8350

M5A88-V EVO

Dual port Intel PRO/1000

I'm a bit limited by the motherboard chipset for my testing- it's an 880G which does not have an IOMMU and that makes PCI passthrough suboptimal, but it should work anyway.

You're correct in your understanding that PCI passthrough hides the device from the host. The guest VM talks through the northbridge chipset directly to the device on the PCIe bus, without going through any translation in the host software stack (except in my case without an IOMMU).

The host will need to have a bridge as well. The virtual machines connect to this bridge in order to talk between each other. The bridge also acts at the host's connection to the network. I believe the virtio-net is a 10G link, so there should be plenty of bandwidth to spare there.

I'm still not entirely sure how the infiniband card will work. I'm not worried about recompiling freebsd or anything like that- I do that pretty much on a daily basis. It just seems like if the gaming rig is only connecting through the infiniband link, there will have to be another bridge in FreeNAS to connect it to the rest of the network. There's nothing wrong with that I guess, it just seems a little weird to use the NAS appliance as a bridge.

Regarding wireless, assuming the USB stick has drivers that work, there's no reason it can't be attached to pfSense. Heck, you might want to run both wlan devices, and that would be perfectly fine.

I should have pfSense and FreeNAS running soon and I can get started setting up a dev environment for building the infiniband module.

Jinxxter · April 9, 2014, 11:49am

Sorry, seems like I missed an arrow there.

The gaming-rig would be connected to the switch via the onboard-LAN and connected to the NAS via Infiniband.

My intention is to seperate internet- and NAS-access from each other.

Freq_Labs · April 10, 2014, 1:54am

Well, I have pfSense and FreeNAS running on top of Xen at the moment. I installed Debian Jessie as the host OS from the netinstall iso. I chose Jessie because it has Xen 4.3, with improved PCI passthrough functionality being the particular feature that swayed my decision. Installing bridge-utils and xen-linux-system-amd64 gave me everything I needed to get up and running. A few commands and config files later, everything seems to be working. I'm not doing any pci passthrough, because I'm using hvm machines, which aren't compatible with pv passthrough. Other than that, I have both machines running, connected to each other, and accessible from the network (on a vlan in my case because I already have a functioning network I don't want to disturb).

I'll get some example configs up here once I sort out a few things. Then I'll try the same setup but with KVM!

Freq_Labs · April 10, 2014, 4:34am

Here's some config hints for you: http://pastebin.com/sm3jDqVg

Freq_Labs · April 13, 2014, 7:26pm

https://bugs.freenas.org/issues/2014

You might have to come up with a different solution for the Infiniband card. If it works in Linux, you could maybe bridge it to the guest? I'll admit, I'm not the least bit familiar with how Infiniband works, so that may or may not be possible. I'm curious though, so I'll look into it.

Freq_Labs · April 13, 2014, 7:47pm

http://www.mellanox.com/pdf/presentations/xs0106_infiniband.pdf

Solution I looks pretty good... at least until IB support finally is functional in FreeNAS.

I don't have any IB hardware to try setting this up, unfortunately.

Jinxxter · April 30, 2014, 2:48pm

Hi there,

sorry for the long absence but work and a serious cold put me out of comission for quite some time and I just recently started to get into my project again.

First of all thank you for your awsome help there so far but I'm having serious trouble getting my thoughts into Linux and I haven't progressed very far at this time.

After fiddling around with Debian and trying to get the virtualization up and running I switched over to CEntOS because there I could have the virtualization installed and kvm aswell as monitoring runs just fine in there.

I currently have FreeNAS up and running and the NIC-passthrough works fine right out of the box aswell as the passthrough of the drives.

Using only GBit-Ehternet I get a transfer-rate between 95MBytes/s and 98MBytes/s which I think is not bad considering it's with a virtual machine.

I haven't looked into getting the Infiniband-adapter to run yet but that will be coming up in the next days.

And when that is up and running I'll move over to pfSense and hope it'll all work out the way I'm hoping it will.

Thanks again and greetings

Jinxxter

Jinxxter · April 30, 2014, 5:36pm

Hmm...

Anyone know what this is?

I'm currently trying to install the OFED-package. Everything went quite well so far.

I downloaded and extracted the package (OFED-3.5-2) and made sure that all required components are installed according to the installation-guide on the Openfabrics-page.

Starting the installation running "./install.pl" starts as described and I am able to pass the first two menus where I first select that I want to install the OFED-package and second which content from the package I want to install.

Upon that moment the installation hangs and I get the following messages repeating theselves very often:

-sh: Row 1: Command not found

-sh: Row 2: Command not found

Anyone have an idea where to start looking why that error occurs?

So far looking through the internet I found nothing in respect to that error.

Thanks in advance

Jinxxter

Freq_Labs · May 1, 2014, 12:02am

Make sure you have git, libtool, autoconf, automake, m4, and wget installed.

Jinxxter · May 1, 2014, 4:28pm

Thanks, seems like m4 and wget were already installed but the otherswere missing next to quite a few other packages which I found out.

I first tried to install the OFED-package in version 3.52 but that was not very communicative in terms of error-messages.

I now am trying out version 1.5.4.1 and I'm getting somewhat forward I believe.

At least I found out that some other packages were missing aswell and I was able to install them.

Starting to install the package works great and I get to the point where the package starts compiing but it then stops at some point.

In the corresponding Log at the end there I find

make[1]: Entering directory `/usr/src/kernels/2.6.32-431.11.2.el6.x86_64' test -e include/linux/autoconf.h -a -e include/config/auto.conf || (        \     echo;                                \     echo "  ERROR: Kernel configuration is invalid.";        \     echo "         include/linux/autoconf.h or include/config/auto.conf are missing.";    \     echo "         Run 'make oldconfig && make prepare' on kernel src to fix it.";    \     echo;                                \     /bin/false) mkdir -p /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/.tmp_versions ; rm -f /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/.tmp_versions/* make -f scripts/Makefile.build obj=/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1 make -f scripts/Makefile.build obj=/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband make -f scripts/Makefile.build obj=/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core   gcc -Wp,-MD,/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/.addr.o.d  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.4.7/include \ -D__OFED_BUILD__ \ -include /lib/modules/2.6.32-431.11.2.el6.x86_64/build/include/linux/autoconf.h \ -include /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/include/linux/autoconf.h \ -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/ \  \  \  \ -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/include \ -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/debug \ -I/usr/local/include/scst \ -I/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/ulp/srpt \ -D__XEN_INTERFACE_VERSION__= \ -I/usr/src/kernels/2.6.32-431.11.2.el6.x86_64/arch/x86/include/mach-xen \ -I/usr/src/kernels/2.6.32-431.11.2.el6.x86_64/arch//include \ -Iinclude \  \ -I/usr/src/kernels/2.6.32-431.11.2.el6.x86_64/arch/x86/include \  -D__KERNEL__ -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Wno-format-security -fno-delete-null-pointer-checks -O2 -m64 -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -maccumulate-outgoing-args -fstack-protector -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -DCONFIG_AS_CFI_SECTIONS=1 -DCONFIG_AS_AVX=1 -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -Wframe-larger-than=2048 -Wno-unused-but-set-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -g -pg -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fno-dwarf2-cfi-asm -fconserve-stack  -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(addr)"  -D"KBUILD_MODNAME=KBUILD_STR(ib_addr)" -D"DEBUG_HASH=45" -D"DEBUG_HASH2=52" -c -o /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/.tmp_addr.o /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/addr.c In file included from include/linux/tracepoint.h:18,                  from include/linux/module.h:18,                  from include/linux/textsearch.h:7,                  from include/linux/skbuff.h:27,                  from include/linux/if_ether.h:129,                  from include/linux/netdevice.h:29,                  from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/netdevice.h:4,                  from include/linux/inetdevice.h:8,                  from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/addr.c:37: /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/rcupdate.h:6:1: warning: "RCU_INIT_POINTER" redefined In file included from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/rcupdate.h:4,                  from include/linux/tracepoint.h:18,                  from include/linux/module.h:18,                  from include/linux/textsearch.h:7,                  from include/linux/skbuff.h:27,                  from include/linux/if_ether.h:129,                  from include/linux/netdevice.h:29,                  from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/netdevice.h:4,                  from include/linux/inetdevice.h:8,                  from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/addr.c:37: include/linux/rcupdate.h:288:1: warning: this is the location of the previous definition In file included from include/linux/netdevice.h:47,                  from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/netdevice.h:4,                  from include/linux/inetdevice.h:8,                  from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/addr.c:37: /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/ethtool.h:15: error: nested redefinition of 'enum ethtool_phys_id_state' /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/ethtool.h:15: error: redeclaration of 'enum ethtool_phys_id_state' /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/ethtool.h:16: error: redeclaration of enumerator 'ETHTOOL_ID_INACTIVE' include/linux/ethtool.h:560: note: previous definition of 'ETHTOOL_ID_INACTIVE' was here /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/ethtool.h:17: error: redeclaration of enumerator 'ETHTOOL_ID_ACTIVE' include/linux/ethtool.h:561: note: previous definition of 'ETHTOOL_ID_ACTIVE' was here /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/ethtool.h:18: error: redeclaration of enumerator 'ETHTOOL_ID_ON' include/linux/ethtool.h:562: note: previous definition of 'ETHTOOL_ID_ON' was here /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/ethtool.h:20: error: redeclaration of enumerator 'ETHTOOL_ID_OFF' include/linux/ethtool.h:564: note: previous definition of 'ETHTOOL_ID_OFF' was here In file included from include/linux/inetdevice.h:8,                  from /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/addr.c:37: /var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/kernel_addons/backport/2.6.32-EL6.1/include/linux/netdevice.h:26: error: static declaration of 'netif_set_real_num_rx_queues' follows non-static declaration include/linux/netdevice.h:1935: note: previous declaration of 'netif_set_real_num_rx_queues' was here make[4]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core/addr.o] Error 1 make[3]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband/core] Error 2 make[2]: *** [/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1/drivers/infiniband] Error 2 make[1]: *** [_module_/var/tmp/OFED_topdir/BUILD/ofa_kernel-1.5.4.1] Error 2 make[1]: Leaving directory`/usr/src/kernels/2.6.32-431.11.2.el6.x86_64'
make: *** [kernel] Error 2
Fehler: Fehler-Status beim Beenden von /var/tmp/rpm-tmp.ut6IH3 (%build)

Fehler beim Bauen des RPM:
 Benutzer vlad existiert nicht - benutze Root
 Gruppe vlad existiert nicht - benutze Root
 Benutzer vlad existiert nicht - benutze Root
 Gruppe vlad existiert nicht - benutze Root
 Fehler-Status beim Beenden von /var/tmp/rpm-tmp.ut6IH3 (%build)

The Log itself is way longer but only from the part I posted it starts showing errors and warnings.

Can you make something of it?

Am I sttill missing something?

Freq_Labs · May 1, 2014, 8:01pm

I'll spin a CentOS VM up and see where I get with this.

EDIT: Oh god, I didn't realize 6.5 was out. Installed from 6.4 discs, updates taking forever...

Freq_Labs · May 1, 2014, 10:45pm

If you can, try following these instructions:

http://www.softpanorama.org/HPC/IB/installing_mellanox_driver_for_rhel65.shtml

You have to be careful to download the right package from Mellanox, the first in the list is for 6.3, counterintuitively. Make sure to get the tar not the iso, as well. Or just try this link:

http://www.mellanox.com/page/mlnx_ofed_eula?mtag=linux_sw_drivers&mrequest=downloads&mtype=ofed&mver=MLNX_OFED-2.1-1.0.6&mname=MLNX_OFED_LINUX-2.1-1.0.6-rhel6.5-x86_64.tgz

RHEL and CentOS are binary compatible, so just ignore that minor difference in the instructions. You don't have to register anything, but do be sure to have the installation up to date.

UPDATE:

The Mellanox package installed fine on my VM. I think that's definitely the route you want to go.

Jinxxter · May 3, 2014, 1:25pm

I used the package you linked and it installed nicely but I'm now stuck with the probkem that it won't mount the adapter.

The problem seems to be related to the fact that my HCA 400Ex are having the MT25208-chip on them but won't be recognized by the driver. lspci is able to identify them properly though.

So I went on and thought that updating the firmware could do the trick but I'm having a hard time finding out which Voltaire-Firmwares and -revisions are matching the corresponding Mellanox-Firmwares and -revisions since on the download-page there are quite a lot to choos from and I don't want to ruin the card because of using a wrong firmware.

Extra problem to that is that it seems like the cards have been fashed with a custom flash necause when I use flint to read out the cards it says

Board-ID: my_VSD (VLT0020010001)

VSD: my_VSD

PSID: VLT0020010001

and so far Mr.Goggle wasn't able to find me a table where I can match PSID's to get forward.

I'll look further into it since I don't want to give it up now. At leas not yet.

Can't thank you enough for your help so far I appreciate it very much.

Freq_Labs · May 3, 2014, 4:55pm

With this I was able to identify your Voltaire card as the Mellanox card MHEL-CF128-T

Then it became a lot easier to find firmware:

http://www.mellanox.com/page/custom_firmware_table

Official fw-25208 package (2007) Release Notes

Some custom firmware stuff? Additional Lion Cub INI Files (2007)

Intermediate release fw-25208 v4.8.300 package (2008) Release Notes

You may have to use the older version OFED from the page I linked to in my previous post if the new one isn't working after you've flashed the cards.

Jinxxter · May 7, 2014, 11:34am

*gnarf*

it can be quite a pain in the **** if you have older hardware which older software only is able to work with.

I've downloaded the corresponding firmware and also set up a clean win xp to use with the ofed-1.1.drivers and this far all seems to work. The adapters are installed poroperly and work even with a win7-computer as counterpart. So far so good.

Thw WinMFT-software is giving me a headache though.

On the Mellanox-site there's only the most recent version available but that doesn't work with my HCA's.

When trying to query the device-ID's I get an error saying old hardware and when I try to burn the firmware regardless I only get an error invalid signature.

Will go and look further maybe I can find an older version of WinMFT.