PFsense large file transfer issues

Networking Networking Software
1

I just recently got pfsense up and running, and when copying an ISO over the network to another PC, my entire network will grind to a halt, using ALL the bandwidth for that ISO transfer. Once the transfer finishes, everything returns to normal, but its very annoying when I'm doing other things and the video I'm watching stops loading. I did some tests and I can't even ping other hosts while the transfer happens. The pfsense web interface won't open either.

What should I do? I'm assuming I need to place a limiter of some sort, but I don't know how to go about doing that...

2

Is the transfer between computers on the same network or on different networks? If it's the same network then this won't have anything to do with pfsense but if it's across networks then it is.

If it's on the same network then assuming your switch supports it you might be able to set up QoS on the switch. Try setting the ports which connect to your router to a higher priority than the other ports, that way any web traffic should take priority over local network traffic.

If it's across multiple networks then it's your router which is the bottleneck. Assuming you can't add more nics to it (one for each network) then your best option is setting up the traffic shaper. I haven't really messed around much with the traffic shaper but I have an idea for you. First you might want to read this, it's a pretty good explanation of the traffic shaper. https://forum.pfsense.org/index.php?topic=11986.0

 

Okay, so if your problem is with pfsense then this is my idea. This is just going to be a simple set up which will prioritise web traffic over local traffic, you can get more complex if you like but for now we'll just keep it simple. Some things to know about the pfsense traffic shaper are that it shapes only the upload traffic for each interface. This is from the perspective of pfsense, so from your computer's perspective it is the download bandwidth. So if you make a traffic shaper rule for an interface then it will only effect the traffic uploaded on that interface, if you want to shape the downloads too then you need to make a rule for the other interfaces. The problem with this is that if you want to set this up to prioritise certain types of web traffic and you have multiple LAN networks then the download shaping will be for each LAN network, as in it won't be a total limit across all your networks. so say your internet speed is 50 down 20 up and you set up your traffic shaper rule so that the WAN upload is set to 20 (so this is your internet upload speed) and the other interfaces are set to 50, then each interface will think it has 50mbps rather than them sharing a total of 50mbps. 

But that problem doesn't really matter in this case, it's just for if you wanted to use the traffic shaper as a QoS on a multi LAN network. For your set up, to prioritise web traffic over local traffic follow these steps:

1) Go to firewall > traffic shaper. You should see all your interfaces on the left

2) For each interface click on the interface and select the enable box at the top

3) choose PRIQ for the scheduler type, this is the simplest one and for this it should work fine.

4) Put in your bandwidth here, you should set it to about 95% of whatever you maximum bandwidth is. It wil probably be 1gbps, but because you have your gigabit nic split across multiple vlans it might be less, so you might need to play around with this value, but for now put in 950mbps.

5) for queue limit try something like 50 and click save at the bottom.

6) Click on the interface again and click add new queue

7) Check the enable box for this queue and name it something like qDefault. This will be the default queue that any traffic which is not put in to another queue will be put it to. with PRIQ you have 7 priority levels with 7 being the highest priority. We're only going to make two queues so choose something like 3 for the default queue. set the queue limit to 50 again. Check the default queue box. Give it a description if you want and click save.

8) Click on the interface again and click add new queue. Do the same as you did for the default queue but name this one something like qLocal and set the priority to 1. Set the queue limit to 50 but don't check any of the boxes. Add a description if you want and click save.

9) do the same thing for each of your local interfaces, make sure you use the same queue names.

10) So now each interface should have a qDefault and qLocal queue. Go to firewall > rules and open up the floating rules tab

11) Create a new rule. Set the action to match. Select all of your local interfaces (everything except WAN) for the interface. Set direction to any. Set TCP/IP version to IPv4 (unless you have ipv6 set up you shouldn't need to make a IPv6 rule). For protocol choose any. For your source address use your local networks alias, and also use it for the destination address. Go down to the advanced options and next to Ackqueue/Queue click advanced leave the first box as none and choose qLocal for the second box. Now click save.

12) So now all traffic passing from one network to another will be put in to the qLocal queue and all other traffic will be in the qDefault queue. Because qLocal is a lower priority than qDefault you should be able to use the internet without problems while transferring files between networks. To check that it's actually working you can go to status > queues and you should be able to see the traffic in each queue.

3

Ok, I think I've got the concept, I'm going to have to play with the values a little bit though. What exactly is the "Queue Limit" for? Is that measured in MB/s, or is it even what it sounds like?

4

I think it's how many packets it will keep in the queue before it starts dropping them. I haven't really messed around with it so I don't know what works well but I don't think you want to set them too high because you need it to drop packets in order for devices to know they need to slow down.

5

Something else to keep in mind is that you're better setting bandwidth values lower than what you measure your bandwidth as. If you set them too high then the traffic shaper won't do anything because you'll be hitting limits on some other device before pfsense sees that you've reached the limit that you set in the traffic shaper.

6

Ok so to clarify, dropping packets shouldn't cause any actual loss of data, its just a way of letting devices know that they need to throttle back a little bit? I'm just trying to make sure if I drop packets during a file transfer then its not going to like corrupt the ISO im uploading or whatever.

7

TCP/IP had built in error correction, so when a packet is dropped or lost or corrupted then it knows about it and sends it again. it also knows that if there are a lot of dropped packets then it needs to slow down to try and avoid dropping packets. TCP/IP is pretty cool on the way that it can get information reliably from point A to B without you needing to know what's going on in the middle. Anyway, the queues acts as a buffer, so pfsense will hang on to stuff in a lower priority queue until higher priority traffic has passed, if the buffer fills up packets start dropping and the connection will slow down anyway. So either way you get the same effect. Play around with the values a bit, I'm just saying you're not trying to never fill up the queue, set it for something which will handle small bursts but will fill up with sustained transfers. 

Also I was thinking, if you're transferring something between your network and your dad's network then you'll need to add his subnet to your local networks alias and add the wan interface to your match rule. 

8

Ok, thanks for clarifying. I got it straightened out, at least for now, what I ended up doing what putting a bandwidth limit on traffic coming into the file server on a certain port, and also a limit on the traffic coming from the WiFi. (I also did some queueing) and now I can watch youtube and upload those files without even noticing.

As far as my dad's network being in the alias, I discovered that myself while trying to create an isolated VLAN to put a virused computer on. (When my neighbor asked me to fix her computer, it was so messed up it wasne even funny, so I put it on its own VLAN to be safe) thanks for the reminder though. What I ended up doing to isolate the secured VLANs was creating a floating rule that applies to all local interfaces, and says that traffic from the alias "isolated" cant go to any place NOT in the "isolated" alias. It works flawlessly.

Thanks again

9

This is unrelated to the traffic shaper, but if my dad sets my router's internal IP as the DMZ, it just basically opens all ports to it as if it were on the public Ip, right? As long as it still goes through his internet filter he's fine with it, so I've got to figure out whether or not that all fits together correctly.

10

Yeah, I'm pretty sure that's how it works, you can test it by opening a port to something and see if you access it remotely. Otherwise you maybe have to open the ports on both routers. Having two NATs can be a pain. Also if you have a dynamic IP for your internet and you want to set up DDNS then you'll probably have to set it on your dad's router as I'm pretty sure that if you do it in pfsense it will use you wan address which isn't your public ip address. 

Remember that the default action of the firewall is to block everything, so you don't need to make a block for the isolated network unless it's included in an allow rule. So if you wanted the isolated network to be able to access the internet but nothing else then all you would need is two rules. The first is allow TCP and UDP, source any, destination isolated address (the address for the interface on pfsense) port 53 (dns). This rule let's the isolated network access the dns server, without it you can't use the internet properly. The second rule is the allow any to not local for accessing the internet. Now the only things it can access is pfsense to make dns requests and the internet. 

Having a rule like block isolated to not isolated is the same as block isolated to any because the router can't firewall traffic on the same network because that traffic doesn't pass through the router, except for traffic going to the pfsense interface, so you're better off using the interface address rather than the whole network. And again, you don't need to have a block isolated to any rule unless you have an allow any rule bellow it, because any traffic which is not expressly allowed is blocked by default. 

I'm just trying to help you understand it, I'm not saying you're doing it wrong, if it works it works after all. 

Also, I'm not really sure about your network layout but if you have vlans which have unrestricted access between each other,  like allow any from vlan1 to vlan2 for example, then you're better off just having them on the same network because it's just creating a bottleneck, especially if you are sharing a single nic for all the local interfaces. 

11

I actually set up an isolated VLAN using a rule set almost exactly the same as the one you mentioned first (neighbor had a really virused laptop she's paying me to fix and I knew it wasn't safe to put on my main network)

 

As far as VLANs being a bottleneck, the only two VLANs that are kind of redundant are my main LAN and my WiFi VLAN, I have captive portal set up on the WiFi though so it needed its own interface for that. The other VLANs I have are all isolated, except my VPN VLAN which only has the Ubuntu OpenVPN access server on it, along with its clients (wanted to be able to assign firewall rules to the VPN clients specifically, and I couldn't figure out how to get OpenVPN working in pfsense so I used OpenVPN AS)

I was wondering if you knew how to set up OpenVPN with pfsense? I got it set up and working with OpenVPN access server, but I couldn't figure it out on pfsense. Also its worth noting that my pfsense box only has 1GB of RAM so if that's not enough I can continue to run it as a VM on the virtualization server.

Thanks for the info about the DMZ, sadly it looks like I won't be doing that because my dad is afraid the PFsense box won't be secure if it has all incoming traffic going to it, plus the netgear website he read about DMZ on was talking about how it was dangerous to use because if the DMZ PC is compromised it can attack the whole network. So I don't think DMZ will be happening any time soon, luckily if I just want to play a LAN game with friends there's my OpenVPN for that.

Thanks again for the tips and the help, and also I saw your post about your neatly organized rack, that's awesome :D

12

I'd say the pfsense box is more secure than a consumer router. There's not much to worry about because until you set up port forwards then all traffic coming in to your WAN interface is blocked. Anyway, a hacker can't exploit a system just because it has an open port, the port has to be connected to something and that something has to be exploitable. If you're running a web server then you're going to get constant attacks from script kiddies and bots but as long as you keep everything up to date it's not really anything to worry about.

So I don't think there's much risk setting up your pfsense box as the DMZ. Things like games and bittorrent and stuff like that which needs an open port can't really be hacked, it's really only things like web servers and mail servers, stuff like that. If you do want to run a web server then do what I do and put it in a separate VLAN with strict rules. The ideal way to set it up would be to forward the ports to the computer (so for a web server this is TCP 80 and 443), this will also create an allow rule on the WAN interface. Then on the web server's VLAN you'd make a rule allowing it to access the internet with your allow to not local rule, if your dad's network is a part of the local alias then this rule will allow the web server to access the internet but not your local network or your dad's network, this way if it were to get hacked then they wouldn't be able to attack other computers on either your network or your dad's. Then you'd make a rule on your local networks allowing them to access the webserver on TCP port 80 and 443. The idea behind this kind of set up is that you open a port from the internet to a separate network, and then you access this network from your trusted network. So you have Trusted network - firewall - DMZ - firewall - Internet.

Do you want to set up pfsense as a VPN server or VPN client? Setting it up as a server is pretty straight forward, as a client is a little tricky. I'll run through the set up for using it as a server, or atleast i'll try to remember.

First thing you need to do is set up certificates. Go to system > cert manager. You should be on the CA tab, create a new CA, change the method to 'create an internal certificate authority' and fill out the details, you can just make stuff up for this but the descriptive name and common name should be the same.

Once you've made the CA go to the certificates tab, you'll need to make a certificate for the openvpn server. Create a new certificate and choose 'create an internal certificate' for the method. For the certificate authority choose the one you just created. for the certificate type choose server certificate. Fill out all the other details like you did for the CA.

Okay, so lets assume you have a CA called VPN-CA, and server certificate called VPN-cert.

Go to system > user manager. You'll need to create users for each vpn client. Create a new users, put in the name and password, you don't need to give them any permissions, check the box that says 'click to create a user certificate'. It'll ask you for a name, put in something like user-cert and choose the CA you created (VPN-CA for example) as the certificate authority then click save. Do this for each user you want to add.

Now go to VPN > OpenVPN and it should open on the server tab. Create a new Server.

Use these settings:

Server Mode: Remote access (SSL/TLS +User auth)

Backend for authentication: Local database

Protocol: UDP

Device Mode: Tun

Interface: WAN

Local port: 1194 (Use whatever you like here, 1194 is the default. If you want to get past a firewall that won't let you connect to a VPN server you can use 443 here, but you won't be able to use this port for a webserver as well as for the VPN server).

Description: Whatever you like

TLS Authentication: Check both boxes

Peer certificate authority: VPN-CA (or whatever you named the CA you created)

Peer ceretificate revocation list: None (unless you create a revocation list, then put that in here)

Server certificate: VPN-Cert (or whatever you named the server certificate you created)

DH parameter length: 1024

Encryption algorithm: AES-128-CBC (you can use a higher bit encryption if you like but it will be slower)

Hardware crypto: No hardware crypto acceleration (unless you have it)

Certificate depth: One (client+server)

 

Strict user/CN matching: Unchecked

IPv4 tunnel network: Put the subnet for the VPN network here in CIDR format for example: 10.1.4.0/24

IPv6 tunnel network: Leave blank unless you need it.

Redirect gateway: I have this unchecked and it works fine, so leave it as unchecked.

IPv4 local networks: Put in the subnets of the local networks you want to be able to access from the VPN, for example 10.1.1.0/24,10.1.2.0/24 etc. This isn't the firewall rule, this just creates the routes, you can put in all your networks here and then use the firewall to control access of you want.

IPv6 local networks: same as above, only use if you need it

Concurrent connections: put in whatever limit you want on the number of concurrent connections.

Compression: Checked

Type-of-service: Up to you, pretty sure this allows QoS to work on the encrypted traffic.

Inter-client communication: Enable if you want clients to be able to access each other, you'll want this for games.

Duplicate connections: enable if you want to be able to connect using the same username from different devices at the same time.

Dynamic IP: checked

Address pool: checked

Topology: I have this unchecked and it works, but I only have one client. 

DNS Default Domain: Enable if you want to use a domain name, use your DDNS address if you want or leave it disabled.

DNS servers: put in the first address of the subnet you chose for the VPN network, so if you have 10.1.4.0/24 as the subnet then put 10.1.4.1 for the DNS server, this will use pfsense's DNS server, you could also use the google DNS servers if you wanted but your local DNS server will work fine.

NTP servers: Enable if you want to set a NTP server.

now click save and it should work. Check that it has made the allow rule on the wan connection, it should do it automatically but if it hasn't then create a rule on the WAN interface to allow UDP any to WAN address port 1194 (or whatever port you decided to use).

If you have the openvpn client export utility package installed then you can go to the client export tab and export a preconfigured config file for each user, or you can just connect manually using the same settings.

13

I forgot to mention, if you use the client export utility then it automatically exports the user certificates but if you set it manually you will need to export those yourself, you can get them from going to system > cert manager, click on the certificates tab and click the third arrow at the end of the certificate you want to export, if you hover over it it will say 'export cert+key in .p12' although you might need to export the cert and key separately. I've only set it up using the client export utility so I'd recommend installing that from the package manager if it isn't already installed.

You can set up openvpn without client certificates aswell, it's easier but it's less secure. To do that go to the server settings and change the server mode to remote access (user auth). Now it will just use a username and password instead of also using client certificates. The rest of the settings will be the same and you still need to set up a CA and server certificate, but now you can create users without a user certificate.

14

Thanks for the OpenVPN tutorial, I'm going to try and read through that tonight.

As far as the pfsense box being more secure, I know, I tried to explain that to my dad, but for whatever reason he thinks that something is going to go wrong and PFsense will become virused and attack the entire network. He means well he just doesn't seem to make sense sometimes with this stuff, but for whatever reason he's afraid of me getting all the incoming traffic because of (like you said) the web servers could be attacked, never mind the fact that im not running a web server on the standard port 80...

Also, while setting up OpenVPN access server (which works great by the way) for whatever reason, my clients can only be pinged by me if they disable their firewalls, and they can't ping hosts on the 192.168.0.0/24 network, but they can access the webmin pages of those servers...any ideas? I think the issue is that its trying to ping a host on the clients local network, because my friends home subnet is the same as that, but then why can they get to the WebMin pages? Is there a certain something I have to set in the client machines to tell it to route ALL traffic through the VPN when connected? I have openvpn AS set to send the clients internet traffic through the VPN, but that doesn't fix this issue...

15

Ok well I followed your instructions, and got 90% of the VPN working. My friend can connect to it with the OpenVPN installer I got from the client export utility, and he gets an IP address, and I can see in PFsense that he is connected. Aside from that though, its useless, because he can't ping ANY other IP addresses, so I think maybe something is messed up in the gateway settings but I dont know where to find that....

The only thing I changed in your tutorial is I enabled the "force all client traffic through the VPN" option because I want EVERYTHING on his PC to go through my network, essentially so its just like he is over at my house, with a wired connection directly to the switch.

Any ideas? I'm lost...

16

Also, instead of creating a whole new subnet of IPs for the VPN clients, is there a way to just simply bridge the VPN clients into an existing interface? So that they are on the same DHCP server, subnet, and everything as me, instead of first going through the separated tunnel network?

17

Clients not being able to access your network: this will have to do with routes, you're server will either have to send the client the route information or they would have to set it up manually. Pfsense does this automatically when you set up the local network settings. Basically you need to tell the clients that all traffic bound for your subnet needs to go through the VPN. 

On pfsense you will also need to make sure that you have a firewall rule to allow traffic to your local networks from the VPN. It should automatically add an openvpn interface tab to the firewall when you set it up but it might be part of the wizard. If it has the tab make a rule allowing any to whichever networks you want to be able to access and do the same on those networks to allow access to the VPN. You could also add the VPN subnet to your local networks alias. 

I'm pretty sure if you enable the force all client traffic through the VPN setting then they won't be able to access their local network, and yeah I don't think it will work properly if your VPN subnet is the same as their local subnet, so pick something that is unlikely to be used. 

I don't think you can set it up to be part of another network. I'm pretty sure when you create the server it's basically setting up a new interface and treats it as such. 

Things to check: make sure you VPN subnet is something different to to what your friends local network uses, otherwise the traffic won't route properly. 

Make sure you have firewall rules to allow the traffic between the VPN and local networks, otherwise it won't go anywhere. 

I think that's all you need to fix in order to get it working, those instructions I gave you before were basically my settings and it all works fine for me. 

18

If you don't have the firewall tab for openvpn then try setting it up again using the wizard. It's been a while since I set mine up and I totally forgot how I did it :P

19

Ok, I think I forgot to set firewall rules on the other interfaces (that didnt occur to me that it would check there too). The VPN subnet is 10.0.0.0/24, my friends local subnet is 192.168.0.0/24, but the catch is that I've also got a subnet on my network that is 192.168.0.0/24. I can change that if need be, but thats got about 8 servers on it so if theres an easy way around that I'd rather not have to reset 8 static IP addresses.

I'm going to try and straighten out the firewall rules here shortly, I'll post again if anything goes wrong. Thanks for the help, there's a lot of things I just keep forgetting to think about (like firewall rules on other interfaces).

20

Wow--- this is a very good thread. Im favoriting this for later. 

Thanks all!