I have used VyOS for several years and Vyatta since about 2010. It is very fast and very reliable. The configuration language is similar to Juniper Networks products. VyOS was created when Brocade stopped providing open source versions of Vyatta, the last open source version was forked into the VyOS project. I have not found anything better on the open source market.
@BarkingMad is correct about the difference of focus, although I cannot speak to the difference in capability between the two since I do not use pfSense. VyOS has a concept of firewall zones and interfaces/networks are assigned to zones (i.e.
private). This does a great job of abstracting the rules from the zones so the administrator can focus on what the rules are supposed to do for a zone separately from which interfaces participate in the zones. At the end of the day, both of them just configure the Linux kernel firewall to behave as you have requested. Neither distribution can exceed the capabilities of the kernel firewall.
One thing that VyOS does that pfSense does not is the ability to configure Quagga/Zebra. This is routing stack that provides routing protocols such as RIP, OSPF, BGP, VRRP and others. An example of using these would be if you had three sites with unreliable VPNs between them. Router protocols send announcements over available links, allowing other routers to know what other sites can be reached at any given time. It's very different than what a firewall does, which is block or allow packets to pass.
The best thing to do is to try both, or at least read the parts of the manual that you believe you will need. Get familiar with the configuration syntax and do some searches for questions on google to see where you find better support for the kinds of questions you will need help with.