Selfhosted

I've dabbled in VyOS and Opnsense, but ran Pfsense for close to a decade (which is basically Opnsense, but more proprietary). Between the two, I think Opnsense is likely your better option.

Updates: There is some nuance, but VyOS doesn't give you a LTS release so you will be taking quarterly upgrades to the latest. You are more or less a beta tester.

Opnsense has I guess one could consider "normal" releases with their community edition and is pretty easy to upgrade. You don't really expect breaking changes.

Tools: CLI is typically faster on VyOS (once you skill up) than the GUI for Opnsense. But Opnsense also provides CLI options to interact with it, so if you decide you want that later on you have it. Both have APIs to interact with too for automation later.

Firewall: VyOS is using netfilter underneath, aka things you may be familiar with like iptables and nftables. It's a fine firewall option, and is similar to what you could find in a normal Linux install (more on that later).

Opnsense I believe, uses the PF package (that's also where the original pfsense got its name). This is a package for FreeBSD, and in my 10+ years of using it, it has been reliable.

Hardware: VyOS being based on Linux will support more hardware out of the box.

Opnsense being based on FreeBSD may run into edge cases of not using your hardware correctly. (I had a thing on Pfsense (also FreeBSD) where it didn't like my built-in NIC and would flap constantly. I gave up trying to use it and used an external NIC instead)

Opinion: Honestly, if I was stuck with just these two as my choices I would probably lean toward Opnsense. It is more popular in homelab community, and there are many many guides on how to manage it.

Another option: You could also just using any other Linux distro and build up the firewall yourself. In the datacenter, it is very common to find big switches or firewalls simply running Debian on them with whatever firewall package the admin wants. It's also a great learning experience.

Offbeat option: Presented with the same inquiry as you, I actually switched to OpenWRT. Many people will tell you that it's just for APs or Routers, but I have been using it for my managed switches and my main firewall for about half a year now. OpenWRT is just Linux underneath, and you get the same CLI access as you would with a normal Linux distro. Plus you do get the option to use LuCI to mange it in the GUI.

It's not as shiny as the *senses but, it is certainly just as capable for this role in a homelab. Remember when I said I was on pfSense for years? I was able to port all of my firewall rules, VLANs, etc into OpenWRT just fine.

Performance on opnsense is fine for anything you'll be doing at home, so I wouldn't factor that into your choice.

Features on opnsense are probably better, just due to all the packages you can install.

Do you prefer being 100% in a CLI all the time? If not I'd say opnsense, the GUI certainly makes things a lot easier when you're not fully familiar with something and spending every day interacting with it.

Never used VyOS so can't help there. I do use OpnSense, TP-Link Omada EAP-650's (with an isolated vLAN for the guest network) multiple vLAN's for cameras, iot, management, Trusted devices, and DMZ, along with Wireguard for remote access and since my ISP only gives me an IPv4 address I use a Wireguard tunnel to Route64 for IPv6 connectivity, a cellular connection for backup internet connectivity, CrowdSec, Intrusion Detection, Caddy, and UnboundDNS.

I used multiple different Router OS' since around 2005 and settled on OpnSense years ago. I stick around because there is rarely an issue and the reporting system makes it easy to visually spot issues.

I would start with opnsense as the router and then maybe play with VyOS in a VM to see if you want to switch down the road.

Opensense is often bound by single-core operations depending on where a network packet may plumb itself through the kernel. A great example is any PPPoE connections on the base OS BSD. Vyos doesn’t have these same limitations since it’s minted on Linux - which supports traffic steering.

Really though, IaC for vyos has always been rock solid. Even though the persistent config in Opnsense is cool, adding new features or configs can be a bit of a pain in the ass

VyOS for performance but if you don't use a network device CLI regularly and it's for home use OPNsense.

OPNsense comes with a lot of pretty sane defaults for a gateway/firewall, VyOS you really do everything from scratch, as it's more geared towards enterprise or service provider.

VyOS does do well as a gateway/firewall and there is a lot you can install to extend features for stuff like what OPNsense has to the point that I have had 0 things I couldn't do when I migrated a few years ago.

Both are just abstractions over primatives and services find in the underlying operating system.

Moving to just native PF or nftables and running your firewall like a server has been on of my best moves. You will learn a tone more, have a lot more flexibility and overall I think it’s a much better experience.

I am a traditional network engineer as well (learned Juno’s/Cisco/etc). Ansible is a good way to do “commits” and config management.