Well according to this documentation NHRP via FRR is not available for FreeBSD. 😞

http://docs.frrouting.org/en/latest/overview.html#feature-matrix

@mdomnis I have since upgraded to 22.01 with FRR version 1.1.1_6. In my preliminary testing, the routes seems to be working closer to what is expected. I still have a weird issue where sometimes the neighbors don't like to peer fully and I have to force restart FRR, but from some quick tests, it looks like at least the route is being added to the table correctly. For now at least.

As an update, I have done some more troubleshooting on the issue:

Switching to static routes over the VTI tunnel works. Using regular tunnel IPv4 also works Its only when we use FRR via OSPF (have not tested BGP) that traffic does not flow between hub and spoke.

Topology is 1 Hub (virtual) with 3 spokes (2 virtual, 1 physical pfSense). Its the physical pfSense spoke that is having issue

Enable IPsec MSS Clamping with different values, 1400, 1350, 1200, etc. on both hub and spoke and no issue. Also adjusted the VTI MTU value as well with no luck

Both sides are using AES-NI CPU Crypto. Enable/Disabling this has no effect

Both sides are using IPsec Asynchronous Cryptography. Enable/Disabling this has no effect

Tried different P2 encryption options but no luck. Currently using

P1: AES128-GCM (128 bits) AES-XCBC via 14 (2048) DH Group P2: ESP AES128-GCM (128 bits) PFS Group: 14 (2048). NO Hash algorithms

It appears another user on Reddit is facing similar issues: https://www.reddit.com/r/PFSENSE/comments/mzab6v/251_and_ipsec_vti/

Any ideas why FRR and OSPF is not sending traffic over the network? What troubleshooting steps can I take to debug this further?

So, nothing did Help that I searched yet...
Well... seems like something within the System was corrupted...
I reinstalled everything from ISO and did the config from ground up. Now its working.

Meanwhile i tried your 2nd suggested workaround, and after a while i got it to work.

What have i done?

turned off redistribution of connected networks (be careful, you might loose access to the device) under "OSPF Areas", i created Area 1 with the ID of 0.0.0.1 entered 10.1.1.0/24 under "Route Summarization" -> "Summary Range" -> "Summary Prefix
", this matches the subnet entered to OpenVPN under "Tunnel Settings" -> IPv4 Tunnel Network under "OSPF Interfaces" i set the ovpn interface to be in Area 1 marked it as "Interface is Passive", because vpn clients do not need to participate in OSPF and i changed the network type from "Not specified (default)" to "Point - multipoint"

With this setting, on the LAN side the Catalyst L3 was able to see 10.1.1.0/24 advertised from the FW, and only that subnet was advertised. The firewall was able to see all advertised routes from LAN from the beginning (after auth and a few basic thing was set up).

If i left the interface type on default or set it to point-to-point, there was nothing advertised from Area 1 , other types seemingly did the trick. From the working ones i picked P-MP which sounds OK for the VPN clients subnet.

If i removed the summary from Area 1 config, and the if type was "p-mp" or any of the working iftypes from aboove, there was only a /32 host route announced with the ovpn server address, despite a few clients were connected. The iftypes which yielded no redistribution, still remained silent irregardless of the value of the summary network.