Proper site to site routed openvpn setup

semiraue

Hi all,

I have two sites connected with single openvpn tunnel as follows.

Pfsense fw hosted in Digital Ocean cloud.
Site A Edgerouter-X behind broadband NAT
Site B Teltonika Rutx11 behind mobile cgnat

I setup this few years back to access the remote site B LAN from my local site A lan. this works without much issues to date.

But as I read this is not the proper setup as routing not properly works between two locations as those are in the same openvpn subnet (192.168.166.0/24)

I.e when I try to ping to either 192.168.55.0/24 or 192.168.77.0/24 networks from mobile phone with VPN on (192.168.166.13/24) I always get ICMP redirect message. which indicates potential routing issues or the some upstream network that I try to reach resides on the same network as I'm in now. Which is correct behavior based on the setup.

And even though this setup is working, I'm still confused how pfsense able to route traffic between gateways resides in the same subnet (192.168.166.2 and 192.168.166.3)

Now my issue comes when i try to do this proper way.

I created two openvpn p2p tunnels from both site A and B to Pfsense box. then I add appropriate static routes to the both ends. After this the routing works perfectly and there is no ICMP redirect messages when I try to ping from my mobile.

SITE A-> PFSENSE (172.18.18.0/30)
SITE B-> PFSENSE (172.19.19.0/30)

But the remote site B is connected to terribly bad mobile network, which is already slow without any VPNs. (I only use split tunneling. not all traffic go through VPN) but with this added site to site routed VPN setup, connections from site A to site B becomes unusable. All my camera live feeds are broken and most of the hosts behind site B router is not reachable or very slow to access.

I double checked everything like static routes/p2p IP ranges/firewall rules from all sites and all are correct. I really cannot get the reason behind this. Is the slowness because of the pfsense inter vpn routing ? or is there any other factor applying ?

What is the best way to accomplished proper routed connection between these two sites without compromising the performance or without involving any NAT ?

Appreciate your inputs

Thanks