I have all networks in firewall, for example:
chain=srcnat action=accept src-address=192.168.20.0/24
This is probably incorrect. You will probably have a source nat rule that masquerades for the outside world. You require some additional rules before this one that excludes traffic bound for your IPSEC reachable LANs. These will have action=accept and you will need to specify the destination address.
e.g.
chain=srcnat action=accept src-address=192.168.1.0/24 dst-address=192.168.111.0/24
chain=srcnat action=src-nat to-addresses=xxx.xxx.xxx.205 to-ports=0-65535 src-address=192.168.1.0/24 out-interface=Internet
In this example, the first rule excludes traffic bound for the remote LAN (192.168.111.0/24), the second rule nats everything else for the Internet, xxx.xxx.xxx.205 is one of my public IPs.
You need to think about routing end-to-end in both directions. Pick a remote destination address and for each client or router in the path, look at the routing table and ask yourself, 'where will this packet be sent next'. Once you get to the remote client successfully, repeat the process in the opposite direction.
Regards
Andrew