tunnel troubleshoot

tomislav91 · Fri Oct 02, 2020 7:29 pm

so guys, i am struggling quite some time with my tunnels(ipsec).
Status is established, but there is no traffic allowed, I must disable/enable to get that working.
So, what is first step, what to do?
What i tried?
I have several tunnels Mikrotik - Mikrotik, and I change it to Mikrotik - PFSense router and seems same.
hq have mikrotik and remote locations also, but those locations are connected through openvpn to pfsense. JUst to have image in head what is where connected and what are those tunnels for.
Also, i can make two MikroTik to have tunnels, one in HQ and another instead of Pfsense is in another HQ, so i can do it like that, but results are the same, in some random time, there is problemwith connecting to remote shops, no ping, cant connect to routers, etc.
So i just disable/enable.
I cant put some scripts to ping, because local subnet in HQ are on core switches.

sindy · Fri Oct 02, 2020 7:53 pm

A blind shot here would be a mismatch of PFS settings, causing the first rekeying of the SAs to fail (so the tunnel would work for just about 25 minutes after establishing if default lifetime=30m is set in /ip ipsec proposal).

Another blind shot would be that pinholes in some external firewall on the path between the two devices expire as no traffic passes through them for an extended period of time, but this would only be relevant if both IPsec peers are running on public IPs so the NAT traversal mechanism is not activated - keepalives are part of that mechanism.

For starters, can you post the configuration exports of the two Mikrotiks which suffer from this issue? Follow the hint in my automatic signature below to remove the sensitive information without breaking the internal logic of the configuration, i.e. substitute eventual public IPs and eventual domain names by the same strings in both exports.

tomislav91 · Sat Oct 03, 2020 12:46 am

A blind shot here would be a mismatch of PFS settings, causing the first rekeying of the SAs to fail (so the tunnel would work for just about 25 minutes after establishing if default lifetime=30m is set in /ip ipsec proposal).

Another blind shot would be that pinholes in some external firewall on the path between the two devices expire as no traffic passes through them for an extended period of time, but this would only be relevant if both IPsec peers are running on public IPs so the NAT traversal mechanism is not activated - keepalives are part of that mechanism.

For starters, can you post the configuration exports of the two Mikrotiks which suffer from this issue? Follow the hint in my automatic signature below to remove the sensitive information without breaking the internal logic of the configuration, i.e. substitute eventual public IPs and eventual domain names by the same strings in both exports.

lifetime on pfsense is 1800sec,and on mikrotik is 30 min.

relevant if both IPsec peers are running on public IPs

yes, both are directly public IPs on interfaces.

I think i hide everything and not copying not important stuff

# oct/02/2020 23:21:32 by RouterOS 6.46.6
# software id = QDC5-RJ3Y
#
# model = CCR1009-8G-1S
# serial number = 606B0516C93B
/interface bridge
add name=bridgeVN
/interface ethernet
set [ find default-name=ether1 ] name=ether1-mgt-backup speed=100Mbps
set [ find default-name=ether2 ] name=ether2-mng-int speed=100Mbps
set [ find default-name=ether3 ] name=ether3_HQCAM speed=100Mbps
set [ find default-name=ether4 ] l2mtu=1576 name="ether4 - Paradox" speed=\
    100Mbps
set [ find default-name=ether5 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full l2mtu=1572 \
    speed=100Mbps
set [ find default-name=ether6 ] speed=100Mbps
set [ find default-name=ether7 ] speed=100Mbps
set [ find default-name=ether8 ] speed=100Mbps
set [ find default-name=sfp1 ] advertise=10M-full,100M-full,1000M-full
/interface vrrp
add interface=ether8 name=vrrp_LAN
add interface=ether1-mgt-backup name=vrrp_WAN_1
add interface=ether2-mng-int name=vrrp_WAN_2
/interface vlan
add disabled=yes interface=ether5 name=vlan2-sw-mgt vlan-id=2
add disabled=yes interface=ether5 name=vlan10 vlan-id=10
add disabled=yes interface=ether5 name=vlan20 vlan-id=20
add disabled=yes interface=ether5 name=vlan30 vlan-id=30
add disabled=yes interface=ether5 name=vlan80 vlan-id=80
add disabled=yes interface=ether5 name=vlan100 vlan-id=100
add disabled=yes interface=ether5 name=vlan200 vlan-id=200
add disabled=yes interface=ether5 name=vlan201 vlan-id=201
add disabled=yes interface=ether5 name=vlan202 vlan-id=202
add disabled=yes interface=ether5 name=vlan210 vlan-id=210
add disabled=yes interface=ether5 name=vlan250 vlan-id=250
add disabled=yes interface=ether5 name=vlan300 vlan-id=300
add comment="   - internet" disabled=yes interface=sfp1 name=vlan301 vlan-id=\
    301
add comment="   - l3 vpn" disabled=yes interface=sfp1 name=vlan303 vlan-id=\
    303
add comment="ex intretnet 2" disabled=yes interface=sfp1 name=vlan307 \
    vlan-id=307
add comment="Mng internet" disabled=yes interface=sfp1 name=vlan337 vlan-id=\
    337
add disabled=yes interface=ether5 name=vlan400 vlan-id=400
add disabled=yes interface=ether5 name=vlan500 vlan-id=500
add disabled=yes interface=ether5 name=vlan600 vlan-id=600
add disabled=yes interface=ether5 name=vlan700 vlan-id=700
add disabled=yes interface=ether5 name=vlan800 vlan-id=800
add disabled=yes interface=ether5 name=vlan900 vlan-id=900
add disabled=yes interface=ether5 name=vlan950 vlan-id=950
add disabled=yes interface=ether5 name=vlan1000 vlan-id=1000
/interface list
add name=WAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip dhcp-server option
add code=43 name=unifi value=0x0104c0a83223
/ip firewall layer7-protocol
add name=torrent-wwws regexp="^.*(get|GET).+(torrent|thepiratebay|1337xto.to|i\
    sohunt|entertane|demonoid|btjunkie|mininova|flixflux|torrentz|vertor|h33t|\
    btscene|bitunity|bittoxic|thunderbytes|entertane|zoozle|vcdq|bitnova|bitso\
    up|meganova|fulldls|btbot|flixflux|seedpeer|fenopy|gpirate|commonbits).*\$\
    "
add name=torrent-dns regexp="^.+(torrent|thepiratebay|isohunt|entertane|demono\
    id|btjunkie|mininova|flixflux|torrentz|vertor|h33t|btscene|bitunity|bittox\
    ic|thunderbytes|entertane|zoozle|vcdq|bitnova|bitsoup|meganova|fulldls|btb\
    ot|flixflux|seedpeer|fenopy|gpirate|commonbits).*\$"
add comment="Block Bit Torrent" name=layer7-bittorrent-exp regexp="^(\\x13bitt\
    orrent protocol|azver\\x01\$|get /scrape\\\?info_hash=get /announce\\\?inf\
    o_hash=|get /client/bitcomet/|GET /data\\\?fid=)|d1:ad2:id20:|\\x08'7P\\)[\
    RP]"
/ip ipsec profile
add dh-group=modp1024 enc-algorithm=3des hash-algorithm=md5 name=profile_1 \
    nat-traversal=no
add dh-group=modp1024 enc-algorithm=3des hash-algorithm=md5 name=profile_2 \
    nat-traversal=no
add enc-algorithm=aes-256,aes-128 name=profile_3 nat-traversal=no
add dh-group=modp1024 enc-algorithm=aes-256,aes-128 name=profile_4
add dh-group=modp1024 dpd-interval=10s dpd-maximum-failures=15 enc-algorithm=\
    3des hash-algorithm=md5 name=profile_10
add enc-algorithm=aes-256,aes-128 name=profile_12 nat-traversal=no
add dh-group=modp1024 dpd-interval=10s enc-algorithm=aes-256 lifetime=8h \
    name=profile_13
add dh-group=modp1024 dpd-interval=10s dpd-maximum-failures=15 enc-algorithm=\
    3des name=profile_14
add dh-group=modp1024 enc-algorithm=aes-256 name=profile_15 nat-traversal=no
/ip ipsec peer
add address=SOME_WAN2/32 disabled=yes local-address=WAN1_ISP_PUBLIC_IP.253 name=\
    peer6 profile=profile_2
add address=SOME_WAN3/32 disabled=yes local-address=WAN1_ISP_PUBLIC_IP.253 name=\
    peer5 profile=profile_1 send-initial-contact=no
add address=routerHQ.main/32 comment=@vn local-address=secondIP.wan name=\
    peer87 profile=profile_15
add address=routerHQ.main/32 local-address=routerHQ.main2 name=peer13 \
    profile=profile_4
add address=routerHQ.main/32 comment=vpn disabled=yes exchange-mode=\
    aggressive name=peer86 profile=profile_14
add address=pfSense.IP/32 name=peer84 profile=profile_13
add address=SOME_WAN/32 name=peer9 profile=profile_3
add comment=Company exchange-mode=aggressive name=peer80 passive=yes profile=\
    profile_10
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-256-cbc,aes-128-cbc,3des \
    lifetime=5m
add auth-algorithms=md5 enc-algorithms=3des lifetime=1d name=proposal1 \
    pfs-group=none
add auth-algorithms=md5 enc-algorithms=3des lifetime=1h name=proposal2 \
    pfs-group=none
add enc-algorithms=3des name=Company
add enc-algorithms=3des name=Company2
add enc-algorithms=aes-256-cbc lifetime=1h name=NS2BG pfs-group=none
add enc-algorithms=aes-256-cbc name=NC-proposal pfs-group=none

/ip firewall connection tracking
set tcp-established-timeout=3h
/ip settings
set rp-filter=loose tcp-syncookies=yes
/interface l2tp-server server
set authentication=mschap1,mschap2 default-profile=l2tp_Company enabled=yes \
    use-ipsec=yes
/interface list member
add interface=vrrp_WAN_1 list=WAN
add interface=vrrp_WAN_2 list=WAN
/interface pptp-server server
set enabled=yes
/ip address
add address=routerHQ.main2/30 comment=INTERNET interface=vrrp_WAN_1 network=\
    WAN1-NETWORK
add address=WAN_PUBLIC IPs.dedicated comment=Public_IP disabled=yes interface=vlan10 \
    network=WAN_PUBLIC IP-NETWORK
add address=10.10.88.1/24 comment=MGT-BACKUP-LOCAL disabled=yes interface=\
    ether1-mgt-backup network=10.10.88.0
add address=hqIPs.126/28 disabled=yes interface=vlan30 network=\
    hqIPs.112
add address=WAN1_ISP_PUBLIC_IP.249/29 disabled=yes network=WAN1_ISP_PUBLIC_IP.248
add address=WAN1_ISP_PUBLIC_IP.253/29 comment=IPSec disabled=yes network=WAN1_ISP_PUBLIC_IP.248
add address=172.16.20.2/24 interface=ether1-mgt-backup network=172.16.20.0
add address=10.99.99.2/24 interface=ether8 network=10.99.99.0
add address=10.99.99.1/24 interface=vrrp_LAN network=10.99.99.0
add address=172.16.10.2/24 interface=ether2-mng-int network=172.16.10.0
add address=secondIP.wan/30 interface=vrrp_WAN_2 network=\
    secondIP.network
add address=192.168.90.1/24 interface=ether3_HQCAM network=\
    192.168.90.0
/ip arp
add address=192.168.50.235 interface=vrrp_LAN mac-address=9E:29:CC:D7:98:F8

/ip dns
set allow-remote-requests=yes servers=1.1.1.1,8.8.8.8
/ip firewall filter
add action=drop chain=forward comment="@Server drop Locations" \
    disabled=yes dst-address=10.0.0.0/8 src-address=192.168.50.144
add action=drop chain=forward comment="@Server drop Locations" \
    dst-address=192.168.200.0/24 src-address=10.98.0.0/24
add action=fasttrack-connection chain=forward dst-port=53 protocol=tcp \
    src-address=192.168.50.0/24
add action=fasttrack-connection chain=forward dst-port=53 protocol=udp \
    src-address=192.168.50.0/24
add action=drop chain=forward disabled=yes src-mac-address=2C:FD:A1:71:E3:B4
add action=drop chain=input disabled=yes src-mac-address=2C:FD:A1:71:E3:B4
add action=drop chain=input disabled=yes packet-size=200-65535 protocol=icmp
add action=accept chain=forward dst-address=10.11.0.0/16 src-address=\
    192.168.30.0/24
add action=accept chain=input dst-address=10.11.0.0/16 src-address=\
    192.168.30.0/24
add action=drop chain=forward disabled=yes packet-size=200-65535 protocol=\
    icmp
add action=jump chain=forward comment="SYN Flood protect FORWARD" \
    connection-state=new disabled=yes jump-target=syn-attack protocol=tcp \
    tcp-flags=syn
add action=jump chain=input comment="SYN Flood protect INPUT" \
    connection-state=new disabled=yes jump-target=syn-attack protocol=tcp \
    tcp-flags=syn
add action=accept chain=syn-attack connection-state=new disabled=yes limit=\
    400,5:packet protocol=tcp tcp-flags=syn
add action=drop chain=syn-attack connection-state=new disabled=yes protocol=\
    tcp tcp-flags=syn
add action=accept chain=forward disabled=yes dst-address-list=Remote_2_IPSEC \
    src-address-list=Local_2_IPSEC
add action=accept chain=forward disabled=yes dst-address-list=Local_2_IPSEC \
    src-address-list=Remote_2_IPSEC
add action=fasttrack-connection chain=forward disabled=yes
add action=drop chain=forward comment="Drop Inv." connection-state=invalid
add action=accept chain=forward dst-address=10.98.0.0/24 src-address=\
    192.168.2.0/24
add action=accept chain=input dst-address=10.98.0.0/24 src-address=\
    192.168.2.0/24
add action=accept chain=input in-interface=vrrp_WAN_1 protocol=icmp
add action=accept chain=input comment=\
    "permit from local ip DNS reuqest udp 53" dst-port=53 protocol=udp \
    src-address-list=local_ip
add action=reject chain=forward comment="@FB FORBID" dst-port=443 protocol=\
    tcp reject-with=icmp-network-unreachable src-address-list=!Social \
    tls-host=*.facebook.com
add action=reject chain=forward comment=@TeamVforbid disabled=yes dst-port=\
    443 protocol=tcp reject-with=icmp-network-unreachable src-address=\
    !192.168.0.0/16 tls-host=*.teamviewer.com
add action=reject chain=forward comment=@Anydesk disabled=yes dst-port=443 \
    protocol=tcp reject-with=icmp-network-unreachable src-address=\
    !192.168.50.0/24 tls-host=*.anydesk.com
add action=accept chain=forward dst-address=192.168.200.0/24 src-address=\
    192.168.23.0/24
add action=accept chain=forward dst-address=192.168.200.0/24 src-address=\
    172.16.2.0/24
add action=accept chain=forward dst-address=192.168.200.0/24 src-address=\
    192.168.30.0/24
add action=accept chain=input dst-address=192.168.200.0/24 src-address=\
    192.168.23.0/24
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=udp \
    reject-with=icmp-network-unreachable
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=tcp \
    reject-with=icmp-network-unreachable
add action=accept chain=forward connection-state=established,related
add action=accept chain=input connection-state=established,related
add action=accept chain=input comment=SSH dst-port=4777 protocol=tcp \
    src-address-list=HQ1addresses
add action=accept chain=input comment=SSH dst-port=4777 protocol=tcp \
    src-address-list=HQ2addresses
add action=accept chain=input comment=WinBox dst-port=8291 protocol=tcp \
    src-address-list=HQ2addresses
add action=accept chain=input comment=WinBox dst-port=8291 protocol=tcp \
    src-address-list=HQ1addresses
add action=accept chain=input comment="Allow ICMP" protocol=icmp \
    src-address-list=HQ1addresses
add action=accept chain=input comment="Allow ICMP" protocol=icmp \
    src-address-list=HQ2addresses
add action=drop chain=input comment="Drop Inv." connection-state=invalid
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=tcp \
    reject-with=icmp-network-unreachable src-address-list=!DNS
add action=reject chain=input comment="Drop !DNS" dst-port=53 protocol=udp \
    reject-with=icmp-network-unreachable src-address-list=!DNS
add action=accept chain=output comment="OUT-Pusti PMTUD" icmp-options=3:4 \
    protocol=icmp
add action=accept chain=input comment="IN-Pusti PMTUD" icmp-options=3:4 \
    protocol=icmp
add action=accept chain=input comment="IN-Propusti ping 1468b do 5 u sekundi" \
    limit=5,1:packet packet-size=1468 protocol=icmp
add action=add-src-to-address-list address-list=pingeri address-list-timeout=\
    1d chain=input comment="IN-Listuj ICMP koji ne matchuje kriterijum" \
    dst-address-list=!HQ2addresses in-interface-list=WAN log-prefix=Ping@IN \
    protocol=icmp src-address-list=!HQ2addresses
add action=add-src-to-address-list address-list=@Services_Phase1 \
    address-list-timeout=30m chain=input comment=IN-Services_Phase1 \
    dst-address-list=!HQ2addresses dst-port=21,22,23,69,80,443,5060,8080 \
    in-interface-list=WAN protocol=tcp src-address-list=!HQ2addresses
add action=add-src-to-address-list address-list=@Services_Phase1 \
    address-list-timeout=30m chain=input comment=IN-Services_Phase1-UDP \
    dst-address-list=!HQ2addresses dst-port=21,22,23,69,80,443,5060,8080 \
    in-interface-list=WAN protocol=udp src-address-list=!HQ2addresses
add action=add-src-to-address-list address-list=@Services_Phase2 \
    address-list-timeout=30m chain=input comment=IN-Services_Phase2 dst-port=\
    21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=tcp \
    src-address-list=@Services_Phase1
add action=add-src-to-address-list address-list=@Services_Phase2 \
    address-list-timeout=30m chain=input comment=IN-Services_Phase2-UDP \
    dst-port=21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=udp \
    src-address-list=@Services_Phase1
add action=add-src-to-address-list address-list=@Services_Phase3 \
    address-list-timeout=1w chain=input comment=IN-Services_Phase3 dst-port=\
    21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=tcp \
    src-address-list=@Services_Phase2
add action=add-src-to-address-list address-list=@Services_Phase3 \
    address-list-timeout=1w chain=input comment=IN-Services_Phase3-UDP \
    dst-port=21,22,23,69,80,443,5060,8080 in-interface-list=WAN protocol=udp \
    src-address-list=@Services_Phase2
add action=drop chain=input comment=IN-Phase3_dropRAW src-address-list=\
    @Services_Phase3
add action=drop chain=input comment="IN-Blokiraj Shodan" src-address-list=\
    shodan
add action=drop chain=input comment="IN-Brani se od pingera" \
    src-address-list=pingeri
add action=jump chain=forward comment="SYN Flood protect FORWARD" \
    connection-state=new disabled=yes jump-target=syn-attack protocol=tcp \
    tcp-flags=syn
add action=jump chain=input comment="SYN Flood protect INPUT" \
    connection-state=new disabled=yes jump-target=syn-attack protocol=tcp \
    tcp-flags=syn
add action=accept chain=input dst-port=161 protocol=udp
add action=accept chain=forward dst-port=161 protocol=udp
add action=accept chain=output dst-port=161 protocol=udp

/ip firewall nat
add action=dst-nat chain=dstnat disabled=yes dst-address=secondIP.wan \
    dst-port=16001 in-interface=ether2-mng-int protocol=udp to-addresses=\
    192.168.90.250 to-ports=16001
add action=dst-nat chain=dstnat disabled=yes dst-port=16001 in-interface=\
    ether2-mng-int protocol=udp to-addresses=192.168.90.250 to-ports=16001
add action=netmap chain=dstnat comment=@SomeServer disabled=yes dst-address=\
    hqIPs.119 protocol=tcp to-addresses=192.168.50.144
add action=dst-nat chain=dstnat dst-address=secondIP.wan dst-port=16001 \
    in-interface=vrrp_WAN_2 protocol=udp to-addresses=192.168.90.250 \
    to-ports=16001
add action=dst-nat chain=dstnat comment="@server!" disabled=yes \
    dst-address=hqIPs.119 dst-port=22 protocol=tcp to-addresses=\
    192.168.50.144 to-ports=22
add action=dst-nat chain=dstnat comment="@server!" disabled=yes \
    dst-address=hqIPs.119 dst-port=3306 protocol=tcp to-addresses=\
    192.168.50.144 to-ports=3306
add action=accept chain=srcnat dst-address=10.11.0.0/16 src-address=\
    192.168.30.0/24
add action=accept chain=srcnat dst-address=192.168.200.0/24 src-address=\
    10.98.0.0/24
add action=netmap chain=srcnat disabled=yes src-address=192.168.90.250 \
    to-addresses=secondIP.wan
add action=accept chain=srcnat dst-address=192.168.200.0/24 src-address=\
    192.168.36.0/24
add action=accept chain=srcnat dst-address=192.168.200.0/24 src-address=\
    192.168.30.0/24
add action=accept chain=srcnat dst-address=192.168.200.0/24 src-address=\
    172.16.2.0/24
add action=dst-nat chain=dstnat dst-address=192.168.0.10 dst-port=10003 \
    protocol=tcp to-addresses=192.168.200.11 to-ports=10003
add action=accept chain=srcnat dst-address=10.100.0.0/16 src-address=\
    192.168.50.0/24
add action=accept chain=srcnat dst-address=192.168.50.0/24 src-address=\
    10.100.0.0/16
add action=accept chain=srcnat dst-address=10.30.0.0/16 src-address=\
    192.168.50.0/24
add action=accept chain=srcnat dst-address=10.40.0.0/16 src-address=\
    192.168.50.0/24
add action=accept chain=srcnat dst-address=192.168.31.0/24 src-address=\
    192.168.90.0/24
add action=accept chain=srcnat dst-address=192.168.41.0/24 src-address=\
    192.168.90.0/24
add action=accept chain=srcnat dst-address=192.168.23.0/24 src-address=\
    192.168.90.0/24
add action=accept chain=srcnat dst-address=192.168.30.0/24 src-address=\
    192.168.90.0/24
add action=accept chain=srcnat dst-address=192.168.50.0/24 src-address=\
    192.168.90.0/24
add action=accept chain=srcnat dst-address=10.11.0.0/16 src-address=\
    192.168.90.0/24
add action=accept chain=srcnat dst-address=10.10.0.0/16 src-address=\
    192.168.90.0/24
add action=accept chain=srcnat disabled=yes dst-address=10.11.0.0/16 \
    src-address=192.168.50.0/24
add action=accept chain=srcnat disabled=yes dst-address=150.10.0.0/16 \
    src-address=192.168.50.0/24
add action=accept chain=srcnat comment=@Company-NS-IT dst-address=172.16.223.9 \
    src-address=192.168.50.0/24
add action=accept chain=srcnat comment=@Company-NS-IT dst-address=\
    192.168.90.0/24 src-address=192.168.50.0/24
add action=accept chain=srcnat comment=@Company-NS-IT dst-address=\
    192.168.50.0/24 src-address=172.16.223.9
add action=accept chain=srcnat comment=BgL2tp-NS_2.3_server dst-address=\
    192.168.5.0/24 src-address=192.168.2.0/24
add action=accept chain=srcnat dst-address=150.11.0.0/16 src-address=\
    192.168.0.0/16
add action=accept chain=srcnat dst-address=150.10.0.0/16 src-address=\
    192.168.0.0/16
add action=accept chain=srcnat dst-address=150.20.0.0/16 src-address=\
    192.168.0.0/16
add action=accept chain=srcnat dst-address=10.10.0.0/16 src-address=\
    192.168.0.0/16
add action=accept chain=srcnat dst-address=10.11.0.0/16 src-address=\
    192.168.0.0/16
add action=accept chain=srcnat dst-address=10.20.0.0/16 src-address=\
    192.168.0.0/16
add action=accept chain=srcnat dst-address=10.100.0.0/16 src-address=\
    192.168.50.0/24
add action=accept chain=srcnat dst-address=10.100.0.0/16 src-address=\
    192.168.23.0/24
add action=accept chain=srcnat dst-address=10.100.0.0/16 src-address=\
    192.168.90.0/24
add action=accept chain=srcnat dst-address=192.168.200.0/24 src-address=\
    192.168.50.0/24
add action=accept chain=srcnat comment="NO NAT LOCAL IP TO LOCAL IP" \
    dst-address-list=local_ip src-address-list=local_ip
add action=dst-nat chain=dstnat in-interface=vrrp_WAN_1 protocol=tcp \
    src-port=58291 to-addresses=10.99.99.3 to-ports=8291
add action=netmap chain=dstnat comment="@Server" dst-address=\
    hqIPs.119 to-addresses=192.168.50.144
add action=netmap chain=dstnat comment="@pfSense.IP - ServerSOME" dst-address=\
    hqIPs.115 to-addresses=192.168.50.20
add action=netmap chain=dstnat comment=mis disabled=yes dst-address=\
    hqIPs.117 to-addresses=192.168.50.32
add action=dst-nat chain=dstnat dst-address=hqIPs.114 to-addresses=\
    192.168.50.123
add action=dst-nat chain=dstnat comment="UnifyAP 4 Remote" dst-address=\
    hqIPs.116 dst-port=8443 protocol=tcp to-addresses=192.168.50.35
add action=dst-nat chain=dstnat comment="UnifyAP 4 Remote" dst-address=\
    hqIPs.116 dst-port=3478 protocol=udp to-addresses=192.168.50.35
add action=dst-nat chain=dstnat comment="UnifyAP 4 Remote" dst-address=\
    hqIPs.116 dst-port=8080 protocol=tcp to-addresses=192.168.50.35
add action=dst-nat chain=dstnat comment=betEx dst-address=hqIPs.118 \
    to-addresses=192.168.50.197
add action=dst-nat chain=dstnat dst-address=routerHQ.main2 dst-port=8080 \
    protocol=tcp to-addresses=192.168.2.3 to-ports=8080
add action=dst-nat chain=dstnat dst-address=routerHQ.main2 dst-port=9182 \
    protocol=tcp to-addresses=192.168.2.3 to-ports=9182
add action=dst-nat chain=dstnat dst-address=routerHQ.main2 dst-port=8081 \
    protocol=tcp to-addresses=192.168.2.3 to-ports=8081
add action=dst-nat chain=dstnat dst-address=routerHQ.main2 dst-port=9080 \
    protocol=tcp to-addresses=192.168.2.3 to-ports=9080
add action=masquerade chain=srcnat out-interface=vrrp_WAN_1
add action=masquerade chain=srcnat out-interface=vrrp_WAN_2
add action=masquerade chain=srcnat src-address=10.10.10.0/24
add action=masquerade chain=srcnat src-address=172.16.0.0/12
add action=masquerade chain=srcnat src-address=192.168.0.0/16
add action=dst-nat chain=dstnat comment=@SomeServer disabled=yes \
    dst-port=16001 in-interface=vrrp_WAN_2 protocol=tcp to-addresses=\
    192.168.90.250 to-ports=16001
add action=dst-nat chain=dstnat disabled=yes dst-port=16001 in-interface=\
    vrrp_WAN_2 protocol=udp to-addresses=192.168.90.250 to-ports=16001
/ip firewall raw
add action=drop chain=prerouting comment="IN-Drop Phase3Raw" disabled=yes \
    log=yes log-prefix=raw src-address-list=@Services_Phase3
/ip ipsec identity
add peer=peer5
add peer=peer6
add peer=peer9
add my-id=address:routerHQ.main2 peer=peer13
add peer=peer86
add peer=peer87
add my-id=address:192.168.50.99 peer=peer84
add peer=peer80
/ip ipsec policy
add comment=l2tp@cms dst-address=192.168.200.0/24 level=unique peer=peer13 \
    proposal=Company sa-dst-address=routerHQ.main sa-src-address=\
    routerHQ.main2 src-address=172.16.2.0/24 tunnel=yes
add comment=l2tp@cms disabled=yes dst-address=192.168.200.0/24 level=unique \
    peer=peer13 proposal=Company sa-dst-address=routerHQ.main \
    sa-src-address=routerHQ.main2 src-address=192.168.100.246/32 tunnel=yes
add dst-address=192.168.200.0/24 level=unique peer=peer13 proposal=Company \
    sa-dst-address=routerHQ.main sa-src-address=routerHQ.main2 src-address=\
    192.168.34.0/24 tunnel=yes
add dst-address=192.168.200.0/24 level=unique peer=peer13 proposal=Company \
    sa-dst-address=routerHQ.main sa-src-address=routerHQ.main2 src-address=\
    192.168.36.0/24 tunnel=yes
add disabled=yes dst-address=192.168.200.0/24 level=unique peer=peer13 \
    proposal=Company sa-dst-address=routerHQ.main sa-src-address=\
    routerHQ.main2 src-address=10.98.0.0/24 tunnel=yes
add comment="IT-SOME_SERVER INTERNAL" dst-address=192.168.47.0/24 level=unique peer=\
    peer13 proposal=Company sa-dst-address=routerHQ.main sa-src-address=\
    routerHQ.main2 src-address=192.168.50.0/24 tunnel=yes
add comment=BG2NS dst-address=10.100.0.0/16 level=unique peer=peer84 \
    proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 \
    src-address=192.168.50.0/24 tunnel=yes
add comment=BG2NS dst-address=10.100.0.0/16 level=unique peer=peer84 \
    proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 \
    src-address=192.168.30.0/24 tunnel=yes
add comment=BG2NS disabled=yes dst-address=10.100.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=10.98.0.0/24 tunnel=yes
add comment=BG2NS dst-address=10.100.0.0/16 level=unique peer=peer84 \
    proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 \
    src-address=192.168.23.0/24 tunnel=yes
add comment=BG2NS dst-address=10.100.0.0/16 level=unique peer=peer84 \
    proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 \
    src-address=192.168.2.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=10.10.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.50.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=10.50.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.30.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=192.168.5.0/24 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.50.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=10.10.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.30.0/24 tunnel=yes
add comment=BG2NS dst-address=192.168.200.0/24 level=unique peer=peer84 \
    proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 \
    src-address=192.168.50.0/24 tunnel=yes
add comment=BG2NS dst-address=192.168.202.0/24 level=unique peer=peer84 \
    proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 \
    src-address=192.168.50.0/24 tunnel=yes
add comment=BG2NS dst-address=192.168.200.0/24 level=unique peer=peer84 \
    proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 \
    src-address=192.168.23.0/24 tunnel=yes
add comment=BG2NS dst-address=192.168.200.0/24 level=unique peer=peer84 \
    proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 \
    src-address=192.168.30.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=10.11.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.50.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=10.11.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.30.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=10.30.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.50.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=10.40.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.50.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=10.20.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.50.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=150.10.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.50.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=150.10.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.30.0/24 tunnel=yes
add comment=BG2NS-IT2Shops disabled=yes dst-address=192.168.200.0/24 \
    level=unique peer=peer84 proposal=NS2BG sa-dst-address=pfSense.IP \
    sa-src-address=routerHQ.main2 src-address=10.98.0.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=10.10.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.2.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=10.11.0.0/16 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.2.0/24 tunnel=yes
add comment=BG2NS-IT2Shops dst-address=192.168.5.0/24 level=unique peer=\
    peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.2.0/24 tunnel=yes
add dst-address=10.10.0.0/16 level=unique peer=peer84 proposal=NS2BG \
    sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 src-address=\
    192.168.90.0/24 tunnel=yes
add dst-address=10.11.0.0/16 level=unique peer=peer84 proposal=NS2BG \
    sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 src-address=\
    192.168.90.0/24 tunnel=yes
add dst-address=150.10.0.0/16 level=unique peer=peer84 proposal=NS2BG \
    sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 src-address=\
    192.168.90.0/24 tunnel=yes
add dst-address=10.100.100.0/24 level=unique peer=peer84 proposal=NS2BG \
    sa-dst-address=pfSense.IP sa-src-address=routerHQ.main2 src-address=\
    192.168.90.0/24 tunnel=yes
add comment=SIP dst-address=172.16.223.9/32 peer=peer9 proposal=NC-proposal \
    sa-dst-address=SOME_WAN sa-src-address=routerHQ.main2 src-address=\
    192.168.0.0/17 tunnel=yes
add comment=OpenBG-MIS disabled=yes dst-address=10.50.0.0/16 level=unique \
    peer=peer84 proposal=NS2BG sa-dst-address=pfSense.IP sa-src-address=\
    routerHQ.main2 src-address=192.168.2.0/24 tunnel=yes
/ip route
add distance=2 gateway=WAN2_IPSS routing-mark=HQCAM
add distance=4 gateway=WAN2_IPSS routing-mark=raspTemp
add distance=1 gateway=PUBLIC_IPS.21 routing-mark=mng-int
add comment=ISP_WAN1 distance=1 gateway=82.117.203.61
add comment=ISP_WAN2 distance=3 gateway=WAN2_IPSS
add distance=1 dst-address=172.16.0.0/24 gateway=172.16.255.2
add distance=1 dst-address=172.29.0.0/24 gateway=10.10.88.2
add distance=1 dst-address=192.168.0.0/24 gateway=192.168.30.10
add distance=1 dst-address=192.168.2.0/28 gateway=10.99.99.254
add distance=1 dst-address=192.168.3.0/24 gateway=192.168.30.10
add distance=1 dst-address=192.168.12.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.21.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.22.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.23.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.24.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.25.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.30.0/24 gateway=10.99.99.254
add disabled=yes distance=1 dst-address=192.168.30.0/24 gateway=192.168.50.99
add distance=1 dst-address=192.168.31.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.32.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.33.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.34.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.35.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.36.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.37.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.41.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.42.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.43.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.44.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.50.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.100.0/24 gateway=10.99.99.254
add distance=1 dst-address=192.168.200.0/24 gateway=192.168.50.99
add distance=1 dst-address=192.168.240.0/20 gateway=192.168.50.2
/ip route vrf
add interfaces=ether2-mng-int,vlan337 routing-mark=mng-int

/queue simple
add disabled=yes max-limit=700M/700M name=internet2 queue=\
    ethernet-default/ethernet-default target=*4 total-queue=ethernet-default


/system identity
set name="Company CCR_Master"
/system leds
set 0 interface=sfp1
/system logging
add disabled=yes topics=ipsec
add action=remote topics=critical
add action=remote topics=warning
add action=remote topics=write
add action=remote disabled=yes topics=ipsec
add topics=critical
add action=echo disabled=yes topics=ipsec
add disabled=yes topics=pptp
/system ntp client
set enabled=yes primary-ntp=176.56.236.23 secondary-ntp=147.91.26.20

peer84 is MikroTik hq1- PFSENSE - most of the problems come from this peer
and peer13 is Mikrotik hq1- Mikrotik hq2 this peer is been used just for some subnets, but in the first time there was a tunnels only from this peer and also didnt work as well

mikahawkins · Sat Oct 03, 2020 6:25 am

Hey Tomislav91,
If you successfully establish both VPN tunnels but still experience connectivity issues, then: Check for network ACLs in your VPC that prevent the attached VPN from establishing a connection. Verify that the security group rules assigned to the EC2 instances in your VPC allow appropriate access.
Regards,
Mika Hawkins

sindy · Sat Oct 03, 2020 3:21 pm

lifetime on pfsense is 1800sec,and on mikrotik is 30 min.

I've only mentioned the lifetime as a troubleshooting hint - if the connections break in 80-100 % of the SA lifetime configured after the connection establishes, it makes sense to look at the PFS settings, as the first rekeying takes place at this time, and on that the mismatch of PFS settings causes the SA to become unusable.

If the pfSense can trigger SA rekey also on number of bytes transported (which RouterOS cannot), it may happen even sooner.

But as you've got pfs-group=none on all /ip ipsec proposal rows in the configuration of the single Mikrotik you've chosen to post, I assume that it's the same at the other Mikrotik(s) and therefore at least at the links between two Mikrotiks, this is not the reason why they fail.

yes, both are directly public IPs on interfaces.

OK. That means that the NAT traversal mechanism is not in use (you have even forbidden its use in the /ip ipsec profile settings), and this practically means the following:

whereas the control packets (IKE) are sent as UDP packets on port 500, the transport packets of the SAs are bare ESP packets
as no pinholes (tracked connections) are expected to need to be forcifully kept open, no keepalives packets are sent

As a consequence, if no traffic is sent using the SA, no ESP packets are sent. So if there is a firewall somewhere between the devices or on one of them which closes the pinhole if no traffic passes through it for some time, the pinhole closes. Since the traffic works for a while after connection re-establishment, ESP traffic in one direction must be capable of opening the pinholes on all the path, otherwise it would not work at all. But if the pinhole is closed and the request in the payload goes in the "wrong" direction, the ESP packet carrying it never makes it to the destination, so no response to that request is ever generated, and thus the pinhole never can be re-created (until eventually a request in the payload would be sent from the "proper" side).

The firewall at the router whose configuration you have posted is complex and leaky, it seems you have missed the fact that the default handling in all chains is "accept", i.e. packets which do not match any rule are accepted. It's best seen in chain=output where you have two action=accept rules but no action=drop one, which just burns CPU on every packet sent by the router but it has no actual effect - everything gets through anyway.

In the context of the above, this means that SA traffic initiated from this router is allowed to be sent (it is accepted in chain=output), creates a tracked connection (pinhole) as there is a couple of rules which refer to connection-state so connection tracking is activated, and therefore the packets in the opposite direction of the SA are accepted by the "accept established,related" rule in chain=input, so even if you added a "drop the rest" rule to the end of chain=input, the firewall at this router would still accept them. As there is no "drop the rest" rule in chain=input, nor any rule selectively dropping the ESP packets from the remote peer, this router's firewall doesn't explain why the IPsec connections break.

But it may not be the case on the other routers. That's why I've asked you for a configuration of both Mikrotiks between which you encounter the issue, not just a single one.

tomislav91 · Sun Oct 04, 2020 10:43 am

lifetime on pfsense is 1800sec,and on mikrotik is 30 min.
I've only mentioned the lifetime as a troubleshooting hint - if the connections break in 80-100 % of the SA lifetime configured after the connection establishes, it makes sense to look at the PFS settings, as the first rekeying takes place at this time, and on that the mismatch of PFS settings causes the SA to become unusable.

If the pfSense can trigger SA rekey also on number of bytes transported (which RouterOS cannot), it may happen even sooner.

But as you've got pfs-group=none on all /ip ipsec proposal rows in the configuration of the single Mikrotik you've chosen to post, I assume that it's the same at the other Mikrotik(s) and therefore at least at the links between two Mikrotiks, this is not the reason why they fail.

yes, both are directly public IPs on interfaces.
OK. That means that the NAT traversal mechanism is not in use (you have even forbidden its use in the /ip ipsec profile settings), and this practically means the following:
whereas the control packets (IKE) are sent as UDP packets on port 500, the transport packets of the SAs are bare ESP packets

as no pinholes (tracked connections) are expected to need to be forcifully kept open, no keepalives packets are sent
As a consequence, if no traffic is sent using the SA, no ESP packets are sent. So if there is a firewall somewhere between the devices or on one of them which closes the pinhole if no traffic passes through it for some time, the pinhole closes. Since the traffic works for a while after connection re-establishment, ESP traffic in one direction must be capable of opening the pinholes on all the path, otherwise it would not work at all. But if the pinhole is closed and the request in the payload goes in the "wrong" direction, the ESP packet carrying it never makes it to the destination, so no response to that request is ever generated, and thus the pinhole never can be re-created (until eventually a request in the payload would be sent from the "proper" side).

The firewall at the router whose configuration you have posted is complex and leaky, it seems you have missed the fact that the default handling in all chains is "accept", i.e. packets which do not match any rule are accepted. It's best seen in chain=output where you have two action=accept rules but no action=drop one, which just burns CPU on every packet sent by the router but it has no actual effect - everything gets through anyway.

In the context of the above, this means that SA traffic initiated from this router is allowed to be sent (it is accepted in chain=output), creates a tracked connection (pinhole) as there is a couple of rules which refer to connection-state so connection tracking is activated, and therefore the packets in the opposite direction of the SA are accepted by the "accept established,related" rule in chain=input, so even if you added a "drop the rest" rule to the end of chain=input, the firewall at this router would still accept them. As there is no "drop the rest" rule in chain=input, nor any rule selectively dropping the ESP packets from the remote peer, this router's firewall doesn't explain why the IPsec connections break.

But it may not be the case on the other routers. That's why I've asked you for a configuration of both Mikrotiks between which you encounter the issue, not just a single one.

But it may not be the case on the other routers

Firewall od the other router are the same, it has just more src or dst nat there, so rules which are for us interesting are the same. But now at this point, tunnel is established thorugh pfsense and there is no much rules on the WAN side except one that we are using for internal purpoeses

2020-10-04 09_37_46-Window.png

dropping the ESP packets from the remote peer, this router's firewall doesn't explain why the IPsec connections break.

How can i upgrade firewall on mikrotik side to tell me more for this problem. I am quite annoyed with this issue, and i am not sure even if tunnel is MikroTik-MikroTik or Mikrotik - PfSense still is a issue of connection to those peers. We are speaking about 350+ peers. Two Subnets are full 10.10 and 10.11 is at half, so you understand the math there. Also with 192.168.200, also is a problem with all subnets from HQ side 192.168.50, 192.168.23, 192.168.30 and so on.
Just to understand better, under MiKroTik L3 core switch and lower are l2 in the branches.

sindy · Sun Oct 04, 2020 12:16 pm

Firewall od the other router are the same, it has just more src or dst nat there, so rules which are for us interesting are the same. But now at this point, tunnel is established thorugh pfsense and there is no much rules on the WAN side except one that we are using for internal purpoeses

What tunnel are you talking about above? You've said that the issue exists on multiple ones or even on all, so why do you now concentrate on one which uses a pfSense?

How can i upgrade firewall on mikrotik side to tell me more for this problem.

action=log firewall rules won't help you much in diagnosing an issue which occurs only randomly. I've mentioned the firewall because I started looking at it just to see whether it might cause the outages after a long period of bi-directional silence, and found that the one you've posted cannot be blamed for this. But I have also found that it is leaky and allows access to the router from outside (you have created a complex state automaton I haven't understood completely to protect ftp,ssh,telnet,tftp etc., but it doesn't deal with the API service and maybe other ones, and none of the services is disabled), and also slightly inefficient as you have several selective action=accept rules before the "accept established&related" one (so every single packet has to be checked against through those rules).

If you can pinpoint a peer where the issue occurs regularly, I would sniff on the public address of that peer as well as the LAN addresses of it on the CCR1009, so that both the transport packets and the payload packets would be sniffed, and do the same at the peer Mikrotik itself. The purpose is to find out whether there is a gap in the payload traffic or some rekey failure or whether the SA just stops working without a visible reason, so due to a software bug or CPU overload on one of the devices.

You can either sniff into a file on the router's flashdisk itself (but the volume of data until the issue occurs may be too high to fit on the disk, so an external flash drive connected via USB is a better option), or you can connect an external PC with tcpdump or Wireshark's dumpcap and use mangle rules with action=sniff-tzsp to copy the packets of interest there.

Sending IPsec logs into a file on the peer in the branch office would also be helpful - or also to the external PC running tcpdump using action=syslog in /system logging. The IPsec logs only log the IKE/IKEv2 processing, not the actual transport of encrypted data, but they are still very verbose and there's no way to restrict the logging to a particular peer. So on the CCR, it has to be done using syslog or to an external USB disk.

We are speaking about 350+ peers.

What's the CPU load (/system profile)? The CCR1009-8G-1S should support hardware accelerated encryption according to the IPsec manual, does it really, i.e. is the H mark shown in /ip ipsec installed-sa print?

Just to understand better, under MiKroTik L3 core switch and lower are l2 in the branches.

What do you mean by "lower are L2"? Are you using the CRS line at the branches, meaning their CPUs are of the SOHO grade, like mipsbe at 400 MHz? That should not matter much if there is just a single tunnel on each of these devices and the traffic volume through the tunnel is in the range of few Mbps. But sure, /tool profile on these should also be looked at.
On what hardware do you run the pfSense?

tomislav91 · Sun Oct 04, 2020 10:55 pm

Firewall od the other router are the same, it has just more src or dst nat there, so rules which are for us interesting are the same. But now at this point, tunnel is established thorugh pfsense and there is no much rules on the WAN side except one that we are using for internal purpoeses
What tunnel are you talking about above? You've said that the issue exists on multiple ones or even on all, so why do you now concentrate on one which uses a pfSense?

How can i upgrade firewall on mikrotik side to tell me more for this problem.
action=log firewall rules won't help you much in diagnosing an issue which occurs only randomly. I've mentioned the firewall because I started looking at it just to see whether it might cause the outages after a long period of bi-directional silence, and found that the one you've posted cannot be blamed for this. But I have also found that it is leaky and allows access to the router from outside (you have created a complex state automaton I haven't understood completely to protect ftp,ssh,telnet,tftp etc., but it doesn't deal with the API service and maybe other ones, and none of the services is disabled), and also slightly inefficient as you have several selective action=accept rules before the "accept established&related" one (so every single packet has to be checked against through those rules).

If you can pinpoint a peer where the issue occurs regularly, I would sniff on the public address of that peer as well as the LAN addresses of it on the CCR1009, so that both the transport packets and the payload packets would be sniffed, and do the same at the peer Mikrotik itself. The purpose is to find out whether there is a gap in the payload traffic or some rekey failure or whether the SA just stops working without a visible reason, so due to a software bug or CPU overload on one of the devices.

You can either sniff into a file on the router's flashdisk itself (but the volume of data until the issue occurs may be too high to fit on the disk, so an external flash drive connected via USB is a better option), or you can connect an external PC with tcpdump or Wireshark's dumpcap and use mangle rules with action=sniff-tzsp to copy the packets of interest there.

Sending IPsec logs into a file on the peer in the branch office would also be helpful - or also to the external PC running tcpdump using action=syslog in /system logging. The IPsec logs only log the IKE/IKEv2 processing, not the actual transport of encrypted data, but they are still very verbose and there's no way to restrict the logging to a particular peer. So on the CCR, it has to be done using syslog or to an external USB disk.

We are speaking about 350+ peers.
What's the CPU load (/system profile)? The CCR1009-8G-1S should support hardware accelerated encryption according to the IPsec manual, does it really, i.e. is the H mark shown in /ip ipsec installed-sa print?

Just to understand better, under MiKroTik L3 core switch and lower are l2 in the branches.
What do you mean by "lower are L2"? Are you using the CRS line at the branches, meaning their CPUs are of the SOHO grade, like mipsbe at 400 MHz? That should not matter much if there is just a single tunnel on each of these devices and the traffic volume through the tunnel is in the range of few Mbps. But sure, /tool profile on these should also be looked at.
On what hardware do you run the pfSense?

I would sniff on the public address of that peer as well as the LAN addresses of it on the CCR1009

So if the public IP of the Mikrotik 1009 "live" on ether1 I will add that ether1 to sniffer? You say LAN addresses, but those vlan are not on CCR, they are on cisco core switch. So I can maybe add VRRP_LAN interface? Where is option to put it on USB. I tried to stream to server from this tutorial https://www.wizzycom.net/traffic-captur ... wireshark/ but nothing happend in wireshark, it is maybe due to fact that when i try to ping my IP 192.168.50.13 there is no ping to this IP?
I noticed that sniffer burn my CPU, so to left it a couple of days to check when problem appear, hm, maybe not good idea, i am not sure. Maybe the best idea is to put it on usb 64GB and let it running and tell support to notice me when problem appears, just cant find the option to put it on external media.

What do you mean by "lower are L2"

So this is our infrastructure:
Two HQ.
One have MikroTik and its our local IPs from where support "go" and solve problem to remote shops. Below that MikroTik are switches, L3 main one(more than one, but thats only for HSRP, not relevant to this problem) and floor switches L2 which have tagged vlans to the offices. One office is most important, it is office from where support works.
So this MikroTik have a tunnel between PfSense from another HQ and on that PfSense is created openvpn servers which are remote shops connection (they are using mikrotiks also, but smaller ones, not CCR). And this is how we communicate with those shops.

Can you help me with commands to put those logs working? Is it good to put it like this

/system logging action add name=usb target=disk disk-file-name=usb1/log

and then just tell ipsec log to go to usb?

ipsec installed-sa print

2020-10-04 21_57_23-Window.png

CPU is at 20% at Hq1 ccr 1009.

sindy · Sun Oct 04, 2020 11:54 pm

Razmišljam da nebi bio razgovor po telefonu mnogo brži...

What do you mean by "lower are L2"
So this MikroTik have a tunnel between PfSense from another HQ and on that PfSense is created openvpn servers which are remote shops connection (they are using mikrotiks also, but smaller ones, not CCR). And this is how we communicate with those shops.

So if I get you right, there is not an IPsec tunnel to each remote shop, but there is one fat pipe to the pfSense machine which concentrates those 350+ OpenVPN tunnels from the remote shops? I was wondering where the 350 IPsec identities went missing from the configuration export :) So when you mentioned "lower are L2", I thought you talk about lower grade Mikrotiks in the remote shops.

I would sniff on the public address of that peer as well as the LAN addresses of it on the CCR1009
So if the public IP of the Mikrotik 1009 "live" on ether1 I will add that ether1 to sniffer? You say LAN addresses, but those vlan are not on CCR, they are on cisco core switch. So I can maybe add VRRP_LAN interface? Where is option to put it on USB. I tried to stream to server from this tutorial https://www.wizzycom.net/traffic-captur ... wireshark/ but nothing happend in wireshark, it is maybe due to fact that when i try to ping my IP 192.168.50.13 there is no ping to this IP?
I noticed that sniffer burn my CPU, so to left it a couple of days to check when problem appear, hm, maybe not good idea, i am not sure. Maybe the best idea is to put it on usb 64GB and let it running and tell support to notice me when problem appears, just cant find the option to put it on external media.

I did mean using the IP addresses, not the interface names, as the sniffing filter. The thing is that you need to have both the payload packets (between local LAN subnet and remote LAN subnet) as well as the IPsec transport packets carrying them in the same .pcap to be able to make some conclusion. So on the HQ machine, you have to filter on the LAN subnet of the remote machine and on the public IP of the remote machine, and vice versa on the remote shop machine (where you need to sniff on any remote address to/from which a traffic may go via the IPsec tunnel). But as there are not 350 small remote machines (one per remote shop) but just the single big one running towards the pfSense, it looks like a mission impossible, as you have to sniff almost all the traffic of the CCR1009 :(
as for Wireshark not receiving the TZSP packets, it is more likely that a firewall/antivirus software blocks the TZSP packets as they are unexpected (there is no related connection initiated by the PC). I had a case where I had to disable AVG, allowing the TZSP destination port in the Windows firewall wasn't sufficient.
sniffing to USB disk would be fine if it was fast enough - as the CCR1009-8G-1S uses the mini AB connector, it has only USB2.0 and hence 480 Mbit/s raw bitrate, which may not be enough depending on the traffic volume to&from the peer (and each packet appears at least twice in the sniff; if they pass through a bridge, then even more times, once from the physical interface and once from the bridge. As you say that the CPU is at 20% during normal operation (no sniffing), I'm afraid the throughput of the USB won't be sufficient unless the traffic to the remote shops is only a fraction of the total one (i.e. unless most of the traffic is inter-VLAN routing at the HQ itself)

Can you help me with commands to put those logs working? Is it good to put it like this
/system logging action add name=usb target=disk disk-file-name=usb1/log and then just tell ipsec log to go to usb?

Yes, this is exactly what I had in mind - first that, and then
/system logging add topics=ipsec,!packet action=usb.
But when defining the logging action, it is probably better to allow more lines per file and more files to rotate than the defaults, as we don't know in advance how big the log will become.

tomislav91 · Mon Oct 05, 2020 10:44 am

Razmišljam da nebi bio razgovor po telefonu mnogo brži...

What do you mean by "lower are L2"
So this MikroTik have a tunnel between PfSense from another HQ and on that PfSense is created openvpn servers which are remote shops connection (they are using mikrotiks also, but smaller ones, not CCR). And this is how we communicate with those shops.
So if I get you right, there is not an IPsec tunnel to each remote shop, but there is one fat pipe to the pfSense machine which concentrates those 350+ OpenVPN tunnels from the remote shops? I was wondering where the 350 IPsec identities went missing from the configuration export :) So when you mentioned "lower are L2", I thought you talk about lower grade Mikrotiks in the remote shops.

I would sniff on the public address of that peer as well as the LAN addresses of it on the CCR1009
So if the public IP of the Mikrotik 1009 "live" on ether1 I will add that ether1 to sniffer? You say LAN addresses, but those vlan are not on CCR, they are on cisco core switch. So I can maybe add VRRP_LAN interface? Where is option to put it on USB. I tried to stream to server from this tutorial https://www.wizzycom.net/traffic-captur ... wireshark/ but nothing happend in wireshark, it is maybe due to fact that when i try to ping my IP 192.168.50.13 there is no ping to this IP?
I noticed that sniffer burn my CPU, so to left it a couple of days to check when problem appear, hm, maybe not good idea, i am not sure. Maybe the best idea is to put it on usb 64GB and let it running and tell support to notice me when problem appears, just cant find the option to put it on external media.

I did mean using the IP addresses, not the interface names, as the sniffing filter. The thing is that you need to have both the payload packets (between local LAN subnet and remote LAN subnet) as well as the IPsec transport packets carrying them in the same .pcap to be able to make some conclusion. So on the HQ machine, you have to filter on the LAN subnet of the remote machine and on the public IP of the remote machine, and vice versa on the remote shop machine (where you need to sniff on any remote address to/from which a traffic may go via the IPsec tunnel). But as there are not 350 small remote machines (one per remote shop) but just the single big one running towards the pfSense, it looks like a mission impossible, as you have to sniff almost all the traffic of the CCR1009 :(

as for Wireshark not receiving the TZSP packets, it is more likely that a firewall/antivirus software blocks the TZSP packets as they are unexpected (there is no related connection initiated by the PC). I had a case where I had to disable AVG, allowing the TZSP destination port in the Windows firewall wasn't sufficient.

sniffing to USB disk would be fine if it was fast enough - as the CCR1009-8G-1S uses the mini AB connector, it has only USB2.0 and hence 480 Mbit/s raw bitrate, which may not be enough depending on the traffic volume to&from the peer (and each packet appears at least twice in the sniff; if they pass through a bridge, then even more times, once from the physical interface and once from the bridge. As you say that the CPU is at 20% during normal operation (no sniffing), I'm afraid the throughput of the USB won't be sufficient unless the traffic to the remote shops is only a fraction of the total one (i.e. unless most of the traffic is inter-VLAN routing at the HQ itself)

Can you help me with commands to put those logs working? Is it good to put it like this
/system logging action add name=usb target=disk disk-file-name=usb1/log and then just tell ipsec log to go to usb?
Yes, this is exactly what I had in mind - first that, and then
/system logging add topics=ipsec,!packet action=usb.
But when defining the logging action, it is probably better to allow more lines per file and more files to rotate than the defaults, as we don't know in advance how big the log will become.

Razmišljam da nebi bio razgovor po telefonu mnogo brži...

Mozemo i tako ako je lakse :D posle ovde okaciti resenje :D

So if I get you right, there is not an IPsec tunnel to each remote shop, but there is one fat pipe to the pfSense machine which concentrates those 350+ OpenVPN tunnels from the remote shops? I was wondering where the 350 IPsec identities went missing from the configuration export :) So when you mentioned "lower are L2", I thought you talk about lower grade Mikrotiks in the remote shops.

firstly, we did it like that, but this for us is better solution, to have one ipsec to pfsense and that pfsense create openvpn servers for those shops. This is not a big deal really, but it is annoying, because i cant figure out where is break point.

it looks like a mission impossible

so, what we can do than?

sindy · Mon Oct 05, 2020 12:02 pm

it looks like a mission impossible
so, what we can do than?

Maybe the best start is to switch on logging of the IPsec and to run a netwatch pinging through the tunnel which will log failures (on-down={:log warning message="ping through tunnel down"}) to see in the logs whether the issue is correlated with a rekey or not.

If it turns out not to be correlated, I'd say the only way ahead is to make the Wireshark work (better using mangle rules with action=sniff-tzsp to send just a small amount of traffic to it first), and once that's done, add more and more traffic until we either have it all or the CPU usage reaches, say, 60 % - after that I think it isn't safe to continue.

Another way to determine the traffic volume in advance is to run /ip ipsec installed-sa print interval=10s where spi=0xsomething for a while and get bytes per second as a 1/10 of the average of the differences between the current-bytes figures - provided that the traffic is more or less even all the time. This is the transport traffic in one direction, and if sniffing also the payload using the mangle rules, we avoid multiplicated packets in case of bridging, so the bandwidth of the sniff will be just double the bandwidth of the IPsec transport packets.

In general, I know there was an issue in Mikrotik's IPsec interworking with Strongswan (but does the pfSense use Strongswan?), I can see cases where the connection fails for some minutes and then starts working again, but I can't tell you right now which Strongswan version I'm running there, and I'm using IKEv2 at that link. Until 6.43.something, even IKEv2 connections between two Mikrotiks were showing similar symptoms, caused by an error in the rekey procedure - the next automatic rekey was fixing it. BTW, the rekey procedure differs in IKE(v1) and IKEv2, so maybe in your case, switching from IKE/aggressive to IKEv2 could help.

tomislav91 · Mon Oct 05, 2020 12:29 pm

it looks like a mission impossible
so, what we can do than?
Maybe the best start is to switch on logging of the IPsec and to run a netwatch pinging through the tunnel which will log failures (on-down={:log warning message="ping through tunnel down"}) to see in the logs whether the issue is correlated with a rekey or not.

If it turns out not to be correlated, I'd say the only way ahead is to make the Wireshark work (better using mangle rules with action=sniff-tzsp to send just a small amount of traffic to it first), and once that's done, add more and more traffic until we either have it all or the CPU usage reaches, say, 60 % - after that I think it isn't safe to continue.

Another way to determine the traffic volume in advance is to run /ip ipsec installed-sa print interval=10s where spi=0xsomething for a while and get bytes per second as a 1/10 of the average of the differences between the current-bytes figures - provided that the traffic is more or less even all the time. This is the transport traffic in one direction, and if sniffing also the payload using the mangle rules, we avoid multiplicated packets in case of bridging, so the bandwidth of the sniff will be just double the bandwidth of the IPsec transport packets.

In general, I know there was an issue in Mikrotik's IPsec interworking with Strongswan (but does the pfSense use Strongswan?), I can see cases where the connection fails for some minutes and then starts working again, but I can't tell you right now which Strongswan version I'm running there, and I'm using IKEv2 at that link. Until 6.43.something, even IKEv2 connections between two Mikrotiks were showing similar symptoms, caused by an error in the rekey procedure - the next automatic rekey was fixing it. BTW, the rekey procedure differs in IKE(v1) and IKEv2, so maybe in your case, switching from IKE/aggressive to IKEv2 could help.

Maybe the best start is to switch on logging of the IPsec

I enabled ipsec logging memory.

and to run a netwatch pinging through the tunnel

i cant ping from mikrotik because vlans are on the cisco below mikrotik, not on router itself.

/ip ipsec installed-sa print interval=10s where spi=0xsomething

where I run this, nothing append

2020-10-05 11_27_44-Window.png

BTW, the rekey procedure differs in IKE(v1) and IKEv2,

on the pfsense it is IKE1, i will try to change to IKEv2

sindy · Mon Oct 05, 2020 1:02 pm

i cant ping from mikrotik because vlans are on the cisco below mikrotik, not on router itself.

How can you forward traffic using IPsec if the Mikrotik isn't configured as a gateway, i.e. if it doesn't have an IP address in the sender's subnet? The 'Tik must first receive the packet in order to match it to a policy and send it via an SA... what am I missing here?

/ip ipsec installed-sa print interval=10s where spi=0xsomething
where I run this, nothing append

sure, you have to replace "something" with the actual SPI value - they are dynamic so you have to look for one first. You can even use spi~"0x(value1|value2)" to watch multiple SAs at a time (it's a normal regexp applied on the parameter value converted to a string)

tomislav91 · Mon Oct 05, 2020 2:54 pm

i cant ping from mikrotik because vlans are on the cisco below mikrotik, not on router itself.
How can you forward traffic using IPsec if the Mikrotik isn't configured as a gateway, i.e. if it doesn't have an IP address in the sender's subnet? The 'Tik must first receive the packet in order to match it to a policy and send it via an SA... what am I missing here?

/ip ipsec installed-sa print interval=10s where spi=0xsomething
where I run this, nothing append
sure, you have to replace "something" with the actual SPI value - they are dynamic so you have to look for one first. You can even use spi~"0x(value1|value2)" to watch multiple SAs at a time (it's a normal regexp applied on the parameter value converted to a string)

so i will first get spi value where src is pfsense and dst is mikrotik and vice versa? two spi?
Which one spi to choose? I have a bunch of spi's with state dying and state mature.
l suppose to use those mature?

sindy · Mon Oct 05, 2020 4:19 pm

Yes, the mature ones are in use. And yes, you need one per direction. The dying ones should not exist for more than a couple of seconds, so if they do, it is already weird (or the traffic volume is so low - the dying SA is normally there after a rekey until the first packet arrives through the new SA and then the dying one can be safely dropped.

The /ip ipsec installed-sa print is not as easy as I've expected, it overwrites the old results on the screen, and if you run it with file=somename append, the interval is ignored. So you have to use a script to run it every 10 seconds (or every minute), indicating the file and the append (so that the contents of the file was not overwritten).

tomislav91 · Mon Oct 05, 2020 6:16 pm

Yes, the mature ones are in use. And yes, you need one per direction. The dying ones should not exist for more than a couple of seconds, so if they do, it is already weird (or the traffic volume is so low - the dying SA is normally there after a rekey until the first packet arrives through the new SA and then the dying one can be safely dropped.

The /ip ipsec installed-sa print is not as easy as I've expected, it overwrites the old results on the screen, and if you run it with file=somename append, the interval is ignored. So you have to use a script to run it every 10 seconds (or every minute), indicating the file and the append (so that the contents of the file was not overwritten).

i will try to make script with this two mature ones, from command you provide to me (value1|value2) and edit this post when i did it

tomislav91 · Tue Oct 13, 2020 11:28 am

Yes, the mature ones are in use. And yes, you need one per direction. The dying ones should not exist for more than a couple of seconds, so if they do, it is already weird (or the traffic volume is so low - the dying SA is normally there after a rekey until the first packet arrives through the new SA and then the dying one can be safely dropped.

The /ip ipsec installed-sa print is not as easy as I've expected, it overwrites the old results on the screen, and if you run it with file=somename append, the interval is ignored. So you have to use a script to run it every 10 seconds (or every minute), indicating the file and the append (so that the contents of the file was not overwritten).
i will try to make script with this two mature ones, from command you provide to me (value1|value2) and edit this post when i did it

i tried to make like this

:set time [/system clock get time]
:local file [/ip ipsec installed-sa print]

:local contents [/file get $file contents]
:set contents ($contents . "\n" . $time)
/file set $file contents=$contents
}

but from some reason, nothing been created in the file section. There is no ipsec file when I run script.

sindy · Tue Oct 13, 2020 12:33 pm

i tried to make like this
:set time [/system clock get time]
:local file [/ip ipsec installed-sa print] <--- this fills a (string) variable called file with the output of the print command

:local contents [/file get $file contents] <--- this tries to extract the contents of a file whose name is the contents of the string variable above, that makes little sense
:set contents ($contents . "\n" . $time)
/file set $file contents=$contents
}
but from some reason, nothing been created in the file section. There is no ipsec file when I run script.

The bad news is that RouterOS is not great when it comes to file manipulation, but the good news is that the timestamp is part of every print "job". So just do
/ip ipsec installed-sa print file=somefilename append
periodically, and each list of SAs will begin with a comment block which will include the timestamp.

But you cannot open the file in RouterOS as soon as it exceeds some size (which is not really big), so you'll have to download it somewhere.

tomislav91 · Tue Oct 13, 2020 1:02 pm

i tried to make like this
:set time [/system clock get time]
:local file [/ip ipsec installed-sa print] <--- this fills a (string) variable called file with the output of the print command

:local contents [/file get $file contents] <--- this tries to extract the contents of a file whose name is the contents of the string variable above, that makes little sense
:set contents ($contents . "\n" . $time)
/file set $file contents=$contents
}
but from some reason, nothing been created in the file section. There is no ipsec file when I run script.
The bad news is that RouterOS is not great when it comes to file manipulation, but the good news is that the timestamp is part of every print "job". So just do
/ip ipsec installed-sa print file=somefilename append
periodically, and each list of SAs will begin with a comment block which will include the timestamp.

But you cannot open the file in RouterOS as soon as it exceeds some size (which is not really big), so you'll have to download it somewhere.

:set time [/system clock get time]
:local file [/ip ipsec installed-sa print file=ipsec append]

:local contents [/file get $file contents]
:set contents ($contents . "\n" . $time)
/file set $file contents=$contents
}

i did like this and i run script mannualy, but there is no ipsec file in the Files?

sindy · Tue Oct 13, 2020 1:14 pm

there should be somefilename.txt if you followed my suggestion literally... what exact command did you type?

tomislav91 · Tue Oct 13, 2020 1:34 pm

there should be somefilename.txt if you followed my suggestion literally... what exact command did you type?

yeah, so this is a script which i created

:set time [/system clock get time]
:local file [/ip ipsec installed-sa print file=ipsec append]

:local contents [/file get $file contents]
:set contents ($contents . "\n" . $time)
/file set $file contents=$contents
}

and just i RUN SCRIPT from GUI.
Is it script ok? I think it is.

2020-10-13 12_34_09-Window.png

sindy · Tue Oct 13, 2020 1:38 pm

Could you please create another script which will contain only the /ip ipsec installed-sa print file=ipsec append part and nothing else, run that new one twice, and see whether the file ipsec.txt is there and what is it contents?

tomislav91 · Tue Oct 13, 2020 1:42 pm

Could you please create another script which will contain only the /ip ipsec installed-sa print file=ipsec append part and nothing else, run that new one twice, and see whether the file ipsec.txt is there and what is it contents?

yeah, its there.

tomislav91 · Fri Oct 16, 2020 11:21 am

i tried to make like this
:set time [/system clock get time]
:local file [/ip ipsec installed-sa print] <--- this fills a (string) variable called file with the output of the print command

:local contents [/file get $file contents] <--- this tries to extract the contents of a file whose name is the contents of the string variable above, that makes little sense
:set contents ($contents . "\n" . $time)
/file set $file contents=$contents
}
but from some reason, nothing been created in the file section. There is no ipsec file when I run script.
The bad news is that RouterOS is not great when it comes to file manipulation, but the good news is that the timestamp is part of every print "job". So just do
/ip ipsec installed-sa print file=somefilename append
periodically, and each list of SAs will begin with a comment block which will include the timestamp.

But you cannot open the file in RouterOS as soon as it exceeds some size (which is not really big), so you'll have to download it somewhere.

i created schjeduler for

/ip ipsec installed-sa print file=ipsec append

every 5 minutes. Waiting now for the tunnel problem and i will post file here.

tomislav91 · Sun Nov 01, 2020 4:21 pm

i tried to make like this
:set time [/system clock get time]
:local file [/ip ipsec installed-sa print] <--- this fills a (string) variable called file with the output of the print command

:local contents [/file get $file contents] <--- this tries to extract the contents of a file whose name is the contents of the string variable above, that makes little sense
:set contents ($contents . "\n" . $time)
/file set $file contents=$contents
}
but from some reason, nothing been created in the file section. There is no ipsec file when I run script.
The bad news is that RouterOS is not great when it comes to file manipulation, but the good news is that the timestamp is part of every print "job". So just do
/ip ipsec installed-sa print file=somefilename append
periodically, and each list of SAs will begin with a comment block which will include the timestamp.

But you cannot open the file in RouterOS as soon as it exceeds some size (which is not really big), so you'll have to download it somewhere.
i created schjeduler for
Code: Select all
/ip ipsec installed-sa print file=ipsec append
every 5 minutes. Waiting now for the tunnel problem and i will post file here.

Tunnel not working but not for all subnets of peer, just for one local subnet, another sees tunnel ip. Thats strange.

This is ipsec sa

new 4.txt

2020-11-01 15_19_03-Window.png

sindy · Sun Nov 01, 2020 8:52 pm

In the past, there was an issue in IKEv2 rekey between two Mikrotiks, where in a few percent of rekeys the peers ended up with different keys for the same SA, hence the receiver was rejecting the packets. This particular issue has been fixed somewhere in late 6.43 version.

You have one policy per each (local subnet, remote subnet) pair, and each SA is rekeyed independently, so if it is a similar issue, it is no surprise that it pops up for a single SA while others remain unaffected. If all SAs handling the same local subnet don't work, it's another thing.

So I'm afraid there may be an issue in rekeying in IKE(v1) between Mikrotik's IPsec implementation and the one used on pfSense. To make sure, you have to get the SA data also from the pfSense while the issue exists. If the failed rekey is the reason, you'll see a difference between the keys shown at both peers for the affected SAs, whilst the working SAs will show the same keys.

tomislav91 · Mon Nov 02, 2020 11:17 am

In the past, there was an issue in IKEv2 rekey between two Mikrotiks, where in a few percent of rekeys the peers ended up with different keys for the same SA, hence the receiver was rejecting the packets. This particular issue has been fixed somewhere in late 6.43 version.

You have one policy per each (local subnet, remote subnet) pair, and each SA is rekeyed independently, so if it is a similar issue, it is no surprise that it pops up for a single SA while others remain unaffected. If all SAs handling the same local subnet don't work, it's another thing.

So I'm afraid there may be an issue in rekeying in IKE(v1) between Mikrotik's IPsec implementation and the one used on pfSense. To make sure, you have to get the SA data also from the pfSense while the issue exists. If the failed rekey is the reason, you'll see a difference between the keys shown at both peers for the affected SAs, whilst the working SAs will show the same keys.

2020-11-02 10_16_23-Window.png

is this helpfull info?

sindy · Mon Nov 02, 2020 11:25 am

Without the actual encryption and authentication keys in use, it is not sufficient, as you can only confirm that it is a rekey issue by comparing the keys at both ends for same SPIs.

Can you show me you /ip ipsec statistics print? There is a counter which grows with each packet coming through the SA whose last rekey failed, but I don't remember which one exactly it is.

tomislav91 · Mon Nov 02, 2020 12:08 pm

Without the actual encryption and authentication keys in use, it is not sufficient, as you can only confirm that it is a rekey issue by comparing the keys at both ends for same SPIs.

Can you show me you /ip ipsec statistics print? There is a counter which grows with each packet coming through the SA whose last rekey failed, but I don't remember which one exactly it is.

yeah, sure

2020-11-02 11_07_42-Window.png

sindy · Mon Nov 02, 2020 12:49 pm

I'm afraid it's the in-state-sequence-errors value - it doesn't sound related, but apparently there is no separate counter for packets encrypted using a wrong key. So whenever this counter increases, there is at least one "miskeyed" SA.

Go to command line of the pfsense and try ip xfrm state. It should show you the same information like /ip ipsec installed-sa print on Mikrotik:

formatted code

src 192.168.12.1 dst 195.201.133.70
        proto esp spi 0x0942b8a2 reqid 2208 mode tunnel
        replay-window 32 flag af-unspec
        auth-trunc hmac(sha256) 0xaf4069a8fe6fac611d203e925c1b0de100b053fabf14dce5a2b81aafd53e46ef 128
        enc cbc(aes) 0x60c4e645bf20c5a321644fcb862271142a2e61622f7d900095d81d1a8afd4fe0
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x0, oseq 0x4d0, bitmap 0x00000000
src 195.201.133.70 dst 192.168.12.1
        proto esp spi 0xc8c37789 reqid 2208 mode tunnel
        replay-window 32 flag af-unspec
        auth-trunc hmac(sha256) 0x5420611fe33c77d00a9a373bd040516cf3b33d397f2bbd869de83fda1aed7736 128
        enc cbc(aes) 0x4628da3a9ec1dd0a02c5ed6b1369d627c511897c1ca57c454f00ae413892c141
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x222, oseq 0x0, bitmap 0xffffffff

With the number of policies you use, it will require to dump the SA information at both machines into a file when the issue appears on one of the policies, and then to do some text processing to convert the files from both machines to a unified form so that you could compare the SA keys per SPI. Be aware that the SPI values are also ephemeral and change with each rekey.

But you may first want to try to switch from exchange mode "main" (which is one of IKE(v1)'s modes) to exchange mode "ike2" (at both the Tik and the pfSense of course); the SA rekey procedure differs for SAs established under control of IKE(v1) and under control of IKEv2, so maybe it will work fine with IKEv2.

If it doesn't help, you'll have to open a case with both Mikrotik and pfSense, as you won't be able to find out on your own which party is responsible for the rekey failure.

tomislav91 · Mon Nov 02, 2020 1:03 pm

I'm afraid it's the in-state-sequence-errors value - it doesn't sound related, but apparently there is no separate counter for packets encrypted using a wrong key. So whenever this counter increases, there is at least one "miskeyed" SA.

Go to command line of the pfsense and try ip xfrm state. It should show you the same information like /ip ipsec installed-sa print on Mikrotik:

formatted code
src 192.168.12.1 dst 195.201.133.70
        proto esp spi 0x0942b8a2 reqid 2208 mode tunnel
        replay-window 32 flag af-unspec
        auth-trunc hmac(sha256) 0xaf4069a8fe6fac611d203e925c1b0de100b053fabf14dce5a2b81aafd53e46ef 128
        enc cbc(aes) 0x60c4e645bf20c5a321644fcb862271142a2e61622f7d900095d81d1a8afd4fe0
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x0, oseq 0x4d0, bitmap 0x00000000
src 195.201.133.70 dst 192.168.12.1
        proto esp spi 0xc8c37789 reqid 2208 mode tunnel
        replay-window 32 flag af-unspec
        auth-trunc hmac(sha256) 0x5420611fe33c77d00a9a373bd040516cf3b33d397f2bbd869de83fda1aed7736 128
        enc cbc(aes) 0x4628da3a9ec1dd0a02c5ed6b1369d627c511897c1ca57c454f00ae413892c141
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x222, oseq 0x0, bitmap 0xffffffff
With the number of policies you use, it will require to dump the SA information at both machines into a file when the issue appears on one of the policies, and then to do some text processing to convert the files from both machines to a unified form so that you could compare the SA keys per SPI. Be aware that the SPI values are also ephemeral and change with each rekey.

But you may first want to try to switch from exchange mode "main" (which is one of IKE(v1)'s modes) to exchange mode "ike2" (at both the Tik and the pfSense of course); the SA rekey procedure differs for SAs established under control of IKE(v1) and under control of IKEv2, so maybe it will work fine with IKEv2.

If it doesn't help, you'll have to open a case with both Mikrotik and pfSense, as you won't be able to find out on your own which party is responsible for the rekey failure.

ip xfrm state

i got ip : command not found tried to find on google but couldnt

But you may first want to try to switch from exchange mode "main" (which is one of IKE(v1)'s modes) to exchange mode "ike2" (at both the Tik and the pfSense of course); the SA rekey procedure differs for SAs established under control of IKE(v1) and under control of IKEv2, so maybe it will work fine with IKEv2.

i tried this earlier and didnt help

sindy · Mon Nov 02, 2020 1:14 pm

Not knowing what linux distribution pfSense is based on, nor which IPsec implementation it uses (openswan, strongswan, something else), I cannot give you a more targeted suggestion. Did you issue that command as a linux user with root privileges, or is there some restricted command line of the pfSense so you first need to get from there to bash?

So try migrating to IKEv2 first and see whether it makes the issue disappear.

tomislav91 · Sun Nov 08, 2020 8:12 pm

Not knowing what linux distribution pfSense is based on, nor which IPsec implementation it uses (openswan, strongswan, something else), I cannot give you a more targeted suggestion. Did you issue that command as a linux user with root privileges, or is there some restricted command line of the pfSense so you first need to get from there to bash?

So try migrating to IKEv2 first and see whether it makes the issue disappear.

So try migrating to IKEv2 first and see whether it makes the issue disappear.

i tried this before, and didnt help.
I will try to find out a pfsense commands for that what are you need for.

tomislav91 · Thu Dec 24, 2020 11:05 pm

I'm afraid it's the in-state-sequence-errors value - it doesn't sound related, but apparently there is no separate counter for packets encrypted using a wrong key. So whenever this counter increases, there is at least one "miskeyed" SA.

Go to command line of the pfsense and try ip xfrm state. It should show you the same information like /ip ipsec installed-sa print on Mikrotik:

formatted code
src 192.168.12.1 dst 195.201.133.70
        proto esp spi 0x0942b8a2 reqid 2208 mode tunnel
        replay-window 32 flag af-unspec
        auth-trunc hmac(sha256) 0xaf4069a8fe6fac611d203e925c1b0de100b053fabf14dce5a2b81aafd53e46ef 128
        enc cbc(aes) 0x60c4e645bf20c5a321644fcb862271142a2e61622f7d900095d81d1a8afd4fe0
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x0, oseq 0x4d0, bitmap 0x00000000
src 195.201.133.70 dst 192.168.12.1
        proto esp spi 0xc8c37789 reqid 2208 mode tunnel
        replay-window 32 flag af-unspec
        auth-trunc hmac(sha256) 0x5420611fe33c77d00a9a373bd040516cf3b33d397f2bbd869de83fda1aed7736 128
        enc cbc(aes) 0x4628da3a9ec1dd0a02c5ed6b1369d627c511897c1ca57c454f00ae413892c141
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x222, oseq 0x0, bitmap 0xffffffff
With the number of policies you use, it will require to dump the SA information at both machines into a file when the issue appears on one of the policies, and then to do some text processing to convert the files from both machines to a unified form so that you could compare the SA keys per SPI. Be aware that the SPI values are also ephemeral and change with each rekey.

But you may first want to try to switch from exchange mode "main" (which is one of IKE(v1)'s modes) to exchange mode "ike2" (at both the Tik and the pfSense of course); the SA rekey procedure differs for SAs established under control of IKE(v1) and under control of IKEv2, so maybe it will work fine with IKEv2.

If it doesn't help, you'll have to open a case with both Mikrotik and pfSense, as you won't be able to find out on your own which party is responsible for the rekey failure.

i found command and here you are result

ipsec statusall
or
swanctl --list-sas
or
setkey -D

HQ2 HQ1
	esp mode=tunnel spi=194053178(0x0b91043a) reqid=179(0x000000b3)
	E: rijndael-cbc  e91d22c7 34c5660b ee38fcb5 617ef027 e51d8664 98026fe0 c2a3f6d7 50324686
	A: hmac-sha1  6c0071c1 7681c0cc 5a6073cf da4d6aae b12cfd84
	seq=0x00e8045a replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:20:35 2020	current: Dec 24 22:03:31 2020
	diff: 2576(s)	hard: 3600(s)	soft: 3589(s)
	last: Dec 24 21:20:35 2020	hard: 0(s)	soft: 0(s)
	current: 18292672176(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 15205466	hard: 0	soft: 0
	sadb_seq=71 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3251491520(0xc1cdd2c0) reqid=179(0x000000b3)
	E: rijndael-cbc  95854d1c 7f9f1b41 9bba52d0 e42a1922 f7b98a9e 5aebd775 b2b08a63 0cb56e22
	A: hmac-sha1  cacfce32 7dc71379 5256cff6 7e10aa2f cb87ce70
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:20:35 2020	current: Dec 24 22:03:31 2020
	diff: 2576(s)	hard: 3600(s)	soft: 3579(s)
	last: Dec 24 21:20:35 2020	hard: 0(s)	soft: 0(s)
	current: 382695806(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 8596284	hard: 0	soft: 0
	sadb_seq=70 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=90744231(0x0568a5a7) reqid=180(0x000000b4)
	E: rijndael-cbc  93f6e98d 6c108df7 c1fe05e8 1398aad2 52e31a3d c7bcfc82 a1c7bd57 63b0d847
	A: hmac-sha1  48cc6406 1db4db69 cfc83235 7627ea44 fa171142
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 22:00:29 2020	current: Dec 24 22:03:31 2020
	diff: 182(s)	hard: 3600(s)	soft: 3587(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=69 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=200505362(0x0bf37812) reqid=180(0x000000b4)
	E: rijndael-cbc  ecaffed1 901ca147 72095182 22809132 6b2b476e 10d7f9d7 c4177edf f1f2a319
	A: hmac-sha1  90ec8e98 14309c03 d7d89773 8da9e88d ab2c499e
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:48:20 2020	current: Dec 24 22:03:31 2020
	diff: 911(s)	hard: 3600(s)	soft: 3586(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=68 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=40106998(0x0263fbf6) reqid=180(0x000000b4)
	E: rijndael-cbc  12ba534a 7a7bbee4 c40f74f0 4ac692cc 9c1db0d0 ffed8f3e 2d9f1140 150e498e
	A: hmac-sha1  b60e0602 1213b0f5 bd6627df cb3dc8d7 5731dad0
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:19:04 2020	current: Dec 24 22:03:31 2020
	diff: 2667(s)	hard: 3600(s)	soft: 3584(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=67 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3284684755(0xc3c84fd3) reqid=180(0x000000b4)
	E: rijndael-cbc  d61bceb6 9cef149f 0f0b875e ada64d8b e69d23af 85825951 36f425f9 b14e569a
	A: hmac-sha1  c5714328 de33e343 fe614677 a7536532 388eb341
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 22:00:29 2020	current: Dec 24 22:03:31 2020
	diff: 182(s)	hard: 3600(s)	soft: 3585(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=66 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3464886564(0xce85f924) reqid=180(0x000000b4)
	E: rijndael-cbc  ada10215 ff34f61b 3f196f8d da7246b7 cdf71b49 5b1afb07 8725f298 578b785c
	A: hmac-sha1  dc7a0217 001c3de4 4a2cc765 8e3efc9d 12d1f0bb
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:48:20 2020	current: Dec 24 22:03:31 2020
	diff: 911(s)	hard: 3600(s)	soft: 3589(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=65 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3290874816(0xc426c3c0) reqid=180(0x000000b4)
	E: rijndael-cbc  e3053811 e32d977c 84a28a15 9dc8e511 8d5649a6 4f53fa82 0f48a721 a4e18042
	A: hmac-sha1  ec526471 d2fb54d0 e0a635d0 7ace8f88 3b30074d
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:19:04 2020	current: Dec 24 22:03:31 2020
	diff: 2667(s)	hard: 3600(s)	soft: 3587(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=64 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=100184661(0x05f8b255) reqid=177(0x000000b1)
	E: rijndael-cbc  51361fba 9581e121 2682dbe1 d315b23d fde1537a 092898d6 f059b8fe 04fae9be
	A: hmac-sha1  272108cc c43dffc2 0342e887 c2893b5e b47cedd3
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 22:00:11 2020	current: Dec 24 22:03:31 2020
	diff: 200(s)	hard: 3600(s)	soft: 3586(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=63 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=13907557(0x00d43665) reqid=177(0x000000b1)
	E: rijndael-cbc  4c850490 43b72603 ce0cb165 4df43b8e 6fa2b544 bb8ff671 0c3ddf1d bfe2641d
	A: hmac-sha1  9c99901d 2b03f4fe 85f53863 9539e47c bf263503
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:49:50 2020	current: Dec 24 22:03:31 2020
	diff: 821(s)	hard: 3600(s)	soft: 3588(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=62 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=95254521(0x05ad77f9) reqid=177(0x000000b1)
	E: rijndael-cbc  f10ba5ab c25c83f4 d6bf8a06 96ee21ac 31167b3d 8a10d256 349f57e1 42312532
	A: hmac-sha1  a480a161 c68242ee 9cb037a7 fd661838 01c2115a
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:20:13 2020	current: Dec 24 22:03:31 2020
	diff: 2598(s)	hard: 3600(s)	soft: 3586(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=61 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3273134059(0xc3180feb) reqid=177(0x000000b1)
	E: rijndael-cbc  b33e978b 2607ee79 e6e308c2 444d9af7 25774a0f bd43bf59 d9a69eb6 8a195792
	A: hmac-sha1  48ae8d5a 5ee789a1 1ca0fc8f b55850e8 9b1d3df1
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 22:00:11 2020	current: Dec 24 22:03:31 2020
	diff: 200(s)	hard: 3600(s)	soft: 3589(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=60 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3243216105(0xc14f8ce9) reqid=177(0x000000b1)
	E: rijndael-cbc  bb69cb03 5f744aa7 48fb6766 9a77cd45 80685183 afed5076 58bc6e16 8db9e410
	A: hmac-sha1  6392010e 398b6de8 61811368 c59b3f88 46cb24c7
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:49:50 2020	current: Dec 24 22:03:31 2020
	diff: 821(s)	hard: 3600(s)	soft: 3580(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=59 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3351502395(0xc7c3de3b) reqid=177(0x000000b1)
	E: rijndael-cbc  0a741595 0609ced4 b9e741ac e1503c99 782064ca 2e7027cf 1211ed45 f2969e10
	A: hmac-sha1  15111446 f7e4b416 9a11b0dc b1be341a afd1420b
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:20:13 2020	current: Dec 24 22:03:31 2020
	diff: 2598(s)	hard: 3600(s)	soft: 3587(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=58 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=16508708(0x00fbe724) reqid=178(0x000000b2)
	E: rijndael-cbc  d152f74d 1c5251de c1792113 45711ebe 1dc1dd7f 1c9e2177 470f2e8d a1b61d5f
	A: hmac-sha1  52d0299e 13727e64 eb869ccd 7fa33802 cb0a1409
	seq=0x0087eb7d replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:38:54 2020	current: Dec 24 22:03:31 2020
	diff: 1477(s)	hard: 3600(s)	soft: 3586(s)
	last: Dec 24 21:38:54 2020	hard: 0(s)	soft: 0(s)
	current: 10107311440(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 8907644	hard: 0	soft: 0
	sadb_seq=57 pid=65019 refcnt=2
HQ2 HQ1
	esp mode=tunnel spi=258189634(0x0f63a942) reqid=178(0x000000b2)
	E: rijndael-cbc  3e5ce00d 1a686859 b1ba7adc df7ef792 8bdaa823 e0b27385 7af4d109 eb598f22
	A: hmac-sha1  5a214de8 20e5eb43 a51aafa6 a38a305e 991472cf
	seq=0x00bb705e replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:06:01 2020	current: Dec 24 22:03:31 2020
	diff: 3450(s)	hard: 3600(s)	soft: 3588(s)
	last: Dec 24 21:06:01 2020	hard: 0(s)	soft: 0(s)
	current: 14053576912(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 12283998	hard: 0	soft: 0
	sadb_seq=56 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3405409504(0xcafa6ce0) reqid=178(0x000000b2)
	E: rijndael-cbc  404450d8 6b2d237b f79cfabb 3fb23909 761c53a9 58208c8d 69972f6c 8856d1c1
	A: hmac-sha1  028e87c7 ac1d78ce b498b4fd ebf41eea f5763c1d
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:38:54 2020	current: Dec 24 22:03:31 2020
	diff: 1477(s)	hard: 3600(s)	soft: 3588(s)
	last: Dec 24 21:38:54 2020	hard: 0(s)	soft: 0(s)
	current: 227913790(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 5115416	hard: 0	soft: 0
	sadb_seq=55 pid=65019 refcnt=2
HQ1 HQ2
	esp mode=tunnel spi=3376674182(0xc943f586) reqid=178(0x000000b2)
	E: rijndael-cbc  82e4cb31 bfe72297 08886a15 c9123576 e9fa00cc fed801f8 faefdc64 0cedb7c2
	A: hmac-sha1  629c5fc3 b4019d8e 380c2a7b 67fb746a 189a501f
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:06:01 2020	current: Dec 24 22:03:31 2020
	diff: 3450(s)	hard: 3600(s)	soft: 3585(s)
	last: Dec 24 21:06:01 2020	hard: 0(s)	soft: 0(s)
	current: 312098311(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 7007371	hard: 0	soft: 0
	sadb_seq=54 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=26192280(0x018fa998) reqid=79(0x0000004f)
	E: rijndael-cbc  8be4236c 936f2747 6fc0377a 7f194da4 81aa7234 04ac7b4d fbe30820 0a58c188
	A: hmac-sha1  823f0faa c971d04b dfc1f451 f1d66532 18d95863
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 22:00:38 2020	current: Dec 24 22:03:31 2020
	diff: 173(s)	hard: 3600(s)	soft: 3587(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=53 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=144768731(0x08a0fedb) reqid=79(0x0000004f)
	E: rijndael-cbc  a990af72 a3442b52 08018d28 2fdae1c2 eadae086 e786045d 8d8e27cf c2d5117f
	A: hmac-sha1  c9c98cd7 b4ffa301 226a673a 02b7f83e 16f82d2f
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:53:52 2020	current: Dec 24 22:03:31 2020
	diff: 579(s)	hard: 3600(s)	soft: 3583(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=52 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=127178844(0x0794985c) reqid=79(0x0000004f)
	E: rijndael-cbc  d4588e3a 204d0b17 21bf204e cbeee408 a89e94b6 a65e1295 14af2e34 eabf6362
	A: hmac-sha1  51c3d07c 67b4f1dd 9962d162 f3aa44aa 78ccb5fb
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:48:33 2020	current: Dec 24 22:03:31 2020
	diff: 898(s)	hard: 3600(s)	soft: 3585(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=51 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=19908652(0x012fc82c) reqid=79(0x0000004f)
	E: rijndael-cbc  97e7be21 891debeb 13ceddf3 a86b55a5 3988fd6b 5c602717 7530e83d 7b72bb03
	A: hmac-sha1  cbc14ca3 63cf7cb1 523ee7ce 453f693d c594b040
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:36:22 2020	current: Dec 24 22:03:31 2020
	diff: 1629(s)	hard: 3600(s)	soft: 3579(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=50 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=254885991(0x0f314067) reqid=79(0x0000004f)
	E: rijndael-cbc  2a944d23 aea728fb a70a27a7 0665f061 55d03006 445a3f97 9c0ab3aa 6a2dd3be
	A: hmac-sha1  f83cf61f 8d38be26 fedbd605 e5a77406 f09bfe1b
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:12:54 2020	current: Dec 24 22:03:31 2020
	diff: 3037(s)	hard: 3600(s)	soft: 3579(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=49 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3409704620(0xcb3bf6ac) reqid=79(0x0000004f)
	E: rijndael-cbc  1b95ec94 a1db9fde 0f574717 bbe4397a 95bf5eab 739ceee7 650453be 682f3419
	A: hmac-sha1  1ae109e4 8c4a5222 1e028f03 35088e29 d8183054
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 22:00:38 2020	current: Dec 24 22:03:31 2020
	diff: 173(s)	hard: 3600(s)	soft: 3584(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=48 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3416480777(0xcba35c09) reqid=79(0x0000004f)
	E: rijndael-cbc  b6c7ac36 c45a6351 4af5f4fc ebdd6d24 f4ce4e32 2c492cd4 9fcf9c8b b874c7c1
	A: hmac-sha1  411c9b98 e00a9e0f 0efef89b b241dcbc f258b122
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:53:52 2020	current: Dec 24 22:03:31 2020
	diff: 579(s)	hard: 3600(s)	soft: 3582(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=47 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3427334997(0xcc48fb55) reqid=79(0x0000004f)
	E: rijndael-cbc  eda58416 5375f1da f61f88e8 b642f8ce b5a3c938 49fecb0d b93b48f5 f94f180c
	A: hmac-sha1  831b07cd 45e77174 2eaed06b 2f23fdd5 9de430f7
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:48:33 2020	current: Dec 24 22:03:31 2020
	diff: 898(s)	hard: 3600(s)	soft: 3585(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=46 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3334325250(0xc6bdc402) reqid=79(0x0000004f)
	E: rijndael-cbc  68cfbd0e 8de7f199 89ce3061 56960074 84c50c25 2c6e1912 365ff3d0 d6b213f9
	A: hmac-sha1  cf0111fa ffa65162 75027035 5beb2155 fc84f6a6
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:36:22 2020	current: Dec 24 22:03:31 2020
	diff: 1629(s)	hard: 3600(s)	soft: 3589(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=45 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3238978861(0xc10ee52d) reqid=79(0x0000004f)
	E: rijndael-cbc  946f8ee3 3ed9e030 04776c61 608542d6 34b0b81d ce8cb4e3 844d161e 88545316
	A: hmac-sha1  5a838db1 f8714dd9 73ee4503 ca2ea3fd 9c182b82
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:12:54 2020	current: Dec 24 22:03:31 2020
	diff: 3037(s)	hard: 3600(s)	soft: 3587(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=44 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=15244991(0x00e89ebf) reqid=58(0x0000003a)
	E: rijndael-cbc  5283f7e6 4f4b571e 7e3f2b82 9baaf1e6 b98b98ad 8f305b03 61f8ccd6 6356d4fb
	A: hmac-sha1  55ac3b41 7e85e5d5 5bde5c55 1584d9eb a788be6c
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 22:01:48 2020	current: Dec 24 22:03:31 2020
	diff: 103(s)	hard: 3600(s)	soft: 3579(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=43 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=234976296(0x0e017428) reqid=58(0x0000003a)
	E: rijndael-cbc  ab517934 e903b97e b6c9ea9b f7fb0985 6dfa689a debc3ca3 c73a0130 a05a124f
	A: hmac-sha1  5c518cfc 3d67def3 81cb03ef bdf2fbf9 1e1382e3
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:45:58 2020	current: Dec 24 22:03:31 2020
	diff: 1053(s)	hard: 3600(s)	soft: 3579(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=42 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=189044906(0x0b4498aa) reqid=58(0x0000003a)
	E: rijndael-cbc  96e3c2a6 d0a365bc 3190a2a5 b084baaa 2c4be94b 6afae290 bdafcb14 fe01e510
	A: hmac-sha1  d3590dbc 05c13cde 5bfa721f a1e59b28 dd18d030
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:19:38 2020	current: Dec 24 22:03:31 2020
	diff: 2633(s)	hard: 3600(s)	soft: 3588(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=41 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3371313563(0xc8f2299b) reqid=58(0x0000003a)
	E: rijndael-cbc  44b44c95 285f40f8 0c1fb1bb 6eeb008c f13faf7c e4469daa ae79fd3f 2570df78
	A: hmac-sha1  f1686b31 5c41abcf 728a6378 2b910418 4ad6a83c
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 22:01:48 2020	current: Dec 24 22:03:31 2020
	diff: 103(s)	hard: 3600(s)	soft: 3587(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=40 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3410313727(0xcb4541ff) reqid=58(0x0000003a)
	E: rijndael-cbc  7e182ca9 cf803578 4704a15c 4d24dc53 7573e608 44446205 8d5f285d b4de2822
	A: hmac-sha1  f402edfe ea3447c9 c112ba14 df7328a3 249086ce
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:45:58 2020	current: Dec 24 22:03:31 2020
	diff: 1053(s)	hard: 3600(s)	soft: 3581(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=39 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3295745067(0xc471142b) reqid=58(0x0000003a)
	E: rijndael-cbc  a74a4d8d 34ab4846 dc619a83 264988a7 cdd55164 c42f773d dc646518 23701cb2
	A: hmac-sha1  04ce0ce6 cc919ed1 6d11bc8e 7666fc91 cc10c016
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:19:38 2020	current: Dec 24 22:03:31 2020
	diff: 2633(s)	hard: 3600(s)	soft: 3581(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=38 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=54996732(0x03472efc) reqid=25(0x00000019)
	E: rijndael-cbc  e8e42a31 aa695dcd 4d255fa6 6355e074 387a816e bad78bbf 3947dec9 274801bf
	A: hmac-sha1  bba93625 6aeab4ad 95b8805e f287a5b9 292a243d
	seq=0x00000064 replay=0 flags=0x00000000 state=mature
	created: Dec 24 22:01:21 2020	current: Dec 24 22:03:31 2020
	diff: 130(s)	hard: 3600(s)	soft: 3583(s)
	last: Dec 24 22:01:22 2020	hard: 0(s)	soft: 0(s)
	current: 15200(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 100	hard: 0	soft: 0
	sadb_seq=37 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=160804156(0x0995ad3c) reqid=25(0x00000019)
	E: rijndael-cbc  d336a553 73439fd4 6ae70c44 763f693b bb3e7645 c6533248 a34838e7 43f86ddc
	A: hmac-sha1  383ef500 e5f3fbb5 738e2151 8b129486 e0447dd9
	seq=0x00000140 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:54:17 2020	current: Dec 24 22:03:31 2020
	diff: 554(s)	hard: 3600(s)	soft: 3583(s)
	last: Dec 24 21:54:17 2020	hard: 0(s)	soft: 0(s)
	current: 48640(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 320	hard: 0	soft: 0
	sadb_seq=36 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=116748749(0x06f571cd) reqid=25(0x00000019)
	E: rijndael-cbc  a42271e7 836b4531 1898f6fb 464f937f 408a434d b1b10f42 9556d764 de7ba6f0
	A: hmac-sha1  ffedee7a eae5e271 d5f39b56 0799bd80 94042d79
	seq=0x0000013b replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:47:16 2020	current: Dec 24 22:03:31 2020
	diff: 975(s)	hard: 3600(s)	soft: 3586(s)
	last: Dec 24 21:47:17 2020	hard: 0(s)	soft: 0(s)
	current: 47880(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 315	hard: 0	soft: 0
	sadb_seq=35 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=49302150(0x02f04a86) reqid=25(0x00000019)
	E: rijndael-cbc  86718e88 35609f47 76c0dc0f 7b8b7558 786d3611 beeaa320 4cf61a40 a07da1c6
	A: hmac-sha1  f3b4d537 a8d1044e 8b305b53 4e983fa9 e79528f4
	seq=0x000005e6 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:13:56 2020	current: Dec 24 22:03:31 2020
	diff: 2975(s)	hard: 3600(s)	soft: 3586(s)
	last: Dec 24 21:13:56 2020	hard: 0(s)	soft: 0(s)
	current: 229520(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 1510	hard: 0	soft: 0
	sadb_seq=34 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3486512649(0xcfcff609) reqid=25(0x00000019)
	E: rijndael-cbc  206942d7 6d5a6196 715b483e e75e952f acf5ee3a fee8a562 1884d7d8 c44f0124
	A: hmac-sha1  0813209e 63f6c4d2 ccd6d938 037b5e2f a8630eb5
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 22:01:21 2020	current: Dec 24 22:03:31 2020
	diff: 130(s)	hard: 3600(s)	soft: 3583(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=33 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3277388473(0xc358fab9) reqid=25(0x00000019)
	E: rijndael-cbc  c8f543d2 474a9c8d 0991bf2b 5ae32e1e 2af18218 abd26691 0e4932b6 7be23ef1
	A: hmac-sha1  e5be2746 91d17af6 c479d0c1 0333ce60 1ddec481
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:54:17 2020	current: Dec 24 22:03:31 2020
	diff: 554(s)	hard: 3600(s)	soft: 3588(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=32 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3360234241(0xc8491b01) reqid=25(0x00000019)
	E: rijndael-cbc  58e1663f f462c931 3f6b1ea1 ddc34267 7236c9b4 4aa81ae6 4c816147 578a0f42
	A: hmac-sha1  72f4e0c1 6baa796e df578ca4 e4f86415 bc4afbbe
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:47:16 2020	current: Dec 24 22:03:31 2020
	diff: 975(s)	hard: 3600(s)	soft: 3579(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=31 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3442901187(0xcd3680c3) reqid=25(0x00000019)
	E: rijndael-cbc  e0c968a6 257ea9f1 0be5546b 946ce7a6 76827da2 2b69d751 168cb2ae dfc06991
	A: hmac-sha1  2d03e453 6e0645e6 f14209ea 32d6f1c8 fe23c5a3
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:13:56 2020	current: Dec 24 22:03:31 2020
	diff: 2975(s)	hard: 3600(s)	soft: 3581(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=30 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=90389729(0x05633ce1) reqid=18(0x00000012)
	E: rijndael-cbc  cc7f0af8 176e42d6 e522cead 1e0b5817 02cc7d95 104887a4 25cd882d 31d9ed35
	A: hmac-sha1  b99535a7 faf72965 14131575 efd192df ff12d49b
	seq=0x0000000e replay=0 flags=0x00000000 state=mature
	created: Dec 24 22:02:22 2020	current: Dec 24 22:03:31 2020
	diff: 69(s)	hard: 3600(s)	soft: 3589(s)
	last: Dec 24 22:02:24 2020	hard: 0(s)	soft: 0(s)
	current: 2352(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 14	hard: 0	soft: 0
	sadb_seq=29 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=262872706(0x0fab1e82) reqid=18(0x00000012)
	E: rijndael-cbc  cb6cc782 907b6f56 551a4cb5 f5719410 558acb6d e8b71956 99160264 37e46ed0
	A: hmac-sha1  e0994b0b c302fd79 3fa3571d 2d098daf dcd3341c
	seq=0x00000062 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:53:46 2020	current: Dec 24 22:03:31 2020
	diff: 585(s)	hard: 3600(s)	soft: 3587(s)
	last: Dec 24 21:53:52 2020	hard: 0(s)	soft: 0(s)
	current: 16464(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 98	hard: 0	soft: 0
	sadb_seq=28 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=196082408(0x0baffae8) reqid=18(0x00000012)
	E: rijndael-cbc  f7aba988 55f2b005 ca80f9f3 021f8589 aa7bf3a4 63530758 f708bd5b c645390d
	A: hmac-sha1  e695b69c 4743570c de655435 2c505692 98eddc7c
	seq=0x0000003c replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:48:31 2020	current: Dec 24 22:03:31 2020
	diff: 900(s)	hard: 3600(s)	soft: 3587(s)
	last: Dec 24 21:48:39 2020	hard: 0(s)	soft: 0(s)
	current: 10080(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 60	hard: 0	soft: 0
	sadb_seq=27 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=33623083(0x02010c2b) reqid=18(0x00000012)
	E: rijndael-cbc  b1a13bbf f9b61c01 c406608e 634b8a8a 67f741d5 fff9eb11 cb128b40 5b380e54
	A: hmac-sha1  1626f6fd 8df02e81 5de96bac d86ff7d5 8f60db5e
	seq=0x000001b5 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:12:22 2020	current: Dec 24 22:03:31 2020
	diff: 3069(s)	hard: 3600(s)	soft: 3581(s)
	last: Dec 24 21:12:27 2020	hard: 0(s)	soft: 0(s)
	current: 72600(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 437	hard: 0	soft: 0
	sadb_seq=26 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3450316315(0xcda7a61b) reqid=18(0x00000012)
	E: rijndael-cbc  c7d768d2 8cbdf398 3328fecf 53b33c8c d4ac4730 c3f33b29 c1ea19a2 10cf8de0
	A: hmac-sha1  d1df22aa 4524f4fd d83f8e25 099d895d 7aee10db
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 22:02:22 2020	current: Dec 24 22:03:31 2020
	diff: 69(s)	hard: 3600(s)	soft: 3580(s)
	last: Dec 24 22:02:24 2020	hard: 0(s)	soft: 0(s)
	current: 957(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 8	hard: 0	soft: 0
	sadb_seq=25 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3295603338(0xc46eea8a) reqid=18(0x00000012)
	E: rijndael-cbc  1d56c65f a93369e5 1294d00a 7bfeb235 172060ae 220aced5 ada1d57e 36e8038f
	A: hmac-sha1  62397368 7d90765e 29d8d60a 970084af 2d972e34
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:53:46 2020	current: Dec 24 22:03:31 2020
	diff: 585(s)	hard: 3600(s)	soft: 3586(s)
	last: Dec 24 21:53:52 2020	hard: 0(s)	soft: 0(s)
	current: 6579(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 53	hard: 0	soft: 0
	sadb_seq=24 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3378150255(0xc95a7b6f) reqid=18(0x00000012)
	E: rijndael-cbc  6dab7c7b 195805fc 58c80467 5be6bb12 d8ce7fac b11f9082 d9e9eb51 a2f677d9
	A: hmac-sha1  d413c7e0 e456a2c9 6d20872a 1654b6e6 8a41b1e7
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:48:31 2020	current: Dec 24 22:03:31 2020
	diff: 900(s)	hard: 3600(s)	soft: 3586(s)
	last: Dec 24 21:48:39 2020	hard: 0(s)	soft: 0(s)
	current: 4010(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 32	hard: 0	soft: 0
	sadb_seq=23 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3266135825(0xc2ad4711) reqid=18(0x00000012)
	E: rijndael-cbc  94d0ef7a ed801680 709ab923 f83751b6 f78bbad9 cccb4f62 0c23be36 1431eb2a
	A: hmac-sha1  60a3241e b33cac0b e4247f0a dcd631da 285ccc61
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:12:22 2020	current: Dec 24 22:03:31 2020
	diff: 3069(s)	hard: 3600(s)	soft: 3581(s)
	last: Dec 24 21:12:27 2020	hard: 0(s)	soft: 0(s)
	current: 34191(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 246	hard: 0	soft: 0
	sadb_seq=22 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=211798274(0x0c9fc902) reqid=37(0x00000025)
	E: rijndael-cbc  bfb98d12 8511db8f 1ab13de9 20e60d53 30a3b597 10a8cee2 b4cd8b32 6395dc4d
	A: hmac-sha1  cb86691f f5874bc9 0edbbfa2 40f8996d 0d5a253c
	seq=0x000002b1 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:46:08 2020	current: Dec 24 22:03:31 2020
	diff: 1043(s)	hard: 3600(s)	soft: 3587(s)
	last: Dec 24 21:46:08 2020	hard: 0(s)	soft: 0(s)
	current: 71656(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 689	hard: 0	soft: 0
	sadb_seq=21 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=200764208(0x0bf76b30) reqid=37(0x00000025)
	E: rijndael-cbc  2f0bc352 add8328d 5c758d00 572a0715 1f5536c0 500e7067 50f3d2b9 265c1fa9
	A: hmac-sha1  72712c47 3d9abaad 0022f83f b5884316 2fc89555
	seq=0x00000159 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:37:41 2020	current: Dec 24 22:03:31 2020
	diff: 1550(s)	hard: 3600(s)	soft: 3579(s)
	last: Dec 24 21:37:41 2020	hard: 0(s)	soft: 0(s)
	current: 35880(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 345	hard: 0	soft: 0
	sadb_seq=20 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=162404294(0x09ae17c6) reqid=37(0x00000025)
	E: rijndael-cbc  1b34d48a e28cd14f 8441cae3 316c8fe3 c91d7b37 6b2ac096 d54f9d7d 3456f85a
	A: hmac-sha1  9345cdca d16bb418 10fae9f2 4e073f3e 304289b0
	seq=0x00000335 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:17:26 2020	current: Dec 24 22:03:31 2020
	diff: 2765(s)	hard: 3600(s)	soft: 3585(s)
	last: Dec 24 21:17:26 2020	hard: 0(s)	soft: 0(s)
	current: 85384(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 821	hard: 0	soft: 0
	sadb_seq=19 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3484264983(0xcfadaa17) reqid=37(0x00000025)
	E: rijndael-cbc  2ea00f57 290814bb 087e0244 f68a94ae 3ab442a1 f22f9e1a 02efc501 54d13fe3
	A: hmac-sha1  7f44f856 477146d4 9ecf2e1b b20b1019 015b5a65
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:46:08 2020	current: Dec 24 22:03:31 2020
	diff: 1043(s)	hard: 3600(s)	soft: 3579(s)
	last: Dec 24 21:46:08 2020	hard: 0(s)	soft: 0(s)
	current: 59220(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 690	hard: 0	soft: 0
	sadb_seq=18 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3253001901(0xc1e4dead) reqid=37(0x00000025)
	E: rijndael-cbc  3c21b35d 850af546 d04fb9d0 be9d8d32 8ecc860b 4a39be08 fc3d638e c63ebb38
	A: hmac-sha1  3f6c4e95 18384338 7c8a54e5 6e641c10 1ede9670
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:37:41 2020	current: Dec 24 22:03:31 2020
	diff: 1550(s)	hard: 3600(s)	soft: 3586(s)
	last: Dec 24 21:37:41 2020	hard: 0(s)	soft: 0(s)
	current: 29621(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 344	hard: 0	soft: 0
	sadb_seq=17 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3358597903(0xc830230f) reqid=37(0x00000025)
	E: rijndael-cbc  e7366fa1 c41d949d 6afea0df 79c13495 ec370b63 9ecbc7da 944adf7c a1dbe146
	A: hmac-sha1  b45167f7 6f32d773 b8ee0f16 9dc1e2e9 971839bf
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:17:26 2020	current: Dec 24 22:03:31 2020
	diff: 2765(s)	hard: 3600(s)	soft: 3585(s)
	last: Dec 24 21:17:27 2020	hard: 0(s)	soft: 0(s)
	current: 70580(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 821	hard: 0	soft: 0
	sadb_seq=16 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=3159326(0x0030351e) reqid=36(0x00000024)
	E: rijndael-cbc  b14c34e4 cbfcdbd5 df7be653 9a28b649 ca7e3718 f5d469cf c3ed1a26 89cc5697
	A: hmac-sha1  67017315 5f180dc3 2297e1da fd522271 a52437a7
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:44:53 2020	current: Dec 24 22:03:31 2020
	diff: 1118(s)	hard: 3600(s)	soft: 3581(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=15 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=183273677(0x0aec88cd) reqid=36(0x00000024)
	E: rijndael-cbc  81a14a2a 67617d60 64fc635d 3bd9ea34 28f5578d eda07e80 1ad2c60c bed0a64d
	A: hmac-sha1  bc24d876 d494f4a6 e7bdad87 5213d213 e8793ee4
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:38:01 2020	current: Dec 24 22:03:31 2020
	diff: 1530(s)	hard: 3600(s)	soft: 3582(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=14 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=188512606(0x0b3c795e) reqid=36(0x00000024)
	E: rijndael-cbc  ffef98d9 5a795c8a 3b23a943 9046114b 0d2dd620 100f5611 fa3690c1 4a57fd7f
	A: hmac-sha1  d94fc47c 6e104bfb ce8bd219 0ae45c14 269f9acc
	seq=0x00000000 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:19:24 2020	current: Dec 24 22:03:31 2020
	diff: 2647(s)	hard: 3600(s)	soft: 3579(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=13 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3242791289(0xc1491179) reqid=36(0x00000024)
	E: rijndael-cbc  023e15cc f0676572 ecbf8f1b 72b4b0f6 3d365c7d 47f2730b 297f1cd2 187bcf17
	A: hmac-sha1  4ecdbda2 7dc997ca 21f9d125 7e9a8213 0dfaf008
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:44:53 2020	current: Dec 24 22:03:31 2020
	diff: 1118(s)	hard: 3600(s)	soft: 3579(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=12 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3416645044(0xcba5ddb4) reqid=36(0x00000024)
	E: rijndael-cbc  d9da1159 039254da bd3f660a aeeda6c6 315cf30d 96d88ded 72ab3ee9 afb004da
	A: hmac-sha1  77e3ed70 0889aa1c cd474fd2 c0aa934f 2515ea22
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:38:01 2020	current: Dec 24 22:03:31 2020
	diff: 1530(s)	hard: 3600(s)	soft: 3584(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=11 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3275300612(0xc3391f04) reqid=36(0x00000024)
	E: rijndael-cbc  45efafcd 09b1a615 79bd2a95 0955eca6 48f633fa dae46530 52a0e3af 0f75bcf9
	A: hmac-sha1  49ec1545 47702fd1 70f2e18b 0356edf8 4d76e5ea
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:19:24 2020	current: Dec 24 22:03:31 2020
	diff: 2647(s)	hard: 3600(s)	soft: 3583(s)
	last:                     	hard: 0(s)	soft: 0(s)
	current: 0(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 0	hard: 0	soft: 0
	sadb_seq=10 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=11914222(0x00b5cbee) reqid=34(0x00000022)
	E: rijndael-cbc  45d9fa43 84bf2cf0 f9b9301d 037e66e0 efc93f00 003385f2 e9264153 35e50a7b
	A: hmac-sha1  bff66e36 3ad97dbd 6e93d57e 449affdb 0a628eda
	seq=0x00000458 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:40:11 2020	current: Dec 24 22:03:31 2020
	diff: 1400(s)	hard: 3600(s)	soft: 3580(s)
	last: Dec 24 21:40:19 2020	hard: 0(s)	soft: 0(s)
	current: 279184(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 1112	hard: 0	soft: 0
	sadb_seq=9 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=30477431(0x01d10c77) reqid=34(0x00000022)
	E: rijndael-cbc  01ed3749 2c6bd3b0 6175355c 9fdb0cc6 8b0ff5b5 76b39b81 6dc403b3 2275938f
	A: hmac-sha1  0dc4ce73 525db503 6b49f5be c64d7967 42b12c46
	seq=0x00000076 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:37:55 2020	current: Dec 24 22:03:31 2020
	diff: 1536(s)	hard: 3600(s)	soft: 3587(s)
	last: Dec 24 21:37:58 2020	hard: 0(s)	soft: 0(s)
	current: 29952(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 118	hard: 0	soft: 0
	sadb_seq=8 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=118553440(0x0710fb60) reqid=34(0x00000022)
	E: rijndael-cbc  6ba99726 bd478d78 f7935a01 e9def5bb 75a69aef d647496c 66d755f2 533d8d08
	A: hmac-sha1  e31d94ea 2f86154b d3b81780 3fc5c5f4 2541e008
	seq=0x000003c2 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:17:46 2020	current: Dec 24 22:03:31 2020
	diff: 2745(s)	hard: 3600(s)	soft: 3587(s)
	last: Dec 24 21:17:50 2020	hard: 0(s)	soft: 0(s)
	current: 241488(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 962	hard: 0	soft: 0
	sadb_seq=7 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3280329269(0xc385da35) reqid=34(0x00000022)
	E: rijndael-cbc  4af4d631 8179b7b8 567de1cf 5d330073 cb112247 c43f6b4d 8eb72743 61a1910a
	A: hmac-sha1  6b196be4 cc59691a 91574952 627e272b 21f0ae19
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:40:11 2020	current: Dec 24 22:03:31 2020
	diff: 1400(s)	hard: 3600(s)	soft: 3586(s)
	last: Dec 24 21:40:19 2020	hard: 0(s)	soft: 0(s)
	current: 104235(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 742	hard: 0	soft: 0
	sadb_seq=6 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3351937984(0xc7ca83c0) reqid=34(0x00000022)
	E: rijndael-cbc  0c7cefe7 1fb9978f 4831b0a4 b36722d9 49e0edb4 13a04067 85d286b1 12efba94
	A: hmac-sha1  94b51112 274267c3 9634cd67 ab68fe9f 98bc45e1
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:37:55 2020	current: Dec 24 22:03:31 2020
	diff: 1536(s)	hard: 3600(s)	soft: 3588(s)
	last: Dec 24 21:37:58 2020	hard: 0(s)	soft: 0(s)
	current: 11180(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 78	hard: 0	soft: 0
	sadb_seq=5 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3392979548(0xca3cc25c) reqid=34(0x00000022)
	E: rijndael-cbc  4c520ca3 bc7ac3e4 adafd01a a9a4e309 df6a76dd 9ed15ee2 c975358d ea79292c
	A: hmac-sha1  6d1ad8eb 59511cda 6a6c1df3 f7c814fa 1a27c0d6
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:17:46 2020	current: Dec 24 22:03:31 2020
	diff: 2745(s)	hard: 3600(s)	soft: 3584(s)
	last: Dec 24 21:17:50 2020	hard: 0(s)	soft: 0(s)
	current: 90200(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 643	hard: 0	soft: 0
	sadb_seq=4 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=234261298(0x0df68b32) reqid=3(0x00000003)
	E: rijndael-cbc  4e3b34ac 9dfed62a ef30151e ded19008 29d0b855 c1d56fa3 6f34c50b 137265cd
	A: hmac-sha1  3548ed38 fd778a41 a856daa4 77855306 b40ff427
	seq=0x000058cd replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:55:45 2020	current: Dec 24 22:03:31 2020
	diff: 466(s)	hard: 3600(s)	soft: 3586(s)
	last: Dec 24 21:55:45 2020	hard: 0(s)	soft: 0(s)
	current: 9857128(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 22733	hard: 0	soft: 0
	sadb_seq=3 pid=65019 refcnt=1
HQ2 HQ1
	esp mode=tunnel spi=264750853(0x0fc7c705) reqid=3(0x00000003)
	E: rijndael-cbc  ab8cf3b1 06ad335e 93ab90c6 14b4b6c6 4f64fb8a b7828ee0 ac3bbd82 58289a68
	A: hmac-sha1  995b130b d2a30f77 2128b957 1f901724 e8f103f4
	seq=0x0001d2f8 replay=0 flags=0x00000000 state=mature
	created: Dec 24 21:12:38 2020	current: Dec 24 22:03:31 2020
	diff: 3053(s)	hard: 3600(s)	soft: 3580(s)
	last: Dec 24 21:12:38 2020	hard: 0(s)	soft: 0(s)
	current: 45426144(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 119544	hard: 0	soft: 0
	sadb_seq=2 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3322104549(0xc6034ae5) reqid=3(0x00000003)
	E: rijndael-cbc  9e7efb6c d33841e4 2061e1a5 e39919d4 b6f081eb 1d6174b4 06722b16 dda503dd
	A: hmac-sha1  b94c24a6 5fdce859 817d419b 17cbfd7e 03c53e78
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:55:45 2020	current: Dec 24 22:03:31 2020
	diff: 466(s)	hard: 3600(s)	soft: 3585(s)
	last: Dec 24 21:55:45 2020	hard: 0(s)	soft: 0(s)
	current: 1868542(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 22712	hard: 0	soft: 0
	sadb_seq=1 pid=65019 refcnt=1
HQ1 HQ2
	esp mode=tunnel spi=3365680657(0xc89c3611) reqid=3(0x00000003)
	E: rijndael-cbc  298f22ec f854490c 587dc067 bbcf5c8c 200e9871 5add2265 0adf8004 0fd31662
	A: hmac-sha1  6252462c 9fbf475e a2fc7cdb f19c90e2 94540ce5
	seq=0x00000000 replay=4 flags=0x00000000 state=mature
	created: Dec 24 21:12:38 2020	current: Dec 24 22:03:31 2020
	diff: 3053(s)	hard: 3600(s)	soft: 3588(s)
	last: Dec 24 21:12:38 2020	hard: 0(s)	soft: 0(s)
	current: 9597881(bytes)	hard: 0(bytes)	soft: 0(bytes)
	allocated: 120641	hard: 0	soft: 0
	sadb_seq=0 pid=65019 refcnt=1

sindy · Fri Dec 25, 2020 7:44 pm

Great, now use this command on the pfsense and /ip ipsec installed-sa print on the Mikrotik almost simultaneously (few seconds of difference usually do not matter).

Once do that while everything works OK, to see what to compare, and then again while the issue exists.

While the issue exists, I assume that the encryption and/or authentication keys will differ between the devices for SAs with the same SPI value.