Upgraded from 750G to RB4011iGS+ and my IKE2 IPSEC doesn't work anymore

Cray7 · Mon Mar 15, 2021 3:42 am

Upgraded from a RouterBOARD 750G r3 to RB4011iGS+, migrated config, but my IPSEC tunnel doesn't want to come up :(.

rtrA is the new RB4011. rtrB is an existing 750G. Logs show that rtrA creates the initial packet, sends it, rtrB receives it, processes, replies to rtrA. Reply reaches rtrA, is detected by packet sniffer but not registered by firewall or ipsec stack. Both routers have public, static IPs.

Has anyone else encountered anything similar? What next steps can I do to debug this? Any help appreciated :).

[rok@rtrA] /system routerboard> /system routerboard print
       routerboard: yes
             model: RB4011iGS+
          revision: r2
     serial-number: XXX
     firmware-type: al2
  factory-firmware: 6.45.9
  current-firmware: 6.45.9
  upgrade-firmware: 6.48.1

[rok@rtrA] /ip ipsec> export
# mar/15/2021 01:33:48 by RouterOS 6.48.1
# software id = 5ZRG-3PM4
# model = RB4011iGS+
# serial number = XXX
/ip ipsec peer
add address=BB.BBB.BBB.BBB/32 exchange-mode=ike2 name=rtrB
/ip ipsec profile
set [ find default=yes ] dh-group=modp2048 dpd-interval=5m enc-algorithm=aes-256 hash-algorithm=sha256 name=profile_1
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc pfs-group=modp2048
/ip ipsec identity
add my-id=fqdn:rtrA peer=rtrB secret=XXX
/ip ipsec policy
set 0 disabled=yes
add dst-address=10.X.X.0/24 peer=rtrB sa-dst-address=BB.BBB.BBB.BBB sa-src-address=0.0.0.0 src-address=10.YY.YY.0/24 tunnel=yes

[rok@rtrA] /ip firewall raw> /tool sniffer packet print
 #    TIME INTERFACE                                      SRC-ADDRESS                                                                  DST-ADDRESS                                                                  IP-PROTOCOL  SIZE CPU FP 
 0   7.052 WAN                                            AA.AAA.AA.AAA:4500                                                           BB.BBB.BBB.BBB:4500                                                          udp           478   3 no 
 1   7.925 WAN                                            BB.BBB.BBB.BBB:4500                                                          AA.AAA.AA.AAA:4500                                                           udp           470   1 no 
 2  12.058 WAN                                            AA.AAA.AA.AAA:4500                                                           BB.BBB.BBB.BBB:4500                                                          udp           478   3 no 
 3  12.112 WAN                                            BB.BBB.BBB.BBB:4500                                                          AA.AAA.AA.AAA:4500                                                           udp           470   1 no 
 4  17.064 WAN                                            AA.AAA.AA.AAA:4500                                                           BB.BBB.BBB.BBB:4500                                                          udp           478   3 no 
 5  17.118 WAN                                            BB.BBB.BBB.BBB:4500                                                          AA.AAA.AA.AAA:4500                                                           udp           470   1 no 
 6  22.059 WAN                                            AA.AAA.AA.AAA:4500                                                           BB.BBB.BBB.BBB:4500                                                          udp           478   3 no 
 7  22.116 WAN                                            BB.BBB.BBB.BBB:4500                                                          AA.AAA.AA.AAA:4500                                                           udp           470   1 no 


[rok@rtrA] /ip firewall raw> /ip firewall raw print stats 
Flags: X - disabled, I - invalid, D - dynamic 
 #    CHAIN                                                                                                                                                                           ACTION                            BYTES         PACKETS
 0  D ;;; special dummy rule to show fasttrack counters
      prerouting                                                                                                                                                                      passthrough                 177 028 927         163 150
 1    prerouting                                                                                                                                                                      accept                                0               0
 2    prerouting                                                                                                                                                                      accept                          115 256             402
[...]

[rok@rtrA] /ip firewall raw> print
Flags: X - disabled, I - invalid, D - dynamic 
 0  D ;;; special dummy rule to show fasttrack counters
      chain=prerouting action=passthrough 

 1    chain=prerouting action=accept src-port=4500 dst-port=4500 protocol=udp src-address=BB.BBB.BBB.BBB 

 2    chain=prerouting action=accept src-address=BB.BBB.BBB.BBB 

[rok@rtrA] /ip firewall raw> /log print where message~"received" or message~"sending" 
01:04:52 ipsec,debug IPSEC: ===== sending 432 bytes from AA.AAA.AA.AAA[4500] to BB.BBB.BBB.BBB[4500] 
01:04:57 ipsec,debug IPSEC: ===== sending 432 bytes from AA.AAA.AA.AAA[4500] to BB.BBB.BBB.BBB[4500] 
01:05:02 ipsec,debug IPSEC: ===== sending 432 bytes from AA.AAA.AA.AAA[4500] to BB.BBB.BBB.BBB[4500] 
01:05:17 ipsec,debug IPSEC: ===== sending 432 bytes from AA.AAA.AA.AAA[4500] to BB.BBB.BBB.BBB[4500] 

[rok@rtrB] /ip ipsec statistics> /log print where message~"received" or message~"sending"
02:08:22 ipsec,debug IPSEC: ===== sending 424 bytes from BB.BBB.BBB.BBB[4500] to AA.AAA.AA.AAA[4500] 
02:08:27 ipsec,debug IPSEC: ===== received 432 bytes from AA.AAA.AA.AAA[4500] to BB.BBB.BBB.BBB[4500] 
02:08:27 ipsec,debug IPSEC: ===== sending 424 bytes from BB.BBB.BBB.BBB[4500] to AA.AAA.AA.AAA[4500] 
02:08:32 ipsec,debug IPSEC: ===== received 432 bytes from AA.AAA.AA.AAA[4500] to BB.BBB.BBB.BBB[4500]

sindy · Mon Mar 15, 2021 2:22 pm

A misconfigured firewall at rtr A can cause this. Show the complete configuration export from rtr A (see the hint on anonymisation in my automatic signature below).

Cray7 · Tue Mar 16, 2021 11:33 pm

Anonymized config following:

>
> /export hide-sensitive
# mar/16/2021 20:10:06 by RouterOS 6.48.1
# software id = 5ZRG-3PM4
#
# model = RB4011iGS+
# serial number = XXX
/interface bridge
add name=GUEST
add name=LAN
/interface ethernet
set [ find default-name=ether3 ] name=GUEST-e3
set [ find default-name=ether4 ] name=GUEST-e4
set [ find default-name=ether5 ] name=GUEST-e5
set [ find default-name=ether6 ] name=LAN-e6
set [ find default-name=ether7 ] name=LAN-e7
set [ find default-name=ether8 ] name=LAN-e8
set [ find default-name=ether9 ] name=LAN-e9
set [ find default-name=ether10 ] name=LAN-e10
set [ find default-name=ether2 ] name=OOB
set [ find default-name=ether1 ] name=WAN
set [ find default-name=sfp-sfpplus1 ] name=WAN-SFP
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec peer
add address=public.ip.rtrB/32 exchange-mode=ike2 name=rtrB
/ip ipsec profile
set [ find default=yes ] dh-group=modp2048 dpd-interval=5m enc-algorithm=aes-256 hash-algorithm=sha256 name=profile_1
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha256 enc-algorithms=aes-256-cbc pfs-group=modp2048
/ip pool
add name=pool-OOB ranges=192.168.88.10-192.168.88.254
add name=pool-LAN ranges=10.33.8.10-10.33.8.254
add name=pool-GUEST ranges=10.33.7.10-10.33.7.254
/ip dhcp-server
add address-pool=pool-OOB authoritative=after-2sec-delay disabled=no interface=OOB name=dhcp-OOB
add address-pool=pool-GUEST authoritative=after-2sec-delay disabled=no interface=GUEST name=dhcp-GUEST
add address-pool=pool-LAN authoritative=after-2sec-delay disabled=no interface=LAN name=dhcp-LAN
/snmp community
set [ find default=yes ] addresses=0.0.0.0/0
/user group
set full policy=local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,dude,tikapp
/interface bridge port
add bridge=GUEST interface=GUEST-e3
add bridge=GUEST interface=GUEST-e4
add bridge=GUEST interface=GUEST-e5
add bridge=LAN interface=LAN-e6
add bridge=LAN interface=LAN-e7
add bridge=LAN interface=LAN-e8
add bridge=LAN interface=LAN-e9
add bridge=LAN interface=LAN-e10
/ip neighbor discovery-settings
set discover-interface-list=all
/ip address
add address=192.168.88.1/24 interface=OOB network=192.168.88.0
add address=10.33.8.1/24 interface=LAN network=10.33.8.0
add address=public.ip.rtrA/29 interface=WAN network=XXXX
add address=10.33.7.1/24 interface=GUEST network=10.33.7.0
/ip dhcp-client
add interface=WAN
/ip dhcp-server lease
add address=10.33.8.2 mac-address=70:10:6F:C7:D7:5E server=dhcp-LAN
/ip dhcp-server network
add address=10.33.7.0/24 dns-server=XXX,XXX gateway=10.33.7.1
add address=10.33.8.0/24 dns-server=10.33.8.2,10.33.8.1 gateway=10.33.8.1
add address=192.168.88.0/24 gateway=192.168.88.1
/ip dns
set allow-remote-requests=yes cache-max-ttl=1d servers=1.1.1.1,8.8.8.8
/ip dns static
add address=192.168.88.1 name=router.lan
/ip firewall address-list
add address=192.168.88.0/24 list=admin
add address=10.33.8.3 list=admin
/ip firewall filter
add action=log chain=input dst-address=public.ip.rtrA dst-port=4500 log-prefix=PKT protocol=udp src-address=public.ip.rtrB src-port=4500
add action=accept chain=input dst-address=public.ip.rtrA dst-port=4500 protocol=udp src-address=public.ip.rtrB src-port=4500
add action=accept chain=input src-address-list=admin
add action=accept chain=input comment="SSH to r0.irl" dst-port=22000 in-interface=WAN protocol=tcp
add action=accept chain=input in-interface=WAN src-address=10.250.100.0/24
add action=accept chain=input in-interface=WAN src-address=public.ip.rtrB
add action=accept chain=input protocol=icmp
add action=accept chain=input connection-state=established,related
add action=drop chain=input in-interface=WAN
add action=accept chain=forward comment="IPSec from rtrB" connection-state=established,related,new,untracked dst-address=10.33.8.0/24 in-interface=WAN src-address=10.250.100.0/24
add action=accept chain=forward connection-state=established,related
add action=drop chain=forward connection-state=invalid
add action=accept chain=forward comment=SMTP connection-state=new dst-address=10.33.8.2 dst-port=25 protocol=tcp
add action=accept chain=forward comment=WWW connection-state=new disabled=yes dst-address=10.33.8.2 dst-port=80 protocol=tcp
add action=accept chain=forward comment=DNS connection-state=new dst-address=10.33.8.2 dst-port=10053 protocol=tcp
add action=accept chain=forward comment=DNS connection-state=new dst-address=10.33.8.2 dst-port=10053 protocol=udp
add action=accept chain=forward comment="SSH to s0.irl" connection-state=new dst-address=10.33.8.2 dst-port=22001 protocol=tcp
add action=reject chain=forward comment="Guests only to iNet" out-interface=!WAN reject-with=icmp-admin-prohibited src-address=10.33.7.0/24
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface=WAN
/ip firewall mangle
add action=change-mss chain=postrouting dst-address=10.250.100.0/24 new-mss=1360 out-interface=WAN protocol=tcp src-address=10.33.8.0/24 tcp-flags=syn tcp-mss=!0-1360
/ip firewall nat
add action=accept chain=srcnat dst-address=10.250.100.0/24 src-address=10.33.8.0/24
add action=masquerade chain=srcnat out-interface=WAN
add action=dst-nat chain=dstnat dst-address=public.ip.rtrA in-interface=WAN port=25 protocol=tcp to-addresses=10.33.8.2 to-ports=25
add action=dst-nat chain=dstnat dst-address=public.ip.rtrA in-interface=WAN port=53 protocol=udp to-addresses=10.33.8.2 to-ports=10053
add action=dst-nat chain=dstnat dst-address=public.ip.rtrA in-interface=WAN port=53 protocol=tcp to-addresses=10.33.8.2 to-ports=10053
add action=dst-nat chain=dstnat dst-address=public.ip.rtrA in-interface=WAN port=80 protocol=tcp to-addresses=10.33.8.2 to-ports=80
add action=dst-nat chain=dstnat dst-address=public.ip.rtrA in-interface=WAN port=443 protocol=tcp to-addresses=10.33.8.2 to-ports=443
add action=dst-nat chain=dstnat dst-address=public.ip.rtrA in-interface=WAN port=22001 protocol=tcp to-addresses=10.33.8.2 to-ports=22
/ip firewall raw
add action=accept chain=prerouting dst-port=4500 protocol=udp src-address=public.ip.rtrB src-port=4500
add action=accept chain=prerouting src-address=public.ip.rtrB
add action=notrack chain=prerouting dst-address=10.33.8.0/24 src-address=10.250.100.0/24
add action=notrack chain=prerouting dst-address=10.250.100.0/24 src-address=10.33.8.0/24
/ip ipsec identity
add my-id=fqdn:rtrA peer=rtrB remote-id=fqdn:rtrB
/ip ipsec policy
set 0 disabled=yes
add dst-address=10.250.100.0/24 peer=rtrB sa-dst-address=public.ip.rtrB sa-src-address=0.0.0.0 src-address=10.33.8.0/24 tunnel=yes
/ip route
add distance=1 gateway=ip.address.gwA
add distance=254 dst-address=10.0.0.0/8 type=unreachable
add distance=1 dst-address=10.250.100.0/24 gateway=ip.address.gwA
add distance=254 dst-address=169.254.0.0/16 type=unreachable
add distance=254 dst-address=172.16.0.0/12 type=unreachable
add distance=254 dst-address=192.168.0.0/16 type=unreachable
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh port=22000
set api disabled=yes
set winbox disabled=yes
set api-ssl disabled=yes
/ip ssh
set forwarding-enabled=remote strong-crypto=yes
/system clock
set time-zone-autodetect=no time-zone-name=UTC
/system identity
set name=rtrA
/system logging
add prefix=IPSEC topics=ipsec
add disabled=yes prefix=PKT topics=packet
/system ntp client
set enabled=yes primary-ntp=129.134.28.123 secondary-ntp=129.134.29.123 server-dns-names=time.facebook.com
/tool sniffer
set filter-interface=WAN filter-ip-address=public.ip.rtrB/32 filter-ip-protocol=udp filter-port=4500 only-headers=yes

sindy · Sat Mar 20, 2021 6:44 pm

I can't see anything wrong in your firewall rules, nor rp-filter=strict that can sometimes surprise.

So the next thing to come to my mind is that the packet sent by RtrB is fragmented as it travels to RtrA, and the second fragment gets dropped by some firewall on the way.

Since the UDP header is present only in the first fragment, the sniffer matching on port=4500 shows that fragment, whereas only reassembled packets reach the firewall.

You can verify this by sniffing into a file at RtrA and opening that file using Wireshark. If the More fragments bit in the Flags field of the IP header is set, this assumption gets confirmed.

How this could be related to a migration from a 750 to a 4011 is unclear to me, though.

Cray7 · Wed Apr 07, 2021 8:22 pm

Sindy, thanks for the hints.

The ipsec tunnel came up without changes to configurations so I think your assumptions about fragmented packets could be right - although I'm puzzled why ISPs would fragment/filter IKE packets and why did it manifest when I replaced the router. Might correlate with the power cycling of ISPs equipment.
In all this time, I was not able to reproduce the error condition. Don't like the "automagically" solved problems as they might reappear without warning - but now I at least have some tips for capturing the traffic for analysis.

Many thanks.

sindy · Wed Apr 07, 2021 9:19 pm

why did it manifest when I replaced the router

I've seen unrelated events to synchronize within tenths of second (not necessarily in networking), so I would not be surprised if something was wrong on the network path. Here, the window was longer, between the last establishment of the tunnel on the 750 and the first attempt with the 4011.

If that was the case, even the failed reassembly of fragmented packets may not have been the actual explanation.

Upgraded from 750G to RB4011iGS+ and my IKE2 IPSEC doesn't work anymore

Upgraded from 750G to RB4011iGS+ and my IKE2 IPSEC doesn't work anymore

Re: Upgraded from 750G to RB4011iGS+ and my IKE2 IPSEC doesn't work anymore

Re: Upgraded from 750G to RB4011iGS+ and my IKE2 IPSEC doesn't work anymore

Re: Upgraded from 750G to RB4011iGS+ and my IKE2 IPSEC doesn't work anymore

Re: Upgraded from 750G to RB4011iGS+ and my IKE2 IPSEC doesn't work anymore

Re: Upgraded from 750G to RB4011iGS+ and my IKE2 IPSEC doesn't work anymore