ipsec tunnel working in 6.37.5, not working in 6.40.8

xt22 · Thu Jun 14, 2018 6:08 pm

Hello,

I have RB1200 in a company connecting to another location via ipsec tunnel, working well. After the vpnfilter etc bugs, I decided to upgrade to last bugfix release 6.40.8, and it completely broke the tunnel - although I am pretty sure I saw something like "established" in ipsec - remote peers after the upgrade, but next morning, the company was totally offline. So I downgraded to 6.37.5, and voila - tunnel up again.

What the hell important has Mikrotik changed between 6.37.5<->6.40.8, that it breaks ipsec tunnels? All I get is

xx.xx.xx.xx peer sent packet for dead phase2
xx.xx.xx.xx failed to pre-process ph2 packet.

I saw some ipsec changes in changelogs, but I have very little space for experimenting, it is production router and I don't want to break it up again - the company's core business depends on this tunnel, so all upgrade/downgrade tests muse be done at weekend nights, but then I don't see the traffic I'd like to see. Has anyone encountered this? Or does anyone know about a change so important, that it breaks ipsec tunnels and must be taken care of?

There was L2TP vpn server before so there are some older settings left, with no effect on the tunnel (at least in 6.37.5), all the tunnel settings are the newer ones under [1].

/ip ipsec policy print
Flags: T - template, X - disabled, D - dynamic, I - inactive, * - default 
 0 T * group=default src-address=::/0 dst-address=::/0 protocol=all 
       proposal=default template=yes 

 1     src-address=192.168.5.0/24 src-port=any dst-address=192.168.1.0/24 
       dst-port=any protocol=all action=encrypt level=require 
       ipsec-protocols=ah-esp tunnel=yes sa-src-address=yy.yy.yy.yy 
       sa-dst-address=xx.xx.xx.xx proposal=xxxxxx priority=0


/ip ipsec peer print      
Flags: X - disabled, D - dynamic 
 0    address=0.0.0.0/0 local-address=0.0.0.0 passive=no port=500 
      auth-method=pre-shared-key secret=xxxxxx 
      generate-policy=port-strict policy-template-group=default 
      exchange-mode=main-l2tp send-initial-contact=yes nat-traversal=yes 
      hash-algorithm=sha1 enc-algorithm=3des dh-group=modp1024 lifetime=1d 
      dpd-interval=2m dpd-maximum-failures=5 

 1    address=xx.xx.xx.xx/32 local-address=0.0.0.0 passive=no port=500 
      auth-method=pre-shared-key secret=xxxxxx generate-policy=no 
      policy-template-group=default exchange-mode=main send-initial-contact=yes 
      nat-traversal=no proposal-check=obey hash-algorithm=sha1 
      enc-algorithm=aes-256 dh-group=modp1024 lifetime=1d lifebytes=0 
      dpd-interval=2m dpd-maximum-failures=5 


/ip ipsec remote-peers print     
 0 local-address=yy.yy.yy.yy remote-address=xx.xx.xx.xx state=established 
   side=responder established=8h53m34s

/ip ipsec proposal print             
Flags: X - disabled, * - default 
 0  * name="default" auth-algorithms=sha1 enc-algorithms=3des lifetime=30m 
      pfs-group=none 

 1    name="xxxxxx" auth-algorithms=sha1 enc-algorithms=aes-256-cbc 
      lifetime=30m pfs-group=modp1024

sindy · Thu Jun 14, 2018 7:06 pm

The parameters of individual components of ipsec configuration do differ between the 6.37.5 and 6.42.x, so it is likely that they also differ between 6.37.5 and 6.40.8 which I don't have anywhere any more.

So the question is how the configuration translation (which is part of the upgrade) works. For example, ah-esp value of ipsec-protocols parameter of /ip ipsec policy is not supported in 6.42.x, either you don't place the parameter there at all which means any of the two can be used, or you specify just one of the protocols, ah or esp.

So the right approach would be to upgrade and then use /ip ipsec export verbose hide-sensitive to analyse (or publish it here after obfuscating the addresses). If you have another Mikrotik, you can use it for the purpose - downgrade it to 6.37.5, use the ipsec-related part of configuration export from the production one on it, of course properly adjusted wrt local addresses if it refers to them, then upgrade the test board and export the ipsec configuration after conversion.

If you don't have a test box capable to run 6.37.5, do the upgrade of the production one on the weekend night, export the configuration to a file and download the file to a PC, and then use the following:

/system logging
add topics=ipsec

/log print follow-only file=ipsec-log topics~"ipsec"

In another terminal window, run

/log print follow-only where !(topics~"debug") && topics~"ipsec"

then, let the remote peer establish the connection, and when you see in the second window several messages you've seen before, mentioning "dead phase 2", stop the logging to file in first window, download ipsec-log.txt and analyse it or publish it here after replacing the local and remote public addresses by l.l.l.l and r.r.r.r, respectively.

xt22 · Mon Jun 18, 2018 1:44 am

Hello Sindy,

thank you for a very complex debug post. I made a test machine from one of my RB2011, compared the ipsec settings and it seems the ah-esp parameter is the only important thing changed. Differences between configs i found so far:

/ip ipsec 6.40.8 (compared to working 6.37.5)
ip ipsec peer:
- missing port=500, added with no effect
- missing local-address=0.0.0.0, not important
- missing lifebytes=0, not important
ip ipsec policy
- in 6.37.5 ipsec-protocols=ah-esp, in 6.40.8 tried both ipsec-protocols=ah, ipsec-protocols=esp with no wffect. Didn't try leaving it empty though.
- missing priotity=0, not important
added /ip ipsec user settings set xauth-use-radius=no - this is missing in 6.37.5

It seems the culprit is in the message "NO-PROPOSAL-CHOSEN" - although there definitely is one selected, I even created new "test" proposal - still with the same error:

log:

/log print follow-only where !(topics~"debug") && topics~"ipsec"
17:45:50 ipsec resent phase2 packet l.l.l.l[500]<=>r.r.r.r[500] 
17:45:51 ipsec r.r.r.r fatal NO-PROPOSAL-CHOSEN notify messsage, phase1 sho
uld be deleted. 
17:46:00 ipsec resent phase2 packet l.l.l.l[500]<=>r.r.r.r[500]
17:46:01 ipsec r.r.r.r fatal NO-PROPOSAL-CHOSEN notify messsage, phase1 sho
uld be deleted. 
17:46:10 ipsec r.r.r.r give up to get IPsec-SA due to time up to wait. 
17:46:10 ipsec IPsec-SA expired: ESP/Tunnel r.r.r.r[500]->l.l.l.l[500
] spi=0xf9442ab 
17:46:11 ipsec acquire for l.l.l.l <=> r.r.r.r 
17:46:11 ipsec suitable policy found: 192.168.5.0/24 <=> 192.168.1.0/24 
17:46:11 ipsec initiate new phase 2 negotiation: l.l.l.l[500]<=>r.r.r.r[500] 
17:46:11 ipsec sent phase2 packet l.l.l.l[500]<=>r.r.r.r[500]
17:46:11 ipsec r.r.r.r fatal NO-PROPOSAL-CHOSEN notify messsage, phase1 sho
uld be deleted. 
-- Ctrl-C to quit. Space prints separator. New entries will appear at bottom.

ip ipsec settings:

/ip ipsec mode-config
set [ find default=yes ] name=request-only
/ip ipsec policy group
set [ find default=yes ] name=default
/ip ipsec proposal
set [ find default=yes ] auth-algorithms=sha1 disabled=yes enc-algorithms=\
    3des lifetime=30m name=default pfs-group=none
add auth-algorithms=sha1 disabled=no enc-algorithms=aes-256-cbc lifetime=30m \
    name=test pfs-group=modp1024
/ip ipsec peer
add address=r.r.r.r/32 auth-method=pre-shared-key dh-group=modp1024 \
    disabled=no dpd-interval=2m dpd-maximum-failures=5 enc-algorithm=aes-256 \
    exchange-mode=main generate-policy=no hash-algorithm=sha1 lifetime=1d \
    nat-traversal=no policy-template-group=default proposal-check=obey \
    send-initial-contact=yes
/ip ipsec policy
set 0 disabled=yes dst-address=::/0 group=default proposal=default protocol=\
    all src-address=::/0 template=yes
add action=encrypt disabled=no dst-address=192.168.1.0/24 dst-port=any \
    ipsec-protocols=esp level=require proposal=test protocol=all \
    sa-dst-address=r.r.r.r sa-src-address=l.l.l.l src-address=\
    192.168.5.0/24 src-port=any tunnel=yes
/ip ipsec user settings
set xauth-use-radius=no

Downgrade to 6.37.5 - it works like a charm again. I also tried 6.42.3, but except it destroyed my master-slave interface settings, it had no effect to ipsec tunnel.
I tried to enable & set the "default" proposal, with no effect.

sindy · Mon Jun 18, 2018 2:42 pm

In your default phase2 proposal configuration I can see only 3des to be permitted, does that match the setting of the remote peer? 3des is quite an old encryption method. What is the remote peer, another Mikrotik or some other device?

Is there the checkbox map of encryption algorithms IP->IPsec->Proposal in 6.37.5 or does it propose everything it can do, without possibility to affect that?

xt22 · Mon Jun 18, 2018 6:45 pm

the default proposal is disabled, I would have removed it but I can't - it says something like "cannot remove ipsec policy (default)".
I have already tried to set and enable the default proposal (sha1, aes-256 cbc like the used proposal), but it has no effect (it is in the last part of my previous post).

Unfortunately I don't know what am I connecting to, the device is not in my posession, it is some kind of voip gateway one of the vendors.

Yes, there is a checkbox map for encryption algorithms, in fact it is identical to 6.40.8 - all checkboxes and fields are the same, there is no difference

sindy · Mon Jun 18, 2018 10:39 pm

In that case I'm afraid the game is nearly over.

While you can see the Phase 1 proposal using Wireshark, as it is sent in plaintext for obvious reasons, Phase 2 proposal is sent encrypted and you cannot choose encryption method null for Phase 1 so that the Phase 2 proposal would be sent in plaintext.

So the only way to see the difference between Phase 2 proposal sent by 6.37.5 and 6.40.8 is to use another Mikrotik as a peer and use ipsec debug on it to see the two proposals in the log.

passarelli · Mon Jun 18, 2018 10:59 pm

Hi xt22, have you created the FILTER and NAT rules? Could print it for us?
One good shot is remove tunnel settings and add it again, but if there is more than 3 or 5 tunnels, this could be awful.

xt22 · Mon Jun 18, 2018 11:14 pm

well, I probably can do that, I have enough CCRs and tons of RB2011s I can use for testing. I can create a working tunnel with two 6.37.5, and then upgrade on of them and see - i probably can even indentify the first version that broke this by upgrading to all the versions between 6.37.5 -- 6.40.8.

I thought this will be a lot easier, I can downgrade & upgrade all day and it is still the same - new versions always don't work, the old 6.37.5 one always does - without any changes, just upgrade/downgrade.. There definitely must be a problem in some version >6.37.5 and up, or at least significant change breaking ipsec/proposal somehow.. I doubt I am the only one on earth using ipsec tunnel with mikrotik, the setting is pretty straightforward and common, actually it is almost the mikrotik example here https://wiki.mikrotik.com/wiki/Manual:I ... sec_tunnel

I'll try to make some more tests and plug two rbs together and write here te results, I'd really like to upgrade the old 6.37.5 version

xt22 · Mon Jun 18, 2018 11:47 pm

Hi Pasarelli,

yes, it has public ip, so it indeed has FILTER and NAT set. But the remote ip is completely allowed, the rule is right after accept related-established and drop-invalid.

filter:
 5    chain=input action=accept connection-state=established 
      in-interface=ether1 log-prefix="" 
 6    chain=input action=accept connection-state=related in-interface=ether1 
      log-prefix="" 
 7    chain=input action=drop connection-state=invalid in-interface=ether1 
      log=no log-prefix="" 
 8    chain=input action=accept src-address=r.r.r.r in-interface=ether1 
      log=yes log-prefix=""  
14    chain=input action=drop src-address=10.0.0.0/8 in-interface=ether1 
      log-prefix="" 
15    chain=input action=drop src-address=172.16.0.0/12 in-interface=ether1 
      log=no log-prefix="" 
16    chain=input action=accept src-address=192.168.5.0/24 in-interface=ether2 
      log=no log-prefix="" 
17    chain=input action=drop src-address=192.168.0.0/16 in-interface=ether1 
      log=no log-prefix="" 
18    chain=input action=drop src-address=127.0.0.0/8 in-interface=ether1 
      log-prefix="" 
19    chain=input action=drop src-address=255.255.255.255 in-interface=ether1 
      log-prefix="" 
20    chain=input action=drop dst-address=0.0.0.0 in-interface=ether1 
      log-prefix="" 
21    chain=input action=drop src-address=224.0.0.0/4 in-interface=ether1 
      log-prefix="" 
22    chain=input action=drop protocol=!udp dst-address=224.0.0.0/4 
      in-interface=ether1 log-prefix="" 
24    chain=input action=drop src-address=240.0.0.0/5 in-interface=ether1 
      log-prefix="" 
25    chain=input action=drop src-address=0.0.0.0/8 in-interface=ether1 
      log-prefix="" 
26    chain=input action=drop src-address=169.254.0.0/16 in-interface=ether1 
      log-prefix="" 
27    chain=input action=drop src-address=192.0.2.0/24 in-interface=ether1 
      log-prefix="" 
30    ;;; ping - echo request
      chain=input action=accept protocol=icmp in-interface=ether1 
      icmp-options=8:0-255 log-prefix="" 
34    chain=input action=accept protocol=udp dst-address=l.l.l.l 
      in-interface=ether1 dst-port=500 log=no log-prefix="" 
36    chain=input action=accept protocol=udp dst-address=l.l.l.l 
      in-interface=ether1 dst-port=4500 log=no log-prefix="" 
38    chain=input action=accept protocol=udp dst-address=l.l.l.l 
      in-interface=ether1 dst-port=1701 log=no log-prefix="" 
40    chain=input action=accept protocol=ipsec-esp dst-address=l.l.l.l 
      in-interface=ether1 log=no log-prefix="" 
41    chain=input action=accept protocol=gre in-interface=ether1 log=no 
      log-prefix="" 
42    chain=forward action=drop p2p=all-p2p log-prefix="" 
43    chain=input action=drop in-interface=ether1 log-prefix=""

nat:
 0    chain=srcnat action=accept src-address=192.168.5.0/24 
      dst-address=192.168.1.0/24 log=no log-prefix="" 
 1    chain=srcnat action=src-nat to-addresses=l.l.l.l 
      src-address=192.168.5.0/24 out-interface=ether1 log=no log-prefix=""

I moved the config to a whole new device, although I did not delete it and create a fresh one - it might be worth a try, maybe the 6.40.8 is some kind of stuck with the settings from 6.37.5.. as I wrote earlier, I can't mess with the device during day, I must wait to weekend night. So I'll try to reproduce it in my office and do some more debugging

sindy · Tue Jun 19, 2018 12:25 am

As it fails on incompatibility of Phase 2 proposal, firewall rules cannot be related - if they were, it would have to fail either much earlier or much later.

tippenring · Wed Jun 20, 2018 5:54 pm

Hello,

I have RB1200 in a company connecting to another location via ipsec tunnel, working well. After the vpnfilter etc bugs, I decided to upgrade to last bugfix release 6.40.8, and it completely broke the tunnel - although I am pretty sure I saw something like "established" in ipsec - remote peers after the upgrade, but next morning, the company was totally offline. So I downgraded to 6.37.5, and voila - tunnel up again.

What the hell important has Mikrotik changed between 6.37.5<->6.40.8, that it breaks ipsec tunnels? All I get is
Code: Select all
xx.xx.xx.xx peer sent packet for dead phase2
xx.xx.xx.xx failed to pre-process ph2 packet.

In my experience so far, this is happening because the peer configs do not match on the subnets to be transported over the tunnel. For example, one side has 192.168.0.0/16, and the other has 192.168.50.0/24. This used to match up to around 6.37.5 (of course only in one direction, but it worked). Now, the mismatch does not work at all. Both sides must match.

rockiechin · Mon Jun 25, 2018 12:15 pm

I got the same problem here. I have upgrade my RB1100AHx4 to 6.42.3 to solve the VPNFilter malware problem, the IPSec tunnel is no longer working.
I have already followed the setup from wiki, the connection is established. I saw I have traffic sent to Dst.,( but no data received from Dst.).
I need to know why as IPSec tunnel is necessary.
Anyone here can help?

Rockie

sindy · Tue Jun 26, 2018 3:53 pm

Anyone can help if you publish the configuration exports and the log as described here.

ipsec tunnel working in 6.37.5, not working in 6.40.8

ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8

Re: ipsec tunnel working in 6.37.5, not working in 6.40.8