Amazon AWS VPN -- A Working Configuration Example and Bug

jimr · Thu Aug 07, 2014 4:45 am

We set out to connect our office LAN to an Amazon AWS Virtual Private Cloud (VPC) using Amazon's VPN facility. After some experimentation, we have a result that is quite usable and secure. Amazon doesn't support Microtik specifically, but they were helpful and not averse to answering questions to help with this setup.

There is a limitation in the Microtik software and a workaround. These will be discussed at the end. Hopefully, Microtik will address this problem in a future software update. In the meantime, it isn't serious obstacle to using AWS VPN.

The network we created is very typical. It could be the "out of the box" default NAT private network behind a single public IP address. The existence of the NAT just makes it interesting.

The first step is to sign-up for AWS. You'll want to create a VPC, which is a prerequisite to using AWS VPN. You'll also want to launch an EC2 machine instance into your VPC so you have something that can respond to "ping" later. The free-tier micro-instance of AWS Linux does this nicely and you never really need to even login or configure it. Just launch it and note the private IP address for testing.

Create a customer gateway and VPN from the web console of AWS. You will supply the public address of your router (or one public address, if you have more than one). In the recipe that follows, I've used x.x.x.x to refer to this address. Use your own!

You also will supply a private AS-number during creation of the VPN and select dynamic routing using BGP. Static routing is another option. For BGP, select any number in the private AS range that doesn't conflict with something you already use. In the example, I selected 65222.

Once AWS has done its magic, your VPN will be "available" and attached to your customer gateway and your VPC. You will then be able to download a configuration file. Choose the generic format (unless you are configuring one of the listed and supported routers). The downloaded file will contain the specific IP-addresses, passwords, and AS-numbers for your VPN. These may differ in small way from what is outlined below. Be sure to substitute your private values for those in the example.

The AWS VPN is implemented as a pair of IPsec tunnels between their facility and your router. The purpose of using two tunnels is to keep traffic flowing during times that IPsec may be negotiating security between the endpoints. Unfortunately, Microtik's limitation partly defeats this goal, but even so, the performance of the resulting setup is quite respectable. More on the limitation and performance at the end.

Referring to the AWS configuration sheet, first assign the customer tunnel endpoints to the public interface of the router. The public address of the router should already be assigned to this interface too.

IP addresses (add):

3   169.254.249.30/30  169.254.249.28  ether1-gateway                                                                     
4   169.254.249.26/30  169.254.249.24  ether1-gateway

Next, we make the default IPsec proposal correspond to AWS requirements. If you don't use these values, you are wasting your own time:

0  * name="default" auth-algorithms=sha1 enc-algorithms=aes-128-cbc lifetime=8m pfs-group=modp1024

Next, you configure five IPsec policies as below. You should be creating six policies, but because of the Microtik limitation, you will have to get along with these five:

0    ;;; AWS Tunnels
      src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

1    src-address=169.254.249.26/32 src-port=any dst-address=169.254.249.25/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0 

2    src-address=169.254.249.30/32 src-port=any dst-address=169.254.249.29/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

3    src-address=169.254.249.25/32 src-port=any dst-address=169.254.249.26/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.119 sa-dst-address=x.x.x.x proposal=default priority=0 

4    src-address=169.254.249.29/32 src-port=any dst-address=169.254.249.30/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.120 sa-dst-address=x.x.x.x proposal=default priority=0

Next, create the IPsec peers that support the Phase-1 handshake and password exchange. Be certain to use the AWS-supplied passwords and don't mix-up the pieces of tunnel #1 and tunnel #2. Use the two public IP-addresses supplied by AWS as a point of reference and keep all the parts for each tunnel together. If you mix them, it won't work.

0   ;;; AWS VPC Tunnel #2
     address=205.251.233.120/32 local-address=x.x.x.x passive=no port=500 auth-method=pre-shared-key 
     secret="<aws-supplied secret>" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no 
     proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s 
     dpd-maximum-failures=3 

1   ;;; AWS VPC Tunnel #1
     address=205.251.233.119/32 local-address=x.x.x.x passive=no port=500 auth-method=pre-shared-key 
     secret="<different aws-supplied secret>" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no 
     proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s 
     dpd-maximum-failures=3

Next, you need some firewall filters to allow IPsec to operate through your public interface. You could abbreviate these a lot, but this long version provides useful counters to provide feedback when it starts working:

1   chain=input action=accept protocol=ipsec-esp src-address=205.251.233.119 dst-address=x.x.x.x in-interface=ether1-gateway 

2   chain=input action=accept protocol=udp src-address=205.251.233.119 dst-address=x.x.x.x in-interface=ether1-gateway src-port=500 
     dst-port=500 

3   chain=input action=accept protocol=ipsec-esp src-address=205.251.233.120 dst-address=x.x.x.x in-interface=ether1-gateway 

4   chain=input action=accept protocol=udp src-address=205.251.233.120 dst-address=x.x.x.x in-interface=ether1-gateway src-port=500 
     dst-port=500

Next, you need a few rules so that BGP from the AWS can talk to your router:

6   chain=input action=accept protocol=tcp src-address=169.254.249.25 dst-address=169.254.249.26 dst-port=179 

7   chain=input action=accept protocol=tcp src-address=169.254.249.29 dst-address=169.254.249.30 dst-port=179

These two rules allow traffic to be routed freely between your AWS VPC and your private network. This is wide open in both directions. The VPC is secure, but you may want to provide a more selective filter.

23   ;;; VPC at AWS us-west-2x
     chain=forward action=accept src-address=172.31.0.0/16 in-interface=ether1-gateway 

24   chain=forward action=accept dst-address=172.31.0.0/16 in-interface=ether2-master-local

Because the local network is private and NAT is running, the following firewall NAT rule allows traffic to arrive at the VPC with the correct (private) source address:

0   ;;; critically important to AWS connectivity that this rule be ahead of "masquerade".
     chain=srcnat action=src-nat to-addresses=192.168.88.0/24 dst-address=172.31.0.0/16

The following firewall NAT filter rule is OPTIONAL and to be used only if the public address you supplied to AWS for your router is different from the public address used to masquerade your private NAT network. Don't use this unless you have two public addresses! And note the exclamation point!!!

1   chain=srcnat action=masquerade src-address=!x.x.x.x out-interface=ether1-gateway

You will need a destination nat rule to defeat any port-mapping rules that may be present in your existing dstnat chain. If none exist, this isn't needed, though it is harmless.

2   ;;; critically important to AWS connectivity that this rule be ahead of any NAT port-mapping
     chain=dstnat action=accept src-address=172.31.0.0/16 in-interface=ether1-gateway

Next, you configure some BGP instances using the addresses from the AWS instructions:

1    name="vgw-1" as=65222 router-id=169.254.249.30 redistribute-connected=no redistribute-static=yes redistribute-rip=no 
      redistribute-ospf=no redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table="" 

2    name="vgw-2" as=65222 router-id=169.254.249.26 redistribute-connected=no redistribute-static=yes redistribute-rip=no 
      redistribute-ospf=no redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table=""

Next, you define some BGP peers so that your BGP knows where to find AWS. The normal periodic behavior of BGP in connection with these peers will startup IPsec because of the handshake going to the far-end tunnel endpoint.

0 E name="awsvpc1" instance=vgw-1 remote-address=169.254.249.29 remote-as=7224 tcp-md5-key="" nexthop-choice=default multihop=no 
     route-reflect=yes hold-time=30s ttl=default in-filter="" out-filter="" address-families=ip update-source=169.254.249.30 
     default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no 

1 E name="awsvpc2" instance=vgw-2 remote-address=169.254.249.25 remote-as=7224 tcp-md5-key="" nexthop-choice=default multihop=no 
     route-reflect=yes hold-time=30s ttl=default in-filter="" out-filter="" address-families=ip update-source=169.254.249.26 
     default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no

Next, you define what local networks you want BGP to advertise to AWS. At minimum, this would be your local private network block. And hopefully, that isn't a 172.31.0.0/16 range network, as these conflict with the monster VPC that AWS allocates to you!

0   network=192.168.88.0/24 synchronize=yes

Assuming BGP is alive and you didn't mix things up, IPsec will install some security associations (SA) that look something like those below. If these don't appear, the tunnels are not working. Check passwords, address, firewall, etc. Also check for the presence of remote-peers, further below. If you see them, then the password and Phase-1 handshake is OK and your problem is firewall or BGP.

0 E  spi=0xF1A1CB0 src-address=205.251.233.120 dst-address=207.38.19.222 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature 
      auth-key="7d08d47b8dd2c46dbc0f8df62b4b899f49bb6a96" enc-key="26e38b12f243133cd8700122443cbf2f" addtime=aug/06/2014 22:20:49 
      expires-in=17m19s add-lifetime=16m/20m current-bytes=2045874 

1 E  spi=0xF7B18F0 src-address=205.251.233.119 dst-address=207.38.19.222 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature 
      auth-key="fd67f15352fae6598c37fa3965ad9990a5354443" enc-key="58c5a47403cb23a77a655135a4ef3263" addtime=aug/06/2014 22:22:10 
      expires-in=18m40s add-lifetime=16m/20m current-bytes=1770 

2 E  spi=0x57469D96 src-address=x.x.x.x dst-address=205.251.233.119 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature 
      auth-key="e2d031639925eb74a270c24941eb70d84b039eaa" enc-key="1dbd99b3b843d62237fe6a2da88b8a17" addtime=aug/06/2014 22:22:10 
      expires-in=18m40s add-lifetime=16m/20m current-bytes=1178 

3 E  spi=0xA099C19C src-address=x.x.x.x dst-address=205.251.233.120 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature 
      auth-key="cba17dab219c1494fe32f0ec034298a2e82274ec" enc-key="1ba56b9109f72a26057cfc5ff2e04afd" addtime=aug/06/2014 22:20:49 
      expires-in=17m19s add-lifetime=16m/20m current-bytes=113976118

Typical IPsec Remote Peers:

0 local-address=x.x.x.x remote-address=205.251.233.119 state=established side=initiator established=2m14s 

1 local-address=x.x.x.x remote-address=205.251.233.120 state=established side=initiator established=3m26s

If BGP is happy, you will see routing advertisements something like these:

PEER     PREFIX               NEXTHOP          AS-PATH                                                                 ORIGIN     LOCAL-PREF
awsvpc1  192.168.88.0/24      169.254.249.30                                                                           igp       
awsvpc2  192.168.88.0/24      169.254.249.26                                                                           igp

With working BGP, you will also see some additional dynamic routes to AWS VPC in your route table as below. Only one such route is allowed to be active at any given instant. Also note that the gateway address for these routes is the far-end address of the IPsec tunnel, indicating that all the communication is secured.

Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          x.x.x.w             1
 1 ADC  192.168.88.0/24    192.168.88.1    ether2-master-l...        0
 2 ADC  169.254.249.24/30  169.254.249.26  ether1-gateway            0
 3 ADC  169.254.249.28/30  169.254.249.30  ether1-gateway            0
 4 ADb  172.31.0.0/16                      169.254.249.25           20
 5  Db  172.31.0.0/16                      169.254.249.29           20
 6 ADC  x.x.x.v/27   x.x.x.x   ether1-gateway            0

If you got this far, you probably have it working.

Don't forget that AWS VPC is a paid service. You pay by the hour for the connection. However, the cost is about US$1 per day, so you're not going to go broke experimenting! If you actually use the tunnel, this is a bargain, especially if you can make it work with a low-cost, high-performance router!

In our case, a 750GL routerboard running v6.15 will see about 50 percent peak CPU utilization (400 MHz) handling sustained encrypted traffic on a 15 megabit/sec link. The effective throughput is about 1.6 Mbyte/sec for long file transfers. Latency is typically below 50 milliseconds between a host on our private network and an instance in our VPC.

The Microtik Limitation Explained

At last, Microtik should fix their little difficulty with IPsec. The problem is, their software does not allow two IPsec policies with identical src- and dst- addresses but different sa-dst-address to be configured. If you attempt to do this, the new policy is declared "invalid" and the whole IPsec policy list becomes unstable.

Following the example above, the missing policy is:

src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0

The presence of this policy would allow the VPN to continue to carry traffic during Phase-2 PFS negotiation periods, which AWS recommends occur every 8 minutes. For our purposes, this is overkill and we've extended the lifetime of Phase-2 a bit so the "pause to refresh" is less frequent. When it happens, the pause is a matter of milliseconds, but even this can be a lot on a fast link or supporting streaming traffic.

It's pretty clear that if you examine the routing table above, only one tunnel is ever actually carrying traffic between your router and AWS. But when Phase-2 timer fires and the active tunnel stops working briefly, the switch to the idle tunnel should be nearly instantaneous and only initiated by the end seeking to transmit (without end-to-end negotiation). The IPsec policy selection employed by Microtik seems to prevent this.

Having said all this the performance of the VPN is not affected, except during the Phase-2 handshake, which is very brief.

Cubo · Fri Sep 12, 2014 1:28 pm

Please indicate the settings downladed from Amazon

KPSI · Mon Sep 29, 2014 10:36 pm

We are tying to set up the same thing and having similar issues. How did you get ahold of Amazon for support question because we can not seem to get to anyone?

P.S. Could you by any chance look over our configuration and see if you could identify the problem?

jimr · Tue Sep 30, 2014 6:46 pm

We are tying to set up the same thing and having similar issues. How did you get ahold of Amazon for support question because we can not seem to get to anyone?

P.S. Could you by any chance look over our configuration and see if you could identify the problem?

I can't advise about AWS support. However, during the time we were getting started with them, we were paying the $100 per month support fee. They were very prompt and helpful with our questions, though they couldn't really help with the Microtik limitation. The two encounters we have were very high quality contacts. They were willing and able to setup screen sharing and watch what I was doing to the Microtik -- even appeared interested. They are not willing to actually configure a customer router, which is understandable.

As for your configuration, it is hard to advise. If you have the AWS configuration file, you would basically follow the steps outlined but substituting your specific addresses and passwords from that file.. Our configuration is still running for us today and it has been very reliable.

sbasnet · Mon Jan 12, 2015 6:32 am

hello, thanks for this explicit configuration. following this example i am able to setup vpn. but in my case, only one BGP comes up. the second one never comes up. but in your example you have shown that both BGP as up. so what am i doing wrong. should the second ipsec policy be also turned on? i have currently this in disabled mode as per your mikrotik limitation explanation.

jimr · Mon Jan 12, 2015 10:34 pm

Yes. Both IPSEC must be enabled to get both BGP relationships to establish.

It has been sometime since I posted those instructions. We have additional experience to share.

Because of the limitation in the Mikrotik, the VPN setup using a Mikrotik router will result in rare application-level issues that are visible to some users. For example, SMB file sharing to a Windows server will occasionally fail if conducted through the tunnel.

To work around this, we recently purchased a used Juniper ICG1000. This is a gigantic router compared to the Mikrotik, but they are readily available in the used market for about $300. The Juniper is an AWS supported configuration, the dual-tunnel limitations aren't present, and the resulting tunnel is rock-solid. Our electric bill may suffer a little, however.

We also tested a Juniper SCG1400. It worked as well and can be purchased used for about $200.

It would be great to see Mikrotik remove this limitation in their implementation of IPSEC.

sbasnet · Fri Jan 23, 2015 3:26 pm

Hello, following your example, the VPN is up. Only one BGP instance is up due to the Mikrotik limitation but BGP is learning prefix advertised by AWS VPC. Referring to the IP addresses used in your example, my PC is having 192.168.88.2/24. From my PC I can ping x.x.x.x, I can ping 169.254.249.29. But no matter what, I cannot ping 172.31.0.3 (the instance IP at AWS). So exactly how do I start troubleshooting the connection? Is it due to my Mikrotik configuration or something at AWS end?? The VPN at AWS is also learning the prefix (192.168.88.0/24) advertised by Mikrotik. I even searched for and added a mangle rule for fragmentation but no luck. Is there some debug in Mikrotik which can display what exactly is happening?
TIA.

jimr · Mon Jan 26, 2015 4:02 pm

On the Mikrotik side, confirm that you have a route through to the 172.31.x.x network. Also be certain that the firewall rules are in place to allow this access. Finally, be certain that you have the rule in place to bypass source address replacement on traffic outbound your local network and destined for AWS. You do not want this replacement. Your local "NAT" addresses must go through the tunnel -- not be replaced by the address of the local WAN interface. To stop this, the AWS destination address range much be the first rule in your NAT-tab (ahead of Masquerade or SRCNAT, depending on how you have it setup).

On the AWS side, be certain you have a route back to your local network (BGP should do this automatically), and make certain that the applicable security rules are in place. They exist in two places, one associated with the VPC (a stateless list of rules) and the other associated with each host instance.

jkarras · Wed Jan 28, 2015 5:33 am

To get around the two SA same policy issue did you try setting each policy with a different priority?

jimr · Wed Jan 28, 2015 6:58 am

Yes. Priority doesn't appear to matter. One of the policies is invariably declared "invalid" by RouterOS. Frankly, I don't think the priorities actually work.

cmurrayis · Wed May 13, 2015 1:39 am

Hi Guys,

Thanks for the guide - BGP is online and routes being exchanged however I am unable to ping from the local network to the VCP.

I can ping the far side of the tunnel 169 address however when I trace the 172 address it fails at the customers gateway and does not show the hops on the tunnel so I assume my rules are incorrect.

Anything you can recommend?

ip firewall nat print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=dstnat action=accept src-address=172.31.0.0/16 in-interface=ether1-wan-brocade-1/1/8 log=no log-prefix=""

1 ;;; AWS bypass NAT
chain=srcnat action=src-nat to-addresses=192.168.0.0/16 dst-address=172.31.0.0/16 log=no log-prefix=""

ip firewall filter print
Flags: X - disabled, I - invalid, D - dynamic
0 chain=forward action=accept protocol=ipsec-esp src-address=54.240.204.X dst-address=103.17.X.X in-interface=ether1-wan-brocade-1/1/8 log=no log-prefix=""

1 chain=forward action=accept protocol=udp src-address=54.240.204.X dst-address=103.17.X.X in-interface=ether1-wan-brocade-1/1/8 src-port=500 dst-port=500 log=no log-prefix=""

2 chain=forward action=accept protocol=ipsec-esp src-address=54.240.204.X dst-address=103.17.X.X in-interface=ether1-wan-brocade-1/1/8 log=no log-prefix=""

3 chain=forward action=accept protocol=udp src-address=54.240.204.X dst-address=103.17.X.X in-interface=ether1-wan-brocade-1/1/8 src-port=500 dst-port=500 log=no log-prefix=""

4 chain=forward action=accept dst-address=172.31.0.0/16 in-interface=ether3-DC log=no log-prefix=""

5 chain=forward action=accept src-address=172.31.0.0/16 in-interface=ether1-wan-brocade-1/1/8 log=no log-prefix=""

6 chain=forward action=accept dst-address=172.31.0.0/16 in-interface=ether2-mpn-brocade-1/1/9 log=no log-prefix=""

7 chain=input action=accept protocol=tcp src-address=169.254.247.17 dst-address=169.254.247.18 dst-port=179 log=no log-prefix=""

8 chain=input action=accept protocol=tcp src-address=169.254.247.22 dst-address=169.254.247.21 dst-port=179 log=no log-prefix=""

ip ipsec policy print
Flags: T - template, X - disabled, D - dynamic, I - inactive, * - default
0 T * group=*2 src-address=::/0 dst-address=::/0 protocol=all proposal=AWS template=yes

1 src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=103.17.X.X sa-dst-address=54.240.204.X proposal=AWS priority=0

2 src-address=169.254.247.18/32 src-port=any dst-address=169.254.247.17/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=103.17.X.X sa-dst-address=54.240.204.X proposal=AWS priority=0

3 src-address=169.254.247.22/32 src-port=any dst-address=169.254.247.21/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=103.17.X.X sa-dst-address=54.240.204.X proposal=AWS priority=0

4 src-address=169.254.247.17/32 src-port=any dst-address=169.254.247.18/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=54.240.204.X sa-dst-address=103.17.X.X proposal=AWS priority=0

5 src-address=169.254.247.21/32 src-port=any dst-address=169.254.247.22/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=54.240.204.X sa-dst-address=103.17.X.X proposal=AWS priority=0

mrz · Wed May 13, 2015 2:13 pm

@jimr

The presence of this policy would allow the VPN to continue to carry traffic during Phase-2 PFS negotiation periods, which AWS recommends occur every 8 minutes. For our purposes, this is overkill and we've extended the lifetime of Phase-2 a bit so the "pause to refresh" is less frequent. When it happens, the pause is a matter of milliseconds, but even this can be a lot on a fast link or supporting streaming traffic.

There shouldn't be any pause because new SAs are re-established when soft-timer expires (old SAs are still active and used for forwarding). Only when new SAs has "mature" state, old ones goes to "dying" state and get removed only after hard-timeout expires.

Test between two MT routers did not show any pause at all during SA renegotiation.

Are you actually see the "pause" until new SAs are established after lifetime expires, or it is just taken from AWS recommendation articles?

Also I do not see this as a requirement for any other implementations on Linux:
http://www.cloudhowto.org/amazon-vpc-wi ... -solution/
http://bleikertz.com/blog/amazon_vpc_with_linux.html
http://www.heitorlessa.com/working-with ... pn-part-2/

jimr · Wed May 27, 2015 5:27 pm

@mrz

Sorry about the delay on this. The notification disappeared in a pile of other e-mail...

I can assure you that the dual-tunnel requirement of AWS cannot be supported by the Mikrotik implementation. I'm not looking at it in the same level of implementation detail as you, but from an outward perspective. The missing rule I described is needed. This rule allows both tunnels to have a common end-point on the Mikrotik side. Any attempt to define this rule creates instability and is declared invalid in RouterOS. Without the rule, the AWS VPN support is crippled. During tunnel renegotiation, most persistent IP connections will time-out (e.g., Windows shares).

Unfortunately, our solution was to substitute a Juniper ScreenOS router that implements the required dual-tunnel perfectly. Juniper is one of about 10 different brands that AWS supports. The ScreenOS tunnel has been running for 116 days without trouble, as of this morning.

I'd suggest that the easiest way for MikroTik to resolve this problem would be to create an AWS VPN to test against. Last time I looked, the cost of one of these VPNs was $15 per month, with no minimum commitment.

@jimr

The presence of this policy would allow the VPN to continue to carry traffic during Phase-2 PFS negotiation periods, which AWS recommends occur every 8 minutes. For our purposes, this is overkill and we've extended the lifetime of Phase-2 a bit so the "pause to refresh" is less frequent. When it happens, the pause is a matter of milliseconds, but even this can be a lot on a fast link or supporting streaming traffic.
There shouldn't be any pause because new SAs are re-established when soft-timer expires (old SAs are still active and used for forwarding). Only when new SAs has "mature" state, old ones goes to "dying" state and get removed only after hard-timeout expires.

Test between two MT routers did not show any pause at all during SA renegotiation.

Are you actually see the "pause" until new SAs are established after lifetime expires, or it is just taken from AWS recommendation articles?

Also I do not see this as a requirement for any other implementations on Linux:
http://www.cloudhowto.org/amazon-vpc-wi ... -solution/
http://bleikertz.com/blog/amazon_vpc_with_linux.html
http://www.heitorlessa.com/working-with ... pn-part-2/

jimr · Wed May 27, 2015 5:36 pm

@mrz

A further comment --

All three of the example implementations cited in your post illustrate the same requirement: a single address on the Mikrotik size is used to terminate both tunnels. This is not possible in RouterOS. Just try it.

The problem exists whether or not one uses BGP for routing (an option with AWS VPN).

My comments about the re-negotiation delay have more to do with why two tunnels are desirable. In RouterOS, the problem is that you can't define two tunnels to one destination that terminate on a common IP-address on the near side.

mrz · Thu May 28, 2015 3:42 pm

My comments about the re-negotiation delay have more to do with why two tunnels are desirable. In RouterOS, the problem is that you can't define two tunnels to one destination that terminate on a common IP-address on the near side.

But according to all examples from link I provided address ranges are different for both policies:

Outside IP Addresses:
  - Customer Gateway:        : 1.2.3.4
  - VPN Gateway              : 72.21.209.225

Inside IP Addresses
  - Customer Gateway         : 169.254.255.2/30
  - VPN Gateway              : 169.254.255.1/30

Outside IP Addresses:
  - Customer Gateway:        : 1.2.3.4
  - VPN Gateway              : 72.21.209.193

Inside IP Addresses
  - Customer Gateway         : 169.254.255.6/30
  - VPN Gateway              : 169.254.255.5/30

Other example too:

Tunnel 1:
Pre-Shared Key : AAAA
Outside IP Addresses:
- Customer Gateway : 1.1.1.1
- Virtual Private Gateway : 2.2.2.2
Inside IP Addresses
- Customer Gateway : 169.254.248.10/30
- Virtual Private Gateway : 169.254.248.9/30
- Customer Gateway ASN : 65000
- Virtual Private Gateway ASN : 7224
- Neighbor IP Address : 169.254.248.9

Tunnel 2:
Pre-Shared Key : BBBB
Outside IP Addresses:
- Customer Gateway : 1.1.1.1
- Virtual Private Gateway : 3.3.3.3
Inside IP Addresses
- Customer Gateway : 169.254.248.14/30
- Virtual Private Gateway : 169.254.248.13/30
- Customer Gateway ASN : 65000
- Virtual Private Gateway ASN : 7224
- Neighbor IP Address : 169.254.248.13

Such configurations works also on RouterOS without any problems, or am I missing something?

riaanhuman · Tue Jun 02, 2015 12:15 pm

Hi Guys

Been reading through this thread as i am also battling with the amazon stuff.

I am configuring this for a client of mine.

My setup is slightly different in the sense that Amazon is not accessing a private ip on my clients network they are accessing a public IP assigned to a windows machine........

I have got the 1 tunnel up and a BGP is established. The subnet of 10.0.0.0/16 is announced back to me and i went as far as setting up the bgp network for the correct subnet range and not announcing a default route back to amazon.

According to BGP the advertising is all good and the BGP session is established.

I cannot ping the Amazon side ( client gave me 10.0.1.42 as the IP )

Because amazon is accessing a public portion of the network i do not need the NAT bypass rule ( altough i did include it ).

Suggestions are VERY welcome please.

giannello · Fri Jun 26, 2015 12:01 am

Hi jimr, and thanks for the great post - it's the most complete I've seen so far about the topic.

There's a couple of things that are not really clear to me:

0    ;;; AWS Tunnels
      src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0

Why is this SA needed? Packets from the 192.168.88.0/24 are src-nat'ed using the Inside IP Address of the Customer Gateway - provided by Amazon. The only SAs should be between the two Inside IP Addresses, as all the traffic is tunneled through them.

also

0   ;;; critically important to AWS connectivity that this rule be ahead of "masquerade".
     chain=srcnat action=src-nat to-addresses=192.168.88.0/24 dst-address=172.31.0.0/16

isn't the to-address supposed to be the Inside IP Address configured on our Mikrotik, so that the packets can be properly SNATed through it?

sthibault · Thu Aug 20, 2015 4:07 pm

Hi Guys

Thanks jimr for the example, I now have a working connection with my VPC using Mikrotik. I will use it for a while to see if the Mikrotik limitation impacts the traffic between our Lan and the VPC.

I hope Mikrotik will take care of the issue in the future.

This is potentially for them a feature that could help get their product known and increase sales. Let me explained, I'm working for SaaS provider and all our infrastructure is hosted on Amazon AWS. When it was the time to acquire a Firewall for our new office, one of the key feature I was looking for was AWS VPN support. Amazon can generate VPN configuration file for the most several vendors such as Cisco, Juniper, Fortinet, Vyatta, etc. (https://aws.amazon.com/vpc/faqs/#C9)

I have been a Mikrotik user for a few years but since it wasn't supported and I had read this thread explaining the limitation with the 2 ipsec tunnel to AWS, I decided agains't purchasing a Cloud Core Router and went with a Juniper SSG-320. This limitation cost Mikrotik a sale at that time and I'm sure that I'm not the only one. Having Mikrotik listed on the Amazon AWS VPN supported device list and included in the automatic generation of config file would be a major advertisement, endorsement and would lead to increase sale.

I went ahead with the purchase of the Juniper SSG, I was please with the ease of setting the AWS VPN connection. It literally takes seconds and it's stable.

I eventually got tired of the complexity of their ScreenOS and decided to order a CCR. I do prefer RouterOS for everything and the support from the community is great. I will keep the SSG and might use it only to connect to our different VPC using several AWS VPN connection, if this doesn't work correctly.

I really hope that Mikrotik look at this issue with another perpective and fix this issue.

Serge

ulysses · Fri Sep 25, 2015 2:31 pm

Year after, this post is still of a great value, thank you very much! Most of the things are straight-forward, but there is one thing that I have struggled to understand for already 20 hours in a row, searched high and low and still can't get my head around it..

First, something minor.

0   ;;; critically important to AWS connectivity that this rule be ahead of "masquerade".
     chain=srcnat action=src-nat to-addresses=192.168.88.0/24 dst-address=172.31.0.0/16

I was puzzled with this notation, since as I understand the idea behind this is that for the BGP routed network we want to avoid masquerading. However the way it is accomplished is rather strange: you change the source address to... itself? Or even, in case the packet arrives from another network that this local router is serving, say, 192.168.89.0/24, it will snat it, mapping to the 88 network, and may introduce conflicts (as I suspect). Instead, I used the return action at this point to simply leave the source address as is. Also, I had to add another rule to allow for the BGP traffic

 2    chain=srcnat action=return dst-address=172.31.0.0/16 log=no log-prefix="" 
 3    chain=srcnat action=return dst-address=169.254.0.0/16 log=yes log-prefix=""

Next, my biggest unresolved question
why do we need to create mirror ipsec policies for the BGP link network, and creating mirror for the main site to site policy breaks everything?
I mean the following snippet, where policies 1-3 and 2-4 are mirroring each other, but when I try to create a mirror policy for 0, even with explicit addresses, it just fails and the packets silently die during the RouthingDecision http://wiki.mikrotik.com/wiki/Manual:Packet_Flow phase (I see the packet on mangle prerouting and nat dstnat, and then it disapears; if I add explicit rule in routing to mark the destination as unreachable then it works by returning the corresponding ICMP message to the issuer). This makes it even harder for me to understand, because it means that changes to the ipsec policy implicitly influences the way the packet is routed, although it is not shown on the packet flow

0    ;;; AWS Tunnels
      src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

1    src-address=169.254.249.26/32 src-port=any dst-address=169.254.249.25/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0 

2    src-address=169.254.249.30/32 src-port=any dst-address=169.254.249.29/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

3    src-address=169.254.249.25/32 src-port=any dst-address=169.254.249.26/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.119 sa-dst-address=x.x.x.x proposal=default priority=0 

4    src-address=169.254.249.29/32 src-port=any dst-address=169.254.249.30/32 dst-port=any protocol=all action=encrypt level=require 
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.120 sa-dst-address=x.x.x.x proposal=default priority=0

I suspect that there is something I don't know about the basics of IPSec implementation in RouterOS, but there is absolutely no information on how it works.
I understand, that one policy should be enough for both encoding and decoding traffic, and in case of decoding it is applied in "reverse", i.e. a policy is working for a received packet where src-address of the policy is matched to the destination address in the received packet. That seems valid assumption since in the examples only one rule is added for a site-to-site tunneling, with single mirrored rule on the other side of the tunnel (different router). But then this should be working the same for the BGP networks! I tried removing rules 3 and 4 above, and everything worked just fine. However the fact that mirror policies for BGP networks work fine, and mirror policy for the main network crashes routing is something that doesn't let me sleep

Now, let's see one big issue with the proposed solution.

The author sets up both VPN tunnels and both BGP instances, but only one site to site policy is installed (active), due to the described RouterOS limitation. But that results in a very bad situation that I have observed.

Since Amazon side sees two working tunnels it's routing at one moment decides to go over the other channel, not the one that is being selected as active on our side. As long as the ipsec policy that would be used to decode the response is in inactive state (NB, "I" stands for inactive, not invalid), or simply missing (like the author suggested), then we observe unanswered ping. At the same time, on the other side our ping is received and even answered, but it returns over the channel that we can't decode. The way I am detecting this, is by monitoring the installed-as current bytes property, which obviously increases with every pair of ICMP messages, and the response goes over the other channel.

The only way to fix this is to shut down one of the BGP instances and the corresponding ipsec policy. Probably a script can be developed that would switch channels.

I am currently investigating options to control the AWS BGP decision on the route selection, so that I can actively control it based on the currently active ipsec policy on my side.

mark2015 · Sat Oct 17, 2015 11:16 pm

I followed all the steps; however, I still cannot ping over the tunnel. AWS says the tunnel are up and everything looks ok from my side. It even seems like the ICMP packet goes out, but nothing ever comes back, and this is from both sides. I can try to ping from AWS to my local network and the router sees the packet, but there's never a response.

ulysses · Mon Oct 19, 2015 10:33 am

the router sees the packet, but there's never a response.

Make sure you accept the packets in the /ip firewall filter forward chain

itcoop · Sun Nov 01, 2015 6:47 pm

Regardless if this thread is stale, it offers some good intelligence. I believe the trouble in your ipsec configuration is this:

level=require

it should be changed to:

level=unique

Mikrotik defaults an ipsec policy to level=require which allows a single pair of bidirectional crypto tunnels (SPI's) to be reused for different ipsec policies; this works when VPN is used between Mikrotik to Mikrotik but not to any other vendor's equipment. Using this level, "unique" SPI's will now be established for every IP range (NAT'd or otherwise).

As a rule-of-thumb for using Mikrotik ipsec VPNs to differing vendors equipment (Cisco, Juniper, AWS, or unknown), I always set level=unique before troubleshooting the tunnel. This simply makes the tunnel much more compliant with the RFC.

Next, you configure five IPsec policies as below. You should be creating six policies, but because of the Microtik limitation, you will have to get along with these five:

0    ;;; AWS Tunnels
      src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=unique 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

1    src-address=169.254.249.26/32 src-port=any dst-address=169.254.249.25/32 dst-port=any protocol=all action=encrypt level=unique 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0 

2    src-address=169.254.249.30/32 src-port=any dst-address=169.254.249.29/32 dst-port=any protocol=all action=encrypt level=unique 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0 

3    src-address=169.254.249.25/32 src-port=any dst-address=169.254.249.26/32 dst-port=any protocol=all action=encrypt level=unique 
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.119 sa-dst-address=x.x.x.x proposal=default priority=0 

4    src-address=169.254.249.29/32 src-port=any dst-address=169.254.249.30/32 dst-port=any protocol=all action=encrypt level=unique
      ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.120 sa-dst-address=x.x.x.x proposal=default priority=0
      
5	src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=unique 
      ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0

ulysses · Sun Nov 01, 2015 11:38 pm

As a rule-of-thumb for using Mikrotik ipsec VPNs to differing vendors equipment (Cisco, Juniper, AWS, or unknown), I always set level=unique before troubleshooting the tunnel. This simply makes the tunnel much more compliant with the RFC.

AWS requires that you aren't creating more than two SA per tunnel, so in case with AWS it is not allowed to set this value to "unique". A quote from http://docs.aws.amazon.com/AmazonVPC/la ... quirements:

You are limited to 1 unique Security Association (SA) pair per tunnel (1 inbound and 1 outbound), and therefore 2 unique SA pairs in total for 2 tunnels (4 SAs). Some devices use policy-based VPN and will create as many SAs as ACL entries. Therefore, you may need to consolidate your rules and then filter so you don't permit unwanted traffic.

dne · Wed Nov 25, 2015 2:27 pm

Quick and dirty but seems to work.

:global activeGatewayAWS [/ip route get [/ip route find dst-address=172.31.0.0/16 bgp active] gateway]
:global saDstAddress
:if ($activeGatewayAWS=ACTIVE_BGP_GATEWAY_ADDR) do={:global saDstAddress ADDR_GW_1} else={:global saDstAddress ADDR_GW_2}
/ip ipsec policy set [/ip ipsec policy find dst-address=172.31.0.0/16] sa-dst-address=$saDstAddress

ulysses · Wed Nov 25, 2015 2:32 pm

Quick and dirty but seems to work.

:global activeGatewayAWS [/ip route get [/ip route find dst-address=172.31.0.0/16 bgp active] gateway]
:global saDstAddress
:if ($activeGatewayAWS=ACTIVE_BGP_GATEWAY_ADDR) do={:global saDstAddress ADDR_GW_1} else={:global saDstAddress ADDR_GW_2}
/ip ipsec policy set [/ip ipsec policy find dst-address=172.31.0.0/16] sa-dst-address=$saDstAddress

What is the trigger for the script? Or it is just to manually switch between the channels?

dne · Wed Nov 25, 2015 3:40 pm

/system scheduler
add interval=1s name=setIPSecPolicy on-event=setIPSecPolicy ...

:global activeGatewayAWS
:global saDstAddress
:global IPSecPeer1 ADDR_GW_1
:global IPSecPeer2 ADDR_GW_2

:for timer from=1 to=6 step=1 do={
:global activeGatewayAWS [/ip route get [/ip route find dst-address=172.31.0.0/16 bgp active] gateway]
:if ($activeGatewayAWS=BGP_GATEWAY_ADDR1) do={:global saDstAddress $IPSecPeer1} else={:global saDstAddress $IPSecPeer2}

:if ( $saDstAddress!=[/ip ipsec policy get [/ip ipsec policy find dst-address=172.31.0.0/16] sa-dst-address]) do={
/ip ipsec policy set [/ip ipsec policy find dst-address=172.31.0.0/16] sa-dst-address=$saDstAddress
}
:delay 100ms
}

but i can´t guarantee anything, because this is my first ros script an i did not use it in production

ulysses · Wed Nov 25, 2015 3:48 pm

Nice..

But it has to be tested well, since I am afraid there may be positive loopback between the activation of the correct IPSec policy and the automatic active BGP route selection. It may even get into constant oscillation switching back and forth. Probably that is the reason you have a loop with 100ms sleep to have it stabilised...

In any case, good work. Maybe I will try this with my tunnel.

dne · Wed Nov 25, 2015 4:02 pm

To prevent flapping between both path we used a bgp filter to set a lower distance to the preferred path.

the 100ms delay is only to switch policy faster then one second.

danielharkerdp · Tue Feb 02, 2016 6:54 pm

Sorry if I am bumping an old thread here, I can't see a date on the thread from my phone... Odd.

Regardless, does setting level=unique get around the MikroTik Limitation??

ulysses · Thu Feb 04, 2016 12:47 pm

Sorry if I am bumping an old thread here, I can't see a date on the thread from my phone... Odd.

Regardless, does setting level=unique get around the MikroTik Limitation??

Nope

. Even otherwise, it is recommended against, since AWS explicitly states that it will only allow two SAs per VPN channel. I mentioned it somewhere up the thread, with a reference to the docs

italotejada · Thu Apr 14, 2016 5:26 pm

Hi,

Thanks for the guide, really is the only one that has worked well using my router Mikrotik. Currently I have three EC2 public instances running within my VPC, and once you establish the IPSec tunnel following the here mentioned steps, everything works perfect, but I can not access the public IP's of this instances, only private IP. If the IPSec tunnel is disabled, i can ping and access to the public IP's, but not private (which seems logical). I need create a route or rule within my Mikrotik to access the public IP's of my instances, once the IPSec tunnel established? I could give a hand with this? Thank you very much

ptofevski · Fri Aug 12, 2016 11:27 am

Hi all,

Has anyone tried the configuration with version 6.36 is there any change to the config, because I have a problem bringing up the tunnels, i can ping 169.x.x.x IP addresses but amazon says tunnel down, and can not ping the IPs inside the VPC.

lordzmd · Wed Sep 28, 2016 11:46 am

Hello all,

I have done the following config on my Mikrotik

ip address add address=169.254.40.38/30 disabled=no interface=WAN_BR network=169.254.40.36

ip ipsec proposal add name="aws" auth-algorithms=sha1 enc-algorithms=aes-128-cbc lifetime=8m pfs-group=modp1024

ip ipsec policy add src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=MyPublicIP sa-dst-address=52.57.15.8 proposal=aws priority=0

ip ipsec policy add src-address=169.254.40.38/32 src-port=any dst-address=169.254.40.37/32 dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes sa-src-address=MyPublicIP sa-dst-address=52.57.15.8 proposal=aws priority=0

ip ipsec peer add address=52.57.15.8/32 local-address=MyPublicIP passive=no port=500 auth-method=pre-shared-key \
secret="ZtkhRAD9Oehw5PvFWMCgxEssgb_Yo97A" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no \
proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s \
dpd-maximum-failures=3

An I do not get any SA installed.

I see although some IPSEC traffic in the logs.

Also I am not ussing BGP but static routes, and indicate my local net on the AWS side when creating VPN.

Also here is my firewall config.

add action=accept chain=input comment="Permit ICMP" protocol=icmp
add action=accept chain=input comment="Permit already established packages" \
    connection-state=established,related
add action=accept chain=input comment=PPTP dst-port=1723 protocol=tcp
add action=accept chain=input protocol=gre
add action=accept chain=input comment="Allow IKE" dst-port=500 in-interface=\
    WAN_BR protocol=udp
add action=accept chain=input comment="Allow IPSec-esp" in-interface=WAN_BR \
    protocol=ipsec-esp
add action=accept chain=input comment="Allow IPSec-ah" protocol=ipsec-ah
add action=accept chain=input dst-port=1234 protocol=udp
add action=drop chain=input comment="Drop all other traffic" in-interface=\
    WAN_BR
add action=accept chain=forward comment=\
    "Accept established connections in Forward" connection-state=\
    established,related
add action=drop chain=forward comment="Drop all Invalid on Forward" \
    connection-state=invalid
add action=drop chain=forward comment="Drop New connections in forward" \
    connection-nat-state=!dstnat connection-state=new in-interface=WAN_BR
/ip firewall nat
add action=masquerade chain=srcnat comment=NAT out-interface=WAN_BR

I specialy did not connect the second tunnel and also can say that Mikrotik is still having the bug mentioned above.

lordzmd · Wed Sep 28, 2016 2:50 pm

The problem of the SA was solved. Now I am looking into the routing problems. I am able to ping machines in my Data Center from the AWS cloud but not able to do the oposite. Investigating.

UPDATE:
Problem solved.Seems Static routing has a slightly configuration.

itcoop · Tue Apr 04, 2017 5:33 am

I've been running a modification of this configuration for a month now. The major revision I made to this config is to terminate each AWS VPN tunnel to separate 1100AHX2's as opposed to only one: this removed the "bug" and, most importantly, allows me to perform Mikrotik maintenance without taking the AWS VPN down.

Today I found that the performance was lacking during a large file transfer. Over a 2Gb raw connection, I was only seeing 30Mb going over the tunnel. Running pcap, I found many retransmissions.

Adding the following mangle rule adjusted the MTU over each tunnel; I'm now getting close to 750Mb on each tunnel:
/ip firewall mangle
add action=change-mss chain=forward dst-address=[my.vpc.net.aws] new-mss=1379 \
passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=!0-1379

The MMS adjustment was suggested in the IPSec Generic document from AWS:

IPSec ESP (Encapsulating Security Payload) inserts additional
headers to transmit packets. These headers require additional space,
which reduces the amount of space available to transmit application data.
To limit the impact of this behavior, we recommend the following
configuration on your Customer Gateway:
- TCP MSS Adjustment : 1379 bytes
- Clear Don't Fragment Bit : enabled
- Fragmentation : Before encryption

mark2015 · Tue Apr 18, 2017 9:57 pm

I have setup a VPN from my Mikrotik router to my AWS VPC as per instruction above (and oddly enough have gotten it work previously). Today however it is not working. I cannot get a full round ping to work from either AWS->myHost or myHost->AWS.

When pinging from AWS->myHost I see that packets are hitting the
1 chain=dstnat action=accept src-address=10.10.0.0/16 in-interface=ether1-gateway
rule as the packet count increases as I keep the pings going; however, they never reach my host (I'm running a tcpdump).

On the other side of the test myHost->AWS, I can actually see the ping hit the ec2 instance (via tcpdump) however, it never arrives back at my source. It feels like the router is dropping packets from AWS destined to my internal network for some reason. Any suggestion on what the issue maybe or how to debug?

Xblackheart · Fri Aug 11, 2017 10:38 pm

I have setup a VPN from my Mikrotik router to my AWS VPC as per instruction above (and oddly enough have gotten it work previously). Today however it is not working. I cannot get a full round ping to work from either AWS->myHost or myHost->AWS.

That is worrisome. I am having a very similar issue. We took our vpn down for business reasons, and now, I'm getting BGP propagation, installed SA's everything that you would think. but no routing, no pings, no ssh.

I've been staring at this all day, At least I'm not the only one.

mark2015 · Mon Oct 09, 2017 1:28 am

I have created a simple parameterized script to help with this whole process. All you need to do is populate the local vars at the top of the script.
The first two vars are specified in CIDR, and all the other vars are just IPs

Code: Select all

:local awsVpcCidr 

:local onPremCidr 



:local tunnelOneInsideIPVirtualPrivateGateway

:local tunnelOneOutsideIPVirtualPrivateGateway

:local tunnelOneInsideIPCustomerGateway



:local tunnelTwoInsideIPCustomerGateway

:local tunnelTwoInsideIPVirtualPrivateGateway

:local tunnelTwoOutsideIPVirtualPrivateGateway



:local myPublicIP [/ip address get [find interface="ether1-gateway"] address];



# add ip address $customer gateway

/ip address add address=$tunnelTwoInsideIPCustomerGateway/30 interface=ether1-gateway 

/ip address add address=$tunnelOneInsideIPCustomerGateway/30 interface=ether1-gateway





/ip ipsec proposal set 0 auth-algorithms=sha1 enc-algorithms=aes-128-cbc lifetime=8m pfs-group=modp1024





# AWS Tunnels - ipsec policy (the first one doesn't work for me) - see if this ever does start to work

#/ip ipsec policy add src-address=0.0.0.0/0 src-port=any dst-address=$awsVpcCidr dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \

#	sa-src-address=$myPublicIP sa-dst-address=$tunnelTwoOutsideIPVirtualPrivateGateway proposal=default priority=0 



# 

/ip ipsec policy add src-address=0.0.0.0/0 src-port=any dst-address=$awsVpcCidr dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \

	sa-src-address=$myPublicIP sa-dst-address=$tunnelOneOutsideIPVirtualPrivateGateway proposal=default priority=0



/ip ipsec policy add src-address=$tunnelOneInsideIPCustomerGateway/32 src-port=any dst-address=$tunnelOneInsideIPVirtualPrivateGateway/32 \

	dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \

	sa-src-address=$myPublicIP sa-dst-address=$tunnelOneOutsideIPVirtualPrivateGateway proposal=default priority=0 



/ip ipsec policy add src-address=$tunnelTwoInsideIPCustomerGateway/32 src-port=any dst-address=$tunnelTwoInsideIPVirtualPrivateGateway/32 \

	dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \

	sa-src-address=$myPublicIP sa-dst-address=$tunnelTwoOutsideIPVirtualPrivateGateway proposal=default priority=0 



/ip ipsec policy add src-address=$tunnelOneInsideIPVirtualPrivateGateway/32 src-port=any dst-address=$tunnelOneInsideIPCustomerGateway/32 \

	dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \

	sa-src-address=$tunnelOneOutsideIPVirtualPrivateGateway sa-dst-address=$myPublicIP proposal=default priority=0 



/ip ipsec policy add src-address=$tunnelTwoInsideIPVirtualPrivateGateway/32 src-port=any dst-address=$tunnelTwoInsideIPCustomerGateway/32 \

	dst-port=any protocol=all action=encrypt level=require ipsec-protocols=esp tunnel=yes \

	sa-src-address=$tunnelTwoOutsideIPVirtualPrivateGateway sa-dst-address=$myPublicIP proposal=default priority=0





# ip sec peers:

#0    AWS VPC Tunnel #2

/ip ipsec peer add address=$tunnelTwoOutsideIPVirtualPrivateGateway/32 local-address=$myPublicIP passive=no port=500 auth-method=pre-shared-key \

     secret="5f5qC810t6mrrKxGdpIbu0gmDQUyI9rk" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no \

     proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s \

     dpd-maximum-failures=3 



#1    AWS VPC Tunnel #1

/ip ipsec peer add address=$tunnelOneOutsideIPVirtualPrivateGateway/32 local-address=$myPublicIP passive=no port=500 auth-method=pre-shared-key \

     secret="26X18YmCMDhybMqh8QpcT7yLqshY4r4c" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no \

     proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s \

     dpd-maximum-failures=3





#firewall filters

/ip firewall filter add chain=input action=accept protocol=ipsec-esp src-address=$tunnelOneOutsideIPVirtualPrivateGateway dst-address=$myPublicIP in-interface=ether1-gateway 

/ip firewall filter add chain=input action=accept protocol=udp src-address=$tunnelOneOutsideIPVirtualPrivateGateway dst-address=$myPublicIP in-interface=ether1-gateway src-port=500 dst-port=500 

/ip firewall filter add chain=input action=accept protocol=ipsec-esp src-address=$tunnelTwoOutsideIPVirtualPrivateGateway dst-address=$myPublicIP in-interface=ether1-gateway 

/ip firewall filter add chain=input action=accept protocol=udp src-address=$tunnelTwoOutsideIPVirtualPrivateGateway dst-address=$myPublicIP in-interface=ether1-gateway src-port=500 dst-port=500



#firewall filter

/ip firewall filter add chain=input action=accept protocol=tcp src-address=$tunnelOneInsideIPVirtualPrivateGateway dst-address=$tunnelOneInsideIPCustomerGateway dst-port=179 

/ip firewall filter add chain=input action=accept protocol=tcp src-address=$tunnelTwoInsideIPVirtualPrivateGateway dst-address=$tunnelTwoInsideIPCustomerGateway dst-port=179



#firewall filter

# VPC at AWS us-west-2x

/ip firewall filter add chain=forward action=accept src-address=$awsVpcCidr in-interface=ether1-gateway 

/ip firewall filter add chain=forward action=accept dst-address=$awsVpcCidr in-interface=ether2-master-local





#nat rule

# critically important to AWS connectivity that this rule be ahead of "masquerade".

/ip firewall nat add chain=srcnat action=src-nat to-addresses=$onPremCidr dst-address=$awsVpcCidr





#The following firewall NAT filter rule is OPTIONAL and to be used only if the public address you supplied to AWS for your router is different from the public address used to masquerade your private NAT network. Don't use this unless you have two public addresses! And note the exclamation point!!!

#i don't have 2 public ip addresses... 

#chain=srcnat action=masquerade src-address=!$myPublicIP out-interface=ether1-gateway





#nat rule

# critically important to AWS connectivity that this rule be ahead of any NAT port-mapping

/ip firewall nat add chain=dstnat action=accept src-address=$awsVpcCidr in-interface=ether1-gateway





#bgp instances

/routing bgp instance add name="vgw-1" as=65530 router-id=$tunnelTwoInsideIPCustomerGateway redistribute-connected=no redistribute-static=yes redistribute-rip=no \

      redistribute-ospf=no redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table="" 



/routing bgp instance add name="vgw-2" as=65530 router-id=$tunnelOneInsideIPCustomerGateway redistribute-connected=no redistribute-static=yes redistribute-rip=no \

      redistribute-ospf=no redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table=""





#bgp peers

/routing bgp peer add name="awsvpc1" instance=vgw-1 remote-address=$tunnelTwoInsideIPVirtualPrivateGateway remote-as=7224 tcp-md5-key="" nexthop-choice=default multihop=no \

     route-reflect=yes hold-time=30s ttl=default in-filter="" out-filter="" address-families=ip update-source=$tunnelTwoInsideIPCustomerGateway \

     default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no 



/routing bgp peer add name="awsvpc2" instance=vgw-2 remote-address=$tunnelOneInsideIPVirtualPrivateGateway remote-as=7224 tcp-md5-key="" nexthop-choice=default multihop=no \

     route-reflect=yes hold-time=30s ttl=default in-filter="" out-filter="" address-families=ip update-source=$tunnelOneInsideIPCustomerGateway \

     default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no





# bpg to advertise network:

/routing bgp network add network=$onPremCidr synchronize=yes

indnti · Thu Jan 24, 2019 12:37 pm

Another solution without mirror-policies and a supernet config to bypass the mikrotik one network limitation on IPsec policies:
http://biplane.com.au/blog/?p=406

nitrag · Fri Jun 21, 2019 10:10 pm

wowsers! spent a long time on troubleshooting routing + timeout issues. Make sure on the AWS side (VPC > Routing Tables > [Your VPC]) that you are propagating the routes back to your router!

Screen Shot 2019-06-21 at 3.06.20 PM.jpg

Also, I updated the script for the dynamic routing config to support 6.44.3 here:
https://github.com/nitrag/aws-vpn-mikro ... ter-config

radrexvalera · Mon Oct 14, 2019 8:03 pm

I'm a total noob on running the github files that I downloaded.

How am I going to run these files ( script )?

Fi011 · Fri Mar 13, 2020 1:03 pm

Did anybody managed to get GRE tunnels up on AWS CHR? They are dead which never happened in my long Mikrotik history.
I want to implement GRE over IPSEC.

ruwi000 · Fri Jun 12, 2020 11:58 am

With version 6.47 I think I got the IPsec to AWS figured out. The new ipsec commands are not (yet) in the documentation, I hope this is fixed soon.
I have to experiment more with this, but it is something to work on.

This seems to work:

/ip ipsec profile
add dh-group=modp1024 dpd-interval=10s dpd-maximum-failures=3 enc-algorithm=aes-128 lifetime=8h name=aws-profile
/ip ipsec peer
add address=3.126.56.73/32 name=peer2 profile=aws-profile
add address=3.123.57.35/32 name=peer1 profile=aws-profile
/ip ipsec proposal
add enc-algorithms=aes-128-cbc lifetime=1h name=aws-proposal
/ip ipsec identity
add peer=peer1 secret=fWKxxxxxxxxxxxxxxxxxxxxxxxxxzQdK
add peer=peer2 secret=N8jxxxxxxxxxxxxxxxxxxxxxxxxxa6l6
/ip ipsec policy
add dst-address=169.254.127.189/32 peer=peer1 proposal=aws-proposal sa-dst-address=3.123.57.35 sa-src-address=x.x.x.x src-address=169.254.127.190/32 tunnel=yes
add dst-address=172.31.0.0/20 peer=peer1,peer2 proposal=aws-proposal sa-dst-address=3.126.56.73 sa-src-address=x.x.x.x src-address=x.x.x.x/24 tunnel=yes
add dst-address=169.254.204.165/32 peer=peer2 proposal=aws-proposal sa-dst-address=3.126.56.73 sa-src-address=x.x.x.x src-address=169.254.204.166/32 tunnel=yes

ulysses · Fri Jun 12, 2020 12:30 pm

Nice!
In the rule with two peers, you are listing the sa-dst-address as for just one of the peers. Will that work with the second route as well?

fcserg · Thu Oct 01, 2020 4:27 pm

Nice!
In the rule with two peers, you are listing the sa-dst-address as for just one of the peers. Will that work with the second route as well?

Did you figure it out?

sindy · Thu Oct 01, 2020 8:40 pm

In fact, the sa-dst-address is inherited from the address of the active peer - you don't need to enter it and it appears anyway. In the manual, it has been described as a Read Only parameter since quite some time ago; it can still be set but it is not necessary to set it (at least for tunnel mode SAs).

yxudous · Thu Feb 18, 2021 6:31 pm

I was able to set up an AWS site to site VPN and get both required ipsec tunnels working following this procedure

https://medium.com/@autogun/aws-site-to ... 77ca5e50ae

A few tips:
It only works on very recent firmware (6.48.1 for me).
Make sure in your BGP instances that the default ASN is set to the same as your AWS customer gateway, eg 65000. (this is not mentioned in the above link)
You might have to fiddle a little (enable +disable) the ipsec policies to get both tunnels up but once you get them up, they will stay up.

I can ssh fine between my local subnet (192.168.2.0/24) and my remote AWS VPC subnet (192.168.88.0/24) but since my remote VPC doesn't have any internet gateway, I would like the remote subnet to to be able to access the internet through the vpn and my local subnet. Somehow that is not working, I have a feeling it has to do with the nat exemption rules but I don't know how to modify the above procedure to get it working...

jrgnwgnr · Thu Mar 11, 2021 10:13 am

I am not a network expert and I am running a Mikrotik hAP ac² and configured AWS Site2Site VPN successfully.

Though sometimes (about once a month), the VPN stopps working, and i don't know why.
I always need to restart, so that everything works again.

It would be very important to reduce downtimes and also to reduce manual actions.
Do you know where i can start? Also may an other hardware may solve this? Maybe also another brand of router?

sindy · Sun Mar 14, 2021 10:55 am

Logging of the VPN operation would be a good start if you can notice the outage quickly enough or if you can connect a USB drive to the hAP ac²'s USB port. Sniffing of network traffic may be necessary if the above doesn't reveal the issue, which may require external hardware.
What kind of VPN is it? Bare IPsec or something else? Posting an anonymized export of your configuration is the best answer.

As for other hardware or brand - the reason may be outside the hAP ac², and if it is inside, it still may be a software issue that can be fixed if reported to the manufacturer. On the other hand, changing the device is a relatively quick way to find out whether the cause is internal or external - if a different hardware running a different operating system doesn't exhibit the same issue for three months, you can assume the issue was caused by the hAP ac².

I use hAP ac²s with IPsec at multiple places and have never suffered a VPN outage that would require a manual intervention to be resolved. But none of my setups has AWS at the remote side. And in more complex setups, where multiple devices at the same site use the same public IP addresses, some specific measures need to be taken so that restarts of individual routers wouldn't affect re-establishment of the tunnels, but I guess that's not your concern.

kuz8 · Wed Aug 25, 2021 1:46 am

I'd appreciate clues on what's wrong with my setup trying to utilize both tunnels between hardware Mikrotik 6.48.3 and AWS VPN in Dynamic routing setup (BGP) and having only random subset of my hosts being able to reach AWS side (tcpdump on AWS side shows it sends replies but my hosts don't receive them when both tunnels are up, when only one tunnel is up all works perfectly but not BGP-redundant):

I've used a combination of these methods:
Sep 2020 https://medium.com/@autogun/aws-site-to ... 77ca5e50ae
Mar 2018 https://kkc.github.io/2018/03/14/AWS-VP ... uterBoard/

I have two test machines on Mikrotik side to reach AWS.

1. It works when only one of 2 tunnels is up (the other one is deliberately down) for both for AWS- and Mikrotik-LAN both ways, but it's not redundant and AWS keeps reminding about it.
2. It doesn't work when both tunnels are up, AWS console shows both as UP, 1 BGP ROUTES, and when I ping from Mikrotik LAN to AWS either of my machines randomly doesn't receive responses from AWS. I see MKT route table showing all routes for both 169.254 AWS-side subnets:

/ip route print
...
 4 ADb  10.14.0.0/22                       169.254.222.5            20
 5  Db  10.14.0.0/22                       169.254.21.45            20
 6 ADb  10.14.8.0/22                       169.254.21.45            20
 7  Db  10.14.8.0/22                       169.254.222.5            20

tcpdump running on AWS side shows it receives all ICMP and sends responses back but I'm randomly not receiving the responses on the Mikrotik LAN machines side when both tunnels are up.

I've tried to have the 1/2 tunnels (10.8.8.0/24 is the whole - split to 0/25 and 128/25) like in the Medium article but it doesn't help:

/ip ipsec policy add dst-address=<<<AWS-PRIVATE-SUBNET-1>>> peer=AWS-1 proposal=aws-proposal src-address=<<<FIRST-HALF-OF-MT-LAN-CIDR>> tunnel=yes
/ip ipsec policy add dst-address=<<<AWS-PRIVATE-SUBNET-1>>> peer=AWS-1 proposal=aws-proposal src-address=<<<SECOND-HALF-OF-MT-LAN-CIDR>>> tunnel=yes

I've also tried to set level="unique" instead of "required" for multi-peer policies, and to change the order of peers, but it also didn't help.

/ip ipsec policy add comment=aws-sandbox-1 dst-address=<<<AWS-PRIVATE-SUBNET-1>>> peer=AWS-1,AWS-2 proposal=aws-proposal src-address=<<<MIKROTIK-PRIVATE-SUBNET>>> tunnel=yes
/ip ipsec policy add comment=aws-sandbox-1 dst-address=<<<AWS-PRIVATE-SUBNET-2>>> peer=AWS-1,AWS-2 proposal=aws-proposal src-address=<<<MIKROTIK-PRIVATE-SUBNET>>> tunnel=yes

My config - some firewall input rules are excessive from the github link, 65000 is exactly the ASN I have set up for my CG on AWS side.

/ip ipsec profile add dh-group=modp1024 dpd-interval=10s dpd-maximum-failures=3 enc-algorithm=aes-128 lifetime=8h name=aws-profile nat-traversal=no
/ip ipsec peer add address=<<<AWS-PUBLIC-IP-2>>> local-address=<<<MIKROTIK-PUBLIC-IP>>> name=AWS-2 profile=aws-profile
/ip ipsec peer add address=<<<AWS-PUBLIC-IP-1>>> local-address=<<<MIKROTIK-PUBLIC-IP>>> name=AWS-1 profile=aws-profile
/ip ipsec proposal add enc-algorithms=aes-128-cbc lifetime=1h name=aws-proposal
/routing bgp instance set default disabled=yes
/routing bgp instance add as=65000 client-to-client-reflection=no name=aws-vgw-1 redistribute-static=yes router-id=<<<169.254-MIKROTIK-SIDE-1>>>
/routing bgp instance add as=65000 client-to-client-reflection=no name=aws-vgw-2 redistribute-static=yes router-id=<<<169.254-MIKROTIK-SIDE-2>>>
/routing bgp network add network=<<<MIKROTIK-PRIVATE-SUBNET>>>
/routing bgp peer add comment=aws-sandbox-1 hold-time=30s instance=aws-vgw-1 keepalive-time=10s name=BGP-vpn-033cc6148d842ce4b-0 remote-address=<<<169.254-AWS-SIDE-1>>> route-reflect=yes
/routing bgp peer add comment=aws-sandbox-2 disabled=yes hold-time=30s instance=aws-vgw-2 keepalive-time=10s name=BGP-vpn-033cc6148d842ce4b-1 remote-address=<<<169.254-AWS-SIDE-2>>> route-reflect=yes
/ip firewall filter add action=accept chain=input comment=AWS dst-address=<<<169.254-MIKROTIK-SIDE-1>>> dst-port=179 protocol=tcp src-address=<<<169.254-AWS-SIDE-1>>>
/ip firewall filter add action=accept chain=input comment=AWS dst-address=<<<169.254-MIKROTIK-SIDE-2>>> dst-port=179 protocol=tcp src-address=<<<169.254-AWS-SIDE-2>>>
/ip firewall filter add action=accept chain=input comment=AWS dst-address=<<<MIKROTIK-PUBLIC-IP>>> in-interface=<<<MIKROTIK-PUBLIC-INTERFACE-NAME>>> protocol=ipsec-esp src-address=<<<AWS-PUBLIC-IP-1>>>
/ip firewall filter add action=accept chain=input comment=AWS dst-address=<<<MIKROTIK-PUBLIC-IP>>> dst-port=500 in-interface=<<<MIKROTIK-PUBLIC-INTERFACE-NAME>>> protocol=udp src-address=<<<AWS-PUBLIC-IP-1>>> src-port=500
/ip firewall filter add action=accept chain=input comment=AWS dst-address=<<<MIKROTIK-PUBLIC-IP>>> in-interface=<<<MIKROTIK-PUBLIC-INTERFACE-NAME>>> protocol=ipsec-esp src-address=<<<AWS-PUBLIC-IP-2>>>
/ip firewall filter add action=accept chain=input comment=AWS dst-address=<<<MIKROTIK-PUBLIC-IP>>> dst-port=500 in-interface=<<<MIKROTIK-PUBLIC-INTERFACE-NAME>>> protocol=udp src-address=<<<AWS-PUBLIC-IP-2>>> src-port=500
/ip firewall nat add action=accept chain=dstnat comment=AWS in-interface=<<<MIKROTIK-PUBLIC-INTERFACE-NAME>>> src-address=<<<AWS-PRIVATE-SUBNET-1>>>
/ip firewall nat add action=accept chain=srcnat comment=AWS dst-address=<<<AWS-PRIVATE-SUBNET-1>>> src-address=<<<MIKROTIK-PRIVATE-SUBNET>>> to-addresses=<<<MIKROTIK-PRIVATE-SUBNET>>>
/ip firewall nat add action=accept chain=srcnat comment=AWS dst-address=<<<AWS-PRIVATE-SUBNET-2>>> src-address=<<<MIKROTIK-PRIVATE-SUBNET>>> to-addresses=<<<MIKROTIK-PRIVATE-SUBNET>>>
/ip ipsec identity add peer=AWS-1 secret=<<<SECRET-1>>>
/ip ipsec identity add peer=AWS-2 secret=<<<SECRET-2>>>
/ip ipsec policy add comment=aws-sandbox-1 dst-address=<<<169.254-AWS-SIDE-1>>> peer=AWS-1 proposal=aws-proposal src-address=<<<169.254-MIKROTIK-SIDE-1>>> tunnel=yes
/ip ipsec policy add comment=aws-sandbox-2 disabled=yes dst-address=<<<AWS-PRIVATE-SUBNET-1>>> peer=AWS-2 proposal=aws-proposal src-address=<<<MIKROTIK-PRIVATE-SUBNET>>> tunnel=yes
/ip ipsec policy add comment=aws-sandbox-1 dst-address=<<<AWS-PRIVATE-SUBNET-1>>> peer=AWS-1 proposal=aws-proposal src-address=<<<MIKROTIK-PRIVATE-SUBNET>>> tunnel=yes
/ip ipsec policy add comment=aws-sandbox-2 disabled=yes dst-address=<<<AWS-PRIVATE-SUBNET-2>>> peer=AWS-2 proposal=aws-proposal src-address=<<<MIKROTIK-PRIVATE-SUBNET>>> tunnel=yes
/ip ipsec policy add comment=aws-sandbox-1 dst-address=<<<AWS-PRIVATE-SUBNET-2>>> peer=AWS-1 proposal=aws-proposal src-address=<<<MIKROTIK-PRIVATE-SUBNET>>> tunnel=yes
/ip ipsec policy add comment=aws-sandbox-2 disabled=yes dst-address=<<<169.254-AWS-SIDE-2>>> peer=AWS-2 proposal=aws-proposal src-address=<<<169.254-MIKROTIK-SIDE-2>>> tunnel=yes

mankomal · Sat Jan 08, 2022 8:51 am

I was able to set up an AWS site to site VPN and get both required ipsec tunnels working following this procedure

https://medium.com/@autogun/aws-site-to ... 77ca5e50ae

A few tips:
It only works on very recent firmware (6.48.1 for me).
Make sure in your BGP instances that the default ASN is set to the same as your AWS customer gateway, eg 65000. (this is not mentioned in the above link)
You might have to fiddle a little (enable +disable) the ipsec policies to get both tunnels up but once you get them up, they will stay up.

I can ssh fine between my local subnet (192.168.2.0/24) and my remote AWS VPC subnet (192.168.88.0/24) but since my remote VPC doesn't have any internet gateway, I would like the remote subnet to to be able to access the internet through the vpn and my local subnet. Somehow that is not working, I have a feeling it has to do with the nat exemption rules but I don't know how to modify the above procedure to get it working...

Did you try it with 6.49 also can you share the configuration once, as I am trying but not been able to do it on 6.49.1

galvesribeiro · Thu Mar 31, 2022 2:00 am

First of all a HUGE THANK YOU to the post Author. I've be battling with AWS CCR2004 and v7.1.5 for a while and I finally got it to work after finding this post with BGP.

I've used a combination of these methods:
Sep 2020 https://medium.com/@autogun/aws-site-to ... 77ca5e50ae
Mar 2018 https://kkc.github.io/2018/03/14/AWS-VP ... uterBoard/

I have two test machines on Mikrotik side to reach AWS.

1. It works when only one of 2 tunnels is up (the other one is deliberately down) for both for AWS- and Mikrotik-LAN both ways, but it's not redundant and AWS keeps reminding about it.
2. It doesn't work when both tunnels are up, AWS console shows both as UP, 1 BGP ROUTES, and when I ping from Mikrotik LAN to AWS either of my machines randomly doesn't receive responses from AWS. I see MKT route table showing all routes for both 169.254 AWS-side subnets:

As mentioned however, we aren't able to keep both tunnels UP. The first reason is because we can't have two policies with the same Src./Dst. as mentioned by the author. The second thing, is that if you add 2 peers to a single Policy, it will fail because AWS only allow a single SA pair to be installed on the same tunnel.

AWS tunnels in order to become both online, they require the customer gateway to support Asymmetric Routing (viewtopic.php?p=922743#p922743). I've discussed a lot with their premier support and this is the only way to have redundancy with both tunnel AFAIK.

That being said, I see the tunnels UP and the BGP routes being propagated however, since the 0.0.0.0/0 -> AWS policy is disabled deliberately on the second tunnel, I have a lot of packets lost when AWS decides to send the response on that tunnel. The same if the CCR decides to send packets thru that tunnel.

In other words, unless we have that missing policy added AND having the support to Asymmetric routing, we are not going to have the full redundancy provided by AWS VPNs. It is a shame that it isn't supported on MikroTik Routers

I hope some day they implement it. I moved away from Ubiquiti's UDM-PRO exactly because of the lack of BGP and two-tunnel support but I guess I made the wrong decision...