Amazon AWS VPN -- A Working Configuration Example and Bug
Posted: Thu Aug 07, 2014 4:45 am
We set out to connect our office LAN to an Amazon AWS Virtual Private Cloud (VPC) using Amazon's VPN facility. After some experimentation, we have a result that is quite usable and secure. Amazon doesn't support Microtik specifically, but they were helpful and not averse to answering questions to help with this setup.
There is a limitation in the Microtik software and a workaround. These will be discussed at the end. Hopefully, Microtik will address this problem in a future software update. In the meantime, it isn't serious obstacle to using AWS VPN.
The network we created is very typical. It could be the "out of the box" default NAT private network behind a single public IP address. The existence of the NAT just makes it interesting.
The first step is to sign-up for AWS. You'll want to create a VPC, which is a prerequisite to using AWS VPN. You'll also want to launch an EC2 machine instance into your VPC so you have something that can respond to "ping" later. The free-tier micro-instance of AWS Linux does this nicely and you never really need to even login or configure it. Just launch it and note the private IP address for testing.
Create a customer gateway and VPN from the web console of AWS. You will supply the public address of your router (or one public address, if you have more than one). In the recipe that follows, I've used x.x.x.x to refer to this address. Use your own!
You also will supply a private AS-number during creation of the VPN and select dynamic routing using BGP. Static routing is another option. For BGP, select any number in the private AS range that doesn't conflict with something you already use. In the example, I selected 65222.
Once AWS has done its magic, your VPN will be "available" and attached to your customer gateway and your VPC. You will then be able to download a configuration file. Choose the generic format (unless you are configuring one of the listed and supported routers). The downloaded file will contain the specific IP-addresses, passwords, and AS-numbers for your VPN. These may differ in small way from what is outlined below. Be sure to substitute your private values for those in the example.
The AWS VPN is implemented as a pair of IPsec tunnels between their facility and your router. The purpose of using two tunnels is to keep traffic flowing during times that IPsec may be negotiating security between the endpoints. Unfortunately, Microtik's limitation partly defeats this goal, but even so, the performance of the resulting setup is quite respectable. More on the limitation and performance at the end.
Referring to the AWS configuration sheet, first assign the customer tunnel endpoints to the public interface of the router. The public address of the router should already be assigned to this interface too.
IP addresses (add):
Next, we make the default IPsec proposal correspond to AWS requirements. If you don't use these values, you are wasting your own time:
Next, you configure five IPsec policies as below. You should be creating six policies, but because of the Microtik limitation, you will have to get along with these five:
Next, create the IPsec peers that support the Phase-1 handshake and password exchange. Be certain to use the AWS-supplied passwords and don't mix-up the pieces of tunnel #1 and tunnel #2. Use the two public IP-addresses supplied by AWS as a point of reference and keep all the parts for each tunnel together. If you mix them, it won't work.
Next, you need some firewall filters to allow IPsec to operate through your public interface. You could abbreviate these a lot, but this long version provides useful counters to provide feedback when it starts working:
Next, you need a few rules so that BGP from the AWS can talk to your router:
These two rules allow traffic to be routed freely between your AWS VPC and your private network. This is wide open in both directions. The VPC is secure, but you may want to provide a more selective filter.
Because the local network is private and NAT is running, the following firewall NAT rule allows traffic to arrive at the VPC with the correct (private) source address:
The following firewall NAT filter rule is OPTIONAL and to be used only if the public address you supplied to AWS for your router is different from the public address used to masquerade your private NAT network. Don't use this unless you have two public addresses! And note the exclamation point!!!
You will need a destination nat rule to defeat any port-mapping rules that may be present in your existing dstnat chain. If none exist, this isn't needed, though it is harmless.
Next, you configure some BGP instances using the addresses from the AWS instructions:
Next, you define some BGP peers so that your BGP knows where to find AWS. The normal periodic behavior of BGP in connection with these peers will startup IPsec because of the handshake going to the far-end tunnel endpoint.
Next, you define what local networks you want BGP to advertise to AWS. At minimum, this would be your local private network block. And hopefully, that isn't a 172.31.0.0/16 range network, as these conflict with the monster VPC that AWS allocates to you!
Assuming BGP is alive and you didn't mix things up, IPsec will install some security associations (SA) that look something like those below. If these don't appear, the tunnels are not working. Check passwords, address, firewall, etc. Also check for the presence of remote-peers, further below. If you see them, then the password and Phase-1 handshake is OK and your problem is firewall or BGP.
Typical IPsec Remote Peers:
If BGP is happy, you will see routing advertisements something like these:
With working BGP, you will also see some additional dynamic routes to AWS VPC in your route table as below. Only one such route is allowed to be active at any given instant. Also note that the gateway address for these routes is the far-end address of the IPsec tunnel, indicating that all the communication is secured.
If you got this far, you probably have it working.
Don't forget that AWS VPC is a paid service. You pay by the hour for the connection. However, the cost is about US$1 per day, so you're not going to go broke experimenting! If you actually use the tunnel, this is a bargain, especially if you can make it work with a low-cost, high-performance router!
In our case, a 750GL routerboard running v6.15 will see about 50 percent peak CPU utilization (400 MHz) handling sustained encrypted traffic on a 15 megabit/sec link. The effective throughput is about 1.6 Mbyte/sec for long file transfers. Latency is typically below 50 milliseconds between a host on our private network and an instance in our VPC.
The Microtik Limitation Explained
At last, Microtik should fix their little difficulty with IPsec. The problem is, their software does not allow two IPsec policies with identical src- and dst- addresses but different sa-dst-address to be configured. If you attempt to do this, the new policy is declared "invalid" and the whole IPsec policy list becomes unstable.
Following the example above, the missing policy is:
The presence of this policy would allow the VPN to continue to carry traffic during Phase-2 PFS negotiation periods, which AWS recommends occur every 8 minutes. For our purposes, this is overkill and we've extended the lifetime of Phase-2 a bit so the "pause to refresh" is less frequent. When it happens, the pause is a matter of milliseconds, but even this can be a lot on a fast link or supporting streaming traffic.
It's pretty clear that if you examine the routing table above, only one tunnel is ever actually carrying traffic between your router and AWS. But when Phase-2 timer fires and the active tunnel stops working briefly, the switch to the idle tunnel should be nearly instantaneous and only initiated by the end seeking to transmit (without end-to-end negotiation). The IPsec policy selection employed by Microtik seems to prevent this.
Having said all this the performance of the VPN is not affected, except during the Phase-2 handshake, which is very brief.
There is a limitation in the Microtik software and a workaround. These will be discussed at the end. Hopefully, Microtik will address this problem in a future software update. In the meantime, it isn't serious obstacle to using AWS VPN.
The network we created is very typical. It could be the "out of the box" default NAT private network behind a single public IP address. The existence of the NAT just makes it interesting.
The first step is to sign-up for AWS. You'll want to create a VPC, which is a prerequisite to using AWS VPN. You'll also want to launch an EC2 machine instance into your VPC so you have something that can respond to "ping" later. The free-tier micro-instance of AWS Linux does this nicely and you never really need to even login or configure it. Just launch it and note the private IP address for testing.
Create a customer gateway and VPN from the web console of AWS. You will supply the public address of your router (or one public address, if you have more than one). In the recipe that follows, I've used x.x.x.x to refer to this address. Use your own!
You also will supply a private AS-number during creation of the VPN and select dynamic routing using BGP. Static routing is another option. For BGP, select any number in the private AS range that doesn't conflict with something you already use. In the example, I selected 65222.
Once AWS has done its magic, your VPN will be "available" and attached to your customer gateway and your VPC. You will then be able to download a configuration file. Choose the generic format (unless you are configuring one of the listed and supported routers). The downloaded file will contain the specific IP-addresses, passwords, and AS-numbers for your VPN. These may differ in small way from what is outlined below. Be sure to substitute your private values for those in the example.
The AWS VPN is implemented as a pair of IPsec tunnels between their facility and your router. The purpose of using two tunnels is to keep traffic flowing during times that IPsec may be negotiating security between the endpoints. Unfortunately, Microtik's limitation partly defeats this goal, but even so, the performance of the resulting setup is quite respectable. More on the limitation and performance at the end.
Referring to the AWS configuration sheet, first assign the customer tunnel endpoints to the public interface of the router. The public address of the router should already be assigned to this interface too.
IP addresses (add):
Code: Select all
3 169.254.249.30/30 169.254.249.28 ether1-gateway
4 169.254.249.26/30 169.254.249.24 ether1-gatewayNext, we make the default IPsec proposal correspond to AWS requirements. If you don't use these values, you are wasting your own time:
Code: Select all
0 * name="default" auth-algorithms=sha1 enc-algorithms=aes-128-cbc lifetime=8m pfs-group=modp1024Code: Select all
0 ;;; AWS Tunnels
src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require
ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0
1 src-address=169.254.249.26/32 src-port=any dst-address=169.254.249.25/32 dst-port=any protocol=all action=encrypt level=require
ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0
2 src-address=169.254.249.30/32 src-port=any dst-address=169.254.249.29/32 dst-port=any protocol=all action=encrypt level=require
ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.120 proposal=default priority=0
3 src-address=169.254.249.25/32 src-port=any dst-address=169.254.249.26/32 dst-port=any protocol=all action=encrypt level=require
ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.119 sa-dst-address=x.x.x.x proposal=default priority=0
4 src-address=169.254.249.29/32 src-port=any dst-address=169.254.249.30/32 dst-port=any protocol=all action=encrypt level=require
ipsec-protocols=esp tunnel=yes sa-src-address=205.251.233.120 sa-dst-address=x.x.x.x proposal=default priority=0Next, create the IPsec peers that support the Phase-1 handshake and password exchange. Be certain to use the AWS-supplied passwords and don't mix-up the pieces of tunnel #1 and tunnel #2. Use the two public IP-addresses supplied by AWS as a point of reference and keep all the parts for each tunnel together. If you mix them, it won't work.
Code: Select all
0 ;;; AWS VPC Tunnel #2
address=205.251.233.120/32 local-address=x.x.x.x passive=no port=500 auth-method=pre-shared-key
secret="<aws-supplied secret>" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no
proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s
dpd-maximum-failures=3
1 ;;; AWS VPC Tunnel #1
address=205.251.233.119/32 local-address=x.x.x.x passive=no port=500 auth-method=pre-shared-key
secret="<different aws-supplied secret>" generate-policy=no exchange-mode=main send-initial-contact=yes nat-traversal=no
proposal-check=obey hash-algorithm=sha1 enc-algorithm=aes-128 dh-group=modp1024 lifetime=8h lifebytes=0 dpd-interval=10s
dpd-maximum-failures=3Code: Select all
1 chain=input action=accept protocol=ipsec-esp src-address=205.251.233.119 dst-address=x.x.x.x in-interface=ether1-gateway
2 chain=input action=accept protocol=udp src-address=205.251.233.119 dst-address=x.x.x.x in-interface=ether1-gateway src-port=500
dst-port=500
3 chain=input action=accept protocol=ipsec-esp src-address=205.251.233.120 dst-address=x.x.x.x in-interface=ether1-gateway
4 chain=input action=accept protocol=udp src-address=205.251.233.120 dst-address=x.x.x.x in-interface=ether1-gateway src-port=500
dst-port=500Next, you need a few rules so that BGP from the AWS can talk to your router:
Code: Select all
6 chain=input action=accept protocol=tcp src-address=169.254.249.25 dst-address=169.254.249.26 dst-port=179
7 chain=input action=accept protocol=tcp src-address=169.254.249.29 dst-address=169.254.249.30 dst-port=179Code: Select all
23 ;;; VPC at AWS us-west-2x
chain=forward action=accept src-address=172.31.0.0/16 in-interface=ether1-gateway
24 chain=forward action=accept dst-address=172.31.0.0/16 in-interface=ether2-master-localCode: Select all
0 ;;; critically important to AWS connectivity that this rule be ahead of "masquerade".
chain=srcnat action=src-nat to-addresses=192.168.88.0/24 dst-address=172.31.0.0/16The following firewall NAT filter rule is OPTIONAL and to be used only if the public address you supplied to AWS for your router is different from the public address used to masquerade your private NAT network. Don't use this unless you have two public addresses! And note the exclamation point!!!
Code: Select all
1 chain=srcnat action=masquerade src-address=!x.x.x.x out-interface=ether1-gatewayYou will need a destination nat rule to defeat any port-mapping rules that may be present in your existing dstnat chain. If none exist, this isn't needed, though it is harmless.
Code: Select all
2 ;;; critically important to AWS connectivity that this rule be ahead of any NAT port-mapping
chain=dstnat action=accept src-address=172.31.0.0/16 in-interface=ether1-gatewayNext, you configure some BGP instances using the addresses from the AWS instructions:
Code: Select all
1 name="vgw-1" as=65222 router-id=169.254.249.30 redistribute-connected=no redistribute-static=yes redistribute-rip=no
redistribute-ospf=no redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table=""
2 name="vgw-2" as=65222 router-id=169.254.249.26 redistribute-connected=no redistribute-static=yes redistribute-rip=no
redistribute-ospf=no redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table=""Next, you define some BGP peers so that your BGP knows where to find AWS. The normal periodic behavior of BGP in connection with these peers will startup IPsec because of the handshake going to the far-end tunnel endpoint.
Code: Select all
0 E name="awsvpc1" instance=vgw-1 remote-address=169.254.249.29 remote-as=7224 tcp-md5-key="" nexthop-choice=default multihop=no
route-reflect=yes hold-time=30s ttl=default in-filter="" out-filter="" address-families=ip update-source=169.254.249.30
default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no
1 E name="awsvpc2" instance=vgw-2 remote-address=169.254.249.25 remote-as=7224 tcp-md5-key="" nexthop-choice=default multihop=no
route-reflect=yes hold-time=30s ttl=default in-filter="" out-filter="" address-families=ip update-source=169.254.249.26
default-originate=never remove-private-as=no as-override=no passive=no use-bfd=noNext, you define what local networks you want BGP to advertise to AWS. At minimum, this would be your local private network block. And hopefully, that isn't a 172.31.0.0/16 range network, as these conflict with the monster VPC that AWS allocates to you!
Code: Select all
0 network=192.168.88.0/24 synchronize=yesCode: Select all
0 E spi=0xF1A1CB0 src-address=205.251.233.120 dst-address=207.38.19.222 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature
auth-key="7d08d47b8dd2c46dbc0f8df62b4b899f49bb6a96" enc-key="26e38b12f243133cd8700122443cbf2f" addtime=aug/06/2014 22:20:49
expires-in=17m19s add-lifetime=16m/20m current-bytes=2045874
1 E spi=0xF7B18F0 src-address=205.251.233.119 dst-address=207.38.19.222 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature
auth-key="fd67f15352fae6598c37fa3965ad9990a5354443" enc-key="58c5a47403cb23a77a655135a4ef3263" addtime=aug/06/2014 22:22:10
expires-in=18m40s add-lifetime=16m/20m current-bytes=1770
2 E spi=0x57469D96 src-address=x.x.x.x dst-address=205.251.233.119 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature
auth-key="e2d031639925eb74a270c24941eb70d84b039eaa" enc-key="1dbd99b3b843d62237fe6a2da88b8a17" addtime=aug/06/2014 22:22:10
expires-in=18m40s add-lifetime=16m/20m current-bytes=1178
3 E spi=0xA099C19C src-address=x.x.x.x dst-address=205.251.233.120 auth-algorithm=sha1 enc-algorithm=aes-cbc replay=4 state=mature
auth-key="cba17dab219c1494fe32f0ec034298a2e82274ec" enc-key="1ba56b9109f72a26057cfc5ff2e04afd" addtime=aug/06/2014 22:20:49
expires-in=17m19s add-lifetime=16m/20m current-bytes=113976118Typical IPsec Remote Peers:
Code: Select all
0 local-address=x.x.x.x remote-address=205.251.233.119 state=established side=initiator established=2m14s
1 local-address=x.x.x.x remote-address=205.251.233.120 state=established side=initiator established=3m26s Code: Select all
PEER PREFIX NEXTHOP AS-PATH ORIGIN LOCAL-PREF
awsvpc1 192.168.88.0/24 169.254.249.30 igp
awsvpc2 192.168.88.0/24 169.254.249.26 igpWith working BGP, you will also see some additional dynamic routes to AWS VPC in your route table as below. Only one such route is allowed to be active at any given instant. Also note that the gateway address for these routes is the far-end address of the IPsec tunnel, indicating that all the communication is secured.
Code: Select all
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 A S 0.0.0.0/0 x.x.x.w 1
1 ADC 192.168.88.0/24 192.168.88.1 ether2-master-l... 0
2 ADC 169.254.249.24/30 169.254.249.26 ether1-gateway 0
3 ADC 169.254.249.28/30 169.254.249.30 ether1-gateway 0
4 ADb 172.31.0.0/16 169.254.249.25 20
5 Db 172.31.0.0/16 169.254.249.29 20
6 ADC x.x.x.v/27 x.x.x.x ether1-gateway 0If you got this far, you probably have it working.
Don't forget that AWS VPC is a paid service. You pay by the hour for the connection. However, the cost is about US$1 per day, so you're not going to go broke experimenting! If you actually use the tunnel, this is a bargain, especially if you can make it work with a low-cost, high-performance router!
In our case, a 750GL routerboard running v6.15 will see about 50 percent peak CPU utilization (400 MHz) handling sustained encrypted traffic on a 15 megabit/sec link. The effective throughput is about 1.6 Mbyte/sec for long file transfers. Latency is typically below 50 milliseconds between a host on our private network and an instance in our VPC.
The Microtik Limitation Explained
At last, Microtik should fix their little difficulty with IPsec. The problem is, their software does not allow two IPsec policies with identical src- and dst- addresses but different sa-dst-address to be configured. If you attempt to do this, the new policy is declared "invalid" and the whole IPsec policy list becomes unstable.
Following the example above, the missing policy is:
Code: Select all
src-address=0.0.0.0/0 src-port=any dst-address=172.31.0.0/16 dst-port=any protocol=all action=encrypt level=require
ipsec-protocols=esp tunnel=yes sa-src-address=x.x.x.x sa-dst-address=205.251.233.119 proposal=default priority=0The presence of this policy would allow the VPN to continue to carry traffic during Phase-2 PFS negotiation periods, which AWS recommends occur every 8 minutes. For our purposes, this is overkill and we've extended the lifetime of Phase-2 a bit so the "pause to refresh" is less frequent. When it happens, the pause is a matter of milliseconds, but even this can be a lot on a fast link or supporting streaming traffic.
It's pretty clear that if you examine the routing table above, only one tunnel is ever actually carrying traffic between your router and AWS. But when Phase-2 timer fires and the active tunnel stops working briefly, the switch to the idle tunnel should be nearly instantaneous and only initiated by the end seeking to transmit (without end-to-end negotiation). The IPsec policy selection employed by Microtik seems to prevent this.
Having said all this the performance of the VPN is not affected, except during the Phase-2 handshake, which is very brief.
. Even otherwise, it is recommended against, since AWS explicitly states that it will only allow two SAs per VPN channel. I mentioned it somewhere up the thread, with a reference to the docs