CHR in Azure

matze1708 · Tue Jan 12, 2021 11:07 am

Good morning,

I have a CHR installed in Azure.
One NIC hangs in a gateway subnet and a 2nd NIC hangs in a VM subnet.

To my local network I have an OpenVPN connection from my RB3011 to the CHR.
I have in the Azure network the subnet 172.17.5.0/24 and in the local one the 10.100.124.0/23.
The OpenVPN endpoints have the addresses 10.99.1.1 and 10.99.1.3.

About my problem.
I can reach from the Azure subnet everything in my local network, but from my local network I can reach only the CHR nothing in the subnet behind.

However, on the CHR I can ping the IP of a VM on the Azure network.
On my RB3011 on the local network, I can ping the IP of the CHR with the IP on the same subnet as the VM on the subnet.

But why can't I get to the subnet from the local network?

[admin@rb3011_alzey] /ip route> print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          192.168.99.1              1
 1 ADC  10.1.1.22/32       10.1.1.1        <ovpn-OVPN_Frei...        0
 2 ADC  10.1.1.23/32       10.1.1.1        <ovpn-OVPN_Main...        0
 3 ADS  10.10.0.0/23                       <ovpn-OVPN_Frei...        1
 4 ADC  10.99.1.0/24       10.99.1.3       ovpn_Azure                0
 5 ADC  10.100.1.0/24      10.100.1.1      br-Gast                   0
 6 ADC  10.100.2.0/24      10.100.2.1      br_Admin                  0
 7 ADC  10.100.124.0/23    10.100.124.1    br-lan                    0
 8 ADS  157.55.198.115/32                  192.168.99.1              0
 9 A S  172.16.0.0/24                      br_Kamera100m             1
10 A S  172.17.5.0/24      10.99.1.3       ovpn_Azure                1
11 A S  172.17.254.0/24    10.99.1.3       ovpn_Azure                1
12 ADC  192.168.99.0/24    192.168.99.100  ether2-DSL                0
13 ADS  192.168.123.0/24                   <ovpn-OVPN_Main...        1

[admin@Mikrotik-CHR-Gateway] /ip route> print
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          172.17.254.1              1
 1 ADC  10.99.1.3/32       10.99.1.1       <ovpn-ovpn_Vere...        0
 2 ADS  10.100.124.0/23                    <ovpn-ovpn_Vere...        1
 3 ADS  168.63.129.16/32                   172.17.254.1              1
 4 ADS  169.254.169.254/32                 172.17.254.1              1
 5 ADC  172.17.5.0/24      172.17.5.254    br_Azure                  0
 6 ADC  172.17.254.0/27    172.17.254.4    ether1                    0

Thanks for your help

sindy · Tue Jan 12, 2021 11:44 am

Is the CHR the default gateway for the VM? If not, is there a route on the VM via CHR to the 3011's LAN subnet?

sindy · Tue Jan 12, 2021 12:33 pm

... and 2), what are the firewall rules at CHR? Since you can establish connections to devices in 3011's LAN from the VMs(?) at Azure, it seems rather like a firewall problem than a routing problem.

matze1708 · Tue Jan 12, 2021 1:08 pm

Hello,
here are the rules of the firewall on the CHR

/ip firewall filter> print
Flags: X - disabled, I - invalid, D - dynamic 
 0    ;;; Netzwerkverkehr forwarding
      chain=forward action=accept connection-state=established,related log=no log-prefix="" 

 1    ;;; Netzwerkverkehr forwarding
      chain=forward action=drop connection-state=invalid log=no log-prefix="" 

 2    ;;; Input Mikrotik Managment P:8291
      chain=input action=accept protocol=tcp in-interface=ether1 dst-port=8291 log=no log-prefix="" 

 3    ;;; Input Mikrotik OVPN P:31194
      chain=input action=accept protocol=tcp in-interface=ether1 dst-port=31194 log=no log-prefix="" 

 4 X  ;;; Input Mikrotik Managment P:500,1701,4500
      chain=input action=accept protocol=udp in-interface=ether1 dst-port=500,1701,4500 log=no log-prefix="" 

 5    chain=input action=accept protocol=ipsec-esp in-interface=ether1 log=no log-prefix="" 

 6    chain=forward action=accept protocol=tcp dst-address=10.100.124.207 dst-port=443 log=no log-prefix="" 

 7    ;;; VPN Verkehr forwarden
      chain=forward action=accept src-address=10.99.1.0/24 log=no log-prefix="" 

 8    ;;; Drop eth1 WAN
      chain=input action=drop in-interface=ether1 log=no log-prefix="" 
      
  [admin@Mikrotik-CHR-Gateway] /ip firewall nat> print
Flags: X - disabled, I - invalid, D - dynamic 
 0 X  chain=srcnat action=accept src-address=10.99.1.0/24 dst-address=172.17.5.0/24 log=no log-prefix="" 

 1    chain=srcnat action=masquerade src-address=172.17.5.0/24 log=no log-prefix="" 

 2    chain=dstnat action=dst-nat to-addresses=10.100.124.207 to-ports=443 protocol=tcp in-interface=ether1 dst-port=443 log=no log-prefix="" 

 3    chain=dstnat action=dst-nat to-addresses=10.100.124.6 to-ports=3000 protocol=tcp in-interface=ether1 dst-port=3000 log=no log-prefix=""

The gateway on the vm is an Azure gateway registered.
However, I have set up a routing table on Azure.
10.100.124.0/23 via 172.17.5.254(CHR)
I also get from the Azure side to my Local network but not reversed.

sindy · Tue Jan 12, 2021 2:59 pm

The firewall on CHR doesn't block it.

So try another thing, sniff at the CHR interface facing to the VMs (/tool sniffer quick interface=br-Azure ip-protocol=icmp ip-address=some.ip.from.10.100.124.0/23) while trying to access some VM from a particular machine in 10.100.124.0/23. This should answer whether the issue is between the Mikrotiks or in the Azure routing.

Off topic, allowing the whole world to access Winbox on the CHR is not a good idea.

matze1708 · Tue Jan 12, 2021 3:31 pm

Hello,
thank you for your answer.

On the CHR I could only detect something when I took the IP to the IP from the VM from Azure.
apparently I come out there with the IP of the openVPN endpoint.

I also ran the sniff on the RB3011.

It seems to arrive yes in the Azure world, but not passed there.

Misterious

[admin@Mikrotik-CHR-Gateway] /tool sniffer> quick interface=br_Azure ip-protocol=icmp ip-address=172.17.5.5    
INTERFACE                     TIME    NUM DIR SRC-MAC           DST-MAC           VLAN   SRC-ADDRESS                         DST-ADDRESS                         PROTOCOL   SIZE CPU FP 
br_Azure                      2.68      1 ->  00:0D:3A:EB:1F:D9 12:34:56:78:9A:BC        10.99.1.3                           172.17.5.5                          ip:icmp      74   0 no 
br_Azure                     7.667      2 ->  00:0D:3A:EB:1F:D9 12:34:56:78:9A:BC        10.99.1.3                           172.17.5.5                          ip:icmp      74   0 no 
br_Azure                    12.666      3 ->  00:0D:3A:EB:1F:D9 12:34:56:78:9A:BC        10.99.1.3                           172.17.5.5                          ip:icmp      74   0 no 
br_Azure                    17.666      4 ->  00:0D:3A:EB:1F:D9 12:34:56:78:9A:BC        10.99.1.3                           172.17.5.5                          ip:icmp      74   0 no 


[admin@rb3011_alzey] /tool sniffer> quick ip-protocol=icmp ip-address=10.100.124.205
INT..     TIME    NUM DI SRC-MAC           DST-MAC           VLAN   SRC-ADDRESS                         DST-ADDRESS                         PROTOCOL  
eth1    3.697      1 <- 00:15:5D:FA:B7:01 4C:5E:0C:9D:F7:5B        10.100.124.205                      172.17.5.5                          ip:icmp   
br-lan    3.697      2 <- 00:15:5D:FA:B7:01 4C:5E:0C:9D:F7:5B        10.100.124.205                      172.17.5.5                          ip:icmp   
eth1    8.681      3 <- 00:15:5D:FA:B7:01 4C:5E:0C:9D:F7:5B        10.100.124.205                      172.17.5.5                          ip:icmp   
br-lan    8.681      4 <- 00:15:5D:FA:B7:01 4C:5E:0C:9D:F7:5B        10.100.124.205                      172.17.5.5                          ip:icmp   
eth1.   13.682      5 <- 00:15:5D:FA:B7:01 4C:5E:0C:9D:F7:5B        10.100.124.205                      172.17.5.5                          ip:icmp   
br-lan   13.682      6 <- 00:15:5D:FA:B7:01 4C:5E:0C:9D:F7:5B        10.100.124.205                      172.17.5.5                          ip:icmp   
eth1    18.68      7 <- 00:15:5D:FA:B7:01 4C:5E:0C:9D:F7:5B        10.100.124.205                      172.17.5.5                          ip:icmp   
br-lan    18.68      8 <- 00:15:5D:FA:B7:01 4C:5E:0C:9D:F7:5B        10.100.124.205                      172.17.5.5                          ip:icmp

EDIT: Thanks for xour information about the Winbox Port. I close this directly when the problem is solved.

sindy · Tue Jan 12, 2021 3:39 pm

If you look to the sniffer results (making the command line as wide as your screen allows before running the command would give you more information about the packets, but this one is sufficient), you can see that the source address of the captured packets is the one of the openvpn client, not an address from 3011's LAN subnet. Hence there is a src-nat or masquerade rule in the 3011's /ip firewall nat which acts not only on traffic leaving via WAN but also on traffic leaving via ovpn-client. Make the rule more selective and try again.

As the routing table in Azure doesn't have a route towards 10.99.1.3, it sends the responses down the default one which is not via CHR.

matze1708 · Tue Jan 12, 2021 4:01 pm

Thanks for your quick help, I think we are getting closer.

I have the following NAT entries.

[admin@rb3011_alzey] /ip firewall nat> print
Flags: X - disabled, I - invalid, D - dynamic 
 0    ;;; kein NAT fuer VPN Clients
      chain=srcnat action=accept src-address=10.1.1.0/24 dst-address=10.100.124.0/23 log=no log-prefix="" 

 1    ;;; kein NAT fuer VPN Clients
      chain=srcnat action=accept src-address=10.1.2.0/24 dst-address=10.100.124.0/23 log=no log-prefix="" 

 2    chain=srcnat action=masquerade src-address=10.100.124.0/23 log=no log-prefix="" 

 3    chain=srcnat action=masquerade out-interface=ether2-DSL log=no log-prefix="" 

 4    ;;; Portweiterleitung DS 5001
      chain=dstnat action=dst-nat to-addresses=10.100.124.14 to-ports=5001 protocol=tcp dst-address=192.168.99.100 dst-port=5001 log=no log-prefix="" 

 5    chain=dstnat action=dst-nat to-addresses=10.100.124.14 to-ports=80 protocol=tcp dst-address=192.168.99.100 dst-port=80 log=yes log-prefix="IP-14-P-80_" 

 6    ;;; Portweiterleitung DS 9910 Kalender APP
      chain=dstnat action=dst-nat to-addresses=10.100.124.14 to-ports=9910 protocol=tcp dst-address=192.168.99.100 dst-port=9910 log=yes log-prefix="IP-14-P-9910_" 

 7    ;;; ;;; Portweiterleitung Kamera T r 4443 -> 443
      chain=dstnat action=dst-nat to-addresses=10.100.124.161 to-ports=443 protocol=tcp dst-address=192.168.99.100 dst-port=4443 log=yes log-prefix="IP-161-P-443_" 

 8    ;;; ;;; Portweiterleitung IIS-RD Gateway 443
      chain=dstnat action=dst-nat to-addresses=10.100.124.207 to-ports=443 protocol=tcp dst-address=192.168.99.100 dst-port=443 log=yes log-prefix="IP-207-P-443_" 

 9    ;;; ;;; Portweiterleitung IIS-RD Gateway UDP 3391
      chain=dstnat action=dst-nat to-addresses=10.100.124.207 to-ports=3391 protocol=udp dst-address=192.168.99.100 dst-port=3391 log=yes log-prefix="IP-207-P-U3391" 

10    ;;; ;;; Portweiterleitung Heizung 5955
      chain=dstnat action=dst-nat to-addresses=10.100.124.30 to-ports=5900 protocol=tcp dst-address=192.168.99.100 dst-port=5955 log=yes log-prefix="IP-30-P-5955_VNC" 

11 X  ;;; ;;; Portweiterleitung Eisb r APP -> 8003
      chain=dstnat action=dst-nat to-addresses=10.100.124.24 to-ports=8003 protocol=tcp dst-address=192.168.99.100 dst-port=8003 log=yes log-prefix="IP-24-P-8003_" 

12 X  ;;; ;;; Portweiterleitung Eisb r APP -> 8004
      chain=dstnat action=dst-nat to-addresses=10.100.124.24 to-ports=8004 protocol=tcp dst-address=192.168.99.100 dst-port=8004 log=yes log-prefix="IP-24-P-8004_"

EDIT:

i have disabled rule

2 chain=srcnat action=masquerade src-address=10.100.124.0/23 log=no log-prefix=""

disabled, now I arrive in Azure with a 10.100.124.0/23 address.

but a ping is still not possible

sindy · Tue Jan 12, 2021 4:08 pm

Your NAT rules don't care about out-interface or out-interface-list, so they src-nat any outgoing connection, no matter where it goes. As it seems you've previously worked for Sicherheitsdienst and only release the configuration line by line, I cannot tell you what to add to rule #2 in the last post to restrict it to act only on outgoing connections leaving through the WAN interface.

matze1708 · Tue Jan 12, 2021 4:19 pm

What information do you need to assess this?

I can't get rid of the thought that something needs to be done to the routing here in Azure.
But what.

matze1708 · Tue Jan 12, 2021 4:35 pm

YES!!! its a azure thing!

I have to enable ip forwarding on the NIC from my CHR VM!

sindy · Tue Jan 12, 2021 4:36 pm

What information do you need to assess this?

Instead of disabling that NAT rule completely, you should add out-interface=something or out-interface-list=something depending on your actual configuration. I cannot guess whether you use interface lists and how you name them if you do, and which interface(s) you use as WAN.

By disabling that NAT rule completely, you've prevented the devices in the relevant subnet from reaching the internet, unless ether2-DSL is your only WAN; if it is, rule #2 was completely unnecessary because rule #3 doesn't care about source address and src-nats everything that goes out via ether2-DSL.

YES!!! its a azure thing!

Congrats.

matze1708 · Wed Jan 13, 2021 4:42 pm

By disabling that NAT rule completely, you've prevented the devices in the relevant subnet from reaching the internet, unless ether2-DSL is your only WAN; if it is, rule #2 was completely unnecessary because rule #3 doesn't care about source address and src-nats everything that goes out via ether2-DSL.

Thank you for your analysis.

I have a supplementary question.
If I want to access an IP on my local network from the Azure Public IP, what should I consider?
I have set up a NAT on the CHR with port forwarding to 10.100.124.207:443.
The port opened in Azure on the NSG.
I am getting traffic over the VPN up to my local network, but the page is not being built.

sindy · Wed Jan 13, 2021 5:40 pm

If I get you right, the web client somewhere in the world is sending a connection request to port 443 at the public IP of the CHR, and the CHR port-forwards that request to 10.100.124.207, so the request goes through the OpenVPN tunnel and reaches the server connected to your 3011.

Now think about the response: the server sends the response to the source IP of the request, which is a public one (if I got you right, see above). But on the 3011, the route via the OpenVPN tunnel is used only for destinations in the 172.x.. range in your Azure subnet, so the response is routed out via the WAN of the 3011, and doesn't even get src-nated (because the response is not the first packet of a session as seen by 3011's connection tracking, so it is not handled by the rules in NAT table), hence even if your ISP doesn't drop it and it makes it to the client, the client ignores it because it doesn't come from the public IP of the CHR to which the request has been sent.

So you have to engage policy routing with connection-mark at the 3011 to make sure that responses to requests that arrived via the OpenVPN tunnel will be sent via the OpenVPN tunnel as well; as such response reaches the CHR, it will get "un-dst-nated" and forwarded to the client from the address the client expects.

The details are here. Read the last paragraph of that post first to understand the relationship to your case. Think about the OpenVPN tunnel as about another WAN of the 3011.

Another approach would be to src-nat the request at the CHR as forwarding it to the tunnel, so the server next to the 3011 would see it as coming from one of your private subnets on Azure, but you would lose the information about the actual address of the client. If you don't need that address, it is way simpler that to implement policy routing for the first time in your life.

matze1708 · Thu Jan 14, 2021 9:49 am

Hello,

now that you gave me the info, it makes sense mentally!

That means I have to mark the connection that goes from the tunnel to my local network.
Then I have to mark everything that has this connection-mark and wants to go out again with a routing-mark!

For this then a route to 0.0.0.0/0 over the VPN but only with routing-mark VPN?

[admin@rb3011_alzey] /ip firewall mangle> print
Flags: X - disabled, I - invalid, D - dynamic 
 0    chain=prerouting action=mark-connection new-connection-mark=con_ovpn_Azure_US passthrough=yes in-interface=ovpn_Azure_US log=no log-prefix="" 

 1    chain=output action=mark-routing new-routing-mark=route_ovpn_Azure_US passthrough=no connection-mark=con_ovpn_Azure_US log=no log-prefix="" 

[admin@rb3011_alzey] /ip route> print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0          10.99.1.3       ovpn_Azure_US             1
 1 ADS  0.0.0.0/0                          192.168.99.1              1

sindy · Thu Jan 14, 2021 10:06 am

Almost correct, except that

the action=mark-routing rule must be in chain prerouting as well - in output, it would handle own traffic of the router, not the one forwarded from LAN
due to the above, the rule must be made to match only traffic not coming from the tunnel itself, as otherwise such traffic would be sent back to the tunnel, so add in-interface=!ovpn_Azure_US to it

matze1708 · Thu Jan 14, 2021 10:19 am

Ok I see,

that's how it works now. Thanks a lot

The necessary firewall rules I make but in the case on the CHR?!

matze1708 · Thu Jan 14, 2021 10:28 am

I have another question for understanding.

If I now call the resource directly via the local wan (3011) the construction of the page is faster, as the construction via Azure and the VPN.

What I have to say, I have selected Azure US! I will try that again.

FurfangosFrigyes · Thu Jan 14, 2021 10:57 am

Do not forget to add custom routes to the Azure Virtual Network. In this case the CHR is working as a soft appliance.

sindy · Thu Jan 14, 2021 12:35 pm

What I have to say, I have selected Azure US! I will try that again.

If the CHR really runs in the USA and the 3011 and the test client in Europe, as seems to be the case, the round-trip delay (client request sending to server response arrival) includes four overseas hops which add tens of milliseconds (about 100 ms each if I remember well), so nothing unexpected. There is some impact of the VPN encapsulation as well (VPN headers occupy some part of the packets so fewer bytes remain available for the original packet, so TCP must send smaller portions of the buffer), but that accounts for up to, say, 10% throughput reduction, but high round trip delay may affect TCP quite a lot (sending is paused every now and then until the ACK arrives).

matze1708 · Thu Jan 14, 2021 1:01 pm

I have now migrated the entire Azure network to Germany.
From a ping with 150ms we are now at about 30ms

Seems to be a bit faster as well.

CHR in Azure

CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Re: CHR in Azure

Who is online