LTE / L2TP/IPSEC tunnel unstable

jamthejame · Sun May 06, 2018 3:05 am

Got a new device with LTE, Siera MC7700. Works fine, however, when i try to set up L2TP/IPSEC client it connects to the server (have no control over the server side).

Then i try to ping but it shows me 'not reachable', and then a few seconds latter it collapses and tries to reconnect again:

Here is the specs of the systems:

[admin@MikroTik] > export
# may/05/2018 19:39:43 by RouterOS 6.42.1
# software id = K0KV-DMB1
#
# model = 953GS-5HnT
# serial number = 49C504E2567E
/interface wireless
set [ find default-name=wlan1 ] band=5ghz-a/n channel-width=20/40mhz-Ce \
    distance=indoors frequency=auto mode=ap-bridge rx-chains=0,1,2 ssid=\
    MikroTik-61AE09 tx-chains=0,1,2 wireless-protocol=802.11
/interface bridge
add admin-mac=******* auto-mac=no comment=BRIDGE name=bridge
/interface ethernet
set [ find default-name=sfp1 ] disabled=yes
set [ find default-name=sfp2 ] disabled=yes
/interface l2tp-client
add add-default-route=yes allow-fast-path=yes connect-to=******** \
    dial-on-demand=yes disabled=no ipsec-secret=****** keepalive-timeout=\
    10 max-mru=1410 max-mtu=1410 name=LiquidVPN password=******* use-ipsec=\
    yes user=********
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface lte apn
add apn=fast.t-mobile.com name="T-Mobile US LITE"
/interface lte
set [ find ] apn-profiles="T-Mobile US LITE" mac-address=************ \
    name=LTE
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp disabled=no interface=bridge name=defconf
/port
set 1 name=usb2
/interface ppp-client
add apn="T-Mobile US LITE" dial-on-demand=no name=ppp-out1 port=usb2
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
    192.168.88.1
/ip dns
set allow-remote-requests=yes servers=208.67.222.222,8.8.8.8
/ip dns static
add address=192.168.88.1 name=router.lan
/ip firewall nat
add action=masquerade chain=srcnat disabled=yes out-interface=LTE
add action=masquerade chain=srcnat out-interface=LiquidVPN
/ip route
add distance=2 gateway=LTE
add distance=1 dst-address=100.204.125.0/24 gateway=LTE
/system routerboard settings
set silent-boot=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
[admin@MikroTik] > ip route print
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          LiquidVPN                 1
 1   S  0.0.0.0/0                          LTE                       2
 2 ADC  10.11.11.0/32      23.227.197.119  LiquidVPN                 0
 3  DS  23.2************                   23.2************          0
 4   S  23.2************                   100.2**********           1
 5 ADC  100.2**********    100.2********** LTE                       0
 6 A S  100.2**********                    LTE                       1
 7 ADC  192.168.88.0/24    192.168.88.1    bridge                    0

Any ideas why that might be happening?

sindy · Sun May 06, 2018 9:52 am

When obfuscating the public IP addresses in the configuration exports and prints, it is essential to do that in such a way that you wouldn't obfuscate also the information about relationship between them.

So e.g. if your public address is 123.45.67.89/25 and the gateway is 123.45.67.1, take care to replace 123.45.67 with something like my.prefix.a systematically in the whole text you publish and keep the last byte and the /25 unchanged. Ctrl-H (find&replace) is your friend, it also helps you not to omit some occurrences of the prefix.

The reason why I write this is that I suspect that you haven't realized one point. When you send a packet to the L2TP client interface, the L2TP encapsulates it into a brand new, locally originated packet and sends the new one to the IP address of the VPN server. At this stage it is not important which route matches this new packet (provided that at least one does), because the IPsec policy created by the L2TP/IPsec astro clock steals the packet and lets the IPsec SA encrypt and encapsulate it again into yet another packet (plain ESP or ESP over UDP). And now comes the point: this packet is routed using the normal routing rules, but normal routing rules say that the default route is the one via L2TP client interface, and no route more specific for the IP address of the VPN server as destination is available. So the double encapsulated packet is sent to the L2TP tunnel again, double encapsulated, sent to the L2TP tunnel again...

Now depending on whether you configure a domain name or an IP address as connect-to of the /interface l2tp-client, a different approach needs to be chosen to fix this. If it is an IP address, simply

/ip route add dst-address=addr.of.vpn.server gateway=LTE

and you should be good.

If the connect-to is a domain name, it means that the actual IP address may change for each new connection, and you never know whether the new one will come from the same subnet or a completely different one, so you don't know how to configure the exception route from the default one.

If this is the case, there are several solutions, none of them perfect. So come back if it turns out you need one of them.

jamthejame · Mon May 07, 2018 8:44 am

Sindy, I'm coming back to thank you again, and breaking down the logic behind it. Unfortunately, in the last few days i was heavily medicated due to spring allergies, so wasnt thinking clearly. Your observation along with suggestion worked 100%. Now, I would create another threat for the question, but they are closely related, so I hope you can extend further knowledge The issue is as follows:

My ultimate goal for this router is to operate in a very tight environment, so only selected IPs to come out and in via L2TP/IPSEC tunnel, along with a set of DNS IPs for pings helping with failover.

Now having input, output and forward and given what you've mentioned above, most of the exceptions should be made for 'forward' chain and for 'input' allow only l2tp/ipsec ports and protocols and the rest to be blocked.

To make sure all traffic goes through LT2TP/IPSEC i set:

/ip firewall filter
add action=drop chain=forward comment=VPN_Only in-interface=LAN out-interface=LTE

Made another exception on 'input' chain for Winbox IP.

However, I tried reproducing the above on 'output' chain and everything seems to crash, i get locked out of Winbox (despite making the exception on 'output' chain for Winbox ip).

My question is, is there a better way to do what i've described above? Any why 'output' chain reacts differently?

sindy · Mon May 07, 2018 10:32 am

Well, here I cannot give a clear answer because the question is not fully clear - what exactly means "I tried reproducing the above on 'output' chain"?

So at this stage, just generic hints:

the roles of the chains in the firewall are clearly visible from this diagram. No in-interface exists for packets handled by chain=output, which means that a rule checking in-interface value in that chain would never match, so it must look different if it could cut you from the world.
it may be a matter of personal opinion, but if I compose a firewall, I prefer the approach "drop everything except what I know must be allowed" to "allow everything except what I know must be dropped", as I'm not Sir Pratchett to imagine even the unimaginable. The rationale behind is that if you by mistake drop something that should not be dropped, your authorized users will quickly let you know; while if you by mistake allow something that should not be allowed, your non-authorized users will never let you know.
In Mikrotik case in particular, this means that the last rule in the chain must be a "drop" one without any conditions, because unlike with iptables, you cannot change the default behaviour of a predefined chain to "drop", it is always "accept".
stateful packet inspection is your best friend. It allows you to concentrate on the initial packet of each connection, and handle the rest using the generic "allow established or related" rule, provided the very last rule in each chain is the "drop" as stated above.
an exception from the above is the chain=output of /ip ifirewall filter. My opinion is that this chain is of little use. The only reason to prevent your own device from actively establishing connections somewhere else by putting restrictions to the output chain of the filter could be to make it impossible for a malware running on the device itself to connect somewhere, but as in our case the malware can easily wipe the firewall completely once it gets in, why bother.

A supercharged introduction to the firewall is here.

sindy · Mon May 07, 2018 2:37 pm

A couple of points specific for L2TP/IPsec:

if both peers run public IP addresses directly on themselves, the session negotiation of the IPsec part runs between their UDP ports 500 and the data transport runs using ESP, which is a layer inside IP at the same hierarchical level as UDP or TCP, and unlike them, it has no notion of ports so it cannot be NATed. If at least one of the peers is behind a NAT, even a 1:1 one or with static port forwarding, the session negotiation on ports 500 detects that and moves both the continuation of itself and the data transport to UDP ports 4500. The ESP is then encapsulated into UDP so it can be NATed.
the L2TP uses the IPsec connection to cipher communication between UDP ports 1700 on the same two addresses of the peers.
if the IPsec tunnel does not establish for some reason, the L2TP may still work without encryption unless you take some counter-measures.

So a box acting as an L2TP/IPsec client can simply be configured not to accept any incoming connection at all via the WAN interface, because it initiates all the above connections from its own side. The magic "accept related, established" in chain=input of /ip firewall filter is all you need to let L2TP/IPsec client be permitted on an otherwise closed firewall.

When configuring an L2TP/IPsec server, you must permit new incoming connections to UDP port 500, but also new incoming connections to UDP port 4500 and new incoming ESP connections, because it is not possible for the firewall to "spy" the need to treat one of these as connection-state=related from the session negotiation as it is already ciphered, nor there is any out-of-band information channel between the IPsec stack and the firewall (at least in RouterOS case). You must also permit new incoming connections to UDP port 1701, but only those matching ipsec-policy=in,ipsec, because otherwise L2TP connections without IPsec, i.e. with much weaker authentication and without any encryption, could be accepted. The rationale behind is that IPsec works very different from other VPN protocols, so the packet decrypted from the received ESP one still appears to the firewall as coming in via the same physical interface as the ESP packet did. The ipsec-policy property of the packet allows you to distinguish between a packet which came in directly (ipsec-policy=in,none) and a packet which has been locally decrypted and decapsulated (ipsec-policy=in,ipsec).

jamthejame · Mon May 07, 2018 6:45 pm

See, Sindy, that exactly that for 'output' chain. I want a client to my router to be only able to initiate connection to selected IPs, the rest of application, windows or linux updates etc to be unable to go online. Therefore, if I just use the 'input' chain, it will not stop the inner network machine to initiate connections. However, you suggestion on 'accept established, related' worked well, as I was able to create 'forward' and 'output' chain 'whitelist' and in the end put the 'accept established, related' and rest just to be dropped. So, seems like a great suggestion, thank you.

BTW, in the past, i used to monitor internet connection via Netwatch as part of some router systems there was basic failover:

/ip firewall filter
add action=drop chain=output comment=Ping_LTE dst-address=8.8.8.8 out-interface=LTE
/tool netwatch
add down-script="/interface lte set [find name=LTE] disabled=yes;\r\
    \n/interface disable VPN;\r\
    \n/ip firewall connection remove [/ip firewall connection find protocol udp and t\
    cp];\r\
    \n/delay 1;\r\
    \n/interface lte set [find name=LTE] disabled=no;\r\
    \n/delay 25;\r\
    \n/interface enable VPN;" host=8.8.8.8 interval=45s

Something that is simple and kind of effective. But this time, I need to do other topological updates, one being able to check if the L2TP/IPSEC tunnel is up on it own. I tried redirecting pinging through Netwatch to VPN interface but not very effective as i remember its icmp and goes from Mik core out which I was not able to redirect. Does that mean that only creating a separate script is the solution, or is there a way to redirect Netwatch pings through the VPN interface?

sindy · Mon May 07, 2018 11:40 pm

I want a client to my router to be only able to initiate connection to selected IPs, the rest of application, windows or linux updates etc to be unable to go online. Therefore, if I just use the 'input' chain, it will not stop the inner network machine to initiate connections.

You didn't get me (or the diagram in the documentation). chain=output has nothing to do with clients on Mikrotik's LAN side, it only processes packets sent by the Mikrotik itself.

To restrict clients, use chain=forward, and permit connection-state=new packets only in only LAN -> VPN (or LAN -> WAN where appropriate) direction, and only to destinations you want to allow the clients to reach. And in chain=input, you may want to prevent the clients from accessing Mikrotik's management (ssh, winbox etc.). You may actually want to permit access to management of the Mikrotik itself only via the VPN if there is a risk that someone could come and connect to an unused port.

this time, I need to do other topological updates, one being able to check if the L2TP/IPSEC tunnel is up on it own. I tried redirecting pinging through Netwatch to VPN interface but not very effective as i remember its icmp and goes from Mik core out which I was not able to redirect. Does that mean that only creating a separate script is the solution, or is there a way to redirect Netwatch pings through the VPN interface?

Two points here.

First, the netwatch privileges have been decreased in the last releases. Some people seem to be really unhappy about that because they cannot use it for configuration adjustments any more, so you may find yourself affected too.

Second, Netwatch pings should be just like any other pings originated by Mikrotik itself (e.g. those checking availability of recursive gateways), so they normally use the default routing table ("main"). One way to change that would be to mark them with some routing-mark in chain=output of /ip firewall mangle and create a default route with the same routing-mark and the gateway you want them to use. So packets without any routing mark would be routed according to the default routing table, and packets with routing mark would be routed according to the routing table named the same.

But I would probably be lazy and instead of exercises with route marking, I would simply set individual routes for one pair of DNS server addresses (such as 8.8.4.4 and 208.67.220.220) via the VPN gateway, and individual routes to another pair (such as 8.8.8.8 and 208.67.222.222) via the WAN gateway, all four in the default routing table, and set firewall rules

/ip firewall address-list
add list=netwatch-vpn-targets address=8.8.4.4
add list=netwatch-vpn-targets address=208.67.220.220
add list=netwatch-wan-targets address=8.8.8.8
add list=netwatch-wan-targets address=208.67.222.222

/ip firewall mangle
add action=jump chain=prerouting icmp-options=0 jump-target=netwatch-monitors protocol=icmp
add action=add-dst-to-address-list address-list=netwatch-vpn-responses address-list-timeout=5s chain=netwatch-monitors src-address-list=netwatch-vpn-targets
add action=add-dst-to-address-list address-list=netwatch-wan-responses address-list-timeout=5s chain=netwatch-monitors src-address-list=netwatch-wan-targets

I would run four netwatch processes, one per each monitored address, and if one of them would get to state down, its on-down script would check the status of both address-lists and depending on it it would take measures. So a temporary loss of one monitored address would not trigger any action at all, and loss of both addresses monitored via VPN would only cause a restart of the VPN if responses from at least one address monitored via WAN would still be present.

And if eventually the netwatch privileges would be too low to let it disable L2TP or power-cycle the LTE, I would schedule a periodical spawn of a script which would checking the two address-lists instead of setting the on-down scripts in netwatch.

Just a remark, whatever you set the address-list-timeout to, the script can still see the address on the list 5 seconds after the timeout has expired (indicating remaining time od 0s all the time).

jamthejame · Tue May 08, 2018 3:16 am

Sindy, you are correct. I didn't get it to the extend as you've explained in the last post, thus thank you. That post helped me resolve certain issues with which I've been struggling earlier today. This is a simplified prototype of firewall built:

Questions#1:

/ip firewall filter
1. add action=drop chain=forward comment=KillSwitchVPN_LTE in-interface=BRIDGE out-interface=LTE
2. add action=drop chain=forward comment=KillSwitchVPN_MGT out-interface=LiquidVPN src-address=192.XXX.XXX.5
3. add action=drop chain=forward comment=KillSwitchVPN_LTE out-interface=LTE src-address=192.XXX.XXX.5
4. add action=drop chain=input comment="Drop Incoming ICMP on LTE" disabled=yes dst-address=100.100.100.100 in-interface=LTE protocol=icmp
5. add action=drop chain=forward comment="Drop packets from BRIDGE that do not have BRIDGE IP" in-interface=BRIDGE log=yes log-prefix=LAN_!BRIDGE src-address=\
    !192.XXX.XXX.75-192.XXX.XXX.77

1. is for making sure no traffic leaves if the tunnel is down 2&3 to make sure MGT has no access to internet 4&5 selfexplanatory.

add action=accept chain=forward comment="Word License#1" dst-address=72.XXX.XXX.X2 dst-port=80 log=yes protocol=tcp
add action=accept chain=forward comment="Word License#2" dst-address=19.XXX.XXX.X4 dst-port=80 log=yes protocol=tcp
add action=accept chain=forward comment="Demo#1" dst-address=20.XXX.XXX.2 dst-port=80 log=yes protocol=tcp
add action=accept chain=forward comment="Demo#2" dst-address=20.XXX.XXX.X2 dst-port=80 log=yes protocol=tcp
add action=accept chain=forward comment="Demo#3" dst-address=20.XXX.XXX.X3 dst-port=443 log=yes protocol=tcp
add action=accept chain=forward comment="Demo#4" dst-address=20.XXX.XXX.X9 dst-port=443 log=yes protocol=tcp
add action=accept chain=input comment="Allow MGT" disabled=yes dst-address=192.XXX.XXX.5 dst-port=***** log=yes protocol=tcp
add action=accept chain=forward comment=DNS dst-port=53 log=yes protocol=tcp
add action=accept chain=input comment="L2TP - IKE v1" dst-port=500 in-interface=LTE log=yes protocol=udp
add action=accept chain=input comment="L2TP - IKE v1" dst-port=4500 in-interface=LTE log=yes protocol=udp
add action=accept chain=input comment="L2TP - Traffic" dst-port=1701 in-interface=LTE log=yes protocol=udp
add action=accept chain=input comment=IPSEC in-interface=LTE log=yes protocol=ipsec-esp
add action=accept chain=input comment="Accept Established, Related Connections" connection-state=established,related
add action=accept chain=forward comment="Accept Established, Related Connections" connection-state=established,related
add action=drop chain=input comment="Drop all INPUT chain LTE" src-address=!192.XXX.XXX.5 in-interface=LTE
add action=drop chain=input comment="Drop all INPUT chain VPN" in-interface=VPN
add action=drop chain=forward comment="Drop all FORWARD chain LTE" in-interface=LTE
add action=drop chain=forward comment="Drop all FORWARD chain VPN" in-interface=VPN
add action=drop chain=input comment="Drop Incoming ICMP on VPN" dst-address=100.100.100.100 in-interface=VPN protocol=icmp
add action=drop chain=input protocol=icmp
add action=drop chain=forward protocol=icmp

In short, have specific IPs that need to be able to connect from Mik client through the tunnel to internet, have DNS and LT2P accept, and as told earlier the 'magic' "Accept Established, Related".
Now, unfortunately, Mik client can still initiate connections on browser etc.. I understand you mentioned the 'output' chain is usually regarding Mik itself, but without it, client can still initiate new connections. Unfortunately, it seems that i'm using 'output' chain unaffectedly as when i add new rule chain=output action=drop in-interface LTE (it doesn't change much) but when i change the interface to VPN, everything locks down, the tunnel collapses etc. Thus, how can i stop outgoing connection and limit only to the ones listed?

Questions#2:
You brought up a good point about quad pings, the thing is that i was thinking on having that setup on failover machine, nevertheless, i used Netwatch because im somewhere of an intermediate level, not advance or export and definitely not Guru. Script writing for me isnt easy because i dont have enough time to sit and get a hold of it or write enough to feel comfortable with it, thus Netwatch was my savior. I like the idea of route marking and I think this can work, but

I would run four netwatch processes, one per each monitored address, and if one of them would get to state down, its on-down script would check the status of both address-lists and depending on it it would take measures. So a temporary loss of one monitored address would not trigger any action at all, and loss of both addresses monitored via VPN would only cause a restart of the VPN if responses from at least one address monitored via WAN would still be present.

And if eventually the netwatch privileges would be too low to let it disable L2TP or power-cycle the LTE, I would schedule a periodical spawn of a script which would checking the two address-lists instead of setting the on-down scripts in netwatch.

Where would I even start if i would to follow your suggestion? Is there a better way than writing a script? I know for you it might be seconds but for someone like me its days if not weeks. Perhaps you know of any examples available online that i can go through and then try to modify, etc? Dont loose patience with me yet

sindy · Tue May 08, 2018 8:56 am

Just a quick update on a few points before the real one hopefully later today.

WRT Question lot #1:

your output rule with in-interface=vpn breaking everything is a mystery to me. I've tried to enter it and got a mid-finger straight away:
Code: Select all
```
[me@MyTik] > ip firewall filter add chain=output in-interface=vlan200 action=log
failure: incoming interface matching not possible in output and postrouting chains
```
So I don't get how you could even enter it.
please post the complete output of /ip firewall export after replacing the public addresses by some meaningful names. The order of rules matters and it is not clear from the two separate lists you've posted.
do you plan the machine in question to be an L2TP/IPsec server? If yes, you haven't followed my recommendation regarding the rule for UDP port 1701. If not, you don't need to care about IPsec-related ports at all as no restrictions in chain=output and "accept established" in chain=input will do the job.

Regarding the /tool netwatch, I've found myself that it is not possible to use the on-down and on-up scripts if you want to use redundancy of monitored points (or ignore short-time outages on lossy links with no alternative, which is the case in one of my deployments). So in addition to fixing the copy-paste mistake I've made in my previous post (updating the address-list on echo requests, icmp-options=8 instead of echo responses, icmp-options=0, once you go that way, you have to use the /tool netwatch only as a controlled generator of periodic ping requests and the evaluation of the results has to be a job of the periodically spawned script. So give me the actions you want to take when only both pings via L2TP fais and when also both pings via LTE fail, I'll put something together.

sindy · Tue May 08, 2018 3:38 pm

So you've made me replace the plain netwatch monitoring on my site where a wireless "backbone" is so overbooked and the air is so noisy that netwatch pings sometimes fail to get responded although the monitored devices themselves are doing fine.

As a reward, here is the script modified for your needs as much as I could, you only have to fill the appropriate corrective actions and the timeouts necessary to give them some time to complete before re-attempting them.

# create the variables if they do not exist yet, be optimistic and set them to green status
if ([/system script environment print count-only where name=lteAlive]=0) do={global lteAlive 1}
if ([/system script environment print count-only where name=vpnAlive]=0) do={global vpnAlive 1}

# compare the status of address-list monitor-lte to the previous one and take action if it became empty
if ([/ip firewall address-list print count-only where list=monitor-lte]=0) do={\
  if ([/system script environment get [find name=lteAlive] value]>0) do={
    global lteAlive 0;
    ... place here the action necessary to kickstart LTE ...;
    /ip firewall address-list add list=monitor-lte address=4.3.2.1 address-list-timeout=... place here the necessary recovery time, such as 2m30s ...
  }
} else={
  if ([/system script environment get [find name=lteAlive] value]=0) do={
    global lteAlive 1
  }
}

# compare the status of address-list monitor-vpn to the previous one and take action if it became empty
if ([/ip firewall address-list print count-only where list=monitor-vpn]=0) do={\
  if ([/system script environment get [find name=vpnAlive] value]>0) do={
    global vpnAlive 0;
    ... place here the action necessary to kickstart L2TP ...;
        /ip firewall address-list add list=monitor-vpn address=4.3.2.1 address-list-timeout=... place here the necessary recovery time, such as 2m30s ...
  }
} else={
  if ([/system script environment get [find name=vpnAlive] value]=0) do={
    global vpnAlive 1
  }
}

You have to paste that script, with your adjustments, as source parameter of a newly created /system script item.

The mangle rules suggested earlier must add some address (source or destination, the address value itself isn't important) to the corresponding address-list (monitor-vpn or monitor-lte) each time they see an icmp echo response from one of the monitored addresses. The address-list-timeout value choice is up to you, but it must be proportionally longer than the pinging interval configured in /tool netwatch. No up-script and down-script values are necessary for any of the four /tool netwatch items

You have to schedule periodic execution of the script with start-time=startup and interval proportional to the times above.

So e.g. if you make the netwatch ping each destination every 15 seconds, then the address-list-timeout in the mangle rules may be set to 65s to tolerate loss of up to three ping responses from each of the two monitored addresses, and the periodicity of script execution may be set to 15s again to react relatively promptly once the failure will have been confirmed.

jamthejame · Tue May 08, 2018 5:28 pm

RE:Question#1:

Regarding 'output'

/ip firewall filter
add action=drop chain=forward comment=KillSwitchVPN_LTE in-interface=BRIDGE out-interface=LTE
add action=drop chain=forward comment=KillSwitchVPN_MGT out-interface=LiquidVPN src-address=192.XXX.XXX.5
add action=drop chain=forward comment=KillSwitchVPN_LTE out-interface=LTE src-address=192.XXX.XXX.5
add action=drop chain=input comment="Drop Incoming ICMP on LTE" disabled=yes dst-address=100.100.100.100 in-interface=LTE protocol=icmp
add action=drop chain=forward comment="Drop packets from BRIDGE that do not have BRIDGE IP" in-interface=BRIDGE log=yes log-prefix=LAN_!BRIDGE src-address=\
    !192.XXX.XXX.75-192.XXX.XXX.77
add action=accept chain=forward comment="Word License#1_Server" dst-address=172.XXX.XXX.12 dst-port=80 log=yes protocol=tcp
add action=accept chain=forward comment="Word License#2_Server" dst-address=191.XXX.XXX.24 dst-port=80 log=yes protocol=tcp
add action=accept chain=forward comment="Demo#1_Server" dst-address=202.XXX.XXX.2 dst-port=80 log=yes protocol=tcp
add action=accept chain=forward comment="Demo#2_Server" dst-address=203.XXX.XXX.12 dst-port=80 log=yes protocol=tcp
add action=accept chain=forward comment="Demo#3_Server" dst-address=204.XXX.XXX.13 dst-port=443 log=yes protocol=tcp
add action=accept chain=forward comment="Demo#4_Server" dst-address=205.XXX.XXX.19 dst-port=443 log=yes protocol=tcp
add action=accept chain=input comment="Allow MGT" disabled=yes dst-address=192.XXX.XXX.5 dst-port=***** log=yes protocol=tcp
add action=accept chain=forward comment=DNS dst-port=53 log=yes protocol=tcp
add action=accept chain=input comment="L2TP - IKE v1" dst-port=500 in-interface=LTE log=yes protocol=udp
add action=accept chain=input comment="L2TP - IKE v1" dst-port=4500 in-interface=LTE log=yes protocol=udp
add action=accept chain=input comment="L2TP - Traffic" dst-port=1701 in-interface=LTE log=yes protocol=udp
add action=accept chain=input comment=IPSEC in-interface=LTE log=yes protocol=ipsec-esp
add action=accept chain=input comment="Accept Established, Related Connections" connection-state=established,related
add action=accept chain=forward comment="Accept Established, Related Connections" connection-state=established,related
add action=drop chain=input comment="Drop all INPUT chain LTE" src-address=!192.XXX.XXX.5 in-interface=LTE
add action=drop chain=input comment="Drop all INPUT chain VPN" in-interface=VPN
add action=drop chain=forward comment="Drop all FORWARD chain LTE" in-interface=LTE
add action=drop chain=forward comment="Drop all FORWARD chain VPN" in-interface=VPN
add action=drop chain=output comment="Drop all OUTPUT chain LTE" in-interface=LTE disabled=yes	
add action=drop chain=output comment="Drop all OUTPUT chain VPN" in-interface=VPN disabled=yes
add action=drop chain=input comment="Drop Incoming ICMP on VPN" dst-address=100.100.100.100 in-interface=VPN protocol=icmp
add action=drop chain=input protocol=icmp
add action=drop chain=forward protocol=icmp

# LTE=WAN Interface
# VPN=L2TP-client interface

1. I'm on the L2TP/IPSEC client side and have no control over the server I'm connecting to
2. all "output" chain rules are disabled because they either both block traffic to the VPN tunnel and in the end collapsing connection
3. Set the rules for port 500, 4500, 1701, before setting up "accept related, established"
4. My only goal is to allow "Word Server & Demo Server" IPs to be connected by the Mik client machine through l2tp/ipsec tunnel. The rest from both sides blocked, nothing else in or out. Managed to do it on the in, but on the out, I'm having troubles as given the above machine can still go and use browser etc.

P.S. Regarding #2, will respond shortly, let me put it together and try running it.

sindy · Tue May 08, 2018 7:47 pm

it is very useful to keep rules belonging to the same chain in one block, because it makes them much easier to read.
what software version are you running and how exactly did you enter those surrealistic output rules? I can only imagine that WinBox or WebFig allow them to be added although they are incorrect, and because it is impossible to match in-interface in output chain, that match condition is ignored and thus the rules match any packet, hence cutting you off the box
blocking icmp completely is a Bad Idea. Ping is just one application of icmp, the majority is network control and if you block that part, you break things like path MTU discovery which relies on icmp. Normally, these flavors of icmp are handled by the magic "accept established,related" rule because the network control icmp messages are considered related to the TCP and/or UDP connections, but you drop icmp already before this rule.
Blocking icmp on an interface is also a bit incompatible with using netwatch over that interface.
You can match icmp requests selectively using protocol=icmp icmp-options=8
neither of your (input, forward) chains ends with a "drop the rest" rule.

So a firewall implementing your requirements looks as follows:

/ip firewall address-list
add list=permitted-http-destinations address=dst-address=172.XXX.XXX.12 comment="Word License#1_Server"
add list=permitted-http-destinations address=dst-address=191.XXX.XXX.24 comment="Word License#2_Server"
add list=permitted-http-destinations address=dst-address=202.XXX.XXX.2 comment="Demo#1_Server"
add list=permitted-http-destinations address=dst-address=203.XXX.XXX.12 comment="Demo#2_Server"

add list=permitted-https-destinations address=dst-address=204.XXX.XXX.13 comment="Demo#1_Server"
add list=permitted-https-destinations address=dst-address=205.XXX.XXX.19 comment="Demo#2_Server"

add list=permitted-local-sources address=192.XXX.XXX.75
add list=permitted-local-sources address=192.XXX.XXX.76
add list=permitted-local-sources address=192.XXX.XXX.77

add list=management-access address=ip.of.management.pc

/ip firewall filter
### chain INPUT - communication with Mikrotik itself ###

add action=accept chain=input connection-state=established,related
# the above rule handles the bulk of all packets towards Mikrotik itself, so it should be the topmost one in each chain so that those packets wouldn't need to be matched by other rules

add action=accept chain=input protocol=udp in-interface=BRIDGE dst-port=53,123 # permit DNS and NTP requests from clients
add action=accept chain=input protocol=tcp in-interface=BRIDGE dst-port=53 # permit DNS requests from clients over TCP
#if you don't want the clients to use Mikrotik as DNS server and/or NTP server, you have to permit access to DNS and/or NTP servers in forward chain instead
# dhcp from clients does not need to be permitted as it is handled before the firewall

add action=accept protocol=tcp in-interface=BRIDGE src-address-list=management-access dst-port=22,80,443,8291
# the above rule permits local management access to the machine, keep only the ports you actually use on the list

# ... place here eventual rules logging various attempts to establish connections to the Mikrotik itself...

add action=drop chain=input disabled=yes
# this rule prevents any other new connections to the Mikrotik itself than those permitted by the previous rules to be established
# from anywhere; only enable it after checking that the "accept management connections" rule just above counts packets, 
# and nevertheless switch on the safe mode before enabling it

### chain FORWARD - communication with Mikrotik itself ###
add action=accept chain=forward connection-state=established,related
# same like in chain INPUT, the above rule should be the topmost one in this chain

add action=accept chain=forward protocol=tcp dst-port=80 in-interface=BRIDGE out-interface=l2tp-out1 dst-address-list=permitted-http-destinations src-address-list=permitted-local-sources
add action=accept chain=forward protocol=tcp dst-port=443 in-interface=BRIDGE out-interface=l2tp-out1 dst-address-list=permitted-https-destinations src-address-list=permitted-local-sources
# the two rules above implement your requirement "allow "Word Server & Demo Server" IPs to be connected by the Mik client machine through l2tp/ipsec tunnel"

# ... place here eventual rules logging what tries to establish new forwarded connections either way...

add action=drop chain=forward
# the above rule drops anything attempting to pass #through# the Mikrotik that didn't match the rules above

jamthejame · Tue May 08, 2018 9:04 pm

RE: Question#2

I'm glad that you found value in helping me through this process. I took yesterday's mangle rule and today. There was one little issue 'address-list-timeout' which the system wasnt taking as it required 'timeout', thus here is the setup that i added:

/ip firewall address-list
add list=netwatch-vpn-targets address=8.8.4.4
add list=netwatch-vpn-targets address=208.67.220.220
add list=netwatch-wan-targets address=8.8.8.8
add list=netwatch-wan-targets address=208.67.222.222

/ip firewall mangle
add action=jump chain=prerouting icmp-options=0 jump-target=netwatch-monitors protocol=icmp
add action=add-dst-to-address-list address-list=netwatch-vpn-responses address-list-timeout=5s chain=netwatch-monitors src-address-list=netwatch-vpn-targets
add action=add-dst-to-address-list address-list=netwatch-wan-responses address-list-timeout=5s chain=netwatch-monitors src-address-list=netwatch-wan-targets

/system script
add name=operation_lte_vpn owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="# create the variables if they \
    do not exist yet, be optimistic and set them to green status\r\
    \nif ([/system script environment print count-only where name=lteAlive]=0) do={global lteAlive 1}\r\
    \nif ([/system script environment print count-only where name=vpnAlive]=0) do={global vpnAlive 1}\r\
    \n# compare the status of address-list monitor-lte to the previous one and take action if it became empty\r\
    \nif ([/ip firewall address-list print count-only where list=monitor-lte]=0) do={\\\r\
    \n  if ([/system script environment get [find name=lteAlive] value]>0) do={\r\
    \n    global lteAlive 0;\r\
    \n    /interface lte set [find name=LTE] disable=yes\r\
    \n\t/interface lte set [find name=LTE] disable=no \r\
    \n\t/ip firewall address-list add list=monitor-lte address=8.8.8.8 timeout=1m5s\r\
    \n  }\r\
    \n} else={\r\
    \n  if ([/system script environment get [find name=lteAlive] value]=0) do={\r\
    \n    global lteAlive 1\r\
    \n  }\r\
    \n}\r\
    \n# compare the status of address-list monitor-vpn to the previous one and take action if it became empty\r\
    \nif ([/ip firewall address-list print count-only where list=monitor-vpn]=0) do={\\\r\
    \n  if ([/system script environment get [find name=vpnAlive] value]>0) do={\r\
    \n    global vpnAlive 0;\r\
    \n        /interface l2tp-client set [find name=VPN] disabled=yes\r\
    \n\t\t/interface l2tp-client set [find name=VPN] disabled=no\r\
    \n        /ip firewall address-list add list=monitor-vpn address=8.8.4.4 timeout=1m5s\r\
    \n  }\r\
    \n} else={\r\
    \n  if ([/system script environment get [find name=vpnAlive] value]=0) do={\r\
    \n    global vpnAlive 1\r\
    \n  }\r\
    \n}"
    
/system scheduler
add interval=1m5s name=operation_lte_vpn on-event="/system script run operation_lte_vpn" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startup

I'm still trying to better understand the logic of the script and certain lines, as this is not 'vanilla' approach, given your level of expertise, however, I'm highly interested in understanding it and learning rather than just plug-n-play with the work someone put in it.

- So, '/system script environment print count-only where name=lteAlive]=0' checks the LTE interface if is enable/disabled, if disabled or 1, the goes to '/system script environment get [find name=lteAlive] value]>0' confirming and defining as lteAlive 0 (disabled) and going into the following command to re-enable it; and with '/ip firewall address-list add list=monitor-lte address=8.8.8.8 timeout=1m5s' gives it another try to see if its enabled and if enabled and goes to let Alive 1 state. Am correct?

- Also, in the above I'm using just 1 IP for each, no sure how the logic would fall for 2 IP for each interface.

- Lastly, I noticed that when L2TP-client is down the pings go out LTE and pass through giving it up status. This script seems to work only for the LTE side, and the netwatch goes from l2tp-client interface to LTE and shows 'up' status and the script keeps on going as business as usual. Not sure if thats suppose to be like it.

sindy · Tue May 08, 2018 9:45 pm

So, '/system script environment print count-only where name=lteAlive]=0' checks the LTE interface if is enable/disabled, if disabled or 1, the goes to '/system script environment get [find name=lteAlive] value]>0' confirming and defining as lteAlive 0 (disabled) and going into the following command to re-enable it; and with '/ip firewall address-list add list=monitor-lte address=8.8.8.8 timeout=1m5s' gives it another try to see if its enabled and if enabled and goes to let Alive 1 state. Am correct?

lteAlive and vpnAlive are status variables created for the sake of detecting a change of interface status. If the variable contains 1 (interface was up at last check) and the address-list tracking the icmp responses coming via this interface is empty during the current run of the script, indicating that the interface is actually down, there is a difference between last known status and the status detected during the current run of the script, so it means the interface went down between the previous and current runs and so the corrective action needs to be taken and the variable updated to remember the current state for comparison in the next run of the script, so that you could notice the change from down to up.

It could actually be completely omited from your script. In my application, I need to send a single e-mail when the destination goes down, and a single e-mail when the destination goes up, and don't need to initiate any corrective actions; in your case, you need to restart the interface as soon as you detect that it is down, and you even want to pretend that it is up (that's why I add the bogus address to the address-list) so that the restart process could complete successfully without being re-triggered over and over again. So the bogus address on the address-list substitutes the variable in your case.

Also, in the above I'm using just 1 IP for each, no sure how the logic would fall for 2 IP for each interface.

The logic is such that the more addresses you ping, the lower the probability that you conclude that the interface is down while actually only the monitored address is down. Even Google DNS can sometimes be down.

Lastly, I noticed that when L2TP-client is down the pings go out LTE and pass through giving it up status. This script seems to work only for the LTE side, and the netwatch goes from l2tp-client interface to LTE and shows 'up' status and the script keeps on going as business as usual. Not sure if thats suppose to be like it.

This must be because you haven't added the individual route(s) to the address(es) used to monitor the state of the L2TP tunnel via the gateway address provided by the L2TP server.
A route with more specific (longer prefix) dst-address always wins over a less specific (shorter prefix) dst-address, regardless the distance value (distance only determines the mutual priority of routes with identical dst-address values. So if you create an individual route to some destination (like the address used to monitor the status of the interface) and that route's gateway goes down, you cannot ping that destination even though a default route's gateway is accessible.

However, I'm not sure whether the route remains active if its gateway property is set to an interface name and the interface is down, but that should not be your case.

jamthejame · Tue May 08, 2018 11:35 pm

So I keep on testing firewall + the failover. So far firewall seems to do what its suppose to. Will keep on working during the night on it and tomorrow to make further adoptions to devices etc. thus truly appreciate you setting my logic and approach to it straight. Also, added your suggestion regarding the route for monitor-vpn, however, noticed something odd.... with the script, I noticed loss of connection on LTE almost every few minutes, but without the script, it seems to be ok, Thus wondering, do you believe because the script is looking for certain changes, it might inadvertently collide with something related to LTE connection itself? Very odd observation and behavior....

/ip firewall address-list
add list=netwatch-vpn-targets address=8.8.4.4
add list=netwatch-vpn-targets address=208.67.220.220
add list=netwatch-wan-targets address=8.8.8.8
add list=netwatch-wan-targets address=208.67.222.222

/ip firewall mangle
add action=jump chain=prerouting icmp-options=0 jump-target=netwatch-monitors protocol=icmp
add action=add-dst-to-address-list address-list=netwatch-vpn-responses address-list-timeout=5s chain=netwatch-monitors src-address-list=netwatch-vpn-targets
add action=add-dst-to-address-list address-list=netwatch-wan-responses address-list-timeout=5s chain=netwatch-monitors src-address-list=netwatch-wan-targets

/ip route
add check-gateway=ping comment=VPN_Ping_Check_1 distance=1 dst-address=8.8.4.4/32 gateway=VPN
add check-gateway=ping comment=VPN_Ping_Check_2 distance=1 dst-address=208.67.220.220 gateway=VPN

/system script
add name=operation_lte_vpn owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="# create the variables if they \
    do not exist yet, be optimistic and set them to green status\r\
    \nif ([/system script environment print count-only where name=lteAlive]=0) do={global lteAlive 1}\r\
    \nif ([/system script environment print count-only where name=vpnAlive]=0) do={global vpnAlive 1}\r\
    \n# compare the status of address-list monitor-lte to the previous one and take action if it became empty\r\
    \nif ([/ip firewall address-list print count-only where list=monitor-lte]=0) do={\\\r\
    \n  if ([/system script environment get [find name=lteAlive] value]>0) do={\r\
    \n    global lteAlive 0;\r\
    \n    /interface lte set [find name=LTE] disable=yes\r\
    \n\t/interface lte set [find name=LTE] disable=no \r\
    \n\t/ip firewall address-list add list=monitor-lte address=8.8.8.8 timeout=1m5s\r\
    \n  }\r\
    \n} else={\r\
    \n  if ([/system script environment get [find name=lteAlive] value]=0) do={\r\
    \n    global lteAlive 1\r\
    \n  }\r\
    \n}\r\
    \n# compare the status of address-list monitor-vpn to the previous one and take action if it became empty\r\
    \nif ([/ip firewall address-list print count-only where list=monitor-vpn]=0) do={\\\r\
    \n  if ([/system script environment get [find name=vpnAlive] value]>0) do={\r\
    \n    global vpnAlive 0;\r\
    \n        /interface l2tp-client set [find name=VPN] disabled=yes\r\
    \n\t\t/interface l2tp-client set [find name=VPN] disabled=no\r\
    \n        /ip firewall address-list add list=monitor-vpn address=8.8.4.4 timeout=1m5s\r\
    \n  }\r\
    \n} else={\r\
    \n  if ([/system script environment get [find name=vpnAlive] value]=0) do={\r\
    \n    global vpnAlive 1\r\
    \n  }\r\
    \n}"

/system scheduler
add interval=1m5s name=operation_lte_vpn on-event="/system script run operation_lte_vpn" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startup

sindy · Wed May 09, 2018 12:08 am

Don't add the addresses of the monitored DNS servers to the /ip firewall address-list manually (or, better to say, remove that part from the configuration), they must be added there dynamically by the mangle rules only, otherwise the whole concept cannot work. I hope the dynamically added ones set the timeout to the ones added statically, but I'm not sure how that works together.

And show me the /tool netwatch export, the mangle rules are adding the addresses to the lists for 5 seconds only, have you set the netwatch for shorter intervals? The trick is that the address-list-timeout must be several times longer than the netwatch ping interval.

jamthejame · Wed May 09, 2018 12:38 am

Haven't added any addresses to DNS-address-list.. and

/ip firewall address-list manually (or, better to say, remove that part from the configuration

can you be more specific?

Ok...so made the change to Mangle=15s, Netwatch=5s and Scrip=1m5s.

It did resolve the disconnection issue... but not it doesnt resolve either LTE or VPN interface when down now. Have I dont too much?

/ip firewall address-list
add list=netwatch-vpn-targets address=8.8.4.4
add list=netwatch-vpn-targets address=208.67.220.220
add list=netwatch-wan-targets address=8.8.8.8
add list=netwatch-wan-targets address=208.67.222.222

/ip firewall mangle
add action=jump chain=prerouting icmp-options=0 jump-target=netwatch-monitors protocol=icmp
add action=add-dst-to-address-list address-list=netwatch-vpn-responses address-list-timeout=15s chain=netwatch-monitors src-address-list=netwatch-vpn-targets
add action=add-dst-to-address-list address-list=netwatch-wan-responses address-list-timeout=15s chain=netwatch-monitors src-address-list=netwatch-wan-targets

/tool netwatch export
add host=8.8.8.8 interval=5s
add host=8.8.4.4 interval=5s
add host=208.67.222.222 interval=5s
add host=208.67.220.220 interval=5s

/ip route
add check-gateway=ping comment=VPN_Ping_Check#1 distance=1 dst-address=8.8.4.4/32 gateway=VPN
add check-gateway=ping comment=VPN_Ping_Check#2 distance=1 dst-address=208.67.220.220 gateway=VPN

/system script
add name=operation_lte_vpn owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="# create the variables if they \
    do not exist yet, be optimistic and set them to green status\r\
    \nif ([/system script environment print count-only where name=lteAlive]=0) do={global lteAlive 1}\r\
    \nif ([/system script environment print count-only where name=vpnAlive]=0) do={global vpnAlive 1}\r\
    \n# compare the status of address-list monitor-lte to the previous one and take action if it became empty\r\
    \nif ([/ip firewall address-list print count-only where list=monitor-lte]=0) do={\\\r\
    \n  if ([/system script environment get [find name=lteAlive] value]>0) do={\r\
    \n    global lteAlive 0;\r\
    \n    /interface lte set [find name=LTE] disable=yes\r\
    \n\t/interface lte set [find name=LTE] disable=no \r\
    \n\t/ip firewall address-list add list=monitor-lte address=8.8.8.8 timeout=1m5s\r\
    \n  }\r\
    \n} else={\r\
    \n  if ([/system script environment get [find name=lteAlive] value]=0) do={\r\
    \n    global lteAlive 1\r\
    \n  }\r\
    \n}\r\
    \n# compare the status of address-list monitor-vpn to the previous one and take action if it became empty\r\
    \nif ([/ip firewall address-list print count-only where list=monitor-vpn]=0) do={\\\r\
    \n  if ([/system script environment get [find name=vpnAlive] value]>0) do={\r\
    \n    global vpnAlive 0;\r\
    \n        /interface l2tp-client set [find name=VPN] disabled=yes\r\
    \n\t\t/interface l2tp-client set [find name=VPN] disabled=no\r\
    \n        /ip firewall address-list add list=monitor-vpn address=8.8.4.4 timeout=1m5s\r\
    \n  }\r\
    \n} else={\r\
    \n  if ([/system script environment get [find name=vpnAlive] value]=0) do={\r\
    \n    global vpnAlive 1\r\
    \n  }\r\
    \n}"

/system scheduler
add interval=1m5s name=operation_lte_vpn on-event="/system script run operation_lte_vpn" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startup

sindy · Wed May 09, 2018 12:48 am

can you be more specific?

Remove this part of configuration:

/ip firewall address-list
add list=netwatch-vpn-targets address=8.8.4.4
add list=netwatch-vpn-targets address=208.67.220.220
add list=netwatch-wan-targets address=8.8.8.8
add list=netwatch-wan-targets address=208.67.222.222

Ok...so made the change to Mangle=15s, Netwatch=5s and Scrip=1m5s.
It did resolve the disconnection issue... but not it doesnt resolve either LTE or VPN interface when down now. Have I dont too much?

Now that's me who doesn't understand what you mean. Disconnection issue solved, interface down issue not solved doesn't make sense to me. Bear in mind that English is not my native language so please describe the two issues which sound the same to me in more detail.

/ip route
add check-gateway=ping comment=VPN_Ping_Check#1 distance=1 dst-address=8.8.4.4/32 gateway=VPN
add check-gateway=ping comment=VPN_Ping_Check#2 distance=1 dst-address=208.67.220.220 gateway=VPN

VPN is interface name or IP address?

Where are the routes for 8.8.8.8/32 and 208.67.222.222? Have you read the post in the "DHCP server stopped" topics?

jamthejame · Wed May 09, 2018 1:07 am

Ok... removed the addresses from 'address-list'.

Initial issue with the script was that i kept on getting disconnected from the internet every few minutes. Than you suggested to check the times for mangle rule:

/ip firewall mangle
add action=jump chain=prerouting icmp-options=0 jump-target=netwatch-monitors protocol=icmp
add action=add-dst-to-address-list address-list=netwatch-vpn-responses address-list-timeout=5s chain=netwatch-monitors src-address-list=netwatch-vpn-targets
add action=add-dst-to-address-list address-list=netwatch-wan-responses address-list-timeout=5s chain=netwatch-monitors src-address-list=netwatch-wan-targets

#changed to

/ip firewall mangle
add action=jump chain=prerouting icmp-options=0 jump-target=netwatch-monitors protocol=icmp
add action=add-dst-to-address-list address-list=netwatch-vpn-responses address-list-timeout=15s chain=netwatch-monitors src-address-list=netwatch-vpn-targets
add action=add-dst-to-address-list address-list=netwatch-wan-responses address-list-timeout=15s chain=netwatch-monitors src-address-list=netwatch-wan-targets

then Netwatch

/tool netwatch export
add host=8.8.8.8 interval=15s
add host=8.8.4.4 interval=15s
add host=208.67.222.222 interval=15s
add host=208.67.220.220 interval=15s

#changed to

/tool netwatch export
add host=8.8.8.8 interval=5s
add host=8.8.4.4 interval=5s
add host=208.67.222.222 interval=5s
add host=208.67.220.220 interval=5s

Now, after the changes, I have stable internet connection BUT, the script is not working. When I tested and manually disabled 'LTE' interface, it didnt see the changed state and kept on being disabled. Same happens when i manually disable VPN interface... it says disabled until i manually enable it again.

/ip route
add check-gateway=ping comment=VPN_Ping_Check#1 distance=1 dst-address=8.8.4.4/32 gateway=VPN
add check-gateway=ping comment=VPN_Ping_Check#2 distance=1 dst-address=208.67.220.220 gateway=VPN

VPN interface, not IP

I thought you only told me to set up routes for VPN as the other two, if VPN is dont get the IP of LTE directly. Basically, when VPN is online all 4 respond to IP of L2TP server. But when L2TP tunnel is down, they still all respond to LTE gw IP, thus all 4 IP always either UP or DOWN

P.S. what is your native language, if you dont mind me asking. English is my second language two.

sindy · Wed May 09, 2018 1:20 am

Now, after the changes, I have stable internet connection BUT, the script is not working. When I tested and manually disabled 'LTE' interface, it didnt see the changed state and kept on being disabled. Same happens when i manually disable VPN interface... it says disabled until i manually enable it again.

I thought you only told me to set up routes for VPN as the other two, if VPN is dont get the IP of LTE directly. Basically, when VPN is online all 4 respond to IP of L2TP server. But when L2TP tunnel is down, they still all respond to LTE gw IP, thus all 4 IP always either UP or DOWN

That's different to my understanding on how it should work. Please show me the output of /ip route print when VPN is up and when it is down. When obfuscating addresses, replace them with meaningful strings.

P.S. what is your native language, if you dont mind me asking. English is my second language two.

Czech. Some other languages supported too

jamthejame · Wed May 09, 2018 1:32 am

#UP
[admin@MikroTik] > ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          VPN                       1
 1   S  0.0.0.0/0                          LTE                       2
 2 A S  8.8.4.4/32                         VPN                       1
 3 ADC  10.11.11.0/32      23.XXX.XXX.119  VPN                       0
 4  DS  23.XXX.XXX.83/32                   23.XXX.XXX.83             0
 5 A S  ;;; VPN Connection
        23.XXX.XXX.83/32                   LTE                       1
 6 ADC  100.XXX.XX1.0/24   100.XXX.XX1.84  LTE                       0
 7 A S  100.XXX.XXX.0/24                   LTE                       1
 8 ADC  192.XXX.XXX.0/24    192.XXX.XXX.1  BRIDGE                    0
 9 A S  208.XXX.XXX.220/32                 VPN                 1

#DOWN 
[admin@MikroTik] > ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          LTE                       2
 1   S  8.8.4.4/32                         VPN                 1
 2 A S  ;;; VPN Connection
        23.XXX.XXX.83/32                   LTE                       1
 3 ADC  100.XXX.XX1.0/24   100.XXX.XX1.84  LTE                       0
 4 A S  100.XXX.XXX.0/24                   LTE                       1
 5 ADC  192.XXX.XXX.0/24    192.XXX.XXX.1  BRIDGE                    0
 6   S  208.XXX.XXX.220/32                 VPN                 1

sindy · Wed May 09, 2018 1:47 am

As expected.

Change the gateways of the individual routes to IP addresses and remove the check-gateway=ping from them. When the VPN interface goes down, the route with it as a gateway becomes inactive too, so it stops shadowing the default one. A route with IP address as a gateway will not go down which is what we need. Never mind that the IP addresses of the VPN's gateway is dynamic, the add-on to the script as described in the DHCP stopped topics addresses that.

And I've realized that as it is physically impossible that VPN would be up while LTE is down, the individual routes for 8.8.8.8 and 208.67.222.222 via LTE's gateway are actually not necessary as it doesn't matter whether these two addresses are pinged via VPN or via LTE, as in both cases if responses are coming back it means that LTE is up.

jamthejame · Wed May 09, 2018 1:59 am

Ok.. so when i change:

/ip route
add distance=1 dst-address=8.8.4.4/32 gateway=VPN check-gateway=ping
add distance=1 dst-address=208.67.220.220/32 gateway=VPN check-gateway=ping

#and change to

add distance=1 dst=address=8.8.4.4/32 gateway=23.XXX.XXX.83
add distance=1 dst=address=208.67.220.220/32 gateway=23.XXX.XXX.83

I get status 'unreachable'. Is that what you meant?

And I've realized that as it is physically impossible that VPN would be up while LTE is down, the individual routes for 8.8.8.8 and 208.67.222.222 via LTE's gateway are actually not necessary as it doesn't matter whether these two addresses are pinged via VPN or via LTE, as in both cases if responses are coming back it means that LTE is up.

Though similar, thats why never added.

UPDATE***: Made all the changes we discussed, however, the script is still not reacting to VPN interface being disabled.

sindy · Wed May 09, 2018 2:11 am

I get status 'unreachable'. Is that what you meant?

No. In my case, adding a route via a nonexistent gateway doesn't make it be reported as unreachable:

[me@MyTik] >  ip route print detail where dst-address=40.113.108.151/32
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 0 A S  dst-address=40.113.108.151/32 gateway=192.168.10.1 gateway-status=192.168.10.1 reachable via  ether1 distance=1 scope=30
        target-scope=10 routing-mark=test

 1 A S  dst-address=40.113.108.151/32 gateway=192.168.10.123 gateway-status=192.168.10.123 reachable via  ether1 distance=1
        scope=30 target-scope=10

But I admit that in my case the nonexistent gateway is in a subnet which is up which likely makes the difference.

I'll come back tomorrow.

jamthejame · Wed May 09, 2018 6:49 am

OK... Do you think there is a way to resolve it, or do we have a logic error? Do you we should look into 'scope'?

Also, have a client device that is connected to the BRIDGE (LAN) and needs ICMP to certain DNS server. I tried adding a rule to allow Pings on the 'forward'' chain, but its not letting me:

/ip firewall filter 
add chain=forward protocol=icmp in-interface=BRIDGE out-interface=VPN src-address-list=permitted-local-resources dst-address-list=approved_dns action accept

The pings are coming from client device but Mikrotik rejects them all. My understanding was that its a forward rule even-though its echo (icmp).

sindy · Wed May 09, 2018 10:58 am

OK... Do you think there is a way to resolve it, or do we have a logic error? Do you we should look into 'scope'?

We have to change the logic but only slightly, instead of sending the pings different ways we'll just be looking which way the responses come in and keep the rest as it was.

And sorry, it was too late here yesterday so I have mixed together two address-lists and asked you to remove one which should have remained there.

So you can delete the dedicated routes via VPN's gateway IP, reduce the number of monitored external addresses if you wish, and and modify the mangle rules which check the source address the following way.

/ip firewall address-list
add address=8.8.8.8 list=netwatch-targets 
# add as many addresses as you want to monitor here, this list must match 1:1 the netwatch items (order doesn't matter of course)

/ip firewall mangle
add action=jump chain=prerouting icmp-options=0 jump-target=netwatch-monitors protocol=icmp
add action=add-dst-to-address-list address-list=netwatch-vpn-responses address-list-timeout=15s chain=netwatch-monitors in-interface=VPN src-address-list=netwatch-targets
add action=add-dst-to-address-list address-list=netwatch-wan-responses address-list-timeout=15s chain=netwatch-monitors src-address-list=netwatch-targets

So icmp echo responses coming from addresses on static (manually configured) address-list netwatch-targets will update the dynamic address-list netwatch-wan-responses regardless which way they got in, and the dynamic address-list netwatch-vpn-responses only if they come in via the VPN interface.

And not only it is enough to have only the two default routes, but their gateways even don't need to be IP addresses if interface names did work for everything except the previous monitoring approach.

Also, have a client device that is connected to the BRIDGE (LAN) and needs ICMP to certain DNS server. I tried adding a rule to allow Pings on the 'forward'' chain, but its not letting me:
Code: Select all
/ip firewall filter 
add chain=forward protocol=icmp in-interface=BRIDGE out-interface=VPN src-address-list=permitted-local-resources dst-address-list=approved_dns action accept
The pings are coming from client device but Mikrotik rejects them all. My understanding was that its a forward rule even-though its echo (icmp).

The rule seems fine to me, so I wonder whether it is not just a typo - I've called the address-list permitted-local-sources in my example and you refer to permitted-local-resources in this rule. Other than that, this rule must be before (above) the final "drop the rest" one in the forward chain, maybe that is the issue? In WinBox and Webfig, you can drag&drop the rules in the table, in command line, use /ip firewall filter print to dynamically assign numbers to the rules, then use /ip firewall filter move N destination=M where N is the number of the rule you want to move and M is the number of the rule before which you want to move it.

jamthejame · Wed May 09, 2018 4:39 pm

Ok... did make the changes, however, when VPN interface is disabled, nothing happens. It doesnt see that change in 'status'. When VPN interface is disabled i do get 'dynamic' timeout in 'address-list' for 'watch-vpn-responses' and the 'address' is LTE interface IP for some reason.

I also change the 'timeout' on scipt is from 1m5s to 35s and 'scheduler' from 1m5s to 35s given that we're only using 2 IPs now 8.8.8.8 and 208.67.222.222. And still nothing. It doesnt see VPN interface 'disabled'. Here is config:
FYI: If i disable VPN interface and have 'in-interface' in mangle, then the rule in mangle for VPN becomes red, thus inactive.

/tool netwatch
add host=8.8.8.8 interval=5s
add disabled=yes host=8.8.4.4 interval=5s
add host=208.67.222.222 interval=5s
add disabled=yes host=208.67.220.220 interval=5s

/ip firewall mangle
add action=jump chain=prerouting comment="Monitor Interfaces VPN & LTE" \
    icmp-options=0:0-255 jump-target=netwatch-monitors protocol=icmp
# VPN not ready
add action=add-dst-to-address-list address-list=watch-vpn-responses \
    address-list-timeout=15s chain=netwatch-monitors in-interface=VPN \
    src-address-list=netwatch-targets
add action=add-dst-to-address-list address-list=watch-vpn-responses \
    address-list-timeout=15s chain=netwatch-monitors src-address-list=\
    netwatch-targets

/system script
add name=operation_lte_vpn owner=admin policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="# c\
    reate the variables if they do not exist yet, be optimistic and set them to \
    green status\r\
    \nif ([/system script environment print count-only where name=lteAlive]=0) d\
    o={global lteAlive 1}\r\
    \nif ([/system script environment print count-only where name=vpnAlive]=0) d\
    o={global vpnAlive 1}\r\
    \n# compare the status of address-list monitor-lte to the previous one and t\
    ake action if it became empty\r\
    \nif ([/ip firewall address-list print count-only where list=monitor-lte]=0)\
    \_do={\\\r\
    \n  if ([/system script environment get [find name=lteAlive] value]>0) do={\
    \r\
    \n    global lteAlive 0;\r\
    \n    /interface lte set [find name=LTE] disable=yes;\r\
    \n/delay 2;\r\
    \n\t/interface lte set [find name=LTE] disable=no; \r\
    \n\t/ip firewall address-list add list=monitor-lte address=208.67.222.222 ti\
    meout=35s\r\
    \n  }\r\
    \n} else={\r\
    \n  if ([/system script environment get [find name=lteAlive] value]=0) do={\
    \r\
    \n    global lteAlive 1\r\
    \n  }\r\
    \n}\r\
    \n# compare the status of address-list monitor-vpn to the previous one and t\
    ake action if it became empty\r\
    \nif ([/ip firewall address-list print count-only where list=monitor-vpn]=0)\
    \_do={\\\r\
    \n  if ([/system script environment get [find name=vpnAlive] value]>0) do={\
    \r\
    \n    global vpnAlive 0;\r\
    \n        /interface l2tp-client set [find name=VPN] disabled=yes;\r\
    \n/delay 2;\r\
    \n\t\t/interface l2tp-client set [find name=VPN] disabled=no;\r\
    \n        /ip firewall address-list add list=monitor-vpn address=8.8.8.8 tim\
    eout=35s\r\
  \n  }\r\
    \n} else={\r\
    \n  if ([/system script environment get [find name=vpnAlive] value]=0) do={\
    \r\
    \n    global vpnAlive 1\r\
    \n  }\r\
    \n}"

/system scheduler
add interval=35s name=operation_lte_vpn on-event=\
    "/system script run operation_lte_vpn" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-time=startup

Also, about the DNS pings from client through Mikrotik, just sharing with you the firewall, maybe you can see something i'm missing:

/tool netwatch
add host=8.8.8.8 interval=5s
add disabled=yes host=8.8.4.4 interval=5s
add host=208.67.222.222 interval=5s
add disabled=yes host=208.67.220.220 interval=5s

/ip firewall mangle
add action=jump chain=prerouting comment="Monitor Interfaces VPN & LTE" \
    icmp-options=0:0-255 jump-target=netwatch-monitors protocol=icmp
# VPN not ready
add action=add-dst-to-address-list address-list=watch-vpn-responses \
    address-list-timeout=15s chain=netwatch-monitors in-interface=VPN \
    src-address-list=netwatch-targets
add action=add-dst-to-address-list address-list=watch-vpn-responses \
    address-list-timeout=15s chain=netwatch-monitors src-address-list=\
    netwatch-targets

/system script
add name=operation_lte_vpn owner=admin policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon source="# c\
    reate the variables if they do not exist yet, be optimistic and set them to \
    green status\r\
    \nif ([/system script environment print count-only where name=lteAlive]=0) d\
    o={global lteAlive 1}\r\
    \nif ([/system script environment print count-only where name=vpnAlive]=0) d\
    o={global vpnAlive 1}\r\
    \n# compare the status of address-list monitor-lte to the previous one and t\
    ake action if it became empty\r\
    \nif ([/ip firewall address-list print count-only where list=monitor-lte]=0)\
    \_do={\\\r\
    \n  if ([/system script environment get [find name=lteAlive] value]>0) do={\
    \r\
    \n    global lteAlive 0;\r\
    \n    /interface lte set [find name=LTE] disable=yes;\r\
    \n/delay 2;\r\
    \n\t/interface lte set [find name=LTE] disable=no; \r\
    \n\t/ip firewall address-list add list=monitor-lte address=208.67.222.222 ti\
    meout=35s\r\
    \n  }\r\
    \n} else={\r\
    \n  if ([/system script environment get [find name=lteAlive] value]=0) do={\
    \r\
    \n    global lteAlive 1\r\
    \n  }\r\
    \n}\r\
    \n# compare the status of address-list monitor-vpn to the previous one and t\
    ake action if it became empty\r\
    \nif ([/ip firewall address-list print count-only where list=monitor-vpn]=0)\
    \_do={\\\r\
    \n  if ([/system script environment get [find name=vpnAlive] value]>0) do={\
    \r\
    \n    global vpnAlive 0;\r\
    \n        /interface l2tp-client set [find name=VPN] disabled=yes;\r\
    \n/delay 2;\r\
    \n\t\t/interface l2tp-client set [find name=VPN] disabled=no;\r\
    \n        /ip firewall address-list add list=monitor-vpn address=8.8.8.8 tim\
    eout=35s\r\
  \n  }\r\
    \n} else={\r\
    \n  if ([/system script environment get [find name=vpnAlive] value]=0) do={\
    \r\
    \n    global vpnAlive 1\r\
    \n  }\r\
    \n}"

	/system scheduler
add interval=35s name=operation_lte_vpn on-event=\
    "/system script run operation_lte_vpn" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-time=startup

sindy · Wed May 09, 2018 5:04 pm

Ok... did make the changes, however, when VPN interface is disabled, nothing happens. It doesnt see that change in 'status'. When VPN interface is disabled i do get 'dynamic' timeout in 'address-list' for 'watch-vpn-responses' and the 'address' is LTE interface IP for some reason.

I also change the 'timeout' on scipt is from 1m5s to 35s and 'scheduler' from 1m5s to 35s given that we're only using 2 IPs now 8.8.8.8 and 208.67.222.222. And still nothing. It doesnt see VPN interface 'disabled'. Here is config:
FYI: If i disable VPN interface and have 'in-interface' in mangle, then the rule in mangle for VPN becomes red, thus inactive.

The fact that the rule becomes inactive doesn't mind as when the interface is down (and thus the rule is inactive), ping responses cannot come through the interface so the items on the address-list gradually time out which is what we want.

However, you haven't changed the address-list names in the script, so the script keeps referring to monitor-lte and monitor-vpn, but you have used different names for them in the mangle rules, and you have made both rules add the addresses to the same address list which means the whole thing cannot work.

so the proper mangle rules are the following ones:

add action=add-dst-to-address-list address-list=monitor-vpn address-list-timeout=15s chain=netwatch-monitors in-interface=VPN src-address-list=netwatch-targets
add action=add-dst-to-address-list address-list=monitor-lte address-list-timeout=15s chain=netwatch-monitors src-address-list=netwatch-targets

After fixing this, if after disabling the VPN manually it does not get re-enabled within a minute or two, check the output of /ip firewall address-list print:

while the VPN is down, you should see only items with list=netwatch-targets and no timeout values, and items with list=monitor-lte and some timeout between 0 and 15 s, but no items with list=monitor-vpn
while the VPN is up, the items with list=monitor-vpn and some timeout between 0 and 15 s should be there as well
when you shut down the LTE, only the items with list=netwatch-targets should print out.

Also, about the DNS pings from client through Mikrotik, just sharing with you the firewall, maybe you can see something i'm missing:

Unfortunately, you have posted exactly the same code as above one more time, which does not contain the firewall rules.

jamthejame · Wed May 09, 2018 6:06 pm

Fantastic... It seems to work now... I'll testing it further and let you know, but so far seems to work for both interfaces as intended.

Also, about the DNS pings from client through Mikrotik, just sharing with you the firewall, maybe you can see something i'm missing:
Unfortunately, you have posted exactly the same code as above one more time, which does not contain the firewall rules.

Foolish me...

I was thinking about making one of my failover Mikrotik devices use the same script to monitor Wan1 and Wan2 status, which is behind LTE Mikrotik. So I tried modifying the script a bit, but I noticed that when i disable Wan1, Netwatch is showing me that both IPs are 'DOWN' or 'UP' (BTW: I'm using 8.8.4.4 and 208.67.220.200) It seems like I'm failing to point Netwatch to the right interface to ping through..
I did add new DNS IPs to address-list of LTE Mikrotik side. Did I make a mistake in the script or its logic?

/ip firewall mangle
add action=jump chain=prerouting comment="Monitor Interfaces WAN_1 & WAN_2" icmp-options=0:0-255 jump-target=netwatch-monitors protocol=icmp
add action=add-dst-to-address-list address-list=monitor-wan_1 address-list-timeout=10m chain=netwatch-monitors in-interface=ether1_WAN_1 src-address-list=netwatch-targets
add action=add-dst-to-address-list address-list=monitor-wan_2 address-list-timeout=10m chain=netwatch-monitors in-interface=ether2_WAN_2 src-address-list=netwatch-targets
	
/system scheduler
add interval=20m10s name=operation_lte_vpn on-event="/system script run operation_wans" policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startup
	
/system script
add name=operation_wans owner=admin policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startup

# create the variables if they do not exist yet, be optimistic and set them to green status
if ([/system script environment print count-only where name=WAN1Alive]=0) do={global WAN1Alive 1}
if ([/system script environment print count-only where name=WAN2Alive]=0) do={global WAN2Alive 1}
# compare the status of address-list monitor-lte to the previous one and take action if it became empty
if ([/ip firewall address-list print count-only where list=monitor-wan_1]=0) do={
if ([/system script environment get [find name=WAN1Alive] value]>0) do={
global WAN1Alive 0;
/ip dhcp-client set [find interface=ether1_WAN_1] disable=yes;
/delay 2;
/ip dhcp-client set [find interface=ether1_WAN_1] disable=no;
/ip firewall address-list add list=monitor-wan_1 address=208.67.220.220 timeout=10m5s
}
} else={
if ([/system script environment get [find name=WAN1Alive] value]=0) do={
global WAN1Alive 1
}
}
#compare the status of address-list monitor-vpn to the previous one and take action if it became empty
if ([/ip firewall address-list print count-only where list=monitor-wan_2]=0) do={
if ([/system script environment get [find name=WAN2Alive] value]>0) do={
global WAN2Alive 0;
/ip dhcp-client set [find interface=ether2WAN2] disable=yes;
/delay 2;
/ip dhcp-client set [find interface=ether2WAN2] disable=no;
/ip firewall address-list add list=monitor-wan_2 address=8.8.4.4 timeout=10m5s
}
} else={
if ([/system script environment get [find name=WAN2Alive] value]=0) do={
global WAN2Alive 1
}
}
	
/tool netwatch
add host=8.8.4.4 interval=5m comment="WAN_2 Interface Monitor"
add host=208.67.220.220 interval=5m comment="WAN_1 Interface Monitor"

By the way, so far so good with the LTE device firewall and script. In the end, I found out that one of the IPs used wasn't on the 'approved' list, which stopped traffic. This is actually great news

P.S. CONGRATULATIONS!.... Writing so much in the last few days, you are a GURU now.

jamthejame · Thu May 10, 2018 2:56 am

Sindy, I was thinking on 'editing' my previous post, but i did that already 4 times today... so the only reason why i leave it, is because of Congratulation on getting "Guru" status. So I did manage to adopt the script into a failover backup, and what I mean by that is, sometime its beneficial to restart the interface and connection, so this actually works great. Plus, also help prevent certain issues of interface not being enabled upon restart etc. This is the script:

/ip firewall mangle
add action=jump chain=prerouting comment="Monitor Interfaces WAN1 & WAN2" icmp-options=0:0-255 jump-target=\
    netwatch-monitors protocol=icmp
add action=add-dst-to-address-list address-list=monitor-wan2 address-list-timeout=15s chain=netwatch-monitors \
    in-interface=ether2_WAN_2 src-address-list=netwatch-targets
add action=add-dst-to-address-list address-list=monitor-wan address-list-timeout=15s chain=netwatch-monitors \
    in-interface=ether1_WAN_1 src-address-list=netwatch-targets

/ip system script/system script
add name=operation_wans owner=***** policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon
# create the variables if they do not exist yet, be optimistic and set them to green status
if ([/system script environment print count-only where name=wanAlive]=0) do={global wanAlive 1}
if ([/system script environment print count-only where name=wan2Alive]=0) do={global wan2Alive 1}
# compare the status of address-list monitor-wan to the previous one and take action if it became empty
if ([/ip firewall address-list print count-only where list=monitor-wan]=0) do={\
if ([/system script environment get [find name=wanAlive] value]>0) do={
global wanAlive 0;
/ip dhcp-client set [find interface=ether1_WAN_1] disable=yes;
/delay 2;
/ip dhcp-client set [find interface=ether1_WAN_1] disable=no; 
/ip firewall address-list add list=monitor-wan address=208.67.222.222 timeout=35s
}
} else={
if ([/system script environment get [find name=wanAlive] value]=0) do={
global wanAlive 1
}
}
# compare the status of address-list monitor-wan2 to the previous one and take action if it became empty
if ([/ip firewall address-list print count-only where list=monitor-wan2]=0) do={\
if ([/system script environment get [find name=wan2Alive] value]>0) do={
global wan2Alive 0;
/ip dhcp-client set [find interface=ether2WAN_2] disable=yes;
/delay 2;
/ip dhcp-client set [find interface=ether2_WAN_2] disable=no;
/ip firewall address-list add list=monitor-wan2 address=8.8.8.8 timeout=35s
}
} else={
if ([/system script environment get [find name=wan2Alive] value]=0) do={
global wan2Alive 1
}
}

/tool netwatch
add comment="WAN_2 Interface Monitor" host=8.8.4.4 interval=5s
add comment="WAN_1 Interface Monitor" host=208.67.220.220 interval=5s

/ip firewall address-list
add address=8.8.4.4 list=netwatch-targets
add address=208.67.220.220 list=netwatch-targets

/system scheduler
add interval=35s name=operation_wans on-event="/system script run operation_wans" policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon start-time=startu

Now, I wanted to ask you, for redundancy purposes, if I would like to add another 2 IPs (1 for each interface to be monitored) as we attempted originally. Is that something that is possible? and if so, how would someone like me go about it?

sindy · Thu May 10, 2018 1:44 pm

Now, I wanted to ask you, for redundancy purposes, if I would like to add another 2 IPs (1 for each interface to be monitored) as we attempted originally. Is that something that is possible? and if so, how would someone like me go about it?

First, monitoring status of two interfaces which can substitute each other, like WAN1 and WAN2 in this scenario, rather than where operation of one of them depends on the other one to be up, like VPN and LTE in the other scenario, does require the individual routes to monitored addresses specific for each interface to ensure that if both interfaces work, the monitoring pings are actually sent out through the interface they monitor and not through the other one.

The method with recursive routing described e.g. here allows the pure failover (where one of the interfaces is preferred and the other one is only used if the other one fails) to work without any scripting and also without use of routing marks. But your tests made me wonder whether it won't provide weird results if one of the WAN interfaces goes physically down, because, as those tests have shown, if an interface is physically down, the route using that interface or an IP address in that interface's subnet as a gateway becomes inactive, so the test pings which should normally use that route take the default route instead, which causes the recursive gateway through the dead interface to seem reachable - unless RouterOS internally checks which way the responses come back. So I have tested that, and RouterOS (at least 6.43rc11) doesn't, but the traffic gets redirected via the other WAN interface anyway, because the routes to auxiliary addresses get redirected from direct routes to recursive routes.

So if backup is your only goal, you can use it as it is; if you need some connections to be bound to a particular interface and not redirected even if that interface goes down, you have to take additional measures.

With the script, the above is not an issue because you verify the in-interface of the icmp echo response, so if the request gets out some other way than expected, the response also comes back via that unexpected way (we talk about static routing and WAN inerfaces in different IP subnets; with dynamic routing protocols or with both WAN interfaces in the same subnet this would not always be the case).

So you have WAN1 and WAN2, associate a set of monitored addresses with each of the WANs (the two sets must not have any address in common of course), create the netwatch rules for all addresses from both sets, create the individual route for each of the monitored addresses via its associated WAN, and only update the status-tracking address lists by icmp responses which come with a proper combination of in-interface and src-address-list. So to make your script properly recognize the WAN status, you only need to

add the individual routes for the monitored addresses associated to each WAN sending the pakets via that WAN,
split the single static address list netwatch-targets into two separate ones, netwatch-targets-wan1 and netwatch-targets-wan2, and modify the mangle rules accordingly, as described above.

jamthejame · Fri May 11, 2018 4:44 pm

Frankly, I had to rethink the whole approach after tonight's "DefCon 4" disaster. In short, people decided to shift from 'DHCP' completely to 'static'. Which created a lot of issue, as all devices where not tested. Bottom line, complete collapse. Additionally, I had false positive with the firewall, as it seems when i was testing i initiated the last rule to 'drop all' in 'forward' chain to be last activated, so the session continued, showing me all well, but when i moved to deployment, ... well, you got the short of it. This is what i have right now:

#DEVICE 1
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface ethernet
set [ find default-name=ether2 ] disabled=yes
set [ find default-name=ether3 ] disabled=yes
set [ find default-name=ether4 ] disabled=yes
set [ find default-name=ether5 ] arp=reply-only
set [ find default-name=ether7 ] disabled=yes
set [ find default-name=ether8 ] disabled=yes
set [ find default-name=ether9 ] disabled=yes
set [ find default-name=ether10 ] disabled=yes
set [ find default-name=sfp1 ] disabled=yes
/interface l2tp-client
add add-default-route=yes allow-fast-path=yes connect-to=23.XXX.XXX.83 dial-on-demand=yes disabled=no ipsec-secret=\
    ***** keepalive-timeout=10 max-mru=1410 max-mtu=1410 name=VPN password=****** use-ipsec=yes user=\
    ******
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip address
add address=192.XXX.XXX.25/24 interface=ether1 network=192.XXX.1.0
add address=10.10.19.1/24 interface=ether5 network=10.10.19.0
/ip arp
add address=10.10.19.145 interface=ether5 mac-address=XX:XX:XX:XX:XX:XX
/ip dns
set allow-remote-requests=yes servers=10.10.10.10,8.8.8.8
/ip firewall address-list
add address=24.XXX.XXX.100 comment=Demo list=permitted-http-destinations-address
add address=75.XXX.XXX.200 comment=Demo1 list=permitted-https-destinations-address
add address=9.9.9.9 comment=DNS list=DNS_Addresses
add address=192.XXX.XXX.2 comment=MGT list=management-access
add address=10.10.19.145 comment="Local Permitted IP Address#1" list=permitted-local-source
/ip firewall filter
add action=accept chain=input comment="INPUT Chain - Packets towards Mikrotik itself" connection-state=\
    established,related
add action=accept chain=input comment="Permit DNS and NTP requests from clients" dst-port=53 in-interface=ether5 \
    protocol=udp
add action=accept chain=input comment="Permit DNS requests from clients over TCP" dst-port=53 in-interface=ether5 \
    protocol=tcp
add action=accept chain=input comment="Winbox Access" dst-port=61345 in-interface=ether6 protocol=tcp src-address-list=\
    management-access
add action=drop chain=input comment="Prevent any other new connection to the Mikrotik itself"
add action=accept chain=forward comment="FORWARD Chain - Packets towards Mikrotik itself" connection-state=\
    established,related
add action=accept chain=input comment="Permit DNS and NTP requests from clients" disabled=yes dst-port=53 in-interface=\
    VPN protocol=udp
add action=accept chain=forward comment="Approved HTTP" dst-address-list=permitted-http-destinations-address dst-port=80 \
    in-interface=ether5 out-interface=VPN protocol=tcp src-address-list=permitted-local-sources
add action=accept chain=forward comment="Approved HTTPS" dst-address-list=permitted-https-destinations-address dst-port=\
    443 in-interface=ether5 out-interface=VPN protocol=tcp src-address-list=permitted-local-sources
add action=accept chain=forward comment="Approved DNS Addresses" dst-address-list=DNS_Addresses in-interface=ether5 \
    out-interface=VPN protocol=icmp src-address-list=permitted-local-sources
add action=drop chain=forward comment=\
    "Drop anything attempting to pass through the Mikrotik that doesn't match the rules above"
/ip firewall nat
add action=masquerade chain=srcnat out-interface=VPN
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip route
add comment=WAN distance=2 gateway=20.20.1.1
add comment=VPN distance=1 dst-address=23.XXX.XXX.83/32 gateway=20.20.1.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set winbox port=XXX
set api-ssl disabled=yes

#DEVICE 2

[admin@MikroTik] > export
/ip pool
add name=LAN ranges=30.30.30.25-30.30.30.28
/ip dhcp-server
add address-pool=LAN disabled=no interface=ether5 name=LAN
/ip address
add address=30.30.30.1/24 interface=ether5 network=30.30.30.0
add address=10.10.19.145/24 interface=ether1 network=192.168.19.0
/ip dhcp-server network
add address=30.30.30.0/24 dns-server=30.30.30.1 gateway=30.30.30.1
/ip dns
set allow-remote-requests=yes servers=10.10.19.1
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
/ip route
add comment=Internet_WAN_1 distance=1 gateway=10.10.19.1
add comment=Internet_WAN_2 distance=2 gateway=10.10.24.1

'Device1' has the firewall we discussed earlier and 'Device2' is there for simple failover between other mikrotiks. However, all works well until the firewall is put in place. Checked the source IP etc. However, i get time out even pinging the 'approved dns list'. Tried moving things around, but this VPN is the one I'm struggling as its not straight in and out. Basically, I'm trying to find out, where is the issue and why it will not pass traffic from the Device2 to Device1.

sindy · Fri May 11, 2018 9:14 pm

Sorry, total confusion.

There are no firewall rules in the export of device 2. So which firewall rules interfere with what exactly?

Also, unless you haven't posted most of the device 2's configuration export, the whole redundancy on device 2 consists in using the route with distance=2 if the interface through which the route with distance=1 goes is down.

jamthejame · Fri May 11, 2018 9:33 pm

Pardon, Sindy. I did not set up the failover on "Device 2' simply because i haven't done it on this new setup. I tested it on the old one and it worked. What I've shows is where everything stops. I figured no need to put 'bla bla' which isn't relevant.

The issue is with the 'Device 1' firewall. To be more exactly, isn't going through, especially when the last 'drop' chain rule is set. When i take it off, it sometimes works, so I wanted to ask if we maybe overlooked something as it doesnt work under the given configuration. Let me know if you need anything else.

sindy · Fri May 11, 2018 9:49 pm

The issue is with the 'Device 1' firewall. To be more exactly, isn't going through, especially when the last 'drop' chain rule is set.

OK, so let's take a look at this one first. It is the simple one, there is again just a typo:

/ip firewall address-list
...
add address=10.10.19.145 comment="Local Permitted IP Address#1" list=permitted-local-source

but

add action=accept chain=forward comment="Approved HTTP" dst-address-list=permitted-http-destinations-address dst-port=80 in-interface=ether5 out-interface=VPN protocol=tcp src-address-list=permitted-local-sources

add action=accept chain=forward comment="Approved HTTPS" dst-address-list=permitted-https-destinations-address dst-port=443 in-interface=ether5 out-interface=VPN protocol=tcp src-address-list=permitted-local-sources

add action=accept chain=forward comment="Approved DNS Addresses" dst-address-list=DNS_Addresses in-interface=ether5 out-interface=VPN protocol=icmp src-address-list=permitted-local-sources

When i take it off, it sometimes works

The sometimes worries me most. If you remove the last action=drop rule in chain=forward, everything must pass through, so if it doesn't, there is a different reason to it than device 1's firewall.

Off topic, what is your first language? My tip is Latvian or Lithuanian.

jamthejame · Fri May 11, 2018 9:57 pm

Those ending issues, I noticed and changed... but still have 'sometimes' part.

P.S. Ukrainian, Russian, some Polish, and some English

UPDATE***: oddly, i can ping all the addresses from DNS List...

BTW: you never answered to me about your specialty..

sindy · Fri May 11, 2018 10:14 pm

BTW: you never answered to me about your specialty..

Тому що ти видалів це питання, коли ти п'ятий раз змінив свій пост

Я побачив одну-дві версії на телефону, але поки я доїхав до комп'ютеру, цього питання вже не було.
I would call myself a network and telco fireman. If something out-of-usual happens with VoIP or networks, they ask me to sort that out.

jamthejame · Fri May 11, 2018 10:38 pm

I'm not even going to try to speculate on the last part, as you sound like someone I spoke earlier today. Last time i remember you mention that you're in Czech? Correct me if I'm wrong.
Yeah, I tend to be sometime trigger happy, especially when i speak with someone competent.

P.S. Seems like google translate, but more of a quick type on the phone or one of those slider devices that finish words quickly.

sindy · Fri May 11, 2018 10:47 pm

I'm not even going to try to speculate on the last part, as you sound like someone I spoke earlier today.

I haven't spoken to anyone in the U.S. nor anyone whose first language was Ukrainian today so it cannot be me

Last time i remember you mention that you're in Czech? Correct me if I'm wrong.

I didn't say I was in Czechia, I only said my native language was Czech. But both is true

P.S. Seems like google translate, but more of a quick type on the phone or one of those slider devices that finish words quickly.

What do you have in mind? The part in Ukrainian? It was genuine, not translated using Google translate, and written on the PC with no auto-completion, but I have to admit it is not the language I know best.

Update: if you can ping the DNS addresses from the clients of Device 2, it is a part of the "sometimes" puzzle, as the firewall on Device 1 doesn't permit icmp in chain forward. So the pings wouldn't get through if they took that path, so they must have reached their destinations some other way.

Update 2: the above is wrong - you permit icmp requests to DNS servers, but you don't permit DNS requests to DNS servers, I didn't expect that so I didn't look carefully

jamthejame · Fri May 11, 2018 10:55 pm

Extremely impressive... you're quite an interesting individual. I dont know if it is forum rules... might get banned, but would share might email, as there are not that many competent individuals around where I'm in USA. Would like to hear how you got into it with Mikrotik.

I haven't spoken to anyone in the U.S. nor anyone whose first language was Ukrainian today so it cannot be me

A true global man...

FYI: you cant say you didnt speak with anyone today who's from US... we've been writing back and forth

sindy · Fri May 11, 2018 10:57 pm

FYI: you cant say you didnt speak with anyone today who's from US... we've been writing back and forth

Correct, but unless you use several identities here, it does not count

jamthejame · Fri May 11, 2018 11:12 pm

jamthejame · Fri May 11, 2018 11:26 pm

Creative.... sent a test one...

By the way, i think its worth mentioning that over the last few days you got this 'forum thread' on 'FIRE", with something like 900 views

jamthejame · Sat May 12, 2018 2:17 pm

Sindy, unfortunately, patience is not my strongest suite, but there is always a price to pay

So, wanted to get your opinion on the last matter about IPs not going through. I attached the one with changed 's', so see if there is something else that might be missing. I do have to say that having the last 'drop' rule in the 'input' chain has done a lot of good, yet i wonder is that also hurt with my L2TP/IPSEC tunnel being down more often than necessary, however, that for later as I'm more concerned as to the firewall issue and listed IPs not going through.

#DEVICE 1
/interface wireless
set [ find default-name=wlan1 ] ssid=MikroTik
/interface ethernet
set [ find default-name=ether2 ] disabled=yes
set [ find default-name=ether3 ] disabled=yes
set [ find default-name=ether4 ] disabled=yes
set [ find default-name=ether5 ] arp=reply-only
set [ find default-name=ether7 ] disabled=yes
set [ find default-name=ether8 ] disabled=yes
set [ find default-name=ether9 ] disabled=yes
set [ find default-name=ether10 ] disabled=yes
set [ find default-name=sfp1 ] disabled=yes
/interface l2tp-client
add add-default-route=yes allow-fast-path=yes connect-to=23.XXX.XXX.83 dial-on-demand=yes disabled=no ipsec-secret=\
    ***** keepalive-timeout=10 max-mru=1410 max-mtu=1410 name=VPN password=****** use-ipsec=yes user=\
    ******
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip address
add address=192.XXX.XXX.25/24 interface=ether1 network=192.XXX.1.0
add address=10.10.19.1/24 interface=ether5 network=10.10.19.0
/ip arp
add address=10.10.19.145 interface=ether5 mac-address=XX:XX:XX:XX:XX:XX
/ip dns
set allow-remote-requests=yes servers=10.10.10.10,8.8.8.8
/ip firewall address-list
add address=24.XXX.XXX.100 comment=Demo list=permitted-http-destinations-address
add address=75.XXX.XXX.200 comment=Demo1 list=permitted-https-destinations-address
add address=9.9.9.9 comment=DNS list=DNS_Addresses
add address=192.XXX.XXX.2 comment=MGT list=management-access
add address=10.10.19.145 comment="Local Permitted IP Address#1" list=permitted-local-sources
/ip firewall filter
add action=accept chain=input comment="INPUT Chain - Packets towards Mikrotik itself" connection-state=\
    established,related
add action=accept chain=input comment="Permit DNS and NTP requests from clients" dst-port=53 in-interface=ether5 \
    protocol=udp
add action=accept chain=input comment="Permit DNS requests from clients over TCP" dst-port=53 in-interface=ether5 \
    protocol=tcp
add action=accept chain=input comment="Winbox Access" dst-port=61345 in-interface=ether6 protocol=tcp src-address-list=\
    management-access
add action=drop chain=input comment="Prevent any other new connection to the Mikrotik itself"
add action=accept chain=forward comment="FORWARD Chain - Packets towards Mikrotik itself" connection-state=\
    established,related
add action=accept chain=input comment="Permit DNS and NTP requests from clients" disabled=yes dst-port=53 in-interface=\
    VPN protocol=udp
add action=accept chain=forward comment="Approved HTTP" dst-address-list=permitted-http-destinations-address dst-port=80 \
    in-interface=ether5 out-interface=VPN protocol=tcp src-address-list=permitted-local-sources
add action=accept chain=forward comment="Approved HTTPS" dst-address-list=permitted-https-destinations-address dst-port=\
    443 in-interface=ether5 out-interface=VPN protocol=tcp src-address-list=permitted-local-sources
add action=accept chain=forward comment="Approved DNS Addresses" dst-address-list=DNS_Addresses in-interface=ether5 \
    out-interface=VPN protocol=icmp src-address-list=permitted-local-sources
add action=drop chain=forward comment=\
    "Drop anything attempting to pass through the Mikrotik that doesn't match the rules above"
/ip firewall nat
add action=masquerade chain=srcnat out-interface=VPN
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes
/ip route
add comment=WAN distance=2 gateway=20.20.1.1
add comment=VPN distance=1 dst-address=23.XXX.XXX.83/32 gateway=20.20.1.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh disabled=yes
set api disabled=yes
set winbox port=XXX
set api-ssl disabled=yes

#DEVICE 2

[admin@MikroTik] > export
/ip pool
add name=LAN ranges=30.30.30.25-30.30.30.28
/ip dhcp-server
add address-pool=LAN disabled=no interface=ether5 name=LAN
/ip address
add address=30.30.30.1/24 interface=ether5 network=30.30.30.0
add address=10.10.19.145/24 interface=ether1 network=192.168.19.0
/ip dhcp-server network
add address=30.30.30.0/24 dns-server=30.30.30.1 gateway=30.30.30.1
/ip dns
set allow-remote-requests=yes servers=10.10.19.1
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
/ip route
add comment=Internet_WAN_1 distance=1 gateway=10.10.19.1
add comment=Internet_WAN_2 distance=2 gateway=10.10.24.1
add comment=Google_DNS distance=1 dst-address=8.8.4.4/32 gateway=192.168.19.1 scope=10
add comment=Quad9_DNS distance=1 dst-address=9.9.9.9/32 gateway=192.168.19.1 scope=10
add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=8.8.4.4 scope=10
add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=208.67.220.220 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=9.9.9.9 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=209.244.0.3 scope=10
add comment=OpenDNS_DNS distance=1 dst-address=208.67.220.220/32 gateway=192.168.24.1 scope=10
add comment=Level3_DNS distance=1 dst-address=209.244.0.3/32 gateway=192.168.24.1 scope=10

Its odd, but the pings through and from Device#2 go through.

sindy · Sat May 12, 2018 2:53 pm

I cannot see anything in the configuration. So please add a rule

action=log chain=forward protocol=tcp connection-state=new

right below the "accept related, established" one in chain=forward on Device 1.

Then, try to establish the http/https connections from the machines behind Device 2.

And then post here the output of /log print where topics~"firewall" on Device 1. Sanitization as usually.

sindy · Sat May 12, 2018 3:44 pm

Wow, just spotted it:

/ip route
add comment=WAN distance=2 gateway=20.20.1.1
add comment=VPN distance=1 dst-address=23.XXX.XXX.83/32 gateway=20.20.1.1

There is no default route via the VPN, so how do you expect anything to have out-interface=vpn?

jamthejame · Sat May 12, 2018 4:39 pm

I think this is the export fault as it didnt print the default routes as the VPN interface is checked with default' route. This is the routing table via 'print':

[admin@MikroTik] > ip route print 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          VPN                     1
 1   S  ;;; WAN
        0.0.0.0/0                          20.20.1.1               2
 2 ADC  10.10.10.0/24      10.10.10.1      ether4                    0
 3 ADC  10.11.11.0/32      23.XXX.XXX.118  VPN                     0
 4 ADS  23.XXX.XXX.83/32                   20.20.1.1               0
 5   S  ;;; VPN
        23.XXX.XXX.83/32                   20.20.1.1               1
 6 ADC  20.20.1.0/24     20.20.1.250   ether1                    0
 7 ADC  20.20.19.0/24    20.20.19.1    ether5                    0

sindy · Sat May 12, 2018 4:46 pm

Well, strictly speaking it is not a fault of the /export, it is a feature because export shows only manual configuration, while /... print shows the dynamic items as well.

So it is rather my fault that I haven't realized this as I was trying to spot someting useful.

What did the log show?

jamthejame · Sat May 12, 2018 6:10 pm

right below the "accept related, established" one in chain=forward on Device 1.

Can you clarify, please

sindy · Sat May 12, 2018 6:17 pm

...
add action=accept chain=forward comment="FORWARD Chain - Packets not towards Mikrotik itself" connection-state=established,related
add action=log chain=forward connection-state=new protocol=tcp
add action=accept chain=input comment="Permit DNS and NTP requests from clients" disabled=yes dst-port=53 in-interface=VPN protocol=udp
add action=accept chain=forward comment="Approved HTTP" dst-address-list=permitted-http-destinations-address dst-port=80 in-interface=ether5 out-interface=VPN protocol=tcp src-address-list=permitted-local-sources
...

jamthejame · Sat May 12, 2018 6:34 pm

So this is great.... My IPS1 just stopped, so my Failover device was support to swtich me to ISP2, but its not... nothing is coming out of it. Worked well with ISP1, but eventhough ISP1 Mik is identical in config and ISP2 (for testing purposes), but traffic not flowing

so now before i do the log, i need to resolve this issue with Device#2

[admin@MikroTik] > export
/interface ethernet
set [ find default-name=ether4 ] arp=reply-only
set [ find default-name=ether5 ] arp=reply-only
/ip address
add address=30.30.30.53.1/24 interface=ether5 network=30.30.30.53.0
add address=40.40.19.145/24 interface=ether1 network=40.40.19.0
add address=50.50.24.173/24 interface=ether2 network=50.50.24.0
/ip arp
add address=30.30.30.53.23 interface=ether5 mac-address=XX.XX.XX.XX.XX.XX
/ip dns
set allow-remote-requests=yes servers=40.40.19.1,50.50.24.1
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
add action=masquerade chain=srcnat out-interface=ether2
/ip route
add distance=1 gateway=10.1.1.1 routing-mark=ISP1
add distance=2 gateway=10.2.2.2 routing-mark=ISP1
add distance=1 gateway=10.2.2.2 routing-mark=ISP2
add distance=2 gateway=10.1.1.1 routing-mark=ISP2
add comment=Internet_WAN_1 distance=1 gateway=40.40.19.1
add comment=Internet_WAN_2 distance=1 gateway=50.50.24.1
add comment=Google#2_DNS distance=1 dst-address=8.8.4.4/32 gateway=40.40.19.1 scope=10
add comment=Quad9_DNS distance=1 dst-address=9.9.9.9/32 gateway=40.40.19.1 scope=10
add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=208.67.220.220 scope=10
add check-gateway=ping distance=1 dst-address=10.1.1.1/32 gateway=8.8.4.4 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=9.9.9.9 scope=10
add check-gateway=ping distance=1 dst-address=10.2.2.2/32 gateway=209.244.0.3 scope=10
add comment=OpenDNS_DNS distance=1 dst-address=208.67.220.220/32 gateway=50.50.24.1 scope=10
add comment=Level3_DNS distance=1 dst-address=209.244.0.3/32 gateway=50.50.24.1 scope=10

i did multiple host checking from the "Advanced Failover" Mikrotik page... but it doesnt seem to move traffic to 'ether2' ... Can't even have pings coming out of it now.... Right now connected directly to IPS2 system to write this message, so it work, as I only left 'input' chain

***UPDATE: I think the issue was 'distance'... had ISP1 w/ distance=1 and ISP2 /w distance=2. Changed the distance for both to 1. Seems to work, let me produce the log now.

sindy · Sat May 12, 2018 7:07 pm

Well, this looks to me as a misunderstanding of the concept of the backup

For each WAN, A and B, you choose two monitored addresses used to check that internet can be reached via that interface's gateway, X1 and X2.

The IP addresses of gateway routers of WAN A and WAN B are GA and GB, respectively.

To make it work even if one of the monitored addresses stops responding, you use an auxiliary address same for both X1 and X2, X0.

And the X0 is then used as the gateway representing WAN X for upper routes.

So the whole scheme looks as follows:
dst-address=A1/32 gateway=GA scope=10
dst-address=A2/32 gateway=GA scope=10
dst-address=A0 gateway=A1/32 scope=10 target=scope=10 check-gateway=ping
dst-address=A0 gateway=A2/32 scope=10 target=scope=10 check-gateway=ping
dst-address=0.0.0.0/0 gateway=A0 is active and routes the traffic via GA on WAN A if at least one of (A1,A2) respond to pings

dst-address=B1/32 gateway=GA scope=10
dst-address=B2/32 gateway=GA scope=10
dst-address=B0 gateway=B1/32 scope=10 target=scope=10 check-gateway=ping
dst-address=B0 gateway=B2/32 scope=10 target=scope=10 check-gateway=ping
dst-address=0.0.0.0/0 gateway=B0 is active and routes the traffic via GB on WAN B if at least one of (B1,B2) respond to pings

Depending on how you prioritize the two routes with dst-address=0.0.0.0/0 within the same routing table (defaut or other) using distance and whether you use both in that routing table, you can use WAN B as a backup of WAN A, or use only one of them, or distribute the traffic evenly between them.

In your configuration, there is a cross-link, because you've linked B0 to B1 and A2, and A0 to A1 and B2. I'm not sure whether that doesn't cause the thing to think that WAN A works even while it actually doesn't.

On top of that, you have default routes only in non-default routing tables, named ISP1 and ISP2. So unless you assign routing-marks ISP1 or ISP2 to packets, they cannot be routed anywhere. Do you assing routing-marks to packets or not?

sindy · Sat May 12, 2018 7:22 pm

My properly working test configuration from Friday rearranged the same way like the one with symbolic names in the previous post:

/ip route

add distance=1 dst-address=8.8.8.8/32 gateway=10.22.6.1 scope=10
add distance=1 dst-address=208.67.220.220/32 gateway=10.22.6.1 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.1/32 gateway=8.8.8.8 scope=10 target=scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.1/32 gateway=208.67.220.220 scope=10 target=scope=10
add distance=1 gateway=100.64.0.1

add distance=1 dst-address=8.8.4.4/32 gateway=10.22.28.1 scope=10
add distance=1 dst-address=208.67.222.222/32 gateway=10.22.28.1 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.2/32 gateway=8.8.4.4 scope=10 target=scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.2/32 gateway=208.67.222.222 scope=10 target=scope=10
add distance=2 gateway=100.64.0.2

Here everything is in the default routing table, so no routing-marks are necessary, and WAN 1 with gateway 10.22.6.1 is the primary uplink, and WAN 2 with gateway 10.22.28.1 is the backup one, because the default route recursively via 10.22.6.1 has distance=1 and the default route recursively via 10.22.28.1 has distance=2.

jamthejame · Sat May 12, 2018 7:25 pm

On top of that, you have default routes only in non-default routing tables, named ISP1 and ISP2. So unless you assign routing-marks ISP1 or ISP2 to packets, they cannot be routed anywhere. Do you assing routing-marks to packets or not?

I'm not, went by as it was written, and was wondering why just leave ISP1 / IPS2 as routing-marks without identifying it.

Should I create a 'mangle' rule for that?

jamthejame · Sat May 12, 2018 7:26 pm

log output for firewall with rules, seems as it is trying to SYN but gets nothing back, this is the address that's relevant '100.200.300.238:443'

12:12:04 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52437->100.200.300.238:443, len 52 
12:12:04 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52438->100.200.300.238:443, len 52 
12:12:06 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52433->100.200.300.238:443, len 48 
12:12:06 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52434->100.200.300.238:443, len 48 
12:12:06 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52439->100.200.300.238:443, len 52 
12:12:08 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52435->200.10.10.177:443, len 48 
12:12:09 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52439->100.200.300.238:443, len 52 
12:12:09 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52436->55.55.55.252:443, len 48 
12:12:10 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52437->100.200.300.238:443, len 48 
12:12:10 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52438->100.200.300.238:443, len 48 
12:12:11 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52440->66.66.66.162:80, len 52 
12:12:11 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52441->100.200.300.238:443, len 52 
12:12:14 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52440->66.66.66.162.162:80, len 52 
12:12:14 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52441->100.200.300.238:443, len 52 
12:12:15 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52439->100.200.300.238:443, len 48 
12:12:15 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), 10.10.10.73:52442->77.77.77.200:80, len 52

sindy · Sat May 12, 2018 7:33 pm

Well, now we come to the point that when obfuscating things, you have to do it in such a way that context is not lost.

The log shows the packets are coming with source address 10.10.10.73, while your configurations say

Device 2:

/ip address
add address=10.10.19.145/24 interface=ether1 network=192.168.19.0

/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1

Device 1:

/ip firewall address-list
add address=10.10.19.145 comment="Local Permitted IP Address#1" list=permitted-local-sources

So I don't know - is it really so that the packets are not src-nated on Device 2 to 10.10.19.145 (maybe because Device 2's ether1 is actually not connected to Device 1's ether5) or is it just an inconsistent obfuscation and the source addresses in the log were actually 10.10.19.145?

(leaving aside that the value of the network parameter is inconsistent with the value of the address parameter on Device 2 on ether1)

jamthejame · Sat May 12, 2018 7:44 pm

Ok... I probably made a mess with obfuscation (actually not 'probably' but actually). If we go back, it will be 10.10.19.145.

Also, taking down the rule from 'forward' chain, allows me to go online, easily... as I'm writing this message from laptop going thru Dev1 and Dev2, but the minute i put that rule, i cant get the system to get online.

So Dev2 seems to move traffic to Dev1 no issue as Dev1 so far is the only one that has the Firewall rules set

jamthejame · Sat May 12, 2018 7:47 pm

Regarding GW failover. Would it make a negative inpact if i would have different distance for GW, as I preferred GW1 to be used all the time if available and GW2 just a failover. Also, I notice, that when both work and traffic moves to 2 and then 1 comes back online, if the session/connection was open on GW2 is continues, until interrupted.

sindy · Sat May 12, 2018 7:48 pm

On top of that, you have default routes only in non-default routing tables, named ISP1 and ISP2. So unless you assign routing-marks ISP1 or ISP2 to packets, they cannot be routed anywhere. Do you assing routing-marks to packets or not?
I'm not, went by as it was written, and was wondering why just leave ISP1 / IPS2 as routing-marks without identifying it.

Should I create a 'mangle' rule for that?

No. Unless you have a reason why you would want some traffic to prefer WAN 1 and other traffic to prefer WAN 2, just completely remove the routes with routing-mark=ISP2, and remove the routing-mark=ISP1 from the routes where it is used. Then all traffic will prefer WAN 1 and only switch over to WAN 2 if internet becomes inaccessible via WAN 1.

And don't forget to fix the crosslink as mentioned above.

sindy · Sat May 12, 2018 7:52 pm

Regarding GW failover. Would it make a negative inpact if i would have different distance for GW, as I preferred GW1 to be used all the time if available and GW2 just a failover.

Yes, if you want the traffic to prefer WAN 1, give the recursive default route through it a lower distance than the one through WAN 2.

Also, I notice, that when both work and traffic moves to 2 and then 1 comes back online, if the session/connection was open on GW2 is continues, until interrupted.

This is normal as the connection tracking and route cache take care of this.

jamthejame · Sat May 12, 2018 7:55 pm

/ip route

add distance=1 dst-address=8.8.8.8/32 gateway=10.22.6.1 scope=10
add distance=1 dst-address=208.67.220.220/32 gateway=10.22.6.1 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.1/32 gateway=8.8.8.8 scope=10 target=scope=10 
add check-gateway=ping distance=1 dst-address=100.64.0.1/32 gateway=208.67.220.220 scope=10 target=scope=10 
add distance=1 gateway=100.64.0.1

add distance=2 dst-address=8.8.4.4/32 gateway=10.22.28.1 scope=10
add distance=2 dst-address=208.67.222.222/32 gateway=10.22.28.1 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.2/32 gateway=8.8.4.4 scope=10 target=scope=10 
add check-gateway=ping distance=1 dst-address=100.64.0.2/32 gateway=208.67.222.222 scope=10 target=scope=10 
add distance=2 gateway=100.64.0.2

Something like this? What about

add comment=Internet_WAN_2 distance=1 gateway=GW1
add comment=Internet_WAN_1 distance=1 gateway=GW2

?

sindy · Sat May 12, 2018 8:02 pm

Something like this? What about
Code: Select all
add comment=Internet_WAN_2 distance=1 gateway=GW1
add comment=Internet_WAN_1 distance=1 gateway=GW2
?

No, the exact reverse. Leave alone the distance parameters of all the routes with dst-addresses A0, A1, A2, B0, B1, B2 (the distance values of these routes are totally irrelevant) but set distance=1 to the route with dst-address=0.0.0.0/0 and gateway=A0, and distance=2 to the route with dst-address=0.0.0.0/0 and gateway=B0.

jamthejame · Sat May 12, 2018 8:07 pm

I believe this is the right way...

/ip route

add distance=1 dst-address=8.8.8.8/32 gateway=10.22.6.1 scope=10
add distance=1 dst-address=208.67.220.220/32 gateway=10.22.6.1 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.1/32 gateway=8.8.8.8 scope=10 target=scope=10 
add check-gateway=ping distance=1 dst-address=100.64.0.1/32 gateway=208.67.220.220 scope=10 target=scope=10 
add distance=1 gateway=100.64.0.1

add distance=1 dst-address=8.8.4.4/32 gateway=10.22.28.1 scope=10
add distance=1 dst-address=208.67.222.222/32 gateway=10.22.28.1 scope=10
add check-gateway=ping distance=2 dst-address=100.64.0.2/32 gateway=8.8.4.4 scope=10 target=scope=10 
add check-gateway=ping distance=2 dst-address=100.64.0.2/32 gateway=208.67.222.222 scope=10 target=scope=10 
add distance=2 gateway=100.64.0.2

add comment=Internet_WAN_2 distance=1 gateway=GW2
add comment=Internet_WAN_1 distance=1 gateway=GW1

By the way, is pinging public DNSs is the best way to go about it? I heard that trying to query website IPs is a better option. Is that true?

sindy · Sat May 12, 2018 8:13 pm

I believe this is the right way...

I believe you must be testing my patience

I suggest you to do something, you do the opposite and ask me to confirm it

Please read my previous post again, and the one regarding consistent obfuscation too. Which addresses the GW1 and GW2 represent in the context of the rest of the code?

jamthejame · Sat May 12, 2018 8:59 pm

I believe you must be testing my patience I suggest you to do something, you do the opposite and ask me to confirm it

Unfortunately, I was told that on a few occasions, which usually happens after too much coffee, thus pardon and glad you brought it up. Sometimes I need someone to remind me of pressing a 'brake' pedal

**Also, lets start from a new page here, because with this obfuscation i created a mess around me and the files.

#Device#1
[admin@MikroTik] > export
/interface ethernet
set [ find default-name=ether4 ] arp=reply-only
set [ find default-name=ether5 ] arp=reply-only
/interface l2tp-client
add add-default-route=yes allow-fast-path=yes connect-to=23.XXX.XXX.83 dial-on-demand=yes disabled=no ipsec-secret=\
    ***** keepalive-timeout=10 max-mru=1410 max-mtu=1410 name=VPN password=**** use-ipsec=yes user=\
    ****
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip address
add address=XX.XX.1.250/24 interface=ether1 network=XX.XX.1.0
add address=XX.XX.19.1/24 interface=ether5 network=XX.XX.19.0
add address=10.10.10.1/24 interface=ether4 network=10.10.10.0
/ip arp
add address=XX.XX.19.145 interface=ether5 mac-address=XX.XX.XX.XX.XX.XX
add address=10.10.10.5 interface=ether4 mac-address=XX.XX.XX.XX.XX.XX
/ip dns
set allow-remote-requests=yes servers=10.10.10.10,8.8.8.8
/ip firewall address-list
add address=24.XXX.XXX.100 comment=Demo list=permitted-http-destinations-address
add address=75.XXX.XXX.200 comment=Demo1 list=permitted-https-destinations-address
add address=9.9.9.9 comment=DNS list=DNS_Addresses
add address=192.XXX.XXX.2 comment=MGT list=management-access
add address=10.10.10.5 comment="Local Permitted IP Address#1" list=permitted-local-sources
/ip firewall filter
add action=accept chain=input comment="INPUT Chain - Packets towards Mikrotik itself" connection-state=\
    established,related
add action=accept chain=input comment=Winbox dst-port=XXX protocol=tcp
add action=accept chain=input comment="Permit DNS and NTP requests from clients" dst-port=53 in-interface=ether5 \
    protocol=udp
add action=accept chain=input comment="Permit DNS requests from clients over TCP" dst-port=53 in-interface=ether5 \
    protocol=tcp
add action=drop chain=input comment="Prevent any other new connection to the Mikrotik itself"
add action=accept chain=forward comment="FORWARD Chain - Packets towards Mikrotik itself" connection-state=\
    established,related disabled=yes
add action=accept chain=input comment="Permit DNS and NTP requests from clients" disabled=yes dst-port=53 in-interface=\
    VPN protocol=udp
add action=accept chain=forward comment="Approved HTTP" disabled=yes dst-address-list=permitted-http-destinations-address \
    dst-port=80 in-interface=ether5 out-interface=VPN protocol=tcp src-address-list=permitted-local-sources
add action=accept chain=forward comment="Approved HTTPS" disabled=yes dst-address-list=\
    permitted-https-destinations-address dst-port=443 in-interface=ether5 out-interface=VPN protocol=tcp \
    src-address-list=permitted-local-sources
add action=accept chain=forward comment="Approved DNS Addresses" disabled=yes dst-address-list=DNS_Addresses \
    in-interface=ether5 out-interface=VPN protocol=icmp src-address-list=permitted-local-sources
add action=drop chain=forward comment=\
    "Drop anything attempting to pass through the Mikrotik that doesn't match the rules above" disabled=yes
add action=drop chain=forward comment="VPN only ether4" in-interface=ether4 out-interface=ether1
add action=drop chain=forward comment="VPN only ether5" in-interface=ether5 out-interface=ether1
add action=drop chain=forward comment="VPN only ether6" in-interface=ether6 out-interface=ether1
add action=drop chain=forward comment="No Internet MGT" in-interface=ether6 out-interface=VPN
/ip firewall nat
add action=masquerade chain=srcnat out-interface=VPN
add action=masquerade chain=srcnat disabled=yes out-interface=ether1
/ip route
add comment=WAN distance=2 gateway=XX.XX.1.1
add comment=VPN distance=1 dst-address=23.XXX.XXX.83/32 gateway=XX.XX.1.1

#Device#2
[admin@MikroTik] > export
/interface ethernet
set [ find default-name=ether4 ] arp=reply-only
set [ find default-name=ether5 ] arp=reply-only
/ip address
add address=XX.XX.53.1/24 interface=ether5 network=XX.XX.53.0
add address=XX.XX.19.145/24 interface=ether1 network=XX.XX.19.0
add address=XX.XX.24.173/24 interface=ether2 network=XX.XX.24.0
/ip arp
add address=XX.XX.53.23 interface=ether5 mac-address=XX.XX.XX.XX.XX.XX
/ip dns
set allow-remote-requests=yes servers=XX.XX.19.1,XX.XX.24.1
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
add action=masquerade chain=srcnat out-interface=ether2
/ip route
add comment=Internet_WAN_2 distance=1 gateway=XX.XX.24.1
add comment=Internet_WAN_1 distance=1 gateway=XX.XX.19.1
add distance=1 gateway=100.64.0.1
add distance=2 gateway=100.64.0.2
add comment=Google#2 distance=1 dst-address=8.8.4.4/32 gateway=XX.XX.19.1 scope=10
add comment=Quad9 distance=1 dst-address=9.9.9.9/32 gateway=XX.XX.24.1 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.1/32 gateway=8.8.4.4 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.1/32 gateway=208.67.220.220 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.2/32 gateway=9.9.9.9 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.2/32 gateway=209.244.0.3 scope=10
add comment=OpenDNS#2 distance=1 dst-address=208.67.220.220/32 gateway=XX.XX.19.1 scope=10
add comment=Level3 distance=1 dst-address=209.244.0.3/32 gateway=192.168.24.1 scope=10

#LOG
12:12:04 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52437->1XX.XX.9.238:443, len 52 
12:12:04 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52438->1XX.XX.9.238:443, len 52 
12:12:06 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52433->1XX.XX.9.238:443, len 48 
12:12:06 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52434->1XX.XX.9.238:443, len 48 
12:12:06 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52439->1XX.XX.9.238:443, len 52 
12:12:08 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52435->7XX.XX.35.177:443, len 48 
12:12:09 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52439->1XX.XX.9.238:443, len 52 
12:12:09 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52436->1XX.XX.194.252:443, len 48 
12:12:10 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52437->1XX.XX.9.238:443, len 48 
12:12:10 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52438->1XX.XX.9.238:443, len 48 
12:12:11 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52440->7XX.XX.65.162:80, len 52 
12:12:11 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52441->1XX.XX.9.238:443, len 52 
12:12:14 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52440->7XX.XX.65.162:80, len 52 
12:12:14 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52441->1XX.XX.9.238:443, len 52 
12:12:15 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52439->1XX.XX.9.238:443, len 48 
12:12:15 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.24.173:52442->7XX.XX.213.200:80, len 52

sindy · Sat May 12, 2018 9:18 pm

OK, so:

Step 1 - remove the two routes below:

add comment=Internet_WAN_2 distance=1 gateway=XX.XX.24.1
add comment=Internet_WAN_1 distance=1 gateway=XX.XX.19.1

These routes would interfere with those tracking the accessibility of internet using 100.64.0.x as gateway)

Step 2 - the source address in the logs from Device 1 is XX.XX.24.173 which is the address assigned to ether2 on Device 2, and in-interface is ether2. This reveals that Device 2's ether2 is connected to Device 1's ether2

However, the address list permitted-local-sources on Device 1 contains a single address 10.10.10.5 which is neither the address of Device 2's ether1 nor the address of Device 2's ether2. So is that an obfuscation error or configuration error?

jamthejame · Sat May 12, 2018 9:56 pm

Unfortunately, I had ISP issues before, so the log i ran on Device#3 which is configured exactly to Device#1. When I was rebuilding, I missed it.
10.10.10.5 was another interface I created to test for another device, which is removed. Thus obfuscation error.

#Device#1
[admin@MikroTik] > export
/interface ethernet
set [ find default-name=ether4 ] arp=reply-only
set [ find default-name=ether5 ] arp=reply-only
/interface l2tp-client
add add-default-route=yes allow-fast-path=yes connect-to=23.XXX.XXX.83 dial-on-demand=yes disabled=no ipsec-secret=\
    ***** keepalive-timeout=10 max-mru=1410 max-mtu=1410 name=VPN password=**** use-ipsec=yes user=\
    ****
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip address
add address=XX.XX.1.250/24 interface=ether1 network=XX.XX.1.0
add address=XX.XX.19.1/24 interface=ether5 network=XX.XX.19.0
add address=10.10.10.1/24 interface=ether4 network=10.10.10.0
/ip arp
add address=XX.XX.19.145 interface=ether5 mac-address=XX.XX.XX.XX.XX.XX
/ip dns
set allow-remote-requests=yes servers=10.10.10.10,8.8.8.8
/ip firewall address-list
add address=24.XXX.XXX.100 comment=Demo list=permitted-http-destinations-address
add address=75.XXX.XXX.200 comment=Demo1 list=permitted-https-destinations-address
add address=9.9.9.9 comment=DNS list=DNS_Addresses
add address=192.XXX.XXX.2 comment=MGT list=management-access
add address=XX.XX.19.145 comment="Local Permitted IP Address#1" list=permitted-local-sources
/ip firewall filter
add action=accept chain=input comment="INPUT Chain - Packets towards Mikrotik itself" connection-state=\
    established,related
add action=accept chain=input comment=Winbox dst-port=XXX protocol=tcp
add action=accept chain=input comment="Permit DNS and NTP requests from clients" dst-port=53 in-interface=ether5 \
    protocol=udp
add action=accept chain=input comment="Permit DNS requests from clients over TCP" dst-port=53 in-interface=ether5 \
    protocol=tcp
add action=drop chain=input comment="Prevent any other new connection to the Mikrotik itself"
add action=accept chain=forward comment="FORWARD Chain - Packets towards Mikrotik itself" connection-state=\
    established,related disabled=yes
add action=accept chain=input comment="Permit DNS and NTP requests from clients" disabled=yes dst-port=53 in-interface=\
    VPN protocol=udp
add action=accept chain=forward comment="Approved HTTP" disabled=yes dst-address-list=permitted-http-destinations-address \
    dst-port=80 in-interface=ether5 out-interface=VPN protocol=tcp src-address-list=permitted-local-sources
add action=accept chain=forward comment="Approved HTTPS" disabled=yes dst-address-list=\
    permitted-https-destinations-address dst-port=443 in-interface=ether5 out-interface=VPN protocol=tcp \
    src-address-list=permitted-local-sources
add action=accept chain=forward comment="Approved DNS Addresses" disabled=yes dst-address-list=DNS_Addresses \
    in-interface=ether5 out-interface=VPN protocol=icmp src-address-list=permitted-local-sources
add action=drop chain=forward comment=\
    "Drop anything attempting to pass through the Mikrotik that doesn't match the rules above" disabled=yes
add action=drop chain=forward comment="VPN only ether4" in-interface=ether4 out-interface=ether1
add action=drop chain=forward comment="VPN only ether5" in-interface=ether5 out-interface=ether1
add action=drop chain=forward comment="VPN only ether6" in-interface=ether6 out-interface=ether1
add action=drop chain=forward comment="No Internet MGT" in-interface=ether6 out-interface=VPN
/ip firewall nat
add action=masquerade chain=srcnat out-interface=VPN
add action=masquerade chain=srcnat disabled=yes out-interface=ether1
/ip route
add comment=WAN distance=2 gateway=XX.XX.1.1
add comment=VPN distance=1 dst-address=23.XXX.XXX.83/32 gateway=XX.XX.1.1

#Device#2
[admin@MikroTik] > export
/interface ethernet
set [ find default-name=ether4 ] arp=reply-only
set [ find default-name=ether5 ] arp=reply-only
/ip address
add address=XX.XX.53.1/24 interface=ether5 network=XX.XX.53.0
add address=XX.XX.19.145/24 interface=ether1 network=XX.XX.19.0
add address=XX.XX.24.173/24 interface=ether2 network=XX.XX.24.0
/ip arp
add address=XX.XX.53.23 interface=ether5 mac-address=XX.XX.XX.XX.XX.XX
/ip dns
set allow-remote-requests=yes servers=XX.XX.19.1,XX.XX.24.1
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
add action=masquerade chain=srcnat out-interface=ether2
/ip route
add distance=1 gateway=100.64.0.1
add distance=2 gateway=100.64.0.2
add comment=Google#2 distance=1 dst-address=8.8.4.4/32 gateway=XX.XX.19.1 scope=10
add comment=Quad9 distance=1 dst-address=9.9.9.9/32 gateway=XX.XX.24.1 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.1/32 gateway=8.8.4.4 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.1/32 gateway=208.67.220.220 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.2/32 gateway=9.9.9.9 scope=10
add check-gateway=ping distance=1 dst-address=100.64.0.2/32 gateway=209.244.0.3 scope=10
add comment=OpenDNS#2 distance=1 dst-address=208.67.220.220/32 gateway=XX.XX.19.1 scope=10
add comment=Level3 distance=1 dst-address=209.244.0.3/32 gateway=192.168.24.1 scope=10

#LOG
12:12:04 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52437->1XX.XX.9.238:443, len 52 
12:12:04 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52438->1XX.XX.9.238:443, len 52 
12:12:06 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52433->1XX.XX.9.238:443, len 48 
12:12:06 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52434->1XX.XX.9.238:443, len 48 
12:12:06 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52439->1XX.XX.9.238:443, len 52 
12:12:08 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52435->7XX.XX.35.177:443, len 48 
12:12:09 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52439->1XX.XX.9.238:443, len 52 
12:12:09 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52436->1XX.XX.194.252:443, len 48 
12:12:10 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52437->1XX.XX.9.238:443, len 48 
12:12:10 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52438->1XX.XX.9.238:443, len 48 
12:12:11 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52440->7XX.XX.65.162:80, len 52 
12:12:11 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52441->1XX.XX.9.238:443, len 52 
12:12:14 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52440->7XX.XX.65.162:80, len 52 
12:12:14 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52441->1XX.XX.9.238:443, len 52 
12:12:15 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52439->1XX.XX.9.238:443, len 48 
12:12:15 firewall,info forward: in:ether2 out:vpn, src-mac XX.XX.XX.XX.XX.XX, proto TCP (SYN), XX.XX.19.145:52442->7XX.XX.213.200:80, len 52

sindy · Sat May 12, 2018 10:15 pm

OK. So now tell me whether with exactly the configuration you've posted the clients can reach the servers or not.

You have disabled all rules in chain=forward of /ip firewall filter, so everything should work. If it doesn't, the reason is not the firewall.

The rule which causes the initial packet of each forwarded connection to be logged is missing completely, so have you taken the log while the posted configuration was active?

It is impossible to debug on logs taken in other configuration than the one you post.

And the log rule was there to find out whether there is no issue with the source address, not to log every packet of every connection because your log would explode, that's why I have restricted the number of logged packets using connection-state=new.

jamthejame · Sun May 13, 2018 6:34 pm

Sindy, sorry about yesterday, but i got to the point with everything, that I was doing more harm than good, so had to walk away and reboot. Furthermore, because I'm not at the central location, don't have all the programs required to get access to servers and their IP, with another layer being that because its a weekend, most of them also offline. I'm only able to test on 1 IP right now, so have to work with that Anyhow, I tried to do what you've asked. But let me first comment on your previous message:

You have disabled all rules in chain=forward of /ip firewall filter, so everything should work. If it doesn't, the reason is not the firewall.

Yes, it is the firewall and it is the 'drop' chain rule, because when it's disabled, everything works fine.

The rule which causes the initial packet of each forwarded connection to be logged is missing completely, so have you taken the log while the posted configuration was active?

The log was taken with the rule being active on Device#3. However, I made rearrangements now, that we'll focus only on device to avoid misunderstandings and issues, encountered yesterday.

It is impossible to debug on logs taken in other configuration than the one you post.

I tried debugging through by adding "add action=disk topics=debug" however, it gave me a bunch VPN encryption logs, and wasn't able to find any related to the 'new connection' state. Is there another way or better way? I dont want to post a bunch of things, to waste your time and effort. How can I be more precise on my side to produce data that can be reviewed?

** Is there a way to 'log' only 1 specific IP and debug it to see what is taking place there?

sindy · Sun May 13, 2018 7:15 pm

Yes, sometimes a time-out is necessary.

When talking about "debug", I did not have in mind that you should start logging debug messages; I only had in mind that I wanted to see what packets are actually arriving and which way, because your action=accept rules check the source address (against an address list), the in-interface, and the out-interface all together, so one mismatch is enough to make the rule ignore the packet. But I never got from you a log and configuration exports from the two devices which would be mutually consistent, so I was never sure what actually happened.

The action=drop rule does what it should do; the fact that you have to disable it to make the Device 2's clients reach the servers means that the other rules don't or, more likely, because they expect something else to arrive than what really arrives.

We also have to concentrate at one thing at a time. So it would be best to have one "category 1" device and one "category 2" device on your table and first resolve the issue with the firewall on category 1 device. Only then it makes sense to deal with the issue of failover on category 2 device.

So my idea would be that you put the logging rule at its place, connect the Device 2 using its proper upink to Device 1, reset rule counters at Device 1, enable all rules except the action=drop chain=forward one, and try to connect the clients to the required servers.

Then you would post the output of
/log print where topics~"firewall"
/ip firewall filter print where chain=forward
/ip firewall filter print stats where chain=forward

and the sanitized configuration export of Device 1.

jamthejame · Sun May 13, 2018 7:47 pm

Sindy, I just tried last time to connect with rebooted Mik and all rules enabled and it connected. Did a few more times thereafter and it seems to connect now. Now sure why. Was thinking maybe it was an issue with a server on the other side. I know it has to be online for sure in 5 hours from now. I will try then, because only then i'll be able to know for use and eliminate the possibility of server either being on maintenance or down etc. I dont want to burn your time and effort if that has nothing to do with Mikrotik but is the issue of another side. I will revert back with my finding in around 5 hours and then if this is Mikrotik, then I'll pursue with the above.