Secondary WAN and failover setup hap ax2 (7.16) for a beginner

bbmikrotik · Wed Oct 02, 2024 5:14 pm

Hello all! I am new to network setup, just basic knowledge, I just bought a Mikrotik hap ax2 , and I want to setup dual WAN failover with ISP modem (already in bridge mode) and 4g/LTE mobile internet modem.

The ISP modem connected to the ethernet1 (internet in) port and I am struggling to setup a secondary WAN to connect the LTE modem and transfer its internet through other ethernet ports. How should I start with it? I have factory defconfig in the Mikrotik, and from the basics, I would like to setup a failover setting in case of ISP internet is out! No special request just want automatic changeover.

I stuck at the setup the ether2-5 port as WAN ports to send internet through them. Do I even have to setup for failover system? I would appreciate any help from the first step from factory defconfig until the end.

Where should I start, where can I find a manual for it?

Thank you in advance!

jaclaz · Wed Oct 02, 2024 7:29 pm

The default is:
ether1:WAN
ether2-5:joined in a bridge LAN

What you have to do is to take ether2 (by convention the lower numbered ports are those that connect to the internet) out of the bridge, so that you will have:
ether1:WAN (1)
ether2:WAN (2)
ether3-5 joined in a bridge:LAN

Be careful to not lock yourself out by categorizing the whole bridge as WAN.

You should have:
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN

and
/interface list member
add interface=ether1 list=WAN
add interface=bridge list=LAN

This latter should become:
/interface list member
add interface=ether1 list=WAN
add interface=ether2 list=WAN
add interface=bridge list=LAN

Then long before attempting to setup failover you will need to separately configure the two interfaces, setting the dhcp client on them (if needed, defconf already has it on ether1) an /ip firewall nat srcnat or masquerade rule for each, and - if not dynamically created - two routes.

Once you will have the one and the other connections working (experiment by enabling and disabling the other route), we can start talking of failover setup.

Personally I would first make just the "normal" connection (via ether1) and test, it, then add the second connection (via ether2).

The easiest (IMHO) failover approach in a user case like yours is this one:
viewtopic.php?t=198999

It would be easier to assist you if you can get minimally familiar with the CLI[1], at least to export and post your current configuration, see if you can follow the instructions here:
viewtopic.php?t=203686#p1051720

[1] while using the GUI interface is much more comfortable to use for changing settings, a textual export or print is the best way to check what the settings are and to communicate them over the forum (much better than a bunch of tiny partial screenshots)

bbmikrotik · Wed Oct 02, 2024 10:14 pm

Hey!

Thank you for being so helpful! I have really basic knowledge of this network setup, near to 0 but trying to learn by myself, it is really hard, so respect your knowledge about this! I would appreciate if you can look in my setup and advise what I have to put in it to set up because I don't know how to proceed with the DHCP Client, it shows invalid, and I have 0 knowledge about firewall rules. I don't want to bother with this just really need some hand-holding for the beginning! I really appreciate your help!

# 2024-10-02 21:04:53 by RouterOS 7.16
# software id = xxxxxxxx
#
# model = C52iG-5HaxD2HaxD
# serial number = xxxxxxxx
/interface bridge
add admin-mac=xx:xx:xx:xx:xx:xx auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] comment=WAN_ISP poe-out=off
set [ find default-name=ether2 ] comment=LTE_WAN
/interface wifi
set [ find default-name=wifi1 ] channel.band=5ghz-ax .skip-dfs-channels=\
    10min-cac .width=20/40/80mhz comment=HomeSSid_5Ghz configuration.mode=ap \
    .ssid=HomeSSid_5Ghz disabled=no security.authentication-types=wpa2-psk \
    .ft=yes .ft-over-ds=yes
set [ find default-name=wifi2 ] channel.band=2ghz-ax .skip-dfs-channels=\
    10min-cac .width=20mhz comment=HomeSSid configuration.mode=ap .ssid=\
    HomeSSid disabled=no security.authentication-types=wpa2-psk .ft=yes \
    .ft-over-ds=yes
add configuration.mode=ap .ssid=SSidGuest mac-address=xxxxxxxxxxxxxx \
    master-interface=wifi1 name=wifi3
add comment="Guest WiFi" configuration.mode=ap .ssid=SSidGuest disabled=no \
    mac-address=xxxxxxxxxxxx master-interface=wifi2 name=wifi4
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
add name=dhcp-pool ranges=192.168.0.50-192.168.0.200
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
# DHCP server can not run on slave interface!
add address-pool=dhcp-pool interface=ether2 name=dhcp1
/disk settings
set auto-media-interface=bridge auto-media-sharing=yes auto-smb-sharing=yes
/interface bridge filter
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward in-interface=wifi3
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward out-interface=wifi3
add action=drop chain=forward in-interface=wifi4
add action=drop chain=forward out-interface=wifi4
/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wifi1
add bridge=bridge comment=defconf interface=wifi2
add bridge=bridge interface=wifi3
add bridge=bridge interface=wifi4
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=all
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
/ip dhcp-client
add comment=defconf interface=ether1
# Interface not active
add comment=LTE interface=ether2
/ip dhcp-server network
add address=192.168.0.0/24 gateway=192.168.0.1
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
    192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan type=A
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" \
    dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/system clock
set time-zone-name=xxxxxx
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

jaclaz · Thu Oct 03, 2024 12:33 am

Don't mistake me for a Mikrotik expert, I am also only a beginner, just knowing enough for basic configurations like yours.

/interface bridge port
add bridge=bridge comment=defconf interface=ether2

Your ether2 is still part of the bridge, hence the DHCP server error, that sees it as slave interface (its master being the bridge).
But it makes no sense to run a DHCP server on ether2, that port/interface is connected ONLY to the LTE modem/router and usually either It has a static IP address assigned or it is set as DHCP client.
Personally I would use a static address, as there is presumably only a single device (the Mikrotik router) connected to the LTE modem/router and both aren't going to be moved around.
But a DHCP client Is fine as well, at the moment is inactive for the same reason, ether2 Is not (yet) a self-standing interface.

bbmikrotik · Thu Oct 03, 2024 8:54 am

Thanks, so how should I proceed with it? Becouse if I set DHCP client shows invalid, should I set up other bridge for ether 2 or something else?

jaclaz · Thu Oct 03, 2024 11:27 am

No, you just need to remove the ether2 from the bridge, leaving in the bridge only ether3,4,5.
Ether2, once outside the bridge, will be able to have a DHCP client enabled on it, just like it is now ether1.

in CLI that would be:

/interface bridge port
remove [ find interface=ether2 ]

Let see if I can help you visualize the setup.

Now your set ports are:

1 - ether1 -> WAN -> DSL modem router (DHCP client enabled)
2 - bridge -> LAN -> other devices (DHCP server running)
     | ether2 <any related setting is ignored because the settings for the bridge prevail>
     | ether3 <any related setting is ignored because the settings for the bridge prevail>
     | ether4 <any related setting is ignored because the settings for the bridge prevail>
     | ether5 <any related setting is ignored because the settings for the bridge prevail>

The first port is "independent", the other four are "assembled together" in what functionally is a switch.

What you want to have is instead:

1 - ether1 -> WAN -> DSL modem router (DHCP client enabled)
2 - ether2 -> WAN -> LTE modem router (DHCP client enabled)
3 - bridge -> LAN -> other devices (DHCP server running)
     | ether3 <any related setting is ignored because the settings for the bridge prevail>
     | ether4 <any related setting is ignored because the settings for the bridge prevail>
     | ether5 <any related setting is ignored because the settings for the bridge prevail>

The first two ports are "independent", the other three remain "assembled together" in what functionally is a switch.

bbmikrotik · Thu Oct 03, 2024 12:57 pm

So now I set as your said, if I connect the internet to that port the invalid disappears, but no internet coming through when I connect ISP to ether2, probably because of the firewall rules, or should I set other things as well?

setup now:

# 2024-10-03 11:52:49 by RouterOS 7.16
# software id = xxxxx
#
# model = C52iG-5HaxD2HaxD
# serial number = xxxxxx
/interface bridge
add admin-mac=xx:xx:xx:xx:xx auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] comment=WAN_ISP poe-out=off
set [ find default-name=ether2 ] comment=WAN_LTE
/interface wifi
set [ find default-name=wifi1 ] channel.band=5ghz-ax .skip-dfs-channels=\
    10min-cac .width=20/40/80mhz comment=HomeSSid_5Ghz configuration.mode=ap \
    .ssid=HomeSSid_5Ghz disabled=no security.authentication-types=wpa2-psk \
    .ft=yes .ft-over-ds=yes
set [ find default-name=wifi2 ] channel.band=2ghz-ax .skip-dfs-channels=\
    10min-cac .width=20mhz comment=HomeSSid configuration.mode=ap .ssid=\
    HomeSSid disabled=no security.authentication-types=wpa2-psk .ft=yes \
    .ft-over-ds=yes
add configuration.mode=ap .ssid=SSidGuest mac-address=xxxxxxx \
    master-interface=wifi1 name=wifi3
add comment="Guest WiFi" configuration.mode=ap .ssid=SSidGuest disabled=no \
    mac-address=xxxxxx master-interface=wifi2 name=wifi4
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
add name=dhcp-pool ranges=192.168.0.50-192.168.0.200
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
/disk settings
set auto-media-interface=bridge auto-media-sharing=yes auto-smb-sharing=yes
/interface bridge filter
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward in-interface=wifi3
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward out-interface=wifi3
add action=drop chain=forward in-interface=wifi4
add action=drop chain=forward out-interface=wifi4
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wifi1
add bridge=bridge comment=defconf interface=wifi2
add bridge=bridge interface=wifi3
add bridge=bridge interface=wifi4
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=all
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
/ip dhcp-client
add comment=defconf interface=ether1
# Interface not active
add comment=LTE interface=ether2
/ip dhcp-server network
add address=192.168.0.0/24 gateway=192.168.0.1
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
    192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan type=A
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" \
    dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/system clock
set time-zone-name=xxxxxx
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

anav · Thu Oct 03, 2024 5:46 pm

1. You are attempting to have two different subnets assigned from the single bridge. There are several approaches, two bridges but not advised or two vlans, very much advised.
You already have two pools.......... just complete the config...... and first add the vlans to the bridge.

/interface vlan
add interface=bridge name=vlan10-home vlan-id=10
add interface=bridge name=vlan20-guest vlan-id=20

/ip address
add address=192.168.88.1/24 interface=vlan10-home network=\
192.168.88.0
add address=192.168.0.1/24 interface=vlan-20-guest network=\
192.168.0.0

/ip dhcp-server
add address-pool=default-dhcp interface=vlan10-home name=defconf
add address-pool=dhcp-pool interface=vlan20-guest name=guestserver

/ip dhcp-server network
add address=192.168.0.0/24 dns-server=192.168.0.1 gateway=192.168.0.1
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
192.168.88.1

2. Add another interface to distinguish guests from home users in terms of general router security.
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
add name=Trusted
/interface list members
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
add comment=defconf interface=vlan10-home list=LAN
add comment=defconf interface=vlan20-guest list=LAN
add interface=vlan10-home list=Trusted

3. GET RID OF ALL BRIDGE FILTERS. These are advanced settings that are rarely used in basic configurations.

4. Make changes as required.
/interface bridge port
add bridge=bridge ingress-filtering=yes frame-types=admit-priority-and-untagged interface=ether3 pvid=10
add bridge=bridge ingress-filtering=yes frame-types=admit-priority-and-untagged interface=ether4 pvid=10
add bridge=bridge ingress-filtering=yes frame-types=admit-priority-and-untagged interface=ether5 pvid=10
add bridge=bridge ingress-filtering=yes frame-types=admit-priority-and-untagged interface=wifi1 pvid=10
add bridge=bridge ingress-filtering=yes frame-types=admit-priority-and-untagged interface=wifi2 pvid=10
add bridge=bridge ingress-filtering=yes frame-types=admit-priority-and-untagged interface=wifi3 pvid=20
add bridge=bridge ingress-filtering=yes frame-types=admit-priority-and-untagged interface=wifi4 pvid=20

5. Add /interface bridge vlan settings
add bridge=bridge tagged=bridge untagged=ether3,ether4,ether5,wifi1,wifi2 vlan-ids=10
add bridge=bridge tagged=bridge untagged=wifi3,wifi4 vlan-ids=20

6. modify neighbours discovery and Set internet detection to none.
/ip neighbor discovery-settings
set discover-interface-list=Trusted
/interface detect-internet
set detect-interface-list=NONE

7. If not using IPV6, set to disabled and remove lists and rules leaving only two rules
add chain=input action=drop
add chain=forward action=drop

8. Modify as per below:
/tool mac-server
set allowed-interface-list=NONE
/tool mac-server mac-winbox
set allowed-interface-list=Trusted

9. MISSING WAN1 information.but why is the Router think its inactive.
So you have some issue with your ISP tha needs to be resolved.

10. Firewall rules are basically default so okay for the default setup but needs to be modified for the changes made.

/ip firewall filter
add action=accept chain=input comment=\
"defconf: accept established,related,untracked" connection-state=\
established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
"defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=accept chain=input in-interface-list=Trusted comment="admin access" { consider further limiting by IP addresses --> firewall address list }
add action=accept chain=input in-interface-list=LAN dst-port=53 protocol=udp comment="users to services"
add action=accept chain=input in-interface-list=LAN dst-port=53 protocol=tcp comment="users to services"
add action=drop chain=input comment="drop all else" { Put this rule in place after all the others so not to lock yourself out }
++++++++++++++++++++++++++++++++++++++++++++
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
"defconf: accept established,related, untracked" connection-state=\
established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
connection-state=invalid
add action=accept chain=forward comment="internet traffic" in-interface-list=LAN out-interface-list=WAN
add action=accept chain=forward comment="port forwarding" connection-nat-state=dstnat { disable or remove if not required }
add action=drop chain=forward comment="Drop all else"

11. For the last step on the bridge and vlan filtering turn vlan filtering on.

/interface bridge
add comment=defconf name=bridge vlan-filtering=yes

jaclaz · Thu Oct 03, 2024 6:19 pm

@anav
there is no reason why OP would want or should have two VLANs.
Very likely the double entry in:
/ip dhcp-server network
is a mistake/accident.
OP is a beginner and is trying to get the hang of a simple home/soho setup, no need for now to make it more complex than needed.

@bbmikrotik
You have to decide which addresses you use on your LAN that are set by the DHCP server on Mikrotik.
The default is 192.168.88.0/24:

/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
192.168.88.1

If you want to use 192.168.0.0/24 instead you disable that one and add the new one.

And you need to add an address in that range in:
/ip address
as well, temporarily you add the new address and disable (as opposed to remove) the defconf one.

Once you are satisfied with the settings LAN side, you connect the ISP modem/router to ether1, the LTE modem/router to ether2 and run:
/ip route print
and post the output, so that we can see which routes are dynamically created.

Anav's point #6 is good advice, independently from VLANs, it is advised to change
/interface detect-internet
set detect-interface-list=all
to:
/interface detect-internet
set detect-interface-list=none

bbmikrotik · Thu Oct 03, 2024 6:52 pm

hey, so I connected my ISP to ether1 and LTE to ether2 as instructed and set up what I could.

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAd+ 0.0.0.0/0 5.187.172.1 1
DAd+ 0.0.0.0/0 192.168.1.1 1
DAc 5.187.172.0/22 ether1 0
DAc 192.168.1.0/24 ether2 0
DAc 192.168.88.0/24 bridge 0

# 2024-10-03 17:50:18 by RouterOS 7.16
# software id = xxxxxx
#
# model = C52iG-5HaxD2HaxD
# serial number = xxxxx
/interface bridge
add admin-mac=xxxxxxxxxxx EB auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] comment=WAN_ISP poe-out=off
set [ find default-name=ether2 ] comment=WAN_LTE
/interface wifi
set [ find default-name=wifi1 ] channel.band=5ghz-ax .skip-dfs-channels=\
    10min-cac .width=20/40/80mhz comment=HomeSSid_5Ghz configuration.mode=ap \
    .ssid=HomeSSid_5Ghz disabled=no security.authentication-types=wpa2-psk \
    .ft=yes .ft-over-ds=yes
set [ find default-name=wifi2 ] channel.band=2ghz-ax .skip-dfs-channels=\
    10min-cac .width=20mhz comment=HomeSSid configuration.mode=ap .ssid=\
    HomeSSid disabled=no security.authentication-types=wpa2-psk .ft=yes \
    .ft-over-ds=yes
add configuration.mode=ap .ssid=SSidGuest mac-address=xxxxxxxxxx \
    master-interface=wifi1 name=wifi3
add comment="Guest WiFi" configuration.mode=ap .ssid=SSidGuest disabled=no \
    mac-address=xxxxxxxx master-interface=wifi2 name=wifi4
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
add name=dhcp-pool ranges=192.168.0.50-192.168.0.200
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
/disk settings
set auto-media-interface=bridge auto-media-sharing=yes auto-smb-sharing=yes
/interface bridge filter
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward in-interface=wifi3
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward out-interface=wifi3
add action=drop chain=forward in-interface=wifi4
add action=drop chain=forward out-interface=wifi4
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wifi1
add bridge=bridge comment=defconf interface=wifi2
add bridge=bridge interface=wifi3
add bridge=bridge interface=wifi4
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
/ip dhcp-client
add comment=defconf interface=ether1
add comment=LTE interface=ether2
/ip dhcp-server network
add address=192.168.0.0/24 gateway=192.168.0.1
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
    192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan type=A
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" \
    dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/system clock
set time-zone-name=xxxxxxxxx
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

anav · Thu Oct 03, 2024 6:54 pm

Im sorry jaclaz, but you clearly missed the boat.
The OP has two subnets, one for the home lan and one for the guest wifi.
Two vlans makes much sense to me.

As for the OP, good luck, advice not followed, your in good hands with jaclaz, out.

bbmikrotik · Thu Oct 03, 2024 7:07 pm

Sorry but I beginner in this stuff, the guest wifi just a setup for somebody who comes and wanted to separate them from the main users, I only want to set up a failover config with a DSL main source and an LTE backup source (a lot of cameras, solar panel system and smart home systems in the house, it could be I will setup Starlink for backup or main in the future), I can disable guest wifi if it is just a pain in the ass. I am feeling lost, this is lots of info! Of course, I appreciate your help! Just don't know exactly what it means if I want dual WAN failover config.

jaclaz · Thu Oct 03, 2024 8:09 pm

Two vlans makes much sense to me.

Not only to you, but there are logical priorities, that you are not taking into account, and as well you are not considering how the OP is a beginner.
Priorities (in my perverted mind):
1) have a decent setup for 1 internet connection
2) add a second connection
3) add a failover of some kind between the two
4) check that basic (single SSID) wireless works (not so given with the complexities of Mikrotik Ax settings)
once the above are done
5) betterings, VLANs, Guest wireless, etc.

As for the OP, good luck, advice not followed, your in good hands with jaclaz, out.

Oww, come off it:
https://www.marriedtothesea.com/030306/gracious.jpg

@bbmikrotik
Ok, so:
Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAd+ 0.0.0.0/0 5.187.172.1 1 <- this one comes from the DHCP server of the ISP 1 modem/router (it is in bridge mode, right?)
DAd+ 0.0.0.0/0 192.168.1.1 1 <- this one comes from the DHCP server of the ISP2 LTE modem/router
DAc 5.187.172.0/22 ether1 0 <- this one comes from the DHCP server of the ISP 1 modem/router
DAc 192.168.1.0/24 ether2 0 <- this one comes from the DHCP server of the ISP2 LTE modem/router
DAc 192.168.88.0/24 bridge 0 <- this one is added automatically by the IP address assigned to the bridge

The two routes to 0.0.0.0/0 (the internet) have the same distance of 1, so they have the + sign as they are in ECMP.

If you disconnect the cable from ether1 you should get the internet from the LTE modem.

If you have not a connection, then this needs to be resolved/fixed.

Right now, if you remove the connection from ether2 and move the cable from ether1 to ether2 you should have the internet connection working as well.

Then, in DHCP client, change the distance of the routes:
/ip dhcp-client
add comment=defconf interface=ether1 default-route-distance=2
add comment=LTE interface=ether2 default-route-distance=1

And check again the /Ip route print
Now the route via LTE should be active, whilst the main one should be inactive (as it has a greater distance)
If you disable the route via LTE, the main one should become active.

And you are ready to follow the linked to failover method:
viewtopic.php?t=198999

bbmikrotik · Thu Oct 03, 2024 8:38 pm

Sadly if I disconnect the DSL or LTE from ether1 and put ether2, no internet connection, either LTE or DSL in ether1 gives back the internet.

This is how it looks like when the internet is connected to ether2:

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAc 192.168.88.0/24 bridge 0

This is what it looks like when the internet is connected to ether1 and ether2 LTE disconnected:

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAd 0.0.0.0/0 5.187.172.1 1
DAc 5.187.172.0/22 ether1 0
DAc 192.168.88.0/24 bridge 0

If I check Bridge-Ports menu only ether3-5 and wifi1-4 are there

jaclaz · Thu Oct 03, 2024 11:25 pm

Well, you had just posted an /ip route print where both ether1 and ether2 got their ip route just fine (so the DHCP client worked fine on both interfaces), now your /ip route print seems to show that it only works on ether1 and not (anymore) on ether2, so something has changed.
The main ISP modem creates a route to 5.187.xxx.yyyy, the LTE creates a route to 192.168.1.1, you posted those results.
You should be able to go back to the situation you were in when you posted that.

bbmikrotik · Fri Oct 04, 2024 4:48 am

Yes the only change I turned off for the second result the LTE modem, if I turn on I will get the same result, the problem is when I disconnect ether1 internet is gone no internet going trough ether2

MTNick · Fri Oct 04, 2024 4:58 am

*deleted. time to take a nap

jaclaz · Fri Oct 04, 2024 11:09 am

Yes the only change I turned off for the second result the LTE modem, if I turn on I will get the same result, the problem is when I disconnect ether1 internet is gone no internet going trough ether2

But the same LTE modem works fine when connected to ether1?
In theory you should be able to connect either modem/router to either ether1 or ether2 and have the connection, the two interfaces are (should be) exactly the same for all that matters.

Since the LTE modem gives the routes, the DHCP server on it and the DHCP client on the Mikrotik are working.

What do you mean you have no internet?
Like you cannot navigate from your PC or you cannot ping (say) 8.8.8.8 (both from your PC and from the Mikrotik terminal)?

The relevant settings of the two interfaces are seemingly identical, and both are WAN, so the masquerade rule applies to both as well as the (default) firewall rules:

/interface ethernet
set [ find default-name=ether1 ] comment=WAN_ISP poe-out=off
set [ find default-name=ether2 ] comment=WAN_LTE
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
/ip dhcp-client
add comment=defconf interface=ether1
add comment=LTE interface=ether2
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN

Still, there is something that I cannot understand in your report(s).
With both devices connected you had (in Green the ether1, in red the ether2 related routes):

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAd+ 0.0.0.0/0 5.187.172.1 1
DAd+ 0.0.0.0/0 192.168.1.1 1
DAc 5.187.172.0/22 ether1 0
DAc 192.168.1.0/24 ether2 0
DAc 192.168.88.0/24 bridge 0

When you have only the ISP modem connected to ether1 you have (correctly):

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAd 0.0.0.0/0 5.187.172.1 1
DAc 5.187.172.0/22 ether1 0
DAc 192.168.88.0/24 bridge 0

When you have only the LTE modem on ether2 you should have:

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAd 0.0.0.0/0 192.168.1.1 1
DAc 192.168.1.0/24 ether2 0
DAc 192.168.88.0/24 bridge 0

If you can reproduce the first one (with both routers connected and the 4 routes, the 2 green and the 2 red ones) what happens when you disconnect ether1?
The two green routes should disappear, but the two red ones should remain.
And if you remove the LTE from ether2 the two routes should disappear, and return, exactly the same, when you connect the LTE to ether1.

bbmikrotik · Fri Oct 04, 2024 3:55 pm

So, the setups in pont by point

1. Ether1 - DSL, Ether2- LTE connected: 

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAd+ 0.0.0.0/0 5.187.172.1 1
DAd+ 0.0.0.0/0 192.168.1.1 1
DAc 5.187.172.0/22 ether1 0
DAc 192.168.1.0/24 ether2 0
DAc 192.168.88.0/24 bridge 0

Internet works on my macbook and my mobile as well, everything is normal as should be, speedtest speed: 600mbit/s

2. Ether1 - LTE, Ether2- DSL connected:

  Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP
* Columns: DST-ADDRESS, GATEWAY, DISTANCE
* DST-ADDRESS GATEWAY DISTANCE
* DAd 0.0.0.0/0 192.168.1.1 1
* DAc 192.168.1.0/24 ether1 0
* DAc 192.168.88.0/24 bridge 0

Internet work as should be of course at limited speed as LTE coverage is moderate 2-3 mbit/s as expected

3. Ether1 - none(disconnected) Ether2- DSL connected:  

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT
* Columns: DST-ADDRESS, GATEWAY, DISTANCE
* DST-ADDRESS GATEWAY DISTANCE
* DAc 192.168.88.0/24 bridge 0

No connection, cannot open web pages etc.

4. Ether1 - none(disconnected) Ether2- LTE connected:  

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP
* Columns: DST-ADDRESS, GATEWAY, DISTANCE
* DST-ADDRESS GATEWAY DISTANCE
* DAd 0.0.0.0/0 192.168.1.1 1
* DAc 192.168.1.0/24 ether2 0
* DAc 192.168.88.0/24 bridge 0

So my biggest surprise, there is internet works perfectly… but when I connect only DSL to ether2 no connection via webpages etc, so no internet, I just guessing why but I am not sure, it could be because of IP pool setup or the setup what previous commenter said (point number 6) -( Anav's point #6 is good advice, independently from VLANs, it is advised to change
/interface detect-internet
set detect-interface-list=all
to:
/interface detect-internet
set detect-interface-list=none)

So my question is correct operation, and if I do not modify my firewall with this setting, is it okay? Am I safe from attacks as before? 

jaclaz · Fri Oct 04, 2024 5:17 pm

/interface detect internet is not part of the firewall, it is something that very few people actually use, let alone *need*, and is known to mess things in some subtle ways in some configurations.
Disabling it is usually advised, see:
viewtopic.php?t=187814

In the best cases it does nothing useful, in the worst it may mess with DHCP (which seems like your case, though it has to be tested if it is actually connected to your issue).

Summing it up, your report says:
DSL works on ether1 but not on ether2.
LTE works on BOTH ether1 and ether2.

This makes me think that the issue is related to *something* in the DSL modem/router that is *somehow* providing the IP address via DHCP to ether1 but not to ether2, (which is what comes out from your /ip route print outputs posted)
Maybe you need to reboot the DSL router after having moved the DSL connection from ether1 to ether2?

The fact that the LTE works indifferently on ether1 and ether2 is a confirmation that (on Mikrotik's side) the two ports are set exactly in the same manner (good).

In any case, you can keep the DSL on ether1 (working) and the LTE on ether2 (working).

bbmikrotik · Fri Oct 04, 2024 5:22 pm

So I can leave live this to set failover? But I am stuck in the beginning with the link you posted tried to setup but it is not that clear to me, how should I proceed?

Is it not a problem if I put the 5.187.172.1 to gateway which is coming from my modem which is in bridge mode? Won't it change? Is it not dynamically changing from time to time what IP address is given to me by the internet provider? Or I can set it for failover setup?

So have this route list:

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, d - DHCP; + - ECMP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
DST-ADDRESS GATEWAY DISTANCE
DAd+ 0.0.0.0/0 192.168.1.1 1
DAd+ 0.0.0.0/0 5.187.172.1 1
DAc 5.187.172.0/22 ether1 0
DAc 192.168.1.0/24 ether2 0
DAc 192.168.88.0/24 bridge 0

Trying to find out from my config what is what from the failover setup forum page:

I am trying to correct the guy addresses with mine, please correct me if I am wrong.

1. Prerequirements:
- Network with DHCP done by MicroTik (in this case: 192.188.1.0/24) - 192.168.88.0/24
- Standard Gateway in DHCP will be the MikroTik (here: 192.168.1.2) - I am not sure about this
- Internet available at (for Example) 192.168.1.1 (in this case DSL) - 5.187.172.0
- Internet available at (for Example) 192.168.1.250 (LTE-Modem) - 192.168.1.0

2. Routing:
- Standard Route 0.0.0.0/0 set to 192.168.1.250 with Distance 1 comment=LTE-Failover -> (keep it DEACTIVATED) - 192.168.1.0
- Standard Route 0.0.0.0/0 set to 192.168.1.1 with Distance 2 - 5.187.172.0

In this setup if I want to change the DAd+ line in IP routes its grey I cannot change the distance, so should I setup my own for this reason or how?

3. Go to ROUTING -> TABLES
- Create a Routing Table named (for Example) "DSL" - check FIB

4. Go To IP -> ROUTES -> Click +
- Dst,Address: 0.0.0.0/0
- Gateway: 192.168.1.1 (your Primary Gateway) - 5.187.172.0
- Routing Table: Select above created ROUTING TABLE (here: "DSL")

5. Go to IP -> FIREWALL -> Tab MANGLE
Create a MANGLE-Rule:
- Tab -> GENERAL
-- Chain: output
-- Dst.Address: 8.8.8.8
-- Protocol: 1 (icmp)
- Tab -> ACTION
-- Action: mark routing
-- New Routing Mark: Select above created ROUTING TABLE (here: "DSL")

6. Go to TOOLS -> NETWATCH
-Tab -> HOST
-- Create a Netwatch Host:
--- Host: 8.8.8.8
--- Type: icmp
--- Interval: 00:00:30
--- Timeout: 5.00

-Tab -> Down
/ip route enable [find comment=LTE-Failover]

-Tab -> Up
/ip route disable [find comment=LTE-Failover]

jaclaz · Fri Oct 04, 2024 6:35 pm

Right now you have both the DSL and the LTE routers connected and the two routes "towards internetet" (the 0.0.0.0/0) active at the same time (the + sign tells us this) because they both have the same distance (of 1 which is the default).
Next step is changing the distance of the two connections to fixed ones:

Then, in DHCP client, change the distance of the routes:
/ip dhcp-client
add comment=defconf interface=ether1 default-route-distance=2
add comment=LTE interface=ether2 default-route-distance=1

And check again the /Ip route print
Now the route via LTE should be active, whilst the main one should be inactive (as it has a greater distance)
If you disable the route via LTE, the main one should become active.

When two routes (to the same address, 0.0.0.0/0) have the same distance, both are "active", when there are two routes to the same address but with different distance only the one with lower distance is active.

Filo's approach is about static routes, while you have dynamic ones, but once you have the two routes with distance of 2 and 1 respectively, it should work the same, but in case there should be issues, we can remove the DHCP setup on ether2 and make it a static route.

What are your difficulties with that approach, theoretical or practical?

The concept is to have (in normal operation) the backup (LTE) connection with a smaller distance BUT deactivated.
So the only route working will be the main (DSL) one as, with its distance of 2, it is the one with the lower distance among the active routes.
The backup route is still there, but deactivated.
Then you add a separate routing table with in it only a route through the DSL connection.
Then you add a mangle rule to mark all the request to (in Filo's example the "usual" google's 8.8.8.8, but it would be better if you choose another one, like 8.8.4.4 - still google's - or 1.1.1.1 - Cloudflare - or something else, as there might be devices that have 8.8.8.8 hardcoded). 8.8.4.4 to look into this new routing table. A mangle rule is a sort of filter that "catches" a given packet and stamps it with a sort of lable that tells the router wich routing table to use for that.
Essentially the chosen site, 8.8.4.4 will only be reachable through the main DSL connection via the separate DSL table, as the mangle rule will force it to take that one.
In normal operation everything will pass through the DSL connection, the 8.8.4.4 because of the specific route in the specific separate table, all the rest because of the general 0.0.0.0/0 route in main table. [1]

The netwatch script runs every 30 seconds and checks if 8.8.4.4 is reachable (through DSL).
If it is not, it enables the LTE route, that since it has a lower distance, becomes the main one.
When 8.8.4.4 returns reachable (still through DSL) it disables again the LTE route.

In winbox you should be able to see the comment near each route, or possibly from cli with /ip route print detail, otherwise instead of comment you can use gateway as selector.

If you have any doubt/question, feel free to ask them, I'll try to answer them as much as I can.

EDIT:
Sorry, cross-posting.
You have almost everything right, BUT your gateways will be:
5.187.172.1
192.168.1.1
The .1 are the actual gateway IP addresses in the routes, the .0 is the network.

I already posted how you can set/change the route distance to a chosen one in /ip dhcp-client.

The commands in the netwatch up and down scripts are based on the (LTE) route having a comment of "LTE-Failover", this has to be checked/adapted in your case, otherwise you can use something *like*:
/ip route enable [find dst-address=0.0.0.0/0 and gateway=192.168.1.1]

[1]There is possibly an alternative to using the separate table and the mangle rule, using the % operator, i.e. putting the route on the main table, as follows:
/ip route
add dst-address=8.8.4.4/32 gateway=5.187.172.1.1%ether1
but it has to be tested, so for the moment is better to try and replicate Filo's method "as is".

(EDIT: corrected typos and omissions)

bbmikrotik · Fri Oct 04, 2024 7:03 pm

I could change the distance to 2 and 1 in respective, now the DSL IP is blue not black like others, I created the DSL routing table with FIG, but cannot change the routing table for Dd 0.0.0.0/0 5.187.172.1 ether1 blue line, it is grayed out for main, or should I add with the + a new one? and delete the other one or how should I proceed with it?

jaclaz · Fri Oct 04, 2024 7:19 pm

I could change the distance to 2 and 1 in respective, now the DSL IP is blue not black like others, I created the DSL routing table with FIG, but cannot change the routing table for Dd 0.0.0.0/0 5.187.172.1 ether1 blue line, it is grayed out for main, or should I add with the + a new one? and delete the other one or how should I proceed with it?

With the route distances changed the DSL one should be not active and the LTE one active.
In Winbox the inactive should become blue, so it's OK.
In /ip route print, the DSL route should not be anymore "DAd" and the LTE one should not have anymore the + sign.
Now, if you disable the LTE one (select the route and click on the red X button) , the DSL one should come back as active.

I am not sure to understand the rest of your report.

You need to add a route to the newly created table (fib, not FIG) no changes in "main" table.
Yes, to add a new one use the + button, making sure to select the "DSL" table, NOT "main" in the dropdown "Routing table".

bbmikrotik · Fri Oct 04, 2024 7:31 pm

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, s - STATIC, d - DHCP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
# DST-ADDRESS GATEWAY DISTANCE
D d 0.0.0.0/0 5.187.172.1 2
DAd 0.0.0.0/0 192.168.1.1 1
DAc 5.187.172.0/22 ether1 0
DAc 192.168.1.0/24 ether2 0
DAc 192.168.88.0/24 bridge 0
0 As 0.0.0.0/0 5.187.172.1 1

Every one of it main except the last line which is in DSL routing table.
This is how my setup looks like, hopefully I am on the good direction with this.

jaclaz · Fri Oct 04, 2024 7:41 pm

Yep, it looks fine to me.

Now you have:
D d 0.0.0.0/0 5.187.172.1 2
DAd 0.0.0.0/0 192.168.1.1 1
The "main" one is NOT Active, the LTE is.
If you disable the LTE, the main should immediately be back to Active (i.e. DAd).

JFYI, when you have more than one table, from CLI you can use a modifier, like :
/ip route print where table=main
and
/ip route print where table=DSL

bbmikrotik · Sat Oct 05, 2024 7:13 pm

So I reach a braking point, when I power on the router in the first 5 minutes it takes the internet from the LTE after the 5 minutes DSL connection kicks in perfectly, LTE disconnects, when I physically disconnect DSL, after 30 seconds the LTE starts again and the internet comes through, but in every 30 seconds the LTE drops again (DSL still disconnected) then after 30 s again starts then 30s working and go on in this order, ON-OFF every 30 s, in netwatch I see the status up-down changing every 30 seconds, what can be the problem? If I keep DSL connected its works and LTE stays diconnected and DSL connected Netwatch shows UP in status.

# 2024-10-05 18:10:21 by RouterOS 7.16
# software id = xxxxx
#
# model = C52iG-5HaxD2HaxD
# serial number = xxxxx
/interface bridge
add admin-mac=xxxx:EB auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] comment=DSL poe-out=off
set [ find default-name=ether2 ] comment=LTE
/interface wifi
set [ find default-name=wifi1 ] channel.band=5ghz-ax .skip-dfs-channels=\
    10min-cac .width=20/40/80mhz comment=HomeSSid_5Ghz configuration.mode=ap \
    .ssid=HomeSSid_5Ghz disabled=no security.authentication-types=wpa2-psk \
    .ft=yes .ft-over-ds=yes
set [ find default-name=wifi2 ] channel.band=2ghz-ax .skip-dfs-channels=\
    10min-cac .width=20mhz comment=HomeSSid configuration.mode=ap .ssid=\
    HomeSSid disabled=no security.authentication-types=wpa2-psk .ft=yes \
    .ft-over-ds=yes
add configuration.mode=ap .ssid=SSidGuest mac-address=xxxxxx \
    master-interface=wifi1 name=wifi3
add channel.band=2ghz-n .width=20mhz comment="Guest WiFi" configuration.mode=\
    ap .ssid=SSidGuest disabled=no mac-address=xxxxxxx \
    master-interface=wifi2 name=wifi4
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
add name=dhcp-pool ranges=192.168.0.50-192.168.0.200
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
/routing table
add disabled=no fib name=DSL
/disk settings
set auto-media-interface=bridge auto-media-sharing=yes auto-smb-sharing=yes
/interface bridge filter
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward in-interface=wifi3
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward out-interface=wifi3
add action=drop chain=forward in-interface=wifi4
add action=drop chain=forward out-interface=wifi4
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wifi1
add bridge=bridge comment=defconf interface=wifi2
add bridge=bridge interface=wifi3
add bridge=bridge interface=wifi4
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=all
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
/ip dhcp-client
add comment=defconf default-route-distance=2 interface=ether1
add comment=LTE interface=ether2
/ip dhcp-server network
add address=192.168.0.0/24 gateway=192.168.0.1
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
    192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan type=A
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-routing chain=output dst-address=8.8.4.4 new-routing-mark=DSL \
    passthrough=yes protocol=icmp
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
/ip route
add disabled=no gateway=5.187.172.1 routing-table=DSL suppress-hw-offload=no
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" \
    dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/system clock
set time-zone-name=xxxxxxx
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool netwatch
add disabled=no down-script=\
    "/ip route enable [find dst-address=0.0.0.0/0 and gateway=192.168.1.1]" \
    host=8.8.4.4 http-codes="" interval=30s test-script="" timeout=5s type=\
    icmp up-script=\
    "/ip route disable [find dst-address=0.0.0.0/0 and gateway=192.168.1.1]"

bbmikrotik · Sat Oct 05, 2024 7:46 pm

Some update, I let the DSL disconnected for a while (10 min) and the LTE settled down and stopped (until - up down up down), now it is works perfectly fine. I am not sure about why, but it can be after router restart it will do every time for the first 10 min than works normal?

Now if I disconnect DSL 30s than goes to LTE if I connect DSL back after 30 sec connects to the DSL and LTE diconnects as should be. Hopefully it will work like that without problem.

I will test it for the next couple of weeks!

THANK YOU VERY MUCH FOR YOUR HELP! I owe you so much! I don't know how should I give this back to you!

Other topic if your are not full of me already

-My current firewall setup is okey for me for home systems to be safe? If the LTE is connected as main in case of outage wont make me vulnerable?

-Should I set up other type of stuff to improve my network? What do you suggest?

-What do you think what can be the problem with the following, if I let my internet modem(Telekom sagemcom ...) in router mode my internet speed via mikrotik (connected to the modem) in my computer via ethernet cable connected to the mikrotik is 800-900 mbit/s ish (1gbit plan is paid for telekom) but if I put the Telekom modem to bridge mode (modem mode) it drops to 540mbit/s. What can be the cause?

jaclaz · Sat Oct 05, 2024 8:48 pm

Good, happy it is more or less working

.

So the links/routes are sometimes "flapping"?
Every thirty seconds look suspiciously linked to the running of the netwatch script.
Try changing the interval to a higher value or - maybe better - try inserting in the netwatch script a small delay *like* another 10-20s

delay 20s

before the enable or disable command.
(I don't think that if you stay offline for a couple minutes when doing the failover to LTE or getting back to main DSL it will be an issue, and in any case you can always later fine tune it ).

There are more sophisticated versions of netwatch settings that may end up being more reliable, the plain one as you have it will trigger the up/down at the first failed ping, and sometimes it can happen that a ping fails.

Additionally, to make the failover and back faster (as perceived by the final user) there might be the need of removing connections, *like*
viewtopic.php?t=103812
viewtopic.php?t=103812#p977354

The real tests however are not connecting and disconnecting the cable from ether1 (which is - I believe - what you are doing) but rather:
1) disconnect the DSL line connector from the ISP router (this simulates an ISP issue on the line, the router is on but cannot reach internet)
2) power off (physically remove plugs from mains) for all three devices and then reinsert them at the same time, it would be easier/better to test if you use one of those multi-plug extensions with a on/off switch (to simulate what happens when a blackout happens)
as these two cases are the most common ones.

About the firewall, the default (defconf) settings are generally considered good enough, though they can be bettered/made more secure.
There is no difference (in defconf firewall) between DSL and LTE (or between ether1 and ether2) as the rules are made for the lists, WAN and LAN.

About the bridge vs. router I don't know what could be the reason, it could also be due to some setting in the ISP router.
Not that I actually know what I am saying but if you set it up as router it is faster, why not setting it up that way?
Purists will start telling you that you have double NAT and that you will increase latency, but as I see it if it works, it works.
And actually it would be much easier if your ISP assigned IP is not really-really static (right now if the ISP changes the assigned IP the route in "DSL" won't work).
If this is the case you will need a script in DHCP client *like*:
viewtopic.php?t=186716
viewtopic.php?t=186716#p939440

You still have to configure more properly your wifi (right now you have an error on wifi3 that I cannot understand the cause, as the settings seem "right" to me).

There is a nice tutorial by tangent on how to setup wifi and Guest Wifi without using VLAN's (simpler) here:
https://tangentsoft.com/mikrotik/wiki?n ... ns%20VLANs

And there is also (of course only if you are game for it) the experiment of modifying the failover method getting rid of the mangle rule and the additional routing table.

You - probably without knowing - entered a very deep rabbit hole ...

bbmikrotik · Sun Oct 06, 2024 3:49 am

Thank you! In this case I will try to set back up the modem as router if it is not a problem, and I am afraid in the future my ISP will change the IP and this whole failover wont work, but if I set back as router my assined IP if I remember right will be 192.168.0.1 like LTE one as I think is static, how should I proceed in this case? Can I change or should I change LTE IP as static or something else? An vica versa for ISP IP address?

But as I undersand this script:

if ($bound=1) do={/ip route set [find where dst-address=0.0.0.0/0 and routing-table="DSL" ] gateway=( $"gateway-address" . "%ether1" ) }

Modifies my DSL table to follow ether1 assined IP comes from the ISP provider kinda stuff?
And where should I insert this script?

My Wifi3 is disabled becouse I use wifi1 5ghz for apple TV for stream from laptop an like this is faster( laptop is connected via cable to mikrotik)

Wifi2 is 2.4 ghz for everything else as the router is behind a wall and 5ghz is too week for it ( not want to increase we got enough radio wave from everywhere) , wifi4 simple guest wifi 2.4ghz

jaclaz · Sun Oct 06, 2024 3:36 pm

That script should go as the DHCP client script (on the ether1 entry), it is a property (just like interface or default-route-distance) of /ip dhcp client.
https://help.mikrotik.com/docs/display/ ... Properties
I.e. this:

/ip dhcp-client
add comment=defconf default-route-distance=2 interface=ether1
add comment=LTE interface=ether2

should become:

/ip dhcp-client
add comment=defconf default-route-distance=2 interface=ether1 script="if ($bound=1) do={/ip route set [find where dst-address=0.0.0.0/0 and routing-table="DSL" ] gateway=( $"gateway-address" . "%ether1" ) }"
add comment=LTE interface=ether2

The condition "if ($bound=1)" should be true every time something changes in what the ISP DHCP server sends, so the script takes the (newly) assigned IP and modifies the (static) route in table "DSL", no matter if the IP address has not changed but the new DHCP lease changed *something else*.

You don't really *need* (in your setup) the . %ether1, as you are not (currently) using it in your route, but it shouldn't create any problem, it is an added specifier "forcing" the connection through ether1, with your current static route:
add disabled=no gateway=5.187.172.1 routing-table=DSL suppress-hw-offload=no
the script will (should) change the 5.187.172.1 to whatever IP the ISP DHCP server provides as gateway, if it is (say) 203.0.13.1, the route should become:
add disabled=no gateway=203.0.13.1%ether1 routing-table=DSL suppress-hw-offload=no

Since you have only one entry in that routing table, you can probably simplify it to:
if ($bound=1) do={/ip route set [find] gateway=( $"gateway-address" . "%ether1" ) }

If possible, it is easier to have two different subnets for the two routes/gateways, as it is easier to read the configuration and distinguish the two.
Right now the DHCP server of your LTE modem leases addresses in the range 192.168.1.0/24, and has itself 192.168.1.1.
When you setup your ISP router/modem you can change its DHCP server to lease addresses in *any* network that is not 192.168.1.0 (which is the LTE) and not 192.168.88.0 (which is your mikrotik LAN.

Personally (but I am way "old school", besides old) I would use static addresses everywhere but of course it is a matter of personal preferences and depends also on intended use, network topology, etc..

In your case I would have (with the ISP modem/router set as router):
192.168.1.1/30 <- the ISP modem router
192.168.1.2/30 <- ether1 on the Mikrotik
10.0.0.1/30 <- the LTE modem router
10.0.0.2/30<- the ether2 on the Mikrotik
So that at first sight you can understand (not now, but in 6 month or 1 year time) which is which (192 is ISP and 10 is LTE) when needing to change something in the configuration.
Using the /30 instead of the "normal" /24 is entirely optional and only a way to underline how these are only single links between two devices (the ISP and Mikrotik and the LTE and Mikrotik).
The DHCP server on the ISP and LTE would then be disabled, and as well the two DHCP clients on Mikrotik.
The configuration will be slightly simplified, and it will be (should be) more robust/secure[1].

[1]only as an example, an hypothetical "evil maid" could gain internet access by just plugging an ethernet cable in one of the bridge LAN ports of either router, this way it would be a bit more complex to do the same.

bbmikrotik · Sun Oct 06, 2024 5:21 pm

1. Sadly still flipping if I disconnect the ISP modem coax or ethernet or just turned off, so how should I proceed with the delay? If I put like this:
/ip route delay 10s enable (find dst-address=0.0.0.0/0 and gateway=192.168.1.1) it not working just stop the script, where should I put the delay command, I tried with :delay before /ip inside and after as well.

2. I turned back on the router mode of the ISP modem now my gateway from that side is 192.168.0.1 everything is adapted for this IP so it is works good except the flipping problem.

bbmikrotik · Sun Oct 06, 2024 6:31 pm

So update about fixed IP adresses:

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, s - STATIC, d - DHCP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
# DST-ADDRESS GATEWAY DISTANCE
DAd 0.0.0.0/0 10.0.0.1 1
D d 0.0.0.0/0 192.168.0.1 2
DAc 10.0.0.0/24 ether2 0
DAc 192.168.0.0/24 ether1 0
DAc 192.168.88.0/24 bridge 0
0 As 0.0.0.0/0 192.168.0.1 1

My way was: set IP address for the modems in the modem setup,BUT when the DHCP off, and I set up fixed IP address for mikrotik in the ISP or LTE modem, just killed the connection between modems(LTE/ISP and Mikrotik) so I just set DHCP for Mikrotik in the ISP modem to 192.168.0.11 and set the DHCP range for 192.168.0.11 - 192.168.0.11 and than fixed the IP address for Mikrotik MAC address. Same for LTE modem but the IP was 10.0.0.100

But I cannot change the /30 in the modem setup.

Other infos:

1. First, when I set the 192.168.1.1 IP for the ISP modem, the internet speed immediately dropped to 450 mbit/sec from the 900 with 192.168.0.1, so I set it back to 192.168.0.1 and the internet speed climbed back to 900 ish, strange behavior but I think is related to the ISP setting which is not reachable by me.

2. The IP firewall script is working as it should (hopefully) I just copied and pasted it as it was in the post, and the changeover is much faster than before.

3. Flipping when ISP out is still there

# 2024-10-06 18:22:46 by RouterOS 7.16
# software id = x
#
# model = C52iG-5HaxD2HaxD
# serial number = x
/interface bridge
add admin-mac=xxxx auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] comment=DSL poe-out=off
set [ find default-name=ether2 ] comment=LTE
/interface wifi
set [ find default-name=wifi1 ] channel.band=5ghz-ax .skip-dfs-channels=\
    10min-cac .width=20/40/80mhz comment=HomeSSid_5Ghz configuration.mode=ap \
    .ssid=HomeSSid_5Ghz disabled=no security.authentication-types=wpa2-psk \
    .ft=yes .ft-over-ds=yes
set [ find default-name=wifi2 ] channel.band=2ghz-ax .skip-dfs-channels=\
    10min-cac .width=20mhz comment=HomeSSid configuration.mode=ap .ssid=\
    HomeSSid disabled=no security.authentication-types=wpa2-psk .ft=yes \
    .ft-over-ds=yes
add configuration.mode=ap .ssid=SSidGuest mac-address=x \
    master-interface=wifi1 name=wifi3
add channel.band=2ghz-n .width=20mhz comment="Guest WiFi" configuration.mode=\
    ap .ssid=SSidGuest disabled=no mac-address=xxx \
    master-interface=wifi2 name=wifi4
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
add name=dhcp-pool ranges=192.168.0.50-192.168.0.200
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
/routing table
add disabled=no fib name=DSL
/disk settings
set auto-media-interface=bridge auto-media-sharing=yes auto-smb-sharing=yes
/interface bridge filter
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward in-interface=wifi3
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward out-interface=wifi3
add action=drop chain=forward in-interface=wifi4
add action=drop chain=forward out-interface=wifi4
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wifi1
add bridge=bridge comment=defconf interface=wifi2
add bridge=bridge interface=wifi3
add bridge=bridge interface=wifi4
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=all
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
/ip dhcp-client
add comment=defconf default-route-distance=2 interface=ether1
add comment=LTE interface=ether2
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
    192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan type=A
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-routing chain=output dst-address=8.8.4.4 new-routing-mark=DSL \
    passthrough=yes protocol=icmp
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.0.1 \
    routing-table=DSL scope=30 suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" \
    dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/system clock
set time-zone-name=Exxxx
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool netwatch
add disabled=no down-script=\
    "/ip route enable [find dst-address=0.0.0.0/0 and gateway=10.0.0.1] \r\
    \n" host=8.8.4.4 http-codes="" interval=30s test-script="" timeout=5s \
    type=icmp up-script="/ip route disable [find dst-address=0.0.0.0/0 and gat\
    eway=10.0.0.1] \r\
    \n"

jaclaz · Sun Oct 06, 2024 7:51 pm

Don't worry about the /30, the /24 is just fine, however JFYI, most non-mikrotik routers use the different notation for network mask:
255.255.255.0=/24 <- 256 values of which 254 usable (first and last are reserved for network and broadcast)
...
255.255.255.248=/29 <- 8 values of which 6 usable
255.255.255.252=/30 <- 4 values of which 2 usable

See;
https://www.calculator.net/ip-subnet-ca ... =Calculate
...
https://www.calculator.net/ip-subnet-ca ... =Calculate
https://www.calculator.net/ip-subnet-ca ... =Calculate

The script would be:
delay 10s
/ip route enable (find dst-address=0.0.0.0/0 and gateway=192.168.1.1)

(the /ip route is the path), delay is a general command instead.

Well you are making a strange mix between DHCP and static address, you are making a DHCP range so narrow that it can only assign one address.

The idea about static IP's is that they are static.

If you connect your PC directly to the ISP modem:
1) disable DHCP on the ISP modem
2) assign to it a static address of (say) 192.168.0.1 with network mask of either /24 or /30 (i.e. either 255.255.255.0 or 255.255.255.252) or /29 (255.255.255.248)
At this point you may lose connection as your PC is set to get its address from the ISP modem DHCP (that you just turned off)
So you might need to manually set your PC to an address in the right range, which is *any* address 192.168.0.x (but not 192.168.0.1) if you chose /24, ONLY 192.168.0.2 if you chose /30, any in 192.168.0.2-192.168.0.6 if you chose /29.

Here it is clear the usefulness (and possibly unwanted side effects) of using a /30 connection, you have only a single IP that can connect to the router.

So at this point you set your PC back to DHCP client, connect it to the Mikrotik (and get an address in the 192.168.88.x range), and:
1) remove/disable the DHCP client on ether1
2) set a static address to ether1:
/ip address
add address=192.168.0.2/30 comment=test_static interface=ether1
(192.168.0.x/24,. etc.)
3) at this point in /ip route print you won't have anymore a "D" route towards internet, so you need to add it manually
/ip route add gateway=192.168.0.1
in /ip route print you should now find the route with dst-address=0.0.0.0/0 as AS (Active and Static)

Maybe I am giving you too much (confusing) information all together at the same time, just take your time to digest the concepts before making further changes.

The flapping is however no good, so I would try to fix that part before making further/other changes to the configuration.

I am not understanding why the link is flapping.

What happens (every 30 seconds) is that the netwatch script attempts to ping 8.8.4.4 via the route in DSL routing table.
If the ping is successful, nothing happens.
If the ping fails, the LTE route is enabled (down-script).
After 30 seconds a new ping is made.
Now, to trigger the up-script (that disables again the LTE) it should mean that the ping this time is successful (which is not possible as the ISP modem/router is still disconnected), the only explanation is that for *some reasons* the ping goes through the LTE interface (i.e. it is *somehow* not "catched" by the mangle rule).
But this even if due to some mistake in the settings or to some "strange" bug should happen always, not only sometimes.

Let's try the alternative.

1) disable (temporarily) the netwatch scripts
2) disable (temporarily) the mangle rule
3) enable (temporarily) the LTE route
2) add a "narrow" route to the "main" table:
/ip route add dst-address=8.8.4.4/32 gateway=192.168.0.1%ether1

Now, if you disconnect the ISP router, if you ping 8.8.8.8 the ping should go through the LTE (Active) and succeed, while if you ping 8.8.4.4 it should go through the (disconnected) ISP and fail, with timeout and/or host unreachable errors.

If this happens, try to reconnect the ISP modem/router and the 8.8.4.4 ping should be successful.

Now, re enable the netwatch script, it should disable the LTE connection.

Test if the routes are flapping again.

P.S.: I see now that you still have:
/interface detect-internet
set detect-interface-list=all

you should really set detect-interface to none.

P.P.S.: Wait a minute, only now I realize

that the netwatch is set to type=icmp, that could be part of the "flapping" issue.
The "default" values for type=icmp (that are not normally visible) could be "wrong" for your connections, see here:
viewtopic.php?t=190925
viewtopic.php?t=194311
viewtopic.php?t=205645
Try - as an experiment - to change the type=icmp to type=simple

bbmikrotik · Sun Oct 06, 2024 8:22 pm

There was cross-post:

Before your post I successfully set the delay, like this:

:delay 100
 /ip route disabled(..)

The thing what happened after was strange... so
1. DSL disconnected waited for 30 sec LTE activated (netwatch 'down') then LTE works good!
2. after I saw the netwatch goes 'up' LTE stayed because the delay is working but the 'UP' stayed there for 100 seconds the in netwatch...
3. after LTE disconnected, 10 seconds or something later 'down' appeared in netwatch and LTE activated again
if I set :delay 30 UP stays there for that much, it stays there until LTE deactivates then check and activates again... Strange

Sorry for my grammar .. hard to explain correctly

I will try what you suggested!!

bbmikrotik · Sun Oct 06, 2024 8:43 pm

Still doing what I posted one before, set delay, first okey second time it waits until LTE goes disconnected, than again activates LTE. Did everything what you suggested.

:# 2024-10-06 19:36:34 by RouterOS 7.16
# software id = Exxxx
#
# model = C52iG-5HaxD2HaxD
# serial number = xxxx
/interface bridge
add admin-mac=xxxxxx auto-mac=no comment=defconf name=bridge
/interface ethernet
set [ find default-name=ether1 ] comment=DSL poe-out=off
set [ find default-name=ether2 ] comment=LTE
/interface wifi
set [ find default-name=wifi1 ] channel.skip-dfs-channels=10min-cac comment=\
    HomeSSid_5Ghz configuration.mode=ap .ssid=HomeSSid_5Ghz disabled=no \
    security.authentication-types=wpa2-psk .ft=yes .ft-over-ds=yes
set [ find default-name=wifi2 ] channel.skip-dfs-channels=10min-cac comment=\
    HomeSSid configuration.mode=ap .ssid=HomeSSid disabled=no \
    security.authentication-types=wpa2-psk .ft=yes .ft-over-ds=yes
add configuration.mode=ap .ssid=SSidGuest mac-address=xxxxxxx \
    master-interface=wifi1 name=wifi3
add channel.band=2ghz-n .width=20mhz comment="Guest WiFi" configuration.mode=\
    ap .ssid=SSidGuest disabled=no mac-address=xxxxx \
    master-interface=wifi2 name=wifi4
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
add name=dhcp-pool ranges=192.168.0.50-192.168.0.200
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
/routing table
add disabled=no fib name=DSL
/disk settings
set auto-media-interface=bridge auto-media-sharing=yes auto-smb-sharing=yes
/interface bridge filter
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward in-interface=wifi3
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward out-interface=wifi3
add action=drop chain=forward in-interface=wifi4
add action=drop chain=forward out-interface=wifi4
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wifi1
add bridge=bridge comment=defconf interface=wifi2
add bridge=bridge interface=wifi3
add bridge=bridge interface=wifi4
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add comment=LTE interface=ether2 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
/ip dhcp-client
# Interface not active
add comment=defconf default-route-distance=2 interface=ether1
add comment=LTE interface=ether2
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
    192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan type=A
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-routing chain=output dst-address=8.8.4.4 new-routing-mark=DSL \
    passthrough=yes protocol=icmp
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.0.1 \
    routing-table=DSL scope=30 suppress-hw-offload=no target-scope=10
add dst-address=8.8.4.4/32 gateway=192.168.0.1%ether1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set ssh disabled=yes
set api disabled=yes
set api-ssl disabled=yes
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" \
    dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/system clock
set time-zone-name=xxxxxx
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool netwatch
add disabled=no down-script=\
    "/ip route enable [find dst-address=0.0.0.0/0 and gateway=10.0.0.1] \r\
    \n" host=8.8.4.4 http-codes="" interval=30s test-script="" timeout=5s \
    type=simple up-script=":delay 40 \r\
    \n/ip route disable [find dst-address=0.0.0.0/0 and gateway=10.0.0.1] \r\
    \n"

Flags: D - DYNAMIC; A - ACTIVE; c - CONNECT, s - STATIC, d - DHCP
Columns: DST-ADDRESS, GATEWAY, DISTANCE
# DST-ADDRESS GATEWAY DISTANCE
DAd 0.0.0.0/0 10.0.0.1 1
D d 0.0.0.0/0 192.168.0.1 2
0 As 8.8.4.4/32 192.168.0.1%ether1 1
DAc 10.0.0.0/24 ether2 0
DAc 192.168.0.0/24 ether1 0
DAc 192.168.88.0/24 bridge 0
1 As 0.0.0.0/0 192.168.0.1 1

jaclaz · Mon Oct 07, 2024 12:17 am

I am still not sure to understand what re-triggers the "up" script.

You are simulating the down by disconnecting ether1, right?

Try to simulate It instead by leaving the ISP router connected and disconnecting the DSL/telephone line from It.

Does It still flap?

If not, the issue could be in the route to 8.8.4.4 that gets disabled/inactive when you disconnect the ether1 and the wider one through ether2 Is used.

But did you try the manual test I proposed earlier? (with netwatch script disabled and LTE router enabled)

Now, if you disconnect the ISP router, if you ping 8.8.8.8 the ping should go through the LTE (Active) and succeed, while if you ping 8.8.4.4 it should go through the (disconnected) ISP and fail, with timeout and/or host unreachable errors.

If this happens, try to reconnect the ISP modem/router and the 8.8.4.4 ping should be successful.

P.S.: to test the alternative, you need to disable the mangle rule, otherwise it will take precedence and the setup will behave as before:

/ip firewall mangle
add action=mark-routing chain=output dst-address=8.8.4.4 new-routing-mark=DSL \
passthrough=yes protocol=icmp

bbmikrotik · Mon Oct 07, 2024 4:23 pm

I tried the second mangle route setting it is the same flipping, about the pinging:

ISP disconnected from the Mikrotik:

/ping 8.8.4.4
SEQ HOST SIZE TTL TIME STATUS
0 8.8.4.4 56 115 38ms822us
1 8.8.4.4 56 115 43ms330us ----'UP' mode with ISP off...
2 8.8.4.4 56 115 42ms819us
3 8.8.4.4 56 115 42ms860us
4 8.8.4.4 56 115 35ms853us
5 8.8.4.4 56 115 39ms971us
6 8.8.4.4 56 115 42ms777us
7 8.8.4.4 56 115 43ms6us
8 no route to host
9 no route to host ------ 'UP' mode ISP off and LTE deactivated by the script
10 no route to host
11 no route to host
12 no route to host
13 no route to host
14 no route to host
15 no route to host
16 no route to host
17 no route to host
18 no route to host
19 no route to host
sent=20 received=8 packet-loss=60% min-rtt=35ms853us avg-rtt=41ms179us max-rtt=43ms330us
SEQ HOST SIZE TTL TIME STATUS
20 no route to host
21 no route to host
22 no route to host
23 no route to host
24 no route to host
25 no route to host
26 no route to host
27 no route to host
28 8.8.4.4 56 115 42ms586us
29 8.8.4.4 56 115 40ms903us - 'DOWN' LTE on again
30 8.8.4.4 56 115 51ms788us
31 8.8.4.4 56 115 46ms956us
32 8.8.4.4 56 115 43ms104us
33 8.8.4.4 56 115 32ms786us

jaclaz · Mon Oct 07, 2024 4:54 pm

Sorry, I do not understand your last report.

Can you try EXACTLY this test:

Let's try the alternative.

1) disable (temporarily) the netwatch scripts
2) disable (temporarily) the mangle rule
3) enable (temporarily) the LTE route
2) add a "narrow" route to the "main" table:
/ip route add dst-address=8.8.4.4/32 gateway=192.168.0.1%ether1

Now, if you disconnect the ISP router, if you ping 8.8.8.8 the ping should go through the LTE (Active) and succeed, while if you ping 8.8.4.4 it should go through the (disconnected) ISP and fail, with timeout and/or host unreachable errors.

If this happens, try to reconnect the ISP modem/router and the 8.8.4.4 ping should be successful.

Now, re enable the netwatch script, it should disable the LTE connection.

Test if the routes are flapping again.

You added the narrow route alright, but you must:

1) disable manually (temporarily) the netwatch scripts
2) disable manually (temporarily) the mangle rule
3) enable manually (temporarily) the LTE route

What I need to understand is how the router behaves in these conditions (please reply with the same numbered tests):

1. With the LTE enabled (manually) and the ISP router connected to ether1, 8.8.8.8 should ping correctly and 8.8.4.4 should show timeout/host unreachable as well be reachable.
2. Now if you try disconnecting the ISP router from ether1, the behaviour should be the same (8.8.8.8 working and 8.8.4.4 not) 8.8.8.8 working and 8.8.4.4 show timeout/host unreachable.
3. If this happens, try re-enabling the netwatch script, the behaviour should remain the same and after the 30s(+delay) or so, the netwatch script might run the "down" script, but nothing should change, still 8.8.8.8 pings should work and 8.8.4.4 not.
4. Wait a couple minutes, there should be no attempts to flapping
5. If you reconnect at this point the ISP router to ether1 and wait the 30s(+delay) or so, the netwatch script should run the "up" script and both 8..8.8.8 and 8.8.4.4 pings should work.
6. Wait a couple minutes, there should be no attempts to flapping.

Then, repeat the same 6 points above, but instead of disconnecting the ISP router from ether1, try disconnecting the telephone/DSL line from the router.

I suspect that for *some reasons* the route to 8.8.4.4 becomes invalid/inactive and it takes the LTE route, making the netwatch check ineffective.
It is possible that (maybe because the ether1 is still using the DHCP client) the Mikrotik "senses" that the ISP router is disconnected in the first series of tests, but it should not in the second set.

EDIT: corrected points #1 and #2 as they were wrong

bbmikrotik · Mon Oct 07, 2024 5:21 pm

So:

1. Both 8.8.4.4 and 8.8.8.8 works, got back all that packages
2. Same as number 1. everything came back
3. so as my previous post, ISP still disconnected from Mikrotik (no hard connection between them ethernet off) until 'DOWN' and LTE is on both 4.4 and 8.8 comes through, if in 'UP' LTE goes off and none of the pings goes through, so like nothing happens, if LTE on both 8.8.4.4 and 8.8.8.8 got pinged correctly so the netwatch turns off the LTE but than nothing comes through because not connection to the internet, because of that LTE goes ON again and this repeating.
4. Waited 10 min with ISP off Netwatch enabled, mangle is still disabled, and a flipping just continues
5. yeah If I reconnect ISP everything works as should be no flapping.

IF I keep the the ISP modem connected to Mikrotik but internet cable before the ISP modem (DSL cable, coax cable disconnected):
1. (Netwatch off, mangle off, ISP modem connected to mikrotik ) 8.8.4.4 time out, 8.8.8.8 good ping
2. (netwatch off, mangle off, ISP modem from mikrotik physically disconnected from mikrotik) 8.8.4.4 coming through and is ok , 8.8.8.8 not working
AND if I connect back the ISP ethernet cable to the mikrotik a number 1 happens 4.4 time out 8.8. ok

What I figured out... IF I physically disconnect ISP from Mikrotik, the flipping starts again, if I just disconnect the main internet cable from the ISP modem, and I put back the connection between the modem and Mikrotik flipping stops and works as should... and I think is the reason is, If I physically disconnect the ISP from Mikrotik the connection between them is disappears from the route list, if I just cut the internet in the ISP but (not between ISP and Mikrotik) stays in the Route list!

Mangle was off in the whole time! I am not sure what is its purpose overall in this case.

jaclaz · Mon Oct 07, 2024 5:35 pm

So: Netwatch disabled, mangle disabled, LTE enabled: both 8.8.4.4 and 8.8.8.8 works., got back all the answers no timeout

With ether1 disconnnected?

Is it the same with ether1 connected but internet not available because dsl line is disconnected from router?

bbmikrotik · Mon Oct 07, 2024 5:55 pm

So:

1. Both 8.8.4.4 and 8.8.8.8 works, got back all that packages
2. Same as number 1. everything came back
3. so as my previous post, ISP still disconnected from Mikrotik (no hard connection between them ethernet off) until 'DOWN' and LTE is on both 4.4 and 8.8 comes through, if in 'UP' LTE goes off and none of the pings goes through, so like nothing happens, if LTE on both 8.8.4.4 and 8.8.8.8 got pinged correctly so the netwatch turns off the LTE but than nothing comes through because not connection to the internet, because of that LTE goes ON again and this repeating.
4. Waited 10 min with ISP off Netwatch enabled, mangle is still disabled, and a flipping just continues
5. yeah If I reconnect ISP everything works as should be no flapping.

IF I keep the the ISP modem connected to Mikrotik but internet cable before the ISP modem (DSL cable, coax cable disconnected):
1. (Netwatch off, mangle off, ISP modem connected to mikrotik ) 8.8.4.4 time out, 8.8.8.8 good ping
2. (netwatch off, mangle off, ISP modem from mikrotik physically disconnected from mikrotik) 8.8.4.4 coming through and is ok , 8.8.8.8 not working until some time than flipping starts again,
AND if I connect back the ISP ethernet cable to the mikrotik a number 1 happens 4.4 time out 8.8. ok, no flipping
3. (netwatch on mangle off, number 2 config) Flipping
4. (netwatch on mangle off, number 1 config) no flipping after some time internet comes back and LTE turns off

What I figured out... IF I physically disconnect ISP from Mikrotik, the flipping starts again, if I just disconnect the main internet cable from the ISP modem, and I put back the connection between the modem and Mikrotik flipping stops and works as should... and I think is the reason is, If I physically disconnect the ISP from Mikrotik the connection between them are disappears from the route list, if I just cut the internet in the ISP (not between ISP and Mikrotik) the 192.168.0.1 will be there

Mangle was off in the whole time! I am not sure what is its purpose overall in this case.

Sorry, it was cross-post, because I tried to manage the stuff that you suggested, so yes there is a different behavior if I cut the internet or cut the connection between ISP modem and Mikrotik.

jaclaz · Mon Oct 07, 2024 7:04 pm

Ok, it starts to make sense, so, for *some reasons* the "narrow" route becomes invalid if the ether1 cable is disconnected. (I suspect it to be due to the ether1 still having the dhcp client and not a static address, but maybe there are other reasons) so the 8.8.4.4 ping takes the "LTE" route and works fine.

Try adding a blackhole route with distance 2. (a blackhole route is something that fails at pinging as it goes nowhere)

Right now you should have in main the "narrow" rule:
/ip route
...
add dst-address=8.8.4.4/32 gateway=192.168.0.1%ether1

add a second route to the same address but with type blackhole and distance 2:
/ip route
add dst-address=8.8.4.4/32 distance=2 type=blackhole
add blackhole dst-address=8.8.4.4 distance=2

The idea is that:
1) when the DSL modem is connected to ether1 AND there is internet on the DSL, the first rule is active (distance=1) and the pings of the netwatch script go through it (and succeed) thus nothing happens
2) when the DSL modem is connected to ether1 BUT there is not internet on the DSL (simulated by disconnecting coax cable) the rule is active (distance=1) and the pings of the netwatch script go through it (but fail) thus switching to LTE
3) when the DSL modem is NOT connected to ether1 AND thus there is not internet on the DSL AND that rule (for whatever reasons) disappears/becomes inactive, the second route will become active (distance=2) and the pings of the netwatch script go through it (but fail) thus doing nothing
4) when the internet is restored on DSL the first rule returns active and the netwatch script disables the LTE

Let's see if this approach works.

EDIT: corrected wrong command line

bbmikrotik · Mon Oct 07, 2024 7:40 pm

There are some problem with the code, there is a red indication in type row, how should I proceed with this, because I don't get the theory behind.

I tested the system, and it works nice, if I simulate an electricity blackout, it first connects to LTE then when the ISP boots up and Netwatch startup 5 min delay elapses connects to the ISP and deactivates LTE, pretty stable, no issues so far if I not disconnect ISP/Mikrotik ethernet cable. Thank you for reaching this setup

I reenabled the mangle rule as well.

In the afternoon I made a factory reset for the whole Mikrotik router and set up everything from the basics without adding the tried rules and experiments.

# 2024-10-07 18:36:25 by RouterOS 7.16
# software id = xxxx
#
# model = C52iG-5HaxD2HaxD
# serial number = xxxx
/interface bridge
add admin-mac=xxxxxx auto-mac=no comment=defconf name=bridge
/interface wifi
set [ find default-name=wifi1 ] channel.band=5ghz-ax .skip-dfs-channels=\
    10min-cac .width=20/40/80mhz configuration.mode=ap .ssid=HomeSSid_5Ghz \
    disabled=no security.authentication-types=wpa2-psk .ft=yes .ft-over-ds=\
    yes
set [ find default-name=wifi2 ] channel.band=2ghz-ax .skip-dfs-channels=\
    10min-cac .width=20mhz configuration.mode=ap .ssid=HomeSSid disabled=no \
    security.authentication-types=wpa2-psk .ft=yes .ft-over-ds=yes
add configuration.mode=ap .ssid=SSidGuest mac-address=4xxxxxx \
    master-interface=wifi1 name=wifi3
add configuration.mode=ap .ssid=SSidGuest disabled=no mac-address=\
    xxxxxxxx master-interface=wifi2 name=wifi4 \
    security.authentication-types=wpa2-psk
/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/ip pool
add name=default-dhcp ranges=192.168.88.10-192.168.88.254
/ip dhcp-server
add address-pool=default-dhcp interface=bridge name=defconf
/routing table
add disabled=no fib name=DSL
/disk settings
set auto-media-interface=bridge auto-media-sharing=yes auto-smb-sharing=yes
/interface bridge filter
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward in-interface=wifi3
# wifi3 not ready
# in/out-bridge-port matcher not possible when interface (wifi3) is not slave
add action=drop chain=forward out-interface=wifi3
add action=drop chain=forward in-interface=wifi4
add action=drop chain=forward out-interface=wifi4
/interface bridge port
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge comment=defconf interface=wifi1
add bridge=bridge comment=defconf interface=wifi2
add bridge=bridge interface=wifi3
add bridge=bridge interface=wifi4
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1 list=WAN
add interface=ether2 list=WAN
/ip address
add address=192.168.88.1/24 comment=defconf interface=bridge network=\
    192.168.88.0
/ip dhcp-client
add comment=defconf default-route-distance=2 interface=ether1
add interface=ether2
/ip dhcp-server network
add address=192.168.88.0/24 comment=defconf dns-server=192.168.88.1 gateway=\
    192.168.88.1
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=192.168.88.1 comment=defconf name=router.lan type=A
/ip firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=accept chain=input comment=\
    "defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
add action=drop chain=input comment="defconf: drop all not coming from LAN" \
    in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" \
    ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" \
    ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related hw-offload=yes
add action=accept chain=forward comment=\
    "defconf: accept established,related, untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
    connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=mark-routing chain=output dst-address=8.8.4.4 new-routing-mark=DSL \
    passthrough=yes protocol=icmp
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" \
    ipsec-policy=out,none out-interface-list=WAN
/ip route
add disabled=no dst-address=0.0.0.0/0 gateway=192.168.0.1 routing-table=DSL \
    suppress-hw-offload=no
add dst-address=8.8.4.4/32 gateway=192.168.0.1%ether1
/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter
add action=accept chain=input comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=\
    invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" \
    dst-port=33434-33534 protocol=udp
add action=accept chain=input comment=\
    "defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=\
    udp src-address=fe80::/10
add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 \
    protocol=udp
add action=accept chain=input comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=input comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=input comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
add action=accept chain=forward comment=\
    "defconf: accept established,related,untracked" connection-state=\
    established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" \
    connection-state=invalid
add action=drop chain=forward comment=\
    "defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment=\
    "defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" \
    hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=\
    icmpv6
add action=accept chain=forward comment="defconf: accept HIP" protocol=139
add action=accept chain=forward comment="defconf: accept IKE" dst-port=\
    500,4500 protocol=udp
add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=\
    ipsec-ah
add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=\
    ipsec-esp
add action=accept chain=forward comment=\
    "defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
add action=drop chain=forward comment=\
    "defconf: drop everything else not coming from LAN" in-interface-list=\
    !LAN
/system clock
set time-zone-name=Exxxxxx
/system note
set show-at-login=no
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool netwatch
add disabled=no down-script=\
    "/ip route enable [find dst-address=0.0.0.0/0 and gateway=10.0.0.1]" \
    host=8.8.4.4 http-codes="" interval=10s test-script="" timeout=5s type=\
    simple up-script=\
    "/ip route disable [find dst-address=0.0.0.0/0 and gateway=10.0.0.1]"

jaclaz · Mon Oct 07, 2024 8:34 pm

My bad, should have been:
/ip route
add blackhole dst-address=8.8.4.4 distance=2

Let's see if I can help you grasp the concepts, the more you get them the more you will be able to correct/fix/tune your setup.

Normally there are two "all internet" rules, the first going through the LTE (disabled) and the second going through the DSL.
We carve an exception to the above EITHER by using a mangle rule and a separate routing table OR by using a very narrow route in the main table to "force" connection to 8.8.4.4 to go ONLY through the DSL, no matter if the DSL or the LTE general rules are active.

The netwatch pings the 8.8.4.4 to switch the LTE on or off.

When you are (more properly) simulating the loss of DSL connection by removing the coax cable) everything works, but when you simulate the the loss of connection by disconnecting the cable from ether1, for *some reasons* the routes pointing to ether1/192.168.0.1, both in the added "DSL" routing table and in the main one are not anymore valid, so the 8.8.4.4 is reached through the "general" rule via the LTE connection.

When the netwatch script tries again to ping 8.8.4.4 can reach it alright, so it disables the LTE route, but at the following next ping (since there is no valid route as the one going through LTE is disabled and the main one through DSL is not active because the link to the DSL router is disconnected) it cannot reach 8.8.4.4 so it enables again the LTE.

Rinse and repeat and you get the flapping.

The idea of the blackhole route (with bigger distance of 2) is that normally this route is not active as there is already another route with the same dst-address but with the lower distance of 1.
When the ether1 is disconnected and the routes through ether1 become invalid, this one becomes active.
Netwatch tries to ping the 8.8.4.4 but since it goes to this blackhole it fails and does not disable the LTE.
When you reconnect the ether1, the 8.8.4.4 route via ether1 becomes again active, the blackhole one becomes inactive and netwatch can ping 8.8.4.4 successfully, so it disables the LTE route.

If you keep the mangle rule enabled, the "other" routing table will be used, and that one has no blackhole, so you will still have the flapping when disconnecting ether1.

It is EITHER mangle rule and separate routing table OR two routes to 8.8.4.4 (one through ether1, one blackhole) in main.

You should try the added blackhole route approach, disabling the mangle rule, as (if it works) it should cover also the possibility of an ethernet cable going bad (ether1 disconnected) or a failure of the ISP modem power supply.

BTW with a static address assigned to ether1 the original approach with the mangle rule or the one with narrow route in main should work also in case of ether1 disconnection, what triggers the mess must be the loss of address of ether1 from the DHCP or however the interrupted connection between ether1 and the DHCP server of the ISP router.

bbmikrotik · Mon Oct 07, 2024 9:20 pm

Ahaa, I started to get the concept, clearer step by step, thank you for your patience!

So, your method is working perfectly

already tried the following tests:
1. ISP modem to Mikrotik modem UTP cable disconnect - no flipping anymore, works as should!

2. Internet coax cable disconnect from ISP modem, ISP modem and Mikrotik connected - no flipping, perfect switch over

3. Blackout simulator (ISP, Mikrotik, LTE modem start simultaneously) - works perfectly after 5 min netwatch delay, ISP takes over, LTE feeds the system with internet until

Thank you very much for this and the education!

Any good advice for my setup?

- I saw there is a bridge drop filter for wifi3/4 guest wifi network, do you think that is enough? I usually connect for that wifi the not trusted stuff for example air purifier, air condition (I don't feel I can trust Chinese-made home stuff, don't want them to reach the home network)
- I want to implement a NAS drive for movies, is it really hard to manage for home use?
- I just want to be more or less safe in my network environment from outside attacks, myself is self-conscious about the usage of the internet so that won't be a problem hopefully.
- Do you think if I just keep my ISP modem in router mode won't make a big difference? What are the pros and contras between the bridge and router mode of the ISP modem?

jaclaz · Mon Oct 07, 2024 11:53 pm

Not that I am actually qualified for giving an expert opinion on bridging vs. routing, so take my advice with a grain or two of salt.
In theory the bridge mode is more direct, so It should be faster, in practice in your case It Is slower, and not a little slower, some 30 percent slower is a lot.
Of course there must be some reasons why this happens, but I cannot find any in your rather simple configuration and - if you think about it - the Mikrotik always works as a router, whether it routes between LAN 192.168.88.x and WAN 5.x.y.z or WAN 192.168.0.x should not affect the speed of the connection, so the issue must be in either the ISP router configuration or in some settings in the ISP source but I have no idea where to look for the cause .
Still in theory the double NAT (one in the Mikrotik and one in the ISP router) might increase the latency, but if the increase is from "very, very little" to "very little" it could be unnoticeable and only measurable by specific tests.

I am not expert enough on wireless setup and related increased security practices to give you valid advice.

The NAS seems to me a non-problem, it is a network connected device like any other, what kind of doubts/needs do you have?
Separating it from some other devices?
In this case you might want to implement VLANs, which Is a rather advanced topic.

The default firewall rules are usually considered "good enough", though they can be made "stricter" following the principle of only explicitly allow what is wanted/needed and then, as last rule of each chain put a "drop all else" one.
Before attempting to modify the default rules one must understand why they are there and what they do, as it is easy to drill big holes in the firewall accidentally (Chesterton's fence principle) or - still accidentally - make everything so "strict" that for one reason or the other you end up being locked out of the router.

Check this recent thread that revolves around these themes:
viewtopic.php?t=209307

bbmikrotik · Tue Oct 08, 2024 6:35 pm

Thank you for everything

jaclaz · Wed Oct 09, 2024 11:41 am

You are welcome

.

In the meantime there is yet another very interesting ongoing thread about firewall,and its rules, JFYI:
viewtopic.php?t=211574

bbmikrotik · Sat Oct 26, 2024 9:45 pm

Hey!

After one month test I found one problem which is not a big deal, just to let you know.

If I restart the LTE modem (ISP ON for the whole time) and LTE modem boots up, routing changes to LTE and and do not want to change to ISP modem anymore, only if I make some changes in netwatch to make the netwatch refresh. (Netwatch shows UP but do not disable LTE modem, just after I force netwatch to refresh)

Larsa · Sat Oct 26, 2024 11:51 pm

Just keep in mind that Netwatch might be pretty unreliable on LTE when using Carrier Aggregation (CA), which is the default mode for most connections.

anav · Sun Oct 27, 2024 2:56 am

Yeah jaclaz that was way simpler than just using two vlans. ;-PP

jaclaz · Sun Oct 27, 2024 9:59 am

Rethinking about it, I think this is "normal" (and had to be expected) due to the dynamic nature of the setup.

It should be tested/checked, but this is what I believe it is happening:

1) Since the route to the LTE is created by the LTE modem DHCP, when you turn it off it simply disappears (poof).

2) The connection continues to go happily through the ISP, Netwatch checks continuously that one, and finds that nothing has changed, so it does nothing.

3) Then you reconnect/switch on the LTE modem, and its DHCP adds a new dynamic route (which is active, with lower distance - 1 - and since the ISP one has 2, the LTE one is used).

4) But netwatch still can ping happily 8.8.4.4 through the ISP one, so it does not do anything.

It would probably make sense to add a static route through 10.0.0.1 and disable the dynamic one (add-default-route=no).

Or use a scheduler script to periodically check that if 8.8.4.4 is reachable through the ISP, then the LTE route (if existing) should be disabled (but this would cause a double switch from ISP to LTE and then from LTE to ISP when you switch off and then on the LTE).

KiwiBloke · Sun Oct 27, 2024 12:02 pm

It have been watching this thread with interest. Thanks to the OP and jaclaz for bringing it up.

I have one question though. Is it better for the LTE device to be in passthrough mode so as not to duplicate settings between devices, merely passing the LTE signal only to the router and let the router take care of the firewall, dhcp settings, etc by itself?

Cheers.

jaclaz · Sun Oct 27, 2024 2:10 pm

@KiwiBloke
I believe the question is up for debate, and possible answers to it can be different depending on use and hardware involved, some devices (ISP router and LTE modem router) might have very good built-in firewalls or may have not any.

If we take for granted that the Mikrotik firewall is the best of the best (if properly configured) the more "direct" and "dumb" are the devices upstream connecting to the internet (bridge mode, etc.) the better.
This will BTW avoid the double NAT issue, provided it is an issue in practice. (In theory it slows down the connection, in practice it depends, if the change is from very, very, very fast to only very, very fast, it is not an issue).

But as we have seen in bbmikrotik's case, his ISP router set in bridge mode is actually giving half the speed it does in router mode, so you never really know.

And also on personal preferences.
Only as an example (and this shows how old I am) I personally like to have as much as possible things "static", I mean you have a "central router" hooked to a ISP modem/router and to a LTE modem/router, all three devices are bolted/screwed to the wall, can you imagine anything more "static" than that?
Every time the ISP router boots, the Mikrotik has to ask a DHCP IP address (and a route) to it (and every time it gets the SAME IP address and route), every time the LTE modem/router boots the Mikrotik has to ask a DHCP IP address (and a route) to it (and every time it gets the SAME IP address and route). It looks a bit vain (besides repetitive), and periodically the DHCP leases end and the questions/answers are re-asked and re-answered.

But on the other hand another scenario would be if you have a LTE portable modem/router that you normally bring with you when you travel but that you hook to the Mikrotik when you are at home (for failover use), it is much more convenient to have it fully configured so that it works when used "standalone" when traveling and when connected to the home router without needing to change anything in its configuration (except maybe turning the wi-fi radio on/off).

anav · Sun Oct 27, 2024 5:37 pm

The onus of the ISP provider is to ensure the difference of their device in bridged mode or Router mode is crystal clear.
Using doublenat, should have no significant slow down in traffic so there is no downside other than a slightly more complicated setup on the MT and possibly the need to forward ports on the ISP device acting as a router. Most here would rather terminate the public IP on the MT device but both ways work almost equally well.

KiwiBloke · Sun Oct 27, 2024 8:22 pm

Thanks for the reply guys. So, what if the LTE device is a Mikrotik along with a Mikrotik router? Would passthrough then be the best option? It would make sense to me to have only the main router do all the heavy lifting, as it were.

bbmikrotik · Tue Oct 29, 2024 9:04 pm

I will try to set static IP addresses for ISP modem and LTE modem, but I struggling with it a little bit, can you please tell me how can I set them up properly? If I set up only in the ISP or LTE modem they disappear from mikrotiks eyes, I would appreciate the help!

jaclaz · Tue Oct 29, 2024 10:52 pm

Of course It depends, but usually DHCP servers (in *any* device) are pre- configured to not cover the whole address space of the network.
As an example, Mikrotik devices are usually configured (defconf) for addresses 192.168.88.10 to 192.168.88.254, while keeping for itself 192.168.88.1. (the wireless wire set Is usually set as192.168.88.2 and 192.168.88.3).
So you have 192.168.88.4 up to 192.168.88.9 that will never be served to any client and won't also be used by default by any device.
Your ISP modem/router could be (say) set to give addresses in the 192.168.1.100 to 192.168.1.250, keeping for itself the 192.168.1.1.
So you need to add to the Mikrotik ether1 a static address, let's say 192.168.1.42, that Is not in the DHCP range but Is still in the 192.168.1.0/24 network.
Rinse and repeat for the LTE router.
Then you disable the DHCP client on ether1 and ether2.
If the router DHCP server leases instead from 192.168.1.2 up to 192.168.1.254, you need to reduce the range to carve out at least one address for the Mikrotik.
This approach has the advantage that you can anytime just connect a laptop to one of the unused ports of the ISP router to configure it (or connect a device bypassing the Mikrotik).
Otherwise, you can disable the DHCP server on the ISP router, set a /30 network, so that there are only two addresses available, one for the ISP router itself and one for the ether1 of the Mikrotik.

bbmikrotik · Wed Oct 30, 2024 9:02 am

So, I made it work!

What I did just for confirmation:
IP address of Mikrotik from ISP: 192.168.0.11
IP address of Mikrotik from LTE: 10.0.0.100
IP address of ISP: 192.168.0.1
IP address of LTE: 10.0.0.1

1. I kept DHCP setting in LTE and ISP modem settings just reduced the DHCP range from 10.0.0.100-10.0.0.100 for LTE and 192.168.0.11-192.168.0.12 for ISP (In ISP and LTE modem settings)
In Mikrotik router winbox: disabled both DHCP client and in IP-Address in Address line I set up:
for ISP modem: 192.168.0.11/24
for LTE modem: 10.0.0.100/24

Then I went to routes-add: set up Gateway:
ISP: 192.168.0.1
LTE: 10.0.0.1

Then went to DNS settings and Set up 8.8.8.8 and 8.8.4.4

And now it seems everything works fine, even the failover setup now works properly, if I restart LTE modem keeps the ISP connected! I will test today if it is stable!

jaclaz · Wed Oct 30, 2024 11:30 am

It seems just fine to me

in the sense that it should work, but is not really what I suggested.

The idea was that the manual addresses given to the Mikrotik interfaces should be outside the range of the addresses the DHCP of the ISP and LTE would lease, so that when/if you connect additionally another device (let's say a laptop) directly to one of the modems/routers (to another LAN port) and it gets an IP address from DHCP there is not the possibility of an IP conflict.

You seem to have setup the addresses so that a conflict/collision in those cases is guranteed.

(but not a problem unless in the case above)

The 8.8.4.4 is used as "canary" (or whatever you want to call it) address for the netwatch script, so there will be situations in which it goes to the blackhole route or however it is unreachable.
Not really an issue as it is the secondary DNS entry, so everything should go to 8.8.8.8 (which has - I believe an uptime of 99.999999999999...9%), but I would put a different DNS server (let's say 1.1.1.1 or 9.9.9.9) as secondary entry.

Who is online