RouterOS 7.5 - failover with recursive routing and PBR

peracchi · Mon Oct 03, 2022 7:50 pm

Hi everbody!

I have the following scenario:

network-diagram.png

I would like your help to get as close as possible to @Anav's "DEFACTO DEFAULT FIREWALL Setup" with failover using recursive routing and policy based routing (PBR).

I don´t know even if this is possible. I´m still learning...

My start point was the new documentation at https://help.mikrotik.com/docs/display/ ... figuration

Failover with recursive routing appears to be working. But I put gateway IP from ISP-2 "manually". I don´t know how to "automate" this task. At some point ISP-2 gateway IP will change.

Need to do PBR and whatever suggestions you have to improve this configuration.

Another thing I don´t fully understand is: using PBR will clients of the failed ISP automatically use the other? And what about clients that aren´t in any of the lists, for example a guest that gets an IP from DHCP pool? Which ISP they will use?

---

# make sure you have absolutely clean router
/system reset-configuration no-defaults=yes skip-backup=yes

# create a new admin account, remove default admin account, disconnect and connect with your new admin account
/user add name=<your_choosen_username> password=<password> group=full
/user remove admin

# rename interfaces for better visualization/understanding
/interface ethernet set [ find default-name=ether1 ] name=ether1_ISP-1
/interface ethernet set [ find default-name=ether2 ] name=ether2_ISP-2
/interface ethernet set [ find default-name=ether3 ] name=ether3_LAN-A
/interface ethernet set [ find default-name=ether4 ] name=ether4_LAN-B
/interface ethernet set [ find default-name=ether5 ] name=ether5_LAN-C

# some interface lists for better visualization/understanding/configuration
/interface list add name=WAN_list
/interface list member add list=WAN_list interface=ether1_ISP-1
/interface list member add list=WAN_list interface=ether2_ISP-2

/interface list add name=LAN_list
/interface list member add list=LAN_list interface=ether3_LAN-A
/interface list member add list=LAN_list interface=ether4_LAN-B
/interface list member add list=LAN_list interface=ether5_LAN-C

/interface list add name=trusted_list
/interface list member add list=trusted_list interface=ether3_LAN-A

# some protection and services configuration
/tool mac-server set allowed-interface-list=trusted_list
/tool mac-server mac-winbox set allowed-interface-list=trusted_list
/tool bandwidth-server set enabled=no
/ip neighbor discovery-settings set discover-interface-list=trusted_list
/ip service disable api,api-ssl,ftp,telnet,www
/ip service set ssh port=<port for SSH>
/ip service set winbox port=<port for WinBox>
/ip proxy set enabled=no
/ip socks set enabled=no
/ip upnp set enabled=no
/ip cloud set ddns-enabled=no update-time=no
/ip ssh set strong-crypto=yes


# set public static IP on ISP-1
/ip address add interface=ether1_ISP-1 address=a.b.c.d/24

# get public DHCP IP from ISP-2
/ip dhcp-client add interface=ether2_ISP-2 disabled=no add-default-route=no use-peer-dns=no use-peer-ntp=no

# set LAN IPs
/ip address add interface=ether3_LAN-A address=10.22.22.10/24 network=10.22.22.0
/ip address add interface=ether4_LAN-B address=172.16.1.172/24 network=172.16.1.0
/ip address add interface=ether5_LAN-C address=192.168.1.192/24 network=192.168.1.0

# disable sfp1 interface
/interface set sfp1 disabled=yes

# set dns
/ip dns set servers=1.1.1.1,8.8.8.8 allow-remote-requests=yes

# define pools for LAN DHCP
/ip pool add name=LAN-A_pool ranges=10.22.22.230-10.22.22.250
/ip pool add name=LAN-B_pool ranges=172.16.1.230-172.16.1.250
/ip pool add name=LAN-C_pool ranges=192.168.1.230-192.168.1.250

# define some static DHCP leases
/ip dhcp-server lease add address=10.22.22.1 mac-address=B8:27:EB:BD:59:19
/ip dhcp-server lease add address=10.22.22.2 mac-address=DC:A6:32:68:FD:5C
/ip dhcp-server lease add address=10.22.22.3 mac-address=DC:A6:32:03:51:6D

# configure DHCP servers
/ip dhcp-server add address-pool=LAN-A_pool interface=ether3_LAN-A lease-time=2d name=DHCP_for_LAN-A
/ip dhcp-server add address-pool=LAN-B_pool interface=ether4_LAN-B lease-time=2d name=DHCP_for_LAN-B
/ip dhcp-server add address-pool=LAN-C_pool interface=ether5_LAN-C lease-time=2d name=DHCP_for_LAN-C

/ip dhcp-server network add address=10.22.22.0/24 gateway=10.22.22.10 dns-server=10.22.22.10
/ip dhcp-server network add address=172.16.1.0/24 gateway=172.16.1.172 dns-server=172.16.1.172
/ip dhcp-server network add address=192.168.1.0/24 gateway=192.168.1.192 dns-server=192.168.1.192

# create ISP routing tables for PBR
/routing table add disabled=no fib name=route_to_ISP-1
/routing table add disabled=no fib name=route_to_ISP-2

# create some addresses lists
/ip firewall address-list add address=10.0.0.0/8 list=RFC1918
/ip firewall address-list add address=172.16.0.0/12 list=RFC1918
/ip firewall address-list add address=192.168.0.0/16 list=RFC1918

/ip firewall address-list add address=10.22.22.1  list=clients_to_ISP-1
/ip firewall address-list add address=10.22.22.2  list=clients_to_ISP-2
/ip firewall address-list add address=10.22.22.3  list=clients_to_ISP-2

# firewall stuff
/ip firewall filter add chain=input connection-state=established,related action=accept
/ip firewall filter add chain=input connection-state=invalid action=drop
/ip firewall filter add chain=input in-interface-list=WAN_list protocol=icmp action=accept
/ip firewall filter add chain=input in-interface-list=WAN_list protocol=tcp port=<port for SSH> action=accept
/ip firewall filter add chain=input in-interface-list=WAN_list protocol=tcp port=<port for WinBox> action=accept
/ip firewall filter add chain=input in-interface-list=WAN_list action=drop

# marks for PBR - disabled for now as I can't get it to work correctly
#/ip firewall mangle add action=mark-routing chain=prerouting dst-address-list=!RFC1918 in-interface-list=LAN_list new-routing-mark=route_to_ISP-1 passthrough=no src-address-list=clients_to_ISP-1
#/ip firewall mangle add action=mark-routing chain=prerouting dst-address-list=!RFC1918 in-interface-list=LAN_list new-routing-mark=route_to_ISP-2 passthrough=no src-address-list=clients_to_ISP-2

/ip firewall nat add chain=srcnat out-interface-list=WAN_list action=masquerade

/ip firewall filter add chain=forward action=fasttrack-connection connection-state=established,related
/ip firewall filter add chain=forward action=accept connection-state=established,related
/ip firewall filter add chain=forward action=drop connection-state=invalid
/ip firewall filter add chain=forward action=drop connection-state=new connection-nat-state=!dstnat in-interface-list=WAN_list

# failover with recursive routing
ip route add dst-address=1.0.0.1 scope=10 gateway=<ISP-1 gateway>
ip route add distance=5 gateway=1.0.0.1 check-gateway=ping target-scope=11

ip route add dst-address=8.8.4.4 scope=10 gateway=<ISP-2 gateway>
ip route add distance=7 gateway=8.8.4.4 check-gateway=ping target-scope=11

# ntp stuff
/system clock set time-zone-autodetect=no time-zone-name=<your time zone>
/system ntp client set enabled=yes
/system ntp client servers add address=<ntp server address>
/system ntp client servers add address=<ntp server address>

Sob · Mon Oct 03, 2022 8:02 pm

I didn't read it all, but DHCP gateway can be easily fixed using lease script:

viewtopic.php?p=899132#p899132

peracchi · Mon Oct 03, 2022 8:06 pm

I didn't read it all, but DHCP gateway can be easily fixed using lease script:

viewtopic.php?p=899132#p899132

Thanks @Sob, will read it right now.

anav · Mon Oct 03, 2022 8:48 pm

I have two dynamic WANIPs, the cable one autogets and puts my new gateway IP everytime.
The other fibre connection does not.

Thus in the DHCP client section (provided on an ISP vlan) is setup to do what you want. Oh and I set the default route number to 255 but dont remember why LOL. Its like a route to nowhere?
The other important thing to not are the highlighted wording. The router is searching on those key words on your IP routes. So they are added to my COMMENTS for each config line. The ones where the gateway IP of the ISP is noted. The reason I have to entries is due to checking the primary ISP with two external DNS entries for my dual wan setup.

:if ($bound=1) do={
:local iface $interface
:local gw [ /ip dhcp-client get [ find interface=$"iface" ] gateway ]
/ip route set [ find comment="PrimaryRecursive" gateway!=$gw ] gateway=$gw
/ip route set [ find comment="SecondaryRecursive" gateway!=$gw ] gateway=$gw
/tool e-mail send to="myinternet@provider.ca" subject=([/system identity
get name]) body=" This is your new gateway IP: $gw";
:local sub3 ([/system clock get time])
/tool fetch "https://api.telegram.org/botxxxxxxxx::C ... nged+WANIP"
:log info "Telegram notification sent VlanBell IP Changed"
}

anav · Mon Oct 03, 2022 8:59 pm

(1) Set this to NONE as should not be used for as access, not secure
/tool mac-server set allowed-interface-list=trusted_list

(2) Some errors and missing things in the config,
suggest you post it complete as an /export minus the serial # and any public WANIP info.

Also detail if your solely concerned about traffic originating on your LANS heading outbound? Was not clear.
Are you trying to ensure traffic originating externally on a WAN from a remote user, goes back out the same WAN?

If so are you running servers on the LANS? Do local users also access the LANS and if so how (direct via LANIP or via a dyndns name)?

It seems the ENTIRE LAN subnets, except two users on one of the LANS should go out ISP1 ???
What is your intent for failover, all ISP1 folks get moved to ISP2 if ISP1 is not available? and the one ISP2 user gets moved to ISP1 if ISP2 is not available?

peracchi · Mon Oct 03, 2022 10:12 pm

.

Also detail ...

.
Homelab here, no animals (or business) will be harmed!

I have more clients, left in config just three RPIs as example.

Goal is to have the two ISPs online, failover with recursive routing.

Clients not in any ISP list should use the "default ISP" (ISP route with lesser distance I suppose).

Clients on ISP lists should use his list ISP except on ISP fail; in that case, they should use the other ISP same way as clients not on any ISP list will do.

I use mainly LAN-A, 10.22.22.0/24.

The other LANs probably will be used for some learning experience. To connect a notebook on ether4 or ether5, to get an address by DHCP and try to learn some more tricks.

I'm not concerned with traffic going from LAN to WAN.

For learning reasons, would like to have LAN-A to LAN-B or LAN-C but not the reverse. LAN-A client can initiate communication with client on LAN-B or LAN-C but not the opposite.

Please let me know if scenario is not clear. I´m not fluent in english nor "networkish"!

I already incorporated @Sob help with ISP using DHCP.
.

suggest you post it complete as an /export minus the serial # and any public WANIP info

.
Export below:
.

# oct/03/2022 16:01:00 by RouterOS 7.5
# software id = TSXI-RLLZ
#
# model = RB760iGS
# serial number = XXXXXXXXXXXX
/interface ethernet
set [ find default-name=ether1 ] name=ether1_ISP-1
set [ find default-name=ether2 ] name=ether2_ISP-2
set [ find default-name=ether3 ] name=ether3_LAN-A
set [ find default-name=ether4 ] name=ether4_LAN-B
set [ find default-name=ether5 ] name=ether5_LAN-C
set [ find default-name=sfp1 ] disabled=yes
/interface list
add name=WAN_list
add name=LAN_list
add name=trusted_list
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=LAN-A_pool ranges=10.22.22.230-10.22.22.250
add name=LAN-B_pool ranges=172.16.1.230-172.16.1.250
add name=LAN-C_pool ranges=192.168.1.230-192.168.1.250
/ip dhcp-server
add address-pool=LAN-A_pool interface=ether3_LAN-A lease-time=2d name=DHCP_for_LAN-A
add address-pool=LAN-B_pool interface=ether4_LAN-B lease-time=2d name=DHCP_for_LAN-B
add address-pool=LAN-C_pool interface=ether5_LAN-C lease-time=2d name=DHCP_for_LAN-C
/port
set 0 name=serial0
/routing table
add disabled=no fib name=route_to_ISP-1
add disabled=no fib name=route_to_ISP-2
/ip neighbor discovery-settings
set discover-interface-list=trusted_list
/interface list member
add interface=ether1_ISP-1 list=WAN_list
add interface=ether2_ISP-2 list=WAN_list
add interface=ether3_LAN-A list=LAN_list
add interface=ether4_LAN-B list=LAN_list
add interface=ether5_LAN-C list=LAN_list
add interface=ether3_LAN-A list=trusted_list
/ip address
add address=a.b.c.d/24 interface=ether1_ISP-1 network=a.b.c.0
add address=10.22.22.10/24 interface=ether3_LAN-A network=10.22.22.0
add address=172.16.1.172/24 interface=ether4_LAN-B network=172.16.1.0
add address=192.168.1.192/24 interface=ether5_LAN-C network=192.168.1.0
/ip cloud
set update-time=no
/ip dhcp-client
add add-default-route=no interface=ether2_ISP-2 script=":if (\$bound=1) do={\r\
    \n  /ip route set [find where comment=\"ISP-2\"] gateway=\$\"gateway-address\" disabled=no\r\
    \n} else={\r\
    \n  /ip route set [find where comment=\"ISP-2\"] disabled=yes\r\
    \n}" use-peer-dns=no use-peer-ntp=no
/ip dhcp-server lease
add address=10.22.22.1 mac-address=B8:27:EB:BD:59:19
add address=10.22.22.2 mac-address=DC:A6:32:68:FD:5C
add address=10.22.22.3 mac-address=DC:A6:32:03:51:6D
/ip dhcp-server network
add address=10.22.22.0/24 dns-server=10.22.22.10 gateway=10.22.22.10
/ip dns
set allow-remote-requests=yes servers=1.1.1.1,8.8.8.8
/ip firewall address-list
add address=10.0.0.0/8 list=RFC1918
add address=172.16.0.0/12 list=RFC1918
add address=192.168.0.0/16 list=RFC1918
add address=10.22.22.1 list=clients_to_ISP-1
add address=10.22.22.2 list=clients_to_ISP-2
add address=10.22.22.3 list=clients_to_ISP-2
/ip firewall filter
add action=accept chain=input connection-state=established,related
add action=drop chain=input connection-state=invalid
add action=accept chain=input in-interface-list=WAN_list protocol=icmp
add action=accept chain=input in-interface-list=WAN_list port=1234 protocol=tcp
add action=accept chain=input in-interface-list=WAN_list port=5678 protocol=tcp
add action=drop chain=input in-interface-list=WAN_list
add action=fasttrack-connection chain=forward connection-state=established,related hw-offload=yes
add action=accept chain=forward connection-state=established,related
add action=drop chain=forward connection-state=invalid
add action=drop chain=forward connection-nat-state=!dstnat connection-state=new in-interface-list=WAN_list
/ip firewall nat
add action=masquerade chain=srcnat out-interface-list=WAN_list
/ip route
add dst-address=1.0.0.1 gateway=a.b.c.1 scope=10
add check-gateway=ping distance=7 gateway=1.0.0.1 target-scope=11
add comment=ISP-2 disabled=yes dst-address=8.8.4.4 gateway=1.2.3.4 scope=10
add check-gateway=ping distance=5 gateway=8.8.4.4 target-scope=11
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www disabled=yes
set ssh port=1234
set api disabled=yes
set winbox port=5678
set api-ssl disabled=yes
/ip ssh
set strong-crypto=yes
/system clock
set time-zone-autodetect=no time-zone-name=America/Sao_Paulo
/system ntp client
set enabled=yes
/system ntp client servers
add address=2001:12ff::8
add address=200.189.40.8
/tool bandwidth-server
set enabled=no
/tool mac-server
set allowed-interface-list=trusted_list
/tool mac-server mac-winbox
set allowed-interface-list=trusted_list

anav · Mon Oct 03, 2022 10:34 pm

Okay if you had a small number of users I would stick with routing rules, or if you could stick all those users into a subnet by themselves.
(how many exceptions in total are talking 3, 5,20?)

Otherwise if you have a mixed of IPs from two or more LANS that you want to assign via firewall address list, then mangling will be the method of choice.

What happens if ISP1 fails, do you want all users on ISP1 to also get to ISP2 ??

peracchi · Mon Oct 03, 2022 11:02 pm

24 MAC addresses fixed in DHCP (sometimes is hard to live with OCD

).

Eventually some guest equipament on Wi-Fi or some VMs for learning/testing.

Forgot LAN-B and LAN-C, think there is only 10.22.22.0/24 and two ISPs.

Some hardware should use (by default) the ISP with more speed (TVs, Chromecast, PS5, etc).

Other should use the slower but more reliable ISP that has static public IP (remote work, some server tests, etc).

But either case, with two ISPs I should not lost Internet connectivity if one of the two ISPs go down.

I know that I must use "mangle" to mark packets. But I done it wrong (look at the commented lines in my first post). I made something wrong but could not see what was.

What happens if ISP1 fails, do you want all users on ISP1 to also get to ISP2 ?

Yes.
And if ISP2 fails, all users on ISP2 should use ISP1.

anav · Mon Oct 03, 2022 11:25 pm

Something like......... Assuming vers 7 firmware..........

/ip route
add check-gateway=ping distance=3 dst-address=0.0.0.0/0 gateway=1.0.0.1 scope=10 target-scope=12 table=main
add distance=3 dst-address=1.0.0.1/32 gateway=PrimaryISP-gatewayIP scope=10 target-scope=11 table=main
..............................................................................................
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30 table=main
add comment=SecondaryISP distance=10 dst-address=0.0.0.0/0 gateway=SecondaryISP-gatewayIP scope=10 target-scope=30 table=ISP2_route

Where the following is the mangling required....... Note all users will go out WAN1, and only designated users on firewall address list will go out WAN2.
/ip firewall mangle
add chain=prerouting src-address-list=clients_to_ISP-2 action=mark-routing new-routing-mark=ISP2_route

With this setup, all users should be sent out WAN1 based on distance alone, except for those on firewall address list marked for exit via WAN2.
WAN1 is checked against a known DNS server, to ensure the WWW is actually available.

BUT ******* The above assumes you have no traffic coming in on the WAN side originating from external users and going to one of the IPs that is supposed to go out WAN2 ????

peracchi · Thu Oct 06, 2022 3:10 am

Hi everbody!

Reading a lot these days, learning some new things but cannot practice too much because of angry wife when Internet is not working at home!

I think I achieved a satisfactory stage, PBR is working fine.

If I disconnect ethernet cable or cut power of one of the ISPs equipment at home, the other handles all traffic. So, clients that prefer ISP-1 go automatically to ISP-2 and vice-versa.

Would like to achieve recursive routing with this setup because the more frequent problem with Internet is something at ISP level (not at home).

If someone wants to try my config just edit some <vars_on_code>, do a big copy and paste and voilà!

# Policy Based Routing with failover on RouterOS v7.5
#
# one ISP with public static IP and other with public dynamic IP (DHCP)
#
# work in progress...
#
# don´t know why things go bad when I enable "fasttrack-connection" on chain forward
#
# I still wants to add recursive routing as method to know when ISPs lost Internet connection
#
# make sure you have absolutely clean router before run all commands below
#
# you can use "/system reset-configuration no-defaults=yes skip-backup=yes" to start from scratch



# rename ethernet interfaces for better visualization/understanding
/interface ethernet
set [ find default-name=ether1 ] name=ether1_ISP-1
set [ find default-name=ether2 ] name=ether2_ISP-2
set [ find default-name=ether3 ] name=ether3_LAN-A
set [ find default-name=ether4 ] name=ether4_LAN-B
set [ find default-name=ether5 ] name=ether5_LAN-C
set [ find default-name=sfp1 ] disabled=yes

# create interface lists
/interface list
add name=WAN_list
add name=LAN_list
add name=trusted_list

# populate interface lists
/interface list member
add interface=ether1_ISP-1 list=WAN_list
add interface=ether2_ISP-2 list=WAN_list
add interface=ether3_LAN-A list=LAN_list
add interface=ether4_LAN-B list=LAN_list
add interface=ether5_LAN-C list=LAN_list
add interface=ether3_LAN-A list=trusted_list

# some reccomended security/optimization settings
/tool bandwidth-server set enabled=no
/tool mac-server set allowed-interface-list=trusted_list
/tool mac-server mac-winbox set allowed-interface-list=trusted_list
/ip neighbor discovery-settings set discover-interface-list=trusted_list
/ip cloud set ddns-enabled=no update-time=no
/ip proxy set enabled=no
/ip socks set enabled=no
/ip upnp set enabled=no
/ip service disable api,api-ssl,ftp,telnet,www
/ip service set ssh port=<your_chosen_ssh_port>
/ip service set winbox port=<your_chosen_winbox_port>
/ip ssh set strong-crypto=yes

# set your static ISP-1 IP address
/ip address
add interface=ether1_ISP-1 address=<your_static_ISP-1_IP_address> network=<your_static_ISP-1_network>

# get your dynamic ISP-2 IP address using DHCP - script will update or disable routes as needed
/ip dhcp-client
add interface=ether2_ISP-2 script=":if (\$bound=1) do={\
    \r\
    \n  :foreach i in [/ip route find] do={\r\
    \n    :if (\".\" . [:find [/ip route get \$i comment] \"ISP-2\"] . \".\"!\
    =\"..\") do={\r\
    \n      /ip route set \$i gateway=\$\"gateway-address\" disabled=no\r\
    \n    }\r\
    \n  }\r\
    \n} else={\r\
    \n  :foreach i in [/ip route find] do={\r\
    \n    :if (\".\" . [:find [/ip route get \$i comment] \"ISP-2\"] . \".\"!\
    =\"..\") do={\r\
    \n      /ip route set \$i disabled=yes\r\
    \n    }\r\
    \n  }\r\
    \n}" add-default-route=no use-peer-dns=no use-peer-ntp=no

# each one of the three remaining ethernet interfaces gets a different intranet address/range
# I mainly use 10.22.22.0/24 network, other interfaces/networks only configured for learning/lab purposes
/ip address
add interface=ether3_LAN-A address=10.22.22.10/24 network=10.22.22.0
add interface=ether4_LAN-B address=172.16.1.172/24 network=172.16.1.0
add interface=ether5_LAN-C address=192.168.1.192/24 network=192.168.1.0

# create pool ranges
/ip pool
add name=LAN-A_pool ranges=10.22.22.230-10.22.22.250
add name=LAN-B_pool ranges=172.16.1.230-172.16.1.250
add name=LAN-C_pool ranges=192.168.1.230-192.168.1.250

# create DHCP servers
/ip dhcp-server
add address-pool=LAN-A_pool interface=ether3_LAN-A lease-time=2d name=DHCP_for_LAN-A
add address-pool=LAN-B_pool interface=ether4_LAN-B lease-time=2d name=DHCP_for_LAN-B
add address-pool=LAN-C_pool interface=ether5_LAN-C lease-time=2d name=DHCP_for_LAN-C

# define some static IPs on DHCP server (used later for PBR)
/ip dhcp-server lease
add address=10.22.22.1 mac-address=<put_your_device_mac_address_here>
add address=10.22.22.2 mac-address=<put_your_device_mac_address_here>
add address=10.22.22.3 mac-address=<put_your_device_mac_address_here>
add address=10.22.22.4 mac-address=<put_your_device_mac_address_here>

# some more DHCP servers definitions
/ip dhcp-server network
add address=10.22.22.0/24  dns-server=10.22.22.10   gateway=10.22.22.10
add address=172.16.1.0/24  dns-server=172.16.1.172  gateway=172.16.1.172
add address=192.168.1.0/24 dns-server=192.168.1.192 gateway=192.168.1.192

# configure DNS server on MikroTik
/ip dns set allow-remote-requests=yes servers=1.1.1.1,8.8.8.8

# addresses that will prefer ISP-1
/ip firewall address-list
add address=10.22.22.1 list=prefer_ISP-1
add address=10.22.22.3 list=prefer_ISP-1

# addresses that will prefer ISP-2
/ip firewall address-list
add address=10.22.22.2 list=prefer_ISP-2
add address=10.22.22.4 list=prefer_ISP-2

# addresses that should not be/go on/to the Internet
/ip firewall address-list
add address=0.0.0.0/8       comment=RFC1122       list=not_in_internet
add address=10.0.0.0/8      comment=RFC1918       list=not_in_internet
add address=100.64.0.0/10   comment=RFC6598       list=not_in_internet
add address=127.0.0.0/8     comment=RFC1122       list=not_in_internet
add address=169.254.0.0/16  comment=RFC3927       list=not_in_internet
add address=172.16.0.0/12   comment=RFC1918       list=not_in_internet
add address=192.0.0.0/24    comment=RFC6890       list=not_in_internet
add address=192.0.2.0/24    comment=RFC5737       list=not_in_internet
add address=192.88.99.0/24  comment=RFC3068       list=not_in_internet
add address=192.168.0.0/16  comment=RFC1918       list=not_in_internet
add address=198.18.0.0/15   comment=RFC2544       list=not_in_internet
add address=198.51.100.0/24 comment=RFC5737       list=not_in_internet
add address=203.0.113.0/24  comment=RFC5737       list=not_in_internet
add address=224.0.0.0/4     comment=RFC1112       list=not_in_internet
add address=233.252.0.0/24  comment=Documentation list=not_in_internet
add address=240.0.0.0/4     comment=RFC6890       list=not_in_internet

# very basic and simple firewall
/ip firewall filter
add action=accept chain=input connection-state=established,related
add action=drop   chain=input connection-state=invalid
add action=accept chain=input in-interface-list=WAN_list protocol=icmp
add action=accept chain=input in-interface-list=WAN_list port=<your_chosen_ssh_port> protocol=tcp
add action=accept chain=input in-interface-list=WAN_list port=<your_chosen_winbox_port> protocol=tcp
add action=drop   chain=input in-interface-list=WAN_list

# why things go bad if I enable "fasttrack-connection" on chain forward???
/ip firewall filter add disabled=yes action=fasttrack-connection chain=forward connection-state=established,related
/ip firewall filter
add action=accept chain=forward connection-state=established,related
add action=drop   chain=forward connection-state=invalid
add action=drop   chain=forward connection-nat-state=!dstnat connection-state=new in-interface-list=WAN_list

# create routing tables for ISPs
/routing table
add disabled=no fib name=prefer_ISP-1
add disabled=no fib name=prefer_ISP-2

# mark packets according to the addresses lists
/ip firewall mangle
add action=mark-routing chain=prerouting dst-address-list=!not_in_internet in-interface-list=LAN_list new-routing-mark=prefer_ISP-1 passthrough=no src-address-list=prefer_ISP-1
add action=mark-routing chain=prerouting dst-address-list=!not_in_internet in-interface-list=LAN_list new-routing-mark=prefer_ISP-2 passthrough=no src-address-list=prefer_ISP-2

# masquerade as needed
/ip firewall nat add action=masquerade chain=srcnat out-interface-list=WAN_list

# default routes for devices not in any ISP preference list
/ip route add routing-table=main comment=ISP-1 distance=1 dst-address=0.0.0.0/0 gateway=<ISP-1_gateway_IP_address> scope=30 target-scope=10
/ip route add routing-table=main comment=ISP-2 distance=2 dst-address=0.0.0.0/0 gateway=<any_IP_script_will_adjust_or_disable> scope=30 target-scope=10

# routes for devices that prefer ISP-1
/ip route add routing-table=prefer_ISP-1 comment=ISP-1 distance=1 dst-address=0.0.0.0/0 gateway=<ISP-1_gateway_IP_address> scope=30 target-scope=10
/ip route add routing-table=prefer_ISP-1 comment=ISP-2 distance=2 dst-address=0.0.0.0/0 gateway=<any_IP_script_will_adjust_or_disable> scope=30 target-scope=10

# routes for devices that prefer ISP-2
/ip route add routing-table=prefer_ISP-2 comment=ISP-2 distance=1 dst-address=0.0.0.0/0 gateway=<any_IP_script_will_adjust_or_disable> scope=30 target-scope=10
/ip route add routing-table=prefer_ISP-2 comment=ISP-1 distance=2 dst-address=0.0.0.0/0 gateway=<ISP-1_gateway_IP_address> scope=30 target-scope=10

# ntp stuff
/system clock set time-zone-autodetect=no time-zone-name=America/Sao_Paulo
/system ntp client set enabled=yes
/system ntp client servers add address=2001:12ff::8
/system ntp client servers add address=200.189.40.8

# clear log, command history and reboot
/system logging action set memory memory-lines=1
/system logging action set memory memory-lines=1000
/console clear-history
/system reboot

Scenario picture again to facilitate to someone that jumps direct to this post.

network-diagram.png

RouterOS 7.5 - failover with recursive routing and PBR

RouterOS 7.5 - failover with recursive routing and PBR

Re: RouterOS 7.5 - failover with recursive routing and PBR

Re: RouterOS 7.5 - failover with recursive routing and PBR

Re: RouterOS 7.5 - failover with recursive routing and PBR

Re: RouterOS 7.5 - failover with recursive routing and PBR

Re: RouterOS 7.5 - failover with recursive routing and PBR

Re: RouterOS 7.5 - failover with recursive routing and PBR

Re: RouterOS 7.5 - failover with recursive routing and PBR

Re: RouterOS 7.5 - failover with recursive routing and PBR

Re: RouterOS 7.5 - failover with recursive routing and PBR