Community discussions

MikroTik App
 
zentavr
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 53
Joined: Tue Nov 05, 2013 2:11 pm

Multiwan setup with Starlink and ip/route check-gateway issue

Wed Oct 25, 2023 5:49 am

Hi,

I'm trying to set up Multi WAN mikrotik router where the second ISP is Starlink with ethernet adapter. The issue is that Starlink provides the default gateway 100.64.0.1 (mac address `00:00:5E:00:01:01`) which does not reply at icmp requests. I guess that they use some kind of a load balancing/or switching when the sattelite is passing the region.

The setup looks like this:
# Adding local subnet to the main table
/routing rule
add action=lookup disabled=no dst-address=172.16.46.0/24 table=main

/routing table
add disabled=no fib name=rtab-wan
add disabled=no fib name=rtab-sat

# Marking the incoming connections from every ISP
/ip firewall mangle
add action=mark-connection chain=prerouting comment=WAN-Routing-Mark connection-mark=no-mark in-interface=bridge_wan new-connection-mark=from-wan-internet passthrough=no
add action=mark-connection chain=prerouting comment=SAT-Routing-Mark connection-mark=no-mark in-interface=bridge_sat new-connection-mark=from-sat-internet passthrough=no

## Outgoing transit traffic rules
add action=mark-routing chain=prerouting comment="WAN Outgoing transit traffic to rtab-wan Routing Table " connection-mark=from-wan-internet dst-address-type=!local in-interface-list=!WAN new-routing-mark=rtab-wan passthrough=no
add action=mark-routing chain=prerouting comment="SAT Outgoing transit traffic to rtab-sat Routing Table " connection-mark=from-sat-internet dst-address-type=!local in-interface-list=!WAN new-routing-mark=rtab-sat passthrough=no

## Outgoing local traffic rules
add action=mark-routing chain=output comment="WAN Outgoing Local traffic to rtab-wan Routing Table " connection-mark=from-wan-internet dst-address-type=!local new-routing-mark=rtab-wan passthrough=no
add action=mark-routing chain=output comment="SAT Outgoing Local traffic to rtab-sat Routing Table " connection-mark=from-sat-internet dst-address-type=!local new-routing-mark=rtab-sat passthrough=no

## Dedicated Routing rules per address lists
add action=mark-routing chain=prerouting comment="Users routed via WAN" dst-address-list=!BOGONS new-routing-mark=rtab-wan passthrough=yes src-address-list=Routed-via-WAN place-before=0
add action=mark-routing chain=prerouting comment="Users routed via SAT" dst-address-list=!BOGONS new-routing-mark=rtab-sat passthrough=yes src-address-list=Routed-via-SAT place-before=0

# Adding emergency default route
/interface bridge add name=br-lo comment="Loopback Routing Interface"
/ip route add distance=254 gateway=br-lo comment="Emergency route"

# Adding the route for the 1st testing IP via WAN
/ip route
add check-gateway=ping comment="For recursion via WAN" distance=1 dst-address=4.2.2.1 gateway=80.92.227.42 scope=11
add check-gateway=ping comment="Unmarked via WAN" distance=1 gateway=4.2.2.1 target-scope=11
add comment="Marked via WAN" distance=1 gateway=4.2.2.1 routing-table=rtab-wan target-scope=11
add comment="Marked via SAT" distance=2 gateway=4.2.2.1 routing-table=rtab-sat target-scope=11

# Route rule from local traffic to internet (WAN)
/routing/rule/add action=lookup comment="From WAN IP to Inet" src-address=8.9.7.41 table=rtab-wan
The DHCP client for starlink bridge is done like this:
/ip dhcp-client add add-default-route=no disabled=no interface=bridge-sat
Also there is a script attached to DHCP clients for `bridge-sat`:
:local ispThis "SAT"
:local ispThisRt "rtab-sat"
:local ispThisGwCheck "arp"
:local ispThisMainRtDistance "2"

:local ispMain "WAN"
:local ispBackup "SAT"

:local checkIp "4.2.2.2"
:local ispMainRt "rtab-wan"
:local ispBackupRt "rtab-sat"

#:local postScript "some-script"

:log info "$ispThis: Bound: $bound, Gateway: $"gateway-address""

:if ($bound=1) do={
    :log info "$ispThis: Removing the old route records"
    /ip route remove [ find gateway="$checkIp" ]
    /ip route remove [ find where dst-address ~"$checkIp" ]

    :log info "$ispThis: Adding actual route records"
    /ip route add check-gateway=$ispThisGwCheck comment="For recursion via $ispThis" distance=1 dst-address=$checkIp gateway=$"gateway-address" scope=11
    /ip route add check-gateway=ping comment="Unmarked via $ispThis" distance=$ispThisMainRtDistance gateway=$checkIp target-scope=11
    /ip route add comment="Marked via $ispBackup" distance=1 gateway=$checkIp routing-table=$ispBackupRt target-scope=11
    /ip route add comment="Marked via $ispMain" distance=2 gateway=$checkIp routing-table=$ispMainRt target-scope=11

    :log info "$ispThis: Setting NAT"
    :if [:tobool ([/ip firewall/nat/ find comment="NAT via $ispThis"])] do={
        :log info "$ispThis:     .... rule had been updated"
        /ip firewall nat set [find comment="NAT via $ispThis"] action=src-nat chain=srcnat ipsec-policy=out,none out-interface=$"interface" to-addresses=$"lease-address"
    } else={
        :log info "$ispThis:     .... rule had been added"
        /ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none out-interface=$"interface" to-addresses=$"lease-address" comment="NAT via $ispThis"
    }

    :log info "$ispThis: Setting routing rules"
    :if [:tobool ([/routing/rule find comment="From $ISP IP to Inet"])] do={
        :log info "$ispThis:     .... rule had been updated"
        /routing/rule/set [find comment="From $ispThis IP to Inet"] action=lookup src-address=$"lease-address" table=$ispThisRt
    } else={
        :log info "$ispThis:     .... rule had been added"
        /routing/rule/add action=lookup comment="From $ispThis IP to Inet" src-address=$"lease-address" table=$ispThisRt
    }

    #:log info "$ispThis: Runnit the post action script"
    #/system script run $postScript

    :log info "$ispThis: Activation is done!"
} else={
    :log info "$ispThis: Removing route records"
    /ip route remove [ find gateway="$checkIp" ]
    /ip route remove [ find where dst-address ~"$checkIp" ]

    :log info "$ispThis: Removing firewall NAT"
    /ip firewall nat remove  [find comment="NAT via $ispThis"]

    :log info "$ispThis: Removing routing rules"
    /routing/rule/remove [find comment="From $ispThis IP to Inet"]

    :log info "$ispThis: Cleaning is done!"
}
The problem is that "check-gateway" feature when there is "ping" value not works (because the Starling gateway not accepts ICMPs). I also tried ARP - that somehow not work as well, but I can see the mac-address of the gateway at IP-ARP via Winbox. Could it be the issue because the gateway is somewhere at the space and the RouterOS times out the arp response? Doing arp ping from the routeros console works as well.

How can I check that starlink is up using Mikrotik's native tools?
Знімок екрана 2023-10-25 о 05.39.47.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 22223
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Multiwan setup with Starlink and ip/route check-gateway issue

Wed Oct 25, 2023 2:55 pm

Could care less about the config because more important are the requirements.
What is WAn1 and what is WAn2 is it primary and failover? Is it PCC load balancing.
What are the expectations if one of the WANs is not available.
What are the expectations for lan users, some to wan1 and some to wan2
Any VPNs??

/export file=anynameyouwish ( minus router serial # and any public WANIP information )
 
User avatar
jvanhambelgium
Forum Guru
Forum Guru
Posts: 1120
Joined: Thu Jul 14, 2016 9:29 pm
Location: Belgium

Re: Multiwan setup with Starlink and ip/route check-gateway issue

Wed Oct 25, 2023 5:29 pm

Put the Starlink in a separate VRF and work from there?
You could issue some health-check to eg. 8.8.8.8 across the Starlink-vrf and make some decisions from there?
 
User avatar
Amm0
Forum Guru
Forum Guru
Posts: 4459
Joined: Sun May 01, 2016 7:12 pm
Location: California
Contact:

Re: Multiwan setup with Starlink and ip/route check-gateway issue

Wed Oct 25, 2023 7:32 pm

I'd think the check-gateway=ping on the 4.2.2.2 is enough with the recursive routing. e.g. check-gateway likely doesn't be on the starlink interface itself. The CGNAT will drop if there is no starlink & 4.2.2.2 recursive route is already checking internet connectivity.

FWIW, the OP's 4.2.2.2 is one of the oldest DNS public servers ... so it's same as 8.8.8.8 used in other recursive routing configs - there a few of them conveniently sequential 4.2.2.1 ... 4.2.2.6, e.g. 4.2.2.1 is ISP1 canary, 4.2.2.2 is ISP2 canary & n.b. using 8.8.8.8 as canary causes anyone using Google DNS to go out only one ISP, while 4.2.2.1...6 are highly unlikely to be used as DNS servers these days
 
zentavr
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 53
Joined: Tue Nov 05, 2013 2:11 pm

Re: Multiwan setup with Starlink and ip/route check-gateway issue

Wed Oct 25, 2023 11:20 pm

Could care less about the config because more important are the requirements.
What is WAn1 and what is WAn2 is it primary and failover?
WAN is the PON connection which might be down because of electricity power loss. Starlink is the backup channel (we have Ecoflow battery for this kind of power outgages).

Is it PCC load balancing.
It is not a PCC load balancing (we actually did not thought about that feature yet :))

What are the expectations if one of the WANs is not available.
So either one or another WAN Provider should handle the internet connections. The fiber channel is preferable if available.

What are the expectations for lan users, some to wan1 and some to wan2
Actually the main requirement the users would be able to connect to the internet (just office stuff like skype, MS Teams, Slack, Zoom, etc...)

Any VPNs??
Mikrotik is an IPSec + OpenVPN server at the WAN1 IS (PON Fiber). It's ok that be unreachable when the main fiber ISP is down.
 
zentavr
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 53
Joined: Tue Nov 05, 2013 2:11 pm

Re: Multiwan setup with Starlink and ip/route check-gateway issue

Wed Oct 25, 2023 11:21 pm

Put the Starlink in a separate VRF and work from there?
You could issue some health-check to eg. 8.8.8.8 across the Starlink-vrf and make some decisions from there?
Frankly speaking I hadn't worked with VRF at all. Would you be so kind to submit any recommendations to start with?
 
zentavr
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 53
Joined: Tue Nov 05, 2013 2:11 pm

Re: Multiwan setup with Starlink and ip/route check-gateway issue

Wed Oct 25, 2023 11:59 pm

I'd think the check-gateway=ping on the 4.2.2.2 is enough with the recursive routing. e.g. check-gateway likely doesn't be on the starlink interface itself. The CGNAT will drop if there is no starlink & 4.2.2.2 recursive route is already checking internet connectivity.

As for the check-gateway:
[zentavr@zentavr-mt-rv] /ip/route> export 
# 2023-10-25 23:22:44 by RouterOS 7.11.2
/ip route
# ...
# This "check-gateway" works pretty well. Either "arp" or "ping" works here
add check-gateway=arp comment="For recursion via WAN" distance=1 dst-address=4.2.2.1/32 gateway=A.B.C.65 routing-table=main scope=11
# ...
# This "check-gateway" does not work at all. "arp" works some time and really flappy, but or "ping" does not work at all. The gateway is marked as "unreachable" in 20 seconds.
add check-gateway=none comment="For recursion via SAT" distance=1 dst-address=4.2.2.2 gateway=100.64.0.1 scope=11
...
FWIW, the OP's 4.2.2.2 is one of the oldest DNS public servers ... so it's same as 8.8.8.8 used in other recursive routing configs - there a few of them conveniently sequential 4.2.2.1 ... 4.2.2.6, e.g. 4.2.2.1 is ISP1 canary, 4.2.2.2 is ISP2 canary & n.b. using 8.8.8.8 as canary causes anyone using Google DNS to go out only one ISP, while 4.2.2.1...6 are highly unlikely to be used as DNS servers these days
These IPs had been selected just because of high SLA.