Connecting to either of 2 WAN links at same time

Tal · Tue Sep 01, 2015 5:31 pm

I have a 493 connected to 2 WAN links. This 493 will be located on a remote site.
I need to be able to access the 493 for management purposes (through winbox) through either public IP.

I setup a route with 2 gateways like this:

/ip route add gateway=1.1.1.1,2.2.2.2 dst-address=9.9.9.9

1.1.1.1 - the gateway for WAN link 1
2.2.2.2 - the gateway for WAN link 2
9.9.9.9 - our office IP - the IP I need to manage the device from

Note: Regular internet usage for customers behind that router is handled by other routes - not concerned about that here

When both WAN links are available, this route works great - I can connect to the device through winbox using either public IP.
When one of the WAN links is not available however, it seems to work about half the time.

Ex.

If I have WAN link 1 down, half the time I'll be able to access the device using the public IP of WAN link 2 using winbox, and a minute or two later I won't be able to.

Any reason it's so flaky? Do I need to use policy based routing to get the desired result, or is there a simple fix for the above configuration?

PS. I know it's fairly easy to configure a route to ping its gateway automatically and fail over to a secondary route over the second WAN link if it fails, but for various complicated reasons, that won't work in my case - I just need to know why the above isn't working.

JJCinAZ · Tue Sep 01, 2015 5:52 pm

When one of the WAN links is not available however, it seems to work about half the time.

That means it is doing exactly what you told it to do. You have effectively setup ECMP routing. Half the time, the reply packets to you are going through WAN1 and the other half, WAN2. Once that decision is made, that route is cached for a time.
What you need to add is some configuration to mark new connections arriving on WAN1 to only be replied to via WAN1 and new connections arriving on WAN2 to only be replied to via WAN2. I won't go over the details here, because quite a few people have created a number of examples in the Wiki and there have been a number of presentations on this at past MUM's. Try searching for "two wan" in the Wiki.

Tal · Tue Sep 01, 2015 6:07 pm

When one of the WAN links is not available however, it seems to work about half the time.
That means it is doing exactly what you told it to do. You have effectively setup ECMP routing. Half the time, the reply packets to you are going through WAN1 and the other half, WAN2. Once that decision is made, that route is cached for a time.
What you need to add is some configuration to mark new connections arriving on WAN1 to only be replied to via WAN1 and new connections arriving on WAN2 to only be replied to via WAN2. I won't go over the details here, because quite a few people have created a number of examples in the Wiki and there have been a number of presentations on this at past MUM's. Try searching for "two wan" in the Wiki.

I was hoping that it was smart enough to do that automatically - whatever IP/interface the WAN packets come in on is the one it leaves on. Guess not.

Yup - I've been reading some posts on dual WAN setups, and they all seem to get into policy based routing - I was hoping the way I have it above was a simple solution. Sounds like I've got some reading to do.

JJCinAZ · Tue Sep 01, 2015 7:05 pm

You don't really want it to do that automatically -- that would reduce your flexibility in the future. Really all you're going to do is mark the new connections coming in on WAN1 and WAN2 and then you're going to look for that Connection Mark and, based on that, you'll be adding Routing Marks to outgoing packets. Then you'll some routes which use those same Routing Marks to route the packets back out WAN1 or WAN2.

That concept of marking the connection and then dealing with the bidirectional traffic of that connection is used for all sorts of control in RouterOS -- queuing, filtering, NAT'ing, QoS, etc. Those "magic" boxes which automatically deal with dual WAN connectivity aren't really magic. They're usually just Linux-based devices where someone figured out the whole connection tracking and marking abilities of Linux and put a UI on it. Once you grok connection tracking and marking, all sorts of magic are available to you too.

Tal · Thu Sep 03, 2015 4:24 pm

Really all you're going to do is mark the new connections coming in on WAN1 and WAN2 and then you're going to look for that Connection Mark and, based on that, you'll be adding Routing Marks to outgoing packets. Then you'll some routes which use those same Routing Marks to route the packets back out WAN1 or WAN2.

This is what I've tried, with explanations as I understand them:

Mark each connection coming in on WAN1 or WAN2 in both directions, to keep track of if it came from WAN1 or WAN2:

/ip firewall mangle
add action=mark-connection chain=prerouting in-interface=WAN1-Interface new-connection-mark=WAN1_conn src-address=9.9.9.9
add action=mark-connection chain=prerouting in-interface=WAN2-Interface new-connection-mark=WAN2_conn src-address=9.9.9.9

Mark just the returning traffic:

/ip firewall mangle
add action=mark-routing chain=output connection-mark=WAN1_conn dst-address=9.9.9.9 new-routing-mark=to_WAN1
add action=mark-routing chain=output connection-mark=WAN2_conn dst-address=9.9.9.9 new-routing-mark=to_WAN2

Setup 2 routes in the routing table. These routes have a route marking, so normal traffic ignores these routes:

/ip route
add distance=1 dst-address=9.9.9.9/32 gateway=1.1.1.1 routing-mark=to_WAN1
add distance=1 dst-address=9.9.9.9/32 gateway=2.2.2.2 routing-mark=to_WAN2

Setup a routing rule that makes traffic with a routing mark use the appropriate routing table:

/ip route rule
add dst-address=9.9.9.9/32 routing-mark=to_WAN1 table=to_WAN1
add dst-address=9.9.9.9/32 routing-mark=to_WAN2 table=to_WAN2

Seems like it should work, but it doesn't. Anyone see a flaw in either my understanding of what a particular step is doing, or what I'm trying to accomplish?

Tal · Thu Sep 03, 2015 7:14 pm

Actually, it seems like if I mark the connections, and then mark the outgoing packets with a routing mark, they automatically get sent through the routes with that same routing mark.

That basically means all I need is:

/ip firewall mangle
add action=mark-connection chain=prerouting in-interface=WAN1-Interface new-connection-mark=WAN1_conn src-address=9.9.9.9
add action=mark-connection chain=prerouting in-interface=WAN2-Interface new-connection-mark=WAN2_conn src-address=9.9.9.9

/ip firewall mangle
add action=mark-routing chain=output connection-mark=WAN1_conn dst-address=9.9.9.9 new-routing-mark=to_WAN1
add action=mark-routing chain=output connection-mark=WAN2_conn dst-address=9.9.9.9 new-routing-mark=to_WAN2

/ip route
add distance=1 dst-address=9.9.9.9/32 gateway=1.1.1.1 routing-mark=to_WAN1
add distance=1 dst-address=9.9.9.9/32 gateway=2.2.2.2 routing-mark=to_WAN2

This lets me connect using either WAN IP when both links are up - I just need to test if it still works when one of the WAN links is down.

JJCinAZ · Thu Sep 03, 2015 8:23 pm

Yes, I don't normally use the routing rules to accomplish that. Normally, I use the "main" routing table for WAN1 and the "WAN2" routing table for WAN2, I would check for new connections only, I'd mark routes for traffic passing through to and from the LAN, and I wouldn't have the source address. Also, if you want your LAN hosts to be able to communicate using WAN2 (via a DST-NAT), you'll need to add your LAN route to the WAN2 routing table. Your config would change to the following in such a case:

/ip firewall mangle
add action=mark-connection chain=prerouting in-interface=WAN1-Interface connection-mark=no-mark new-connection-mark=WAN1_conn
add action=mark-connection chain=prerouting in-interface=WAN2-Interface connection-mark=no-mark new-connection-mark=WAN2_conn
add action=mark-routing chain=output connection-mark=WAN2_conn new-routing-mark=WAN2
add action=mark-routing chain=prerouting connection-mark=WAN2_conn new-routing-mark=WAN2 comment="forwarded traffic"

/ip route
add distance=1 dst-address=0.0.0.0/0 gateway=2.2.2.2 routing-mark=WAN2
add dst-address=<lan-subnet> gateway=<lan-intf> routing-mark=WAN2

If you want the LAN to be able to work through the second WAN, you either need an entry in the WAN2 routing table for the LAN or you would need some routing rules (the rules are more complex and more sensitive to different versions in my experience).

Tal · Thu Sep 03, 2015 8:37 pm

My previous post seems to work

For LAN, I have a single default route in the main routing table.

There's a script I have that uses a separate routing table to check if the 2 WAN links are up. If both are up, it uses the one I set as the primary one. If only one is up, it will delete the current default route in the main routing table and add a default route that points to the gateway of the WAN link which is up. Works great.

While I am happy with this approach for LAN, I thought it would be nice to be able to connect to the device using winbox through either one of its 2 WAN IPs - not through whatever IP happens to be in use at the time. Seems that it all works now. I'll have to do some more through testing to be sure I didn't miss anything, but right now, seems to be perfect.

Thanks for the help.

Connecting to either of 2 WAN links at same time

Connecting to either of 2 WAN links at same time

Re: Connecting to either of 2 WAN links at same time

Re: Connecting to either of 2 WAN links at same time

Re: Connecting to either of 2 WAN links at same time

Re: Connecting to either of 2 WAN links at same time

Re: Connecting to either of 2 WAN links at same time

Re: Connecting to either of 2 WAN links at same time

Re: Connecting to either of 2 WAN links at same time