We added some additional information and renamed one of the Wiki topic about ECMP:
http://wiki.mikrotik.com/wiki/ECMP_load ... masquerade

Some users complained about connection interruptions every 10-30min. After looking deeper into problem we were able to determine that Linux Kernel initiates a routing table flush every 10min. So after that connections are assigned to gateways once again and may or may not be on the same gateway.

If you have fully routed network (clients address can be routed via all available gateway), change of the gateway will have no ill effect, but in case you use masquerade, change of the gateway will result in change of the packet's source address and connection will be dropped.

This flush was introduced by Linux Kernel developers to eliminate possibility of DoS attacks on your routers.

So ... currently it's not really possible to make a fully working ECMP setup with Masquerade due to a Kernel design issue. As an alternative you can use this: http://wiki.mikrotik.com/wiki/NTH_load_ ... masquerade

Thank you for the update!

Will be testing it soon.

One question:

How will we modify this for a 3 or more WAN link scenario?

The reason I ask is that most of my sites have 3 or more Internet links per MT box.

Kind Regards,
Arno

Hi Normis, thank for the information re ECMP. I have changed my ECMP config to NTH load balancing with masquerade as per the Wiki. Everything seems to work fine except that the load balancing is very skewed towards the "odd" connection.

Below find the config that I have implemented on ROS3.22:

/ip firewall mangle
add action=mark-connection chain=prerouting comment="" disabled=no in-interface=local new-connection-mark=odd passthrough=yes src-address-list=odd
add action=mark-routing chain=prerouting comment="" disabled=no in-interface=local new-routing-mark=odd passthrough=no src-address-list=odd
add action=mark-connection chain=prerouting comment="" disabled=no in-interface=local new-connection-mark=even passthrough=yes src-address-list=even
add action=mark-routing chain=prerouting comment="" disabled=no in-interface=local new-routing-mark=even passthrough=no src-address-list=even
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=odd nth=2,1 passthrough=yes
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=odd nth=2,1 passthrough=yes src-address-list=!even
add action=add-src-to-address-list address-list=odd address-list-timeout=1d chain=prerouting comment="" connection-mark=odd disabled=no in-interface=local
add action=mark-routing chain=prerouting comment="" connection-mark=odd disabled=no in-interface=local new-routing-mark=odd passthrough=no
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=even nth=2,2 passthrough=yes
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no in-interface=local new-connection-mark=even nth=2,2 passthrough=yes src-address-list=!odd
add action=add-src-to-address-list address-list=even address-list-timeout=1d chain=prerouting comment="" connection-mark=even disabled=no in-interface=local
add action=mark-routing chain=prerouting comment="" connection-mark=even disabled=no in-interface=local new-routing-mark=even passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL1
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL2

/ip route
add check-gateway=arp comment="Route odd ToADSL1" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL1 routing-mark=odd
add check-gateway=arp comment="Route even ToADSL2" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL2 routing-mark=even
add check-gateway=arp comment="Route router traffic ToADSL2" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL2

If I investigate my firewall connections, address-lists and interface loads I see the following:

admin@MikroTik] /ip firewall address-list> print
Flags: X - disabled, D - dynamic
# LIST ADDRESS
0 D odd 192.168.254.2
1 D odd 192.168.0.101
2 D odd 192.168.0.62
3 D even 192.168.0.34
4 D odd 192.168.0.66
5 D odd 192.168.0.32
6 D even 192.168.254.1
7 D odd 192.168.0.15
8 D odd 192.168.0.31
9 D odd 192.168.0.10
10 D even 192.168.0.5
11 D odd 192.168.0.40
12 D odd 192.168.0.63
13 D odd 192.168.0.250

Most of the IP's are assigned to the "odd address list" i.e only 3 out of 14 to even. As a result the load on my odd interface is also much higher than the even interface. Am I missing something or what can be the reason for the "skew allocation"?

Nice to know that I was not smoking something funny back in Feb. when I reported this.


http://forum.mikrotik.com/viewtopic.php?f=2&t=29412

I will give this setup a try this weekend and see if it does better. Thanks for finding out the root of the problem.

I have changed my config to the one listed below in order to "resolve my problem" where my "odd address list" contained about 3-4 times more addresses than my "even address list"
With the above config I do not get any duplication of addresses in the different address lists and I had a split of 8/7 (i.e. out of a total of 15 IP's) addresses respectively in my odd/even address lists.

Yes, this seems to be doing much better. I successfully downloaded an entire ISO without dropping. Obviously the flushing of the routing table was causing my issues. This solution is Ok, but the ECMP was better in that opening multiple connections from one host gave more "bandwidth" to that host since some of those connections would go over both links. I have not fully completed my setup but it does split the connections based on IP. I used the above posters mangle rules but alas I have less hosts at this location so could not really see if I was getting odd or even marks distributed better. Once I have this locked down I will try one of my other test sites and report back.


Will there be any changes made by Mikrotik to enable us to use the ECMP solution with masqueraded connections again? I really liked the connections getting balanced. Thanks for tearing it to this more deeply.

Does it affect the TCP performance when the routing table is flushed? Do we have packet drops or packet delays because of it?

To our Latvian friends: I see you have added my "Connections to the router itself" config that we developed with mcgaiver, to the ECMP WiKi but the editing is disabled so I was not able to add my other part of the config that works around the flushing problem so connections stay on their proper gateways even after flushed....

Oh and by the way, could we use Private Messages on this forum again please?

If you wanted to post it I would be happy to test it. My only problem with ECMP was the routing table flushing. If it was posted in your other thread, I missed it .

Is there a way to simulate flushing the routes?

hi,
i have rb1000 and i have 2 wan and 1 local ip .... i have tried these setting but all the connection was odd or even there is no balancing and only one wan working ...
i dont want to balance between them because one wan is 4Mb and the other is 1Mb
is there any setting to do this ???
thanks.

I think the best way to go is to use ECMP which can balance between different links (4M and 1M) distributing across all links perfectly, and implement the workaround of the routing table flushing that I am using since months already. In the setups of all my remote clients (that contact me via givememorebandwidth AT gmail DOT com with WinBox login info) it works perfectly, so it is tested.


Setting up a proper ECMP route gives you controlled way of balance, you can balance 4/5 of all connections to one gateway and 1/5 to another. It also provides good fail-over - it can detect whether the gateway IP replies to icmp or arp requests, it can work with gateway interfaces instead of gateway IP addresses, making it perfect in the case with one ISP/same gateway/multiple connections. Another good thing about it is that when one user downloads with more TCP connections, they are distributed across the gateways so he can have ALL the bandwidth for himself when he needs it.

Thank you NetworkPro for your assistance and thank you for your fast reply

I actually am using gateway names with the nth config too. This configuration has been solid. I will be rolling this to a few test sites and then going back to the ECMP configuration. I will figure out the way to work around the flushes and post it for everyone.

I have come across another nth load balancing example that was posted by Janisk:

http://wiki.mikrotik.com/wiki/NTH_load_ ... pproach%29

I have also implemented his approach. The distribution between the IP addresses with this example is better than the http://wiki.mikrotik.com/wiki/NTH_load_ ... masquerade example.

I agree that nth load balancing is not as granular as ECMP. However, it does not have any of the issues that requires one to implement policy routing to make ECMP work properly. Furthermore once you have forced certain connections via a particular interface with policy routing your load balacing is anyway skewed and then you are still uncertain whether you have catered for all "exceptions". Also as far as I'm concerned multiple PPTP connections will not work properly with ECMP because of the issues with policy routing PPTP.
Also please note that the order of mangle rules in any of the nth load blancing examples are important. It will not work properly if you change the order of some the rules.

By the way, PPTP works with ECMP + route table flush workarounds, but L2TP does not. I mean from the router itself to a external router somewhere in the wild. Tested. I wonder how come L2TP does not work, what is so special about it that is different from PPTP for example? L2TP gets hit by the flush, the tunnel drops each 10 minutes +/- 2 or 3 or ?. I guess the connection-state=new does not catch it.

By the way, PPTP works with ECMP + route table flush workarounds, but L2TP does not. I mean from the router itself to a external router somewhere in the wild. Tested. I wonder how come L2TP does not work, what is so special about it that is different from PPTP for example? L2TP gets hit by the flush, the tunnel drops each 10 minutes +/- 2 or 3 or ?. I guess the connection-state=new does not catch it.

Are you useing l2tp/ipsec? if so then yes it would be affected by the flush since it uses higher level protocol than the routing protocols. Thus when the routes are flushed everything in the upper layers have to reconnect. Since the ipsec is considered lvl4/5 (depending on who you talk to) it will have to reconnect on a route flush since that happens on lvl3.

The mangle rules that we use (route table flush workarounds as I call em) fix the TCP connections (no reconnects, no loss in performance (not 100% analyzed for perf)) so they should fix everything. But L2TP somehow we miss. Maybe it is missed by connection-state=new as I said. Not sure.

can we see that rules?..

The mangle rules that we use (route table flush workarounds as I call em) fix the TCP connections (no reconnects, no loss in performance (not 100% analyzed for perf)) so they should fix everything. But L2TP somehow we miss. Maybe it is missed by connection-state=new as I said. Not sure.

l2tp is UDP, and therefore has a very short conntrack entry.

Normis,

Any update as to if and when mikrotik will be releasing a fix or work around other than using the nth for ecmp?

changing Kernel? will they do it at all?..

Well they could patch the kernel to get rid of the route table flushing or make it an option, so we can turn it off when we use masq/local ip addresses for customers.

By the way, me and a friend recently needed to route certain connections over interfaces in the same broadcast domain with the same gateway. Something like ECMP with the same gateway, but without PPPoE or PPTP tunnels to simplify things. MikroTik RouterOS currently can not handle this scenario. This is very sad.

I have been trying to get ECMP to work with this config on a RB150. Would your config work on that NetworkPro?

MikroTik RouterOS currently can not handle this scenario. This is very sad.

ECMP - cannot. use http://wiki.mikrotik.com/wiki/NTH_load_ ... masquerade

Well they could patch the kernel to get rid of the route table flushing or make it an option, so we can turn it off when we use masq/local ip addresses for customers.

By the way, me and a friend recently needed to route certain connections over interfaces in the same broadcast domain with the same gateway. Something like ECMP with the same gateway, but without PPPoE or PPTP tunnels to simplify things. MikroTik RouterOS currently can not handle this scenario. This is very sad.

Nth Can not as well

I don't understand your config. please, in more details =)

interface routing didn't help?

I believe this to be the problem, this is the "route" that is at fault...:


The gateway IP address is the same on ether1 ether2 ether3 ether4. When I try to set up the route without an IP address for gateway, it does not work at all. When I input the gateway IP, RouterOS always sends through ether1, despite "prefsrc" field. When I add the IP and the interface - the route works for a while, until table flush occurs.. even with the rules that I use, since they rely on route mark and the route on the picture routes marked traffic.

When routing is through PPPoE, PPTP etc, there is no such problem, when used with my flushing workaround rules, all is flawless.

Continuing the ECMP discussion..

I the past days i have had complains about the ECMP problem (mostly with disconnection of MSN messenger) by 2-3 of our clients. The strange thing is that one other client that is using the ECMP with 6 external lines (mathematically speaking he should have disconnections more often) doesn't report disconnections. I checked his router and he has version 3.10, the others instead have newer ones. I downgraded to 3.10 one of the other routers and something surely changed. I don't see disconnections anymore (at least not as often as before). I'll have to test this in the upcoming days but for sure the downgrade has improved things.
Can anyone confirm this?? Maybe Normis can tell us when the Kernel writers have introduced the "Flushing" instruction, so we can use older versions of RouterOS for the clients needing ECMP.

The other problem is that other Open Source Linux Based Routers doesn't report this problem with the ECMP. I'm going to test one in the upcoming days, but it would be great if anyone has tested successfully ECMP with something else.

@ GuJack20
can u pls upload MTv3.10 to test it with u ?
i can't found it

just google

all_packages-x86-3.10.zip
or all_packages-mipsbe-3.10.zip if you have a routerboard.

The ECMP problem still exist but there is a slight improvement.

just google

all_packages-x86-3.10.zip
or all_packages-mipsbe-3.10.zip if you have a routerboard.

The ECMP problem still exist but there is a slight improvement.

ok thanks
i will test 3.17 i think that it will doing great , i will back with result

3.23 arrived
http://www.mikrotik.com/download.html
so ECMP bug fixed or not ?

...
By the way, me and a friend recently needed to route certain connections over interfaces in the same broadcast domain with the same gateway. Something like ECMP with the same gateway, but without PPPoE or PPTP tunnels to simplify things. MikroTik RouterOS currently can not handle this scenario. This is very sad.

This problem is attacked with routing-test package. Will test in matter of hours and post results + send supout to support e-mail. edit: no success.

NetworkPro. Can't you explain here the workaround of the ECMP problem? Or by email. I surely will do as discussed..

same problem in 3.23
messenger dissconnect every 30 min
i tried to routing mark msn & yahoo messenger but the same prob too

this is my rules
/ip firewall mangle
add action=mark-routing chain=prerouting comment="Y! routing mark" disabled=\
no dst-port=5050 new-routing-mark=Yahoo passthrough=no protocol=tcp
add action=mark-routing chain=prerouting comment="MSN routing mark" disabled=\
no dst-port=1863 new-routing-mark=MSN passthrough=no protocol=tcp

/ip route
add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
WAN-1 routing-mark=Yahoo scope=30 target-scope=10
add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
WAN-1 routing-mark=MSN scope=30 target-scope=10

is this rules right ? or is there any way to selove this prob ?

I do not think this is related to the "flush of connections by kernel", at least in some cases.
For example, when you visit a page which would ask your identification, or would ask
for a confiramtion by entering a code, it will keep on recycling you every time you click
on login, bouncing back at the login or confirmation page. In this case we are aproximatly
returning to this page every 30 seconds, so it's obvious it has nothing to do with the flush of
connections. I tested this with http pages or https, same behaviour.
Take for example http://cqcounter.com/whois/ you will see what I mean

Now as gujack mentioned, donwgrading to version 3.10 this doesn't hapen any more.
So I think this is a problem realted to ecmp itself.

Any update on patching the kernel or a fix for the ecmp normis?

In this case we are aproximatly
returning to this page every 30 seconds, so it's obvious it has nothing to do with the flush of
connections.

A single web page load will sometimes use 20-30 connections... it's not just a single connection for each page. You are probably loading the captcha image on one connection and hitting their form on another.

A single web page load will sometimes use 20-30 connections... it's not just a single connection for each page.

didn't you forget about keep-alive connections?.. all modern browsers use them

C:\Documents and Settings\Chups>nslookup
> cqcounter.com
Name: cqcounter.com
Addresses: 75.160.72.234, 63.231.252.74

they use load balancing via DNS. maybe just browser sends request to different IP, so ECMP uses another gateway?

Chupaka... have you tried anything else than RouterOS for ECMP?? Any clue for a stable open source system?

no, sorry, I'm RouterOS fan =)
and I used only 'Nth' (it was 'Random' really =) ), no ECMP

Let's have an organized idea about how ECMP works.
So, when I attempt to connect to MSN Server for example, my computer initiates
a connection with MSN destination. It should look like this:

src-address=192.168.1.10:23000 dst-address=90.90.90.90:1863, right?

once this connection is established, RouterOS would keep it alive according to
the specified value in the conntrack for established TCP connection.
Now, we are told that the kernel flushes these established connections, which
would lead my computer to initiate another connection, sth. like this:

src-address=192.168.1.10:23001 dst-address=90.90.90.90:1863, right?

From the perspective of RouterOS this should be considered a "related connection".
Can't we make this related connection to go through the same gateway? Does conntrack
keeps track of gateway used? It looks not. Can we make a mangle to keeps track
of gateway used?
Do you think this is an approach to the solution, or is it completely wrong ?

'make a mangle to keeps track of gateway used' means 'use Nth example from Wiki' =)

kernel flushes connections table, not closes all connections. your connection just can now use another gateway, so if you use src-nat, it brokes

The nth will not keep gateway per connection but per IP.
It is not what I meant, I am hopping for any solution that will
keep gateway for connection.
Anyway it doesn't look it can be done.

The problem I have with nth is that first you can experience different
internet perfomance with two different adjacent computers, next,
most important I think, is you need to configure some extra scripts
for failover.

no, sorry, I'm RouterOS fan =)
and I used only 'Nth' (it was 'Random' really =) ), no ECMP

so ur using nth
i tried to use it but it didn't works fine .
all connections used only 1 wan .
so is there any changes u did to make NTH works fine ?
and if there is changed can u share it with us pls ?

another thing what is the best way ?
http://wiki.mikrotik.com/wiki/NTH_load_ ... masquerade
or http://wiki.mikrotik.com/wiki/NTH_load_ ... pproach%29

The problem I have with nth is that first you can experience different
internet perfomance with two different adjacent computers, next,
most important I think, is you need to configure some extra scripts
for failover.

well, if you have two computers - yes, you need ECMP. if you have two hundred computers, Nth is quite enough. I love you, the law of large numbers =)

failover is just adding a route with gateway of another WAN and bigger distance

so ur using nth
i tried to use it but it didn't works fine .
all connections used only 1 wan .
so is there any changes u did to make NTH works fine ?
and if there is changed can u share it with us pls ?

actually, it was not Nth, it was my own setup developed from scratch. it worked fine while we use it =) and as far as I can see it was using the same principles as Nth in Wiki

so ur using nth
i tried to use it but it didn't works fine .
all connections used only 1 wan .
so is there any changes u did to make NTH works fine ?
and if there is changed can u share it with us pls ?

actually, it was not Nth, it was my own setup developed from scratch. it worked fine while we use it =) and as far as I can see it was using the same principles as Nth in Wiki

now i'm using nth
but there is prob ..
proxy doesn't work


so web proxy doesn't works with NTH or there is something wrong in my rules ?

as I can see by 'Sent To Client', proxy is working. but is not caching =) did you enabled web cache at all?

as I can see by 'Sent To Client', proxy is working. but is not caching =) did you enabled web cache at all?

yes
settings :

enabled: yes
src-address: 0.0.0.0
port: 8080
parent-proxy: 0.0.0.0
parent-proxy-port: 0
cache-administrator: "master"
max-cache-size: none
cache-on-disk: yes
max-client-connections: 1000
max-server-connections: 1000
max-fresh-time: 3d
serialize-connections: no
always-from-cache: yes
cache-hit-dscp: 4
cache-drive: primary-master

firewall :

chain=dstnat action=redirect to-ports=8080 protocol=tcp dst-port=80

change max-cache-size: none no max-cache-size: unlimited

Added: but you cannot use Nth to load-balance transparent proxy. use ECMP for Web-Proxy or wait for PCC...

change max-cache-size: none no max-cache-size: unlimited

Added: but you cannot use Nth to load-balance transparent proxy. use ECMP for Web-Proxy or wait for PCC...

i think it works now
and will test it if there is prob i will disable proxy and wait for PCC
thanks

if there is prob i will disable proxy and wait for PCC

well, now all your proxy traffic should go through one gateway. but for proxy you may use ECMP - it's not long-time connections like instant messengers, etc.
just add two gateways for your main default route, and leave one default gateway to every routing table in your Nth config

@MrIC, The problem is probably that the proxy does not have a proper storage for the cache. It needs a drive that is partitioned that is initialized for it, has the proper folder structure. Maybe you could somehow try to rebuild the cache or make it use a secondary drive or microSD card... And dont forget to upgrade RouterOS version.

Are there any fix or news about ECMP problem?

what problem?..

routing table flush every 10min

also I had some problems with NAT:

I configured ECMP load balancing with masquerade, but when I'm switching from route 2 to route 3, I can't connect to RDC through WAN1, but WAN2 works. When I'm switching back to route 2 WAN1 work's without problems.
I'm planing to try PCC but i'm still curious

routing table flush every 10min

it won't be fixed by MikroTik, it's in Kernel

about RDC: you need to track, which interface the connection came from and then route it back via the same connection. PCC example in Wiki do that

Are there any fix or news about ECMP problem?

There are quite a few people who would like to see the ECMP connection table flush reversed. Don't expect it to happen any time soon. Right now PCC is best option using only mikrotik.

flush is security/usability measure, I don't believe it will be removed

Maybe some mangling can help with the ecmp/flush issue. Use add-src-to-address-list when a client is first routed through a gateway, and later, after the flush you can restore the route by matching from src-in-address-list and using route marks.

that is why PPC was introduced, it does that for you in a manner you configure it, so you do not have to do tedious task of shuffling and mangling endless user connections by yourself. Just set up PPC and it will make short work of it, and connections will be recreated to same gateway they where before the flash. So you got security measure and working configuration that can be easily configured, IMHO, win-win situation.

janisk: what you say is right, i haven't used PCC though it does look really good. Anyway, does it work along with ECMP or replaces it completely? I am confused as to the situations where it's best to use ECMP or NTH or Random or PCC for loadbalancing/failover.

For now what i see is:

ECMP the "kernel routing/rpdb" way of loadbalancing. Automatic failover, has the flush issue.
PCC the "netfilter way", only mikrotik +3.24 and more customizable than ECMP alone on how you will sort connections, don't know if it handles its failover automatically as ECMP or has to be scripted.
NTH/Random: netfilter way, can do almost as everything as PCC with more work on designing the mangles, and otherwise can be used to support ECMP. Failover is scripted.

Am i missing something?

PCC with 'both-addresses' classifier is like ECMP
PCC with 'both-addresses-and-ports' classifier is like Nth (well, varies)
PCC with 'src-address' classifier is the best way =)

about failover: to use ECMP-like failover on PCC, you need just add several routes (main and backups) with different distances for each routing table. also, look at http://wiki.mikrotik.com/wiki/Advanced_ ... _Scripting

What about netmap to public pool (BGP prefix) and two assymetric uplinks with BGP advertising this prefix on both links? I suppose I can use ECMP in this case, am I right?

It's no problem to use PCC either, but because of assymetric uplinks (ratio 15:4) it looks much cleaner to me to use ECMP with 19 gateways (15+4) than to have 19 PCC rules and divide PCC identifier by 19.

Heads-up - If at each "table flush" your upload packets switch path this could negatively impact protocol performance.

actually, ~5 rules will be enough: 4 for one uplink, and 1 for all the rest (w/o pcc)

but indeed, it would be nice to have a possibility to set a range of reminders (like in ports or addresses)

p.s. I don't think 1:4 will hurt you compared to 4:15