ECMP - Load balancing not working properly

nicopretorius · Thu Jan 29, 2009 7:00 am

As many users in the forums I have run into problems with the load balancing example in the wiki: http://wiki.mikrotik.com/wiki/Load_Balancing_Persistent. I have configured a basic setup as per the example with the exception that I'm using interface routing to add the defaults gateways rather than IP addresses. The reason for this is that I'm using ADSL with PPPoE for which the internet addresses are assigned via DHCP.

The problems that I experience are:
1. https connections does not work properly. I tried using Internet banking and the server logs me out the moment a packet is detected from another IP.
2. SMTP connections do not work properly. Mails are dropped halfway while being sent.
2. I cannot establish a PPTP connection from the PPTP client on my router (ROS3.19) to my PPTP server.

The moment I disable one of the PPPoE interfaces without changing anything else in the config, everything works 100%.

The Wiki example in my opnion is either incomplete or ECMP does not work properly for load balancing purposes. The problems that others have experienced as per the postings in this forum confirms this. It should either be fixed or completely removed from the Wiki because it creates more problems than what is solves in its current format.

I will therefore rather try my luck with http://wiki.mikrotik.com/wiki/Per-Traff ... _Balancing than waste further time on something that does not work.

If anybody has other examples that work well for them, please post them here.

JJCinAZ · Thu Jan 29, 2009 7:14 pm

ECMP does exactly what it is supposed to do. Maybe the Wiki article needs to have more warnings that you have to understand protocols better. The real problem is up at Layer 7 where applications make assumptions about source IP addresses which are incongruent with IP networking. As a network engineer, it's your job to reconcile the lower layers of IP with the applications using it. I know none of that helps you, so here are some comments which might help.

First, try getting away from dynamic public IP addresses. Things get much easier.

Two, SMTP connections should not be affected in their basic form. The process of sending a message to an MTA should be a single TCP connection on port 25. When I say single connection I mean atomic for at least one whole email message. Now you may run into a problem with SPF or reverse DNS but given that you are a dynamic public IP, you pretty much can't use SPF and the reverse DNS is never going to point to your MX. You should sniff the outbound traffic during a failure case and confirm that an SMTP session with a server stays on a single interface until the TCP session is closed. Note that the very next SMTP session to the same or to a different server can go out either ECMP route.

Three, PPTP is composed of two IP sessions, a TCP session on port 1723 for session control and encapsulated data in using the GRE protocol. You have to keep these on the same outbound interface using policy routing to get PPTP to work. You could try setting up a different type of tunneling which uses a single IP session or is session-less.

nicopretorius · Sat Feb 07, 2009 5:20 pm

The config works better now that I have changed my config to do src-nat on interfaces as per the latest wiki config rather on the src-address and also based on what has been posted in http://forum.mikrotik.com/viewtopic.php?f=2&t=29464.

I use a RB600 as my "ECMP router" and I used the default route as per the detail below rather than what has been suggested in the post listed above.

/ip route
add check-gateway=ping comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL1,ADSL2
add comment="" disabled=no distance=1 dst-address=192.168.0.0/24 gateway=192.168.254.2 scope=30 target-scope=10

I then use a second router (RB532) which connects to the ECMP router as indicated in the attached picture. The reason for this is to get away from the problem where traffic that originates from the router itself (such as PPTP client from the ECMP router itself) is not handled correctly and creates all sorts of issues.

This config generally works well except that the tunnels that I create on the RB532 disconnect and re-connect at regular intervals (every 10 minutes or multiples thereof - i.e. 20 minutes).

I also intermittently experienced the following problems:

With a particular https site (Internet banking) once or twice during a period of a week the site logged me out for no reason, i.e. 99% of the time the site works without issues.
I also experienced a problem twice where emails failed halfway through being sent

Failover works without any problems. Both DSL connections are to the same ISP and even though the one route (via ADSL1) shows as being not active (as shown below), traffic is routed via both ADSL interfaces.

[admin@MikroTik] /ip route> print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY-STATE GATEWAY DISTANCE INTERFACE
0 A S 0.0.0.0/0 reachable ADSL1 1 ADSL1
reachable ADSL2 ADSL2
1 A S 192.168.0.0/24 reachable 192.168.254.2 1 local
2 ADC 192.168.254.0/24 192.168.254.1 0 local
3 ADC 196.x.y.1/32 196.x.z.66 0 ADSL2
4 DC 196.x.y.1/32 196.x.z.93 0 ADSL1

Has anybody else experienced something similar with PPTP and ECMP?

NetworkPro · Sun Feb 08, 2009 1:44 am

For the problem of connecting to the router from Internet and maybe connections from the router itself to Internet:

....
- *Policy routing everything coming in a certain interface, to the Router itself, to go out the same (connection-mark @ input, routing mark @ output)
....

*

/ip firewall mangle
add action=mark-connection chain=input connection-state=new in-interface=ADSL2 new-connection-mark=ADSL2Con2R passthrough=yes
add action=mark-connection chain=input connection-state=new in-interface=ADSL1 new-connection-mark=ADSL1Con2R passthrough=yes
add action=mark-routing chain=output connection-mark=ADSL2Con2R new-routing-mark=ToADSL2 passthrough=yes
add action=mark-routing chain=output connection-mark=ADSL1Con2R new-routing-mark=ToADSL1 passthrough=yes

Then you must specify the default route for the ToADSL1 and ToADSL2 markings. You can see how I have done it a couple of days ago here: http://forum.mikrotik.com/viewtopic.php ... 68#p143568 - check out the screenshot.

For the PPTP problem:

ECMP does exactly what it is supposed to do. Maybe the Wiki article needs to have more warnings that you have to understand protocols better. The real problem is up at Layer 7 where applications make assumptions about source IP addresses which are incongruent with IP networking. As a network engineer, it's your job to reconcile the lower layers of IP with the applications using it. I know none of that helps you, so here are some comments which might help.
...

Two, SMTP connections should not be affected in their basic form. The process of sending a message to an MTA should be a single TCP connection on port 25. When I say single connection I mean atomic for at least one whole email message. Now you may run into a problem with SPF or reverse DNS but given that you are a dynamic public IP, you pretty much can't use SPF and the reverse DNS is never going to point to your MX. You should sniff the outbound traffic during a failure case and confirm that an SMTP session with a server stays on a single interface until the TCP session is closed. Note that the very next SMTP session to the same or to a different server can go out either ECMP route.

Three, PPTP is composed of two IP sessions, a TCP session on port 1723 for session control and encapsulated data in using the GRE protocol. You have to keep these on the same outbound interface using policy routing to get PPTP to work. You could try setting up a different type of tunneling which uses a single IP session or is session-less.

This means: mangle policy route them over one of the ADSLs.

I wonder what the mangle rules will be for the PPTP to go through only one of the interfaces to work properly... If anyone has any suggestions and pointers... So the port is 1723 tcp and then theres a GRE along with it... hmm.. should work....

knects · Mon Feb 09, 2009 9:28 pm

So I have followed along with this thread in hopes of making this work. I have most functionality setup as you guys do with a few odd instances.

Hosted services behind the network do not always get connections.

i.e. stmp, imap and ssh sessions forwarded to hosts behind the nat are very sparadic in accepting the connections. When I do get replys and attempt to auth ssh authenticates but seems to not know how to get back to me from internet. Sometimes is does but then times out after a few minutes and drops me.

All connection attempts are on the same ADSL interface and public IP.

I have this setup with 2 PPPOE connections over bridged modems. They are different providers with different public addresses and networks.

Outbound connections work well and get balanced okay with a few nuances. i.e sip traffic is confused from asterisk box. Will probably force this over one route with routing marks but then lose failover. I need to look at sip_nat.conf and see if I can setup both public addresses with success.

Should this setup give me persistent connections? And if so is there a time out? I have adjusted the Generic Timeout in the Firewall Connection Tracking with no success. I need to sniff the packets abit but iTunes and online gaming sessions get dropped after about 10 mins and have to be restarted.

RB433 ROS 3.20

/ip route
add check-gateway=arp comment="Route All ToADSL1" disabled=no distance=1 \
dst-address=0.0.0.0/0 gateway=ADSL1 routing-mark=ToADSL1
add check-gateway=ping comment="" disabled=yes distance=1 dst-address=\
0.0.0.0/0 gateway=ADSL1,ADSL2
add check-gateway=arp comment="Route All ToADSL2" disabled=no distance=1 \
dst-address=0.0.0.0/0 gateway=ADSL2 routing-mark=ToADSL2
add check-gateway=arp comment="ECMP Test - BAD" disabled=no distance=1 \
dst-address=0.0.0.0/0 gateway=ADSL1,ADSL2
add check-gateway=arp comment="Route All Else to ADSL1" disabled=no distance=\
2 dst-address=0.0.0.0/0 gateway=ADSL1

/ip firewall nat
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL1
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL2
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=22 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=22
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=25 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=25
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=80 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=80
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=443 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=443
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=587 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=587
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=993 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=993
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=5004-5037 \
in-interface=ADSL1 protocol=udp to-addresses=xx.xx.xx.7 to-ports=\
5004-5037
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=5039-5082 \
in-interface=ADSL1 protocol=udp to-addresses=xx.xx.xx.7 to-ports=\
5039-5082
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=10000-20000 \
in-interface=ADSL1 protocol=udp to-addresses=xx.xx.xx.7 to-ports=\
10000-20000

/ip firewall mangle
add action=mark-connection chain=input comment=\
"Policy Routing All connections from ADSL1 to Router back to ADSL1" \
connection-state=new disabled=no in-interface=ADSL1 new-connection-mark=\
ADSL1Con2R passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=ADSL1Con2R \
disabled=no new-routing-mark=ToADSL1 passthrough=yes
add action=mark-connection chain=input comment=\
"Policy Routing All connections from ADSL2 to Router back to ADSL2" \
connection-state=new disabled=no in-interface=ADSL2 new-connection-mark=\
ADSL2Con2R passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=ADSL2Con2R \
disabled=no new-routing-mark=ToADSL2 passthrough=yes

I have also added the following rule in to see if it drops the connection for iTunes and such and either way the apps still disconnect. And of course If I disable ADSL2 pppoe connection everything works just fine.

/ip firewall filter
add action=drop chain=forward connection-state=invalid

Update:: I have verified with torch that the forwarded packets coming in ADSL1 from internet through nat are at times getting routed back through ADSL2.

shadowteller · Mon Feb 16, 2009 10:44 pm

I have followed the same tutorial and was pretty excited that it got updated as the old one seemed to be broken.

My 2 links that I am trying to load balance are a 1.5M T1 and a 3M DSL. I understand that adding the 3M pipe in under gateway=WAN1,WAN2,WAN2 will force the router to make multiple use of that particular gateway to "weight" the higher throughput of the DSL line.

I have also implemented the same configuration with the suggested changes. and am seeing pretty much the same problems as what knects is seeing. When I used a Download Accelerator I see pretty close to the 4.5M of traffic that would be expected. As well as when I do an update in Debian.

However, my project based Webserver and email server seem to sporadically miss connections. As if I am using some sort of round-robin bonding without the other side being bonded. itunes and Zune Market place seem to also break and can stay sometimes connected for as little as 10 minutes, but as much as 1 hour. Everquest and WoW suffer from this as well and are playable for a short time, but then drop. It appears as if at some point the router just stiops following the routing marks.
I also still see the original of HTTPS, SSH and SMTP failing as indicated by the very first post and the very last post.

Does anybody have any idea or is this supposed to just be a port 80 only load balancer? No point posting the config as they probably all look the same. Someone Please help?

Otherwise I am thinking of using my overly priced Edimax BR724 Load Balancer.

Please Help I am desperate.

Preston

NetworkPro · Tue Feb 17, 2009 12:06 am

So I have followed along with this thread in hopes of making this work. I have most functionality setup as you guys do with a few odd instances.

Hosted services behind the network do not always get connections.

i.e. stmp, imap and ssh sessions forwarded to hosts behind the nat are very sparadic in accepting the connections. When I do get replys and attempt to auth ssh authenticates but seems to not know how to get back to me from internet. Sometimes is does but then times out after a few minutes and drops me.

All connection attempts are on the same ADSL interface and public IP.

I have this setup with 2 PPPOE connections over bridged modems. They are different providers with different public addresses and networks.

Outbound connections work well and get balanced okay with a few nuances. i.e sip traffic is confused from asterisk box. Will probably force this over one route with routing marks but then lose failover. I need to look at sip_nat.conf and see if I can setup both public addresses with success.

Should this setup give me persistent connections? And if so is there a time out? I have adjusted the Generic Timeout in the Firewall Connection Tracking with no success. I need to sniff the packets abit but iTunes and online gaming sessions get dropped after about 10 mins and have to be restarted.

RB433 ROS 3.20

/ip route
add check-gateway=arp comment="Route All ToADSL1" disabled=no distance=1 \
dst-address=0.0.0.0/0 gateway=ADSL1 routing-mark=ToADSL1
add check-gateway=ping comment="" disabled=yes distance=1 dst-address=\
0.0.0.0/0 gateway=ADSL1,ADSL2
add check-gateway=arp comment="Route All ToADSL2" disabled=no distance=1 \
dst-address=0.0.0.0/0 gateway=ADSL2 routing-mark=ToADSL2
add check-gateway=arp comment="ECMP Test - BAD" disabled=no distance=1 \
dst-address=0.0.0.0/0 gateway=ADSL1,ADSL2
add check-gateway=arp comment="Route All Else to ADSL1" disabled=no distance=\
2 dst-address=0.0.0.0/0 gateway=ADSL1

/ip firewall nat
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL1
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ADSL2
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=22 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=22
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=25 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=25
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=80 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=80
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=443 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=443
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=587 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=587
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=993 \
in-interface=ADSL1 protocol=tcp to-addresses=xx.xx.xx.21 to-ports=993
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=5004-5037 \
in-interface=ADSL1 protocol=udp to-addresses=xx.xx.xx.7 to-ports=\
5004-5037
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=5039-5082 \
in-interface=ADSL1 protocol=udp to-addresses=xx.xx.xx.7 to-ports=\
5039-5082
add action=dst-nat chain=dstnat comment="" disabled=no dst-port=10000-20000 \
in-interface=ADSL1 protocol=udp to-addresses=xx.xx.xx.7 to-ports=\
10000-20000

/ip firewall mangle
add action=mark-connection chain=input comment=\
"Policy Routing All connections from ADSL1 to Router back to ADSL1" \
connection-state=new disabled=no in-interface=ADSL1 new-connection-mark=\
ADSL1Con2R passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=ADSL1Con2R \
disabled=no new-routing-mark=ToADSL1 passthrough=yes
add action=mark-connection chain=input comment=\
"Policy Routing All connections from ADSL2 to Router back to ADSL2" \
connection-state=new disabled=no in-interface=ADSL2 new-connection-mark=\
ADSL2Con2R passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=ADSL2Con2R \
disabled=no new-routing-mark=ToADSL2 passthrough=yes

I have also added the following rule in to see if it drops the connection for iTunes and such and either way the apps still disconnect. And of course If I disable ADSL2 pppoe connection everything works just fine.

/ip firewall filter
add action=drop chain=forward connection-state=invalid

It's cool that you find our work useful.

Update:: I have verified with torch that the forwarded packets coming in ADSL1 from internet through nat are at times getting routed back through ADSL2.

So Policy Route them too

NetworkPro · Tue Feb 17, 2009 12:22 am

I have followed the same tutorial and was pretty excited that it got updated as the old one seemed to be broken.

My 2 links that I am trying to load balance are a 1.5M T1 and a 3M DSL. I understand that adding the 3M pipe in under gateway=WAN1,WAN2,WAN2 will force the router to make multiple use of that particular gateway to "weight" the higher throughput of the DSL line.

I have also implemented the same configuration with the suggested changes. and am seeing pretty much the same problems as what knects is seeing. When I used a Download Accelerator I see pretty close to the 4.5M of traffic that would be expected. As well as when I do an update in Debian.

DA dl-ing from same server or different mirrors!??

However, my project based Webserver and email server seem to sporadically miss connections. As if I am using some sort of round-robin bonding without the other side being bonded. itunes and Zune Market place seem to also break and can stay sometimes connected for as little as 10 minutes, but as much as 1 hour. Everquest and WoW suffer from this as well and are playable for a short time, but then drop. It appears as if at some point the router just stiops following the routing marks.
I also still see the original of HTTPS, SSH and SMTP failing as indicated by the very first post and the very last post.

Does anybody have any idea or is this supposed to just be a port 80 only load balancer? No point posting the config as they probably all look the same. Someone Please help?

Otherwise I am thinking of using my overly priced Edimax BR724 Load Balancer.

Please Help I am desperate.

Preston

Problem is probably due to the broken applications using multiple connections, some TCP some UDP some to different servers. Please Policy Route them correctly! over one of the links and Please report back here. Thank you.

P.S. How to know if for example a game makes multiple connections and how to know which they are - simply close all other apps and make mangle rules to log the new connections, etc.. etc.. Good luck.

P.S. 2. I myself policy route A-Bunch-of-Client-IPs over one line and Another-Bunch over the other, since I do all this, including ADSL modem bridging etc. etc. over the Internet, without the need for me to be behind their router. (Which is not best practice but I can handle it).

knects · Tue Feb 17, 2009 12:42 am

If by policy routing you mean something like this setup http://wiki.mikrotik.com/wiki/Per-Traff ... _Balancing. I have tried to set this up for my SSH sessions but have been unsuccessful. (One service at a time thing).I will be trying again this week. I also have thought about just setting up the services in a DMZ port on a 493 and see what happens.

I have 2 scenarios that I am working here. 1. Just load balencing for 2 connections and the other is load balancing with hosted services behind as in my current config.

I actually have been able to keep the applications up for more than 10 minutes but not more than an hour it is random.

If this is not what you mean then I guess I need a pointer

.

Thanks so much for the help. Would really love to get this working. Glad MT did something with the wiki those other configs were uh, interesting.

janisk · Tue Feb 17, 2009 12:22 pm

basic idea for ECMP load ballancing to work is
1) make sure that incoming connections from internet gets out the same route they came in - mangle with routing marks, so if something fails, check what connections it has, that policy route it.
2) special connections are routed to one interface, like pptp tunnels, different chat clients.

because, when you create connection it is assigned to one of the gateways and it stays there, forever. But when based on this connection another one is established, usually the other end is expecting that connection is coming from same ip address/host, it can be assigned to other gateway (as it is how ECMP works) , there for address changes and your both connections are dropped

IMHO - this covers all the basic knowledge that needs to be is covered

knects · Sat Feb 21, 2009 10:16 pm

Ok, so policy routing has taken care of most issues but am still having problems with connections. Long file transfers via http, sftp, scp or ftp time out. If I use clients such as transmission or such they seem to work just fine. Downloading large files will time out after a period of time. This has happened from various hosts and servers. Seems to be random as to time or amount transfered. Files are > 500MB. Any ideas?

NetworkPro · Sun Feb 22, 2009 12:02 am

Could be unstable links... too much congestion.... problem in upstream provider....

knects · Sun Feb 22, 2009 1:05 am

Ahh, but if I disable one of the links then everything is fine across each of them. Only when using the ECMP setup does this happen.

NetworkPro · Sun Feb 22, 2009 1:44 am

Ok, so policy routing has taken care of most issues but am still having problems with connections. Long file transfers via http, sftp, scp or ftp time out. If I use clients such as transmission or such they seem to work just fine. Downloading large files will time out after a period of time. This has happened from various hosts and servers. Seems to be random as to time or amount transfered. Files are > 500MB. Any ideas?

Can you view the connection log of the download program? Maybe there is a reconnect and resume, that is not obvious, unless you peek in the transfer log? So you tried this with what programs/browser/downloaders and what files did you download? Usually I test with Linux distribs ISO files. If you feel it will help, you could post a screenshot or two.

knects · Sun Feb 22, 2009 2:18 pm

Yeah, I was doing a sftp of a linux iso from a box at my office. It ran for about 15 minutes then stalled and eventually timed out. I will try to get a log from it and post it. I do all my testing the same as you with linux iso files. I was having the issues via http doing the same. I have an RB600 with daughterboard at my office. It has a 6MB link not doing load balancing. I have an SSH connection open to the same box and it does not time out. Let me work on getting some log files. Right now my AppleTV is timing out trying to download movies. iTunes has been working just fine since policy routing the random ports. Forcing all http traffic over one link does not really help me "load balance". At least at this point it is working "well enough" that we can start testing with customers.

Browsers - Firefox for OSX and Safari for HTTP downloads
Using Transmission for OSX for the torrent downloads of the linux iso's.

pace · Thu Mar 12, 2009 7:45 am

I'm having the same issue as knects. Connections that last longer than 10 minutes or so tend to fail. Also a RB433, I've tried 3.20 and 4.0Beta1 with no change in results. All of my policy routing for PPTP and IPSec are working fine. Also if I tag a particular internal system to use a particular route, that works too.

I've contacted Mikrotik support and their best solution was to tell me to just set some of my internal clients to use one connection and some to use another (not really what I wanted to hear).

I'd be happy to try something out if anyone has any ideas...

pace

knects · Tue Mar 17, 2009 2:02 pm

So far this morning upgrading to 3.22 has helped alot with my problem. I did not see anything specific in the changelog. I will update this after more thorough testing. I have made it well past any download sizes than I did on 3.20.

EDIT:

Spoke too soon. Hung at 325MB before I was only getting to around 80. Better but not complete. I will test some more. If I stop the download in Safari and resume it continues. I am trying my bittorrent client. Guess I am busting out wireshark.

janisk · Tue Mar 17, 2009 4:24 pm

so you are saying you have ECMP configured and you cannot download largr files, create pptp tunnel from client in your network, through ECMP router to server somewhere out in the wild, etc...

well..i tried simple configuration, without fancy configuration, simple basics:

router with ecmp:
set up 2 or more uplinks, masquerade them all (out-interface, rscnat, masquerade one rule for each of them)

tunnel was up for several hours, no reconnects, no nothing, traffic passed through without problems.

if possible, i would suggest you to make simple configuration and see what changes when and will continue to test configuration with ECPM

knects · Tue Mar 17, 2009 6:10 pm

My only problem on 3.20 was downloading larger files i.e. linux iso's via scp, sftp, wget, firefox/safari or bittorrent. I upgraded to 3.22 this morning and started trying agin. I was successful to about 300+ meg till it hung, If I stop it and resume it continues. I forgot to update the boot loader, I have now done that and am trying it again.
PPTP has been fine for me.....

BTW thank you for fix on 493.

pace · Wed Mar 18, 2009 3:52 am

I've not tried 3.22 yet, but large file downloads stop after 10-20 minutes of downloading. Like knects, pausing and unpausing the download causes it to start going again. I also have problems with long ssh sessions being open.

Anyone out there with a 433 that has ECMP working?

pace

ssffzzxx · Wed Mar 18, 2009 11:51 am

Dear All,
Currently I use mikrotik RB450 with OS 3.22. I have been struggling to figure it out how to have load balancing and fail over. is anyone in this forum has a script that run on this router?

I create fail over with load balancing just like in the wiki.
The problem is, when i unplug my ISP cable, the traffic can not find another gateway.
for example with ping applicatuion from client PC with option -t
ex: ping www.linux.org -t

if i unplug one of the ISP gateway cable, I got time out.. but It could NOT get reply again.
I had to close the cmd window and do type ping , than i get reply using another gateway.

I really need a script that run smoothly, that can rediret trafic to another gateway.
Thanks in advance.

Best Regards

NetworkPro · Wed Mar 18, 2009 2:05 pm

Dear All,
Currently I use mikrotik RB450 with OS 3.22. I have been struggling to figure it out how to have load balancing and fail over. is anyone in this forum has a script that run on this router?

I create fail over with load balancing just like in the wiki.
The problem is, when i unplug my ISP cable, the traffic can not find another gateway.
for example with ping applicatuion from client PC with option -t
ex: ping http://www.linux.org -t

if i unplug one of the ISP gateway cable, I got time out.. but It could NOT get reply again.
I had to close the cmd window and do type ping , than i get reply using another gateway.

I really need a script that run smoothly, that can rediret trafic to another gateway.
Thanks in advance.

Best Regards

Please try to play around with Gateway Distance. When RouterOS detects a gateway is not active due to interface beind down, it will route right away to the next active gateway with the next distance.

ssffzzxx · Thu Mar 19, 2009 4:20 am

Hello Network Pro,
Thanks for you respond.
Actually I read your post here : http://forum.mikrotik.com/viewtopic.php ... 64&start=0
may I know that your configuration for load balancing and fail over work properly?

These the script that i'll put to my RB450:

'''/ ip address'''
add address=192.168.0.1/24 network=192.168.0.0 broadcast=192.168.0.255 interface=Local
add address=10.111.0.2/24 network=10.111.0.0 broadcast=10.111.0.255 interface=wlan2
add address=10.112.0.2/24 network=10.112.0.0 broadcast=10.112.0.255 interface=wlan1

'''/ ip route'''
add dst-address=0.0.0.0/0 gateway=10.111.0.1,10.112.0.1 check-gateway=ping

'''/ ip firewall nat'''
add chain=srcnat out-interface=wlan1 action=masquerade
add chain=srcnat out-interface=wlan2 action=masquerade

Should I add :

mark-connections in mangle chain input
mark-routing in mangle chain output
create a routes for routing-marks

/ip firewall filter
add action=drop chain=forward connection-state=invalid

This is a script from your discussion with macgaiver. Is it really work for you?
Thanks and Regards

NetworkPro · Thu Mar 19, 2009 1:42 pm

Q: Should I add...
A: Yes.

Back then I routed UsersA via one interface and UsersB via the other and the ECMP route is currently disabled on that router. Now, on another one, I am trying to accomplish slightly more complex scenario with more PPPoE interfaces + IP addresses in the ECMP route.

ssffzzxx · Fri Mar 20, 2009 3:56 am

Dear All, Network Pro

I think, I finally have my script like this :

'''/ ip address'''
add address=192.168.0.1/24 interface=ether1
add address=x.y.z.g/24 interface=ether2
add address=a.b.c.k/28 interface=ether3

'''/ ip firewall nat'''
add chain=srcnat out-interface=ether2 action=masquerade
add chain=srcnat out-interface=ether3 action=masquerade

‘’’/ ip firewall mangle’’’
Add action=mark-connection chain=input comment”policy routing all conenction from isp1 to router back to isp1” connection-state=new in-interfaces=ether2 new-conenction-mark=ether2con2R passtrough=yes

Add action=mark-routing chain=output comment=”” connection-mark=ether2con2R new-routing-mark=toether2 passtrough=yes

Add action=mark-conenction chain=input comment=”policy routing all connection from isp2 to router back to isp2” connection-state=new in-interfaces=ether3 new-connection-mark=ether3con2R passtrough=yes

Add action=mark-routing chain=output comment=”” connection-mark=ether3con2R new-routing-mark=toether3 passtrough=yes

‘’’/ip route’’’
Add dst-address=0.0.0.0/0 check-gateway=arp routing-mark=toether2 gateway=x.y.z.c distance=1 disabled=no

Add dst-address=0.0.0.0/0 check-gateway=arp routing-mark=toether3 gateway=a.b.c.d distance=1 disabled=no

Add dst-address=0.0.0.0/0 check-gateway=arp gateway=x.y.z.c,a.b.c.d distance=1 disabled=no

Add dst-address=0.0.0.0/0 check-gateway=arp gateway=x.y.z.c distance=2 disabled=no

‘’’/ip firewall rule’’’
Add action=drop chain=forward connection-state=invalid

There is a little problem, when I use 'ping' application, using -t parameters, and i unpluged the isp gateway cable, I got 'request time out' and after i wait for several minutes i didn't get reply.. ( I still got time out ) It couldn't find another gateway.

But, if I using the connection to browse internet, after i unpluged one ISP gateway cable, after doing refresh on the browser, i still able to reach the web address.

is this the correct result of this 'ECMP and failover' script?
is this script realy doing/working to perform load balancing?

Thanks in advance for you all.
Best Regards

NetworkPro · Fri Mar 20, 2009 1:10 pm

Improved:

/ ip address
add address=192.168.0.1/24 interface=ether1
add address=x.y.z.g/24 interface=ether2
add address=a.b.c.k/28 interface=ether3

/ ip firewall nat
add chain=srcnat out-interface=ether2 action=masquerade
add chain=srcnat out-interface=ether3 action=masquerade

/ ip firewall mangle
add action=mark-connection chain=input comment="policy routing all conenction from isp1 to router back to isp1" \
connection-state=new in-interfaces=ether2 new-conenction-mark=ether2con2R passtrough=yes
add action=mark-routing chain=output connection-mark=ether2con2R new-routing-mark=toether2 passtrough=yes

add action=mark-conenction chain=input comment="policy routing all connection from isp2 to router back to isp2" \
connection-state=new in-interfaces=ether3 new-connection-mark=ether3con2R passtrough=yes
add action=mark-routing chain=output connection-mark=ether3con2R new-routing-mark=toether3 passtrough=yes

/ip route
add check-gateway=arp routing-mark=toether2 gateway=x.y.z.c distance=1
add check-gateway=arp routing-mark=toether3 gateway=a.b.c.d distance=1
add check-gateway=arp gateway=x.y.z.c,a.b.c.d distance=2 comment="ECMP Route"
add check-gateway=arp gateway=x.y.z.c distance=3 disabled=yes comment="Failover but may not be needed at all! since ECMP Route already does it well."

/ip firewall rule
add action=drop chain=forward connection-state=invalid comment="Drop packets of invalid connections"

There is a little problem, when I use 'ping' application, using -t parameters, and i unpluged the isp gateway cable, I got 'request time out' and after i wait for several minutes i didn't get reply.. ( I still got time out ) It couldn't find another gateway. 1

But, if I using the connection to browse internet, after i unpluged one ISP gateway cable, after doing refresh on the browser, i still able to reach the web address.

is this the correct result of this 'ECMP and failover' script?
is this script realy doing/working to perform load balancing?

Thanks in advance for you all.
Best Regards

1: The ICMP connection is already established and is managed by ConnectionTracking mechanisms of RouterOS. Because of this, it must be disconnected and re-established for it to go the working gateway. This is why refreshing the browser works - it makes new tcp connections.

to MikroTik officials, is this right? Is it possible for the ping to get routed right away? Is it possible if you make changes to RouterOS for it? Maybe a feature request?

changeip · Fri Mar 20, 2009 5:58 pm

this would be a great addition, I want the same thing, but I am not sure that it could be accomplished. How would the router know which connections in the conn-track table need to be removed ? There are too many variables to know which ones should be removed automatically, I mean you wouldn't want to kill all open connections because that would kick off the working ones as well.

If you have specific cases that you know of, like a ping -t, maybe you can write a script under netwatch that removes connections with those specific entries. They would be specific to whatever source IP you were using. Try this:

/ip firewall connection> remove [find protocol=icmp]

That would kill off any outstanding icmp connections.

:put [:len [/ip firewall connection find protocol="icmp"]]

That would show you if there were even any to kill.

pace · Fri Mar 20, 2009 8:27 pm

I've not tried 3.22 yet, but large file downloads stop after 10-20 minutes of downloading. Like knects, pausing and unpausing the download causes it to start going again. I also have problems with long ssh sessions being open.

Anyone out there with a 433 that has ECMP working?

Just FYI, I've tried 4.0b2 (which supposedly has 3.22 bug fixes in it) and I've still got my ECMP download problem.

My question is still out there: does anyone has a 433 with ECMP working without assigning particular routes to individual devices behind the 433?

pace

NetworkPro · Fri Mar 20, 2009 8:41 pm

By the way, I can connect to your routers via the Internet via WinBox and see for myself and compare the config with my several configs. PM me with login details.

ValeriyZ · Sat Mar 21, 2009 9:08 am

Load Balancing. Where my error?

Upgraded to the version 3.20. Router executes one task only - Load Balancing ½ and to redirect ICMP on a certain interface (gateway). Before balancing was on principle:

/ip firewall mangle 
add action=mark-connection chain=prerouting comment= Inet_1 connection-state=new disabled=no in-interface=ether3 new-connection-mark= Inet_1 nth=3,1 passthrough= yes 
add action=mark-routing chain=prerouting comment="" connection-mark=Inet_1 disabled=no in-interface=ether3 new-routing-mark= Inet_1 passthrough=no
….

Now did according to the sample from wiki (http://wiki.mikrotik.com/wiki/Load_Balancing_Persistent) BUT ~ 10 minutes goes precipices of beginning to swing on the channel (gateway) of ether2.

Dear All. prompt me where I assumed an error.

/ip address 
add address=192.168.28.2/24 broadcast=192.168.28.255 comment=Inet_Modem_1 disabled=no interface=ether1 network=192.168.28.0 
add address=192.168.1.3/24 broadcast=192.168.1.255 comment=Inet_Modem_2 disabled=no interface=ether2 network=192.168.1.0 
add address=172.168.0.1/24 broadcast=172.168.0.255 comment=Output disabled=no interface=ether3 network=172.168.0.0

/ip firewall mangle
add action=mark-connection chain=prerouting comment="ICMP Redirect" disabled=no new-connection-mark=ICMP passthrough=yes protocol=icmp src-address=  172.168.0.0/24
add action=mark-packet chain=prerouting comment="" connection-mark=ICMP disabled=no new-packet-mark=ICMP passthrough=yes protocol=icmp src-address=172.168.0.0/24
add action=mark-routing chain=prerouting comment="" connection-mark=ICMP disabled=no new-routing-mark=ICMP packet-mark=ICMP passthrough=no protocol=icmp src-address=172.168.0.0/24

add action=mark-connection chain=input comment= Inet_1 connection-state=new disabled=no in-interface= ether1  new-connection-mark= Inet_1 passthrough=yes
add action=mark-routing chain=output comment="" connection-mark= Inet_1 disabled=no new-routing-mark= Inet_1 passthrough=yes

add action=mark-connection chain=input comment= Inet_2 connection-state=new disabled=no in-interface=ether2  new-connection-mark= Inet_2 passthrough=yes
add action=mark-routing chain=output comment="" connection-mark= Inet_2 disabled=no new-routing-mark= Inet_2 passthrough=yes


/ip firewall nat
add action=masquerade chain=srcnat comment="ICMP Redirect" connection-mark= ICMP disabled=no src-address=172.168.0.0/24

add action=masquerade chain=srcnat comment="L B 2" disabled=no out-interface= ether3
add action=masquerade chain=srcnat comment="" disabled=no out-interface=ether2

add action=masquerade chain=srcnat comment="L B 2" disabled=no src-address=172.168.0.0/24

/ip route
add check-gateway=arp comment="ICMP Redirect" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.1 routing-mark=ICMP scope=30 target-scope=10

add check-gateway=arp comment="LB 2" disabled=yes distance=2 dst-address=0.0.0.0/0 gateway=192.168.28.1,192.168.28.1,192.168.1.1 scope=30 target-scope=10

add check-gateway=arp comment="" disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=192.168.28.1 routing-mark= Inet_2 scope=30 target-scope=10
add check-gateway=arp comment="" disabled=yes distance=1 dst-address=  0.0.0.0/0 gateway=192.168.1.1 routing-mark= Inet_1 scope=30 target-scope=10

NetworkPro · Sat Mar 21, 2009 9:46 am

Dear ValeriyZ, I think I Fixed it, I also cleaned it from what was not needed:

/ip address
add address=192.168.28.2/24 comment=Inet_Modem_1 interface=ether1
add address=192.168.1.3/24 comment=Inet_Modem_2 interface=ether2
add address=172.168.0.1/24 comment=Output interface=ether3

/ip firewall mangle
add action=mark-connection chain=prerouting comment="ICMP Redirect" new-connection-mark=ICMP passthrough=yes protocol=icmp src-address=172.168.0.0/24 connection-state=new

add action=mark-routing chain=prerouting connection-mark=ICMP new-routing-mark=ICMP passthrough=yes protocol=icmp

add action=mark-connection chain=input comment="Inet_1 to Router" connection-state=new in-interface=ether1 new-connection-mark=Inet_1 passthrough=yes
add action=mark-routing chain=output connection-mark=Inet_1 new-routing-mark=Inet_1 passthrough=yes

add action=mark-connection chain=input comment="Inet_2 to Router" connection-state=new in-interface=ether2 new-connection-mark=Inet_2 passthrough=yes
add action=mark-routing chain=output connection-mark=Inet_2 new-routing-mark= Inet_2 passthrough=yes

/ip firewall nat
add action=masquerade chain=srcnat comment="L B 2" out-interface= ether3
add action=masquerade chain=srcnat comment="This NAT rule will masq the ICMP correctly!" out-interface=ether2

/ip route
add check-gateway=arp comment="ICMP Redirect" distance=1 gateway=192.168.1.1 routing-mark=ICMP
add check-gateway=arp comment="The ECMP Route LB 2" disabled=no distance=2 gateway=192.168.28.1,192.168.28.1,192.168.1.1

add check-gateway=arp comment="Route all connections to Router from Inet1 back to Inet1" disabled=no distance=1 gateway=192.168.28.1 routing-mark=Inet_1
add check-gateway=arp comment="Route all connections to Router from Inet2 back to Inet2" disabled=no distance=1 gateway=192.168.1.1 routing-mark=Inet_2

So, ValeriyZ, PM me , I will give you my bank account info if You Want to donate =) or you could give me login details to config it for you

ValeriyZ · Sat Mar 21, 2009 10:45 am

Dear NetworkPro greater thank you that you answered quickly.
I believe in to that you brought in correct corrections however able I will be to apply them only tomorrow early in the morning when little users will be. As an experiment with the use of new rules of Load Balancing made off a failure, I disabled them, to what being of «disabled=yes» is related in many rules. Tomorrow in the morning I necessarily will be bequeathed concerning your changes.

Now I carry out Load Balancing by an old method through mark-connection, nth and mark-routing.

BUT if in a code:

/ip firewall mangle
add action=mark-connection chain=prerouting comment="ICMP Redirect" new-connection-mark=ICMP passthrough=yes protocol=icmp src-address=172.168.0.0/24 connection-state=new

add action=mark-routing chain=prerouting connection-mark=ICMP new-routing-mark=ICMP [b]passthrough=yes[/b] protocol=icmp

passthrough=yes:

Reply from 81.19.70.3: bytes=32 time=31ms TTL=47
Reply from 81.19.70.3: bytes=32 time=99ms TTL=52
Reply from 81.19.70.3: bytes=32 time=35ms TTL=47
Reply from 81.19.70.3: bytes=32 time=193ms TTL=52
Reply from 81.19.70.3: bytes=32 time=21ms TTL=47
Reply from 81.19.70.3: bytes=32 time=109ms TTL=52
Reply from 81.19.70.3: bytes=32 time=108ms TTL=52
Reply from 81.19.70.3: bytes=32 time=36ms TTL=47

If passthrough=no to all is OK.

NetworkPro · Sat Mar 21, 2009 10:57 am

Don't worry about it. After implementing what I have suggested as a whole - with the ECMP, this problem will either be fixed by itself or will be easily fixed, via WinBox.

ValeriyZ · Sun Mar 22, 2009 9:14 am

NetworkPro

Your recommendations did not help unfortunately. Made those alterations that you recommended. Turned off the rules of ICMP. Ruter overloaded even. Result of totzhe, from 10 beginnings to swing On-line of films in a flow 10 mines were torn off all.

I will be upgrade on 3.22 or wait more stable version and then will apply your recommendations.

NetworkPro · Sun Mar 22, 2009 10:52 am

No problem mate. Just check what you did wrong or what is different in your setup, and correct and adjust the config. If you can.

NetworkPro · Thu Mar 26, 2009 5:20 pm

At a router near Sofia with 6+ ADSL lines we are currently testing ECMP + New connection marking in postrouting + routing marking those connections in prerouting, to try and ensure they get routed out the interface they originally got out to. Try for yourself, could be either the right solution or a good enough workaround.

ssffzzxx · Wed Apr 01, 2009 7:31 am

Hi All, Network Pro
Have any of you found better configuration for ECMP / load balancing / fail over?

Regards

ssffzzxx · Wed Apr 01, 2009 11:41 am

Dear ValeriyZ, I think I Fixed it, I also cleaned it from what was not needed:

/ip firewall mangle

add action=mark-connection chain=prerouting comment="ICMP Redirect" new-connection-mark=ICMP passthrough=yes protocol=icmp src-address=172.168.0.0/24 connection-state=new

add action=mark-routing chain=prerouting connection-mark=ICMP new-routing-mark=ICMP passthrough=yes protocol=icmp

Dear Network Pro,
From script above,
how does the connection knows how to route back to to certain ISP ( route the connection from ISP A and route back to ISP A).
Because usually we should make a mark in chain input and also chain output :

Add action=mark-connection chain=input comment”policy routing all conenction from isp1 to router back to isp1” connection-state=new in-interfaces=ether2 new-conenction-mark=ether2con2R passtrough=yes

Add action=mark-routing chain=output comment=”” connection-mark=ether2con2R new-routing-mark=toether2 passtrough=yes

CMIIW
Best Regards

NetworkPro · Wed Apr 01, 2009 12:37 pm

Why does this board have Personal Messages disabled?

Whoever still has load balancing, ECMP, etc, issues: Contact me via givememorebandwidth @
gmail dot com for solutions.

tured · Sat Apr 04, 2009 5:04 am

we being trying to make ecmp work with no luck we have 3 adsl but most traffic goes ut 1 interface and then when that one is full it shares the load betwenn the other 2 , we wanted to share evenly the load among all adsl, can that be posible, can you help please

nicopretorius · Sun Apr 05, 2009 9:31 am

I have added the configs as detailed below to my original posting and also upgraded all routers involved to 3.22. I still experienced issues with my PPTP connections. Specifically, the disconnection issues related to my second PPTP connection (from the same client to my secondary VPN server) which would disconnect at regular intervals. On inspection of my firewalls connections I saw that some of the connections for the second VPN would remain in the "TCP-wait state".

I solved this problem by policy routing any connection to my PPTP servers via one of my ADSL interfaces.

/ip firewall mangle
add action=mark-connection chain=input comment="" connection-state=new disabled=no in-interface=ADSL2 new-connection-mark=ADSL2Con2R passthrough=yes
add action=mark-connection chain=input comment="" connection-state=new disabled=no in-interface=ADSL1 new-connection-mark=ADSL1Con2R passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=ADSL2Con2R disabled=no new-routing-mark=ToADSL2 passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=ADSL1Con2R disabled=no new-routing-mark=ToADSL1 passthrough=yes

/ip route
add check-gateway=arp comment="Route All ToADSL1" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL1 routing-mark=ToADSL1
add check-gateway=arp comment="Route All ToADSL2" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ADSL2 routing-mark=ToADSL2
add check-gateway=arp comment="ECMP Route" disabled=no distance=2 dst-address=0.0.0.0/0 gateway=ADSL1,ADSL2

The questions/problems that I still have is the following:

1. It is problematic to know which are all the applications that one will need to policy route.
2. I'm not sure that the code as shown above is working 100% to force connections exiting via one interface to return via the same interface. The code works perfect for connections that are initiated from devices on the "outside" that connect to the router or "hosted devices" behind the router. I.e. in "/ip firewall connection" I can see the connections that are marked ADSL1Con2R and ADSL2Con2R. If I look at connections initiated from the inside, however, I very seldom see any of the connections being marked ADSL1Con2R and ADSL2Con2R which tells me the above mangle rules are not forcing/maintaining connections via a specific interface unless I add additional policy routing.
3. I tried to policy route all PPTP via a specific interface with the following:

/ip firewall mangle
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no dst-port=1723 new-connection-mark=PPTP passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting comment="" connection-state=new disabled=no new-connection-mark=GRE passthrough=yes protocol=gre
add action=mark-routing chain=prerouting comment="" connection-mark=PPTP disabled=no new-routing-mark=ToADSL1 passthrough=yes
add action=mark-routing chain=prerouting comment="" connection-mark=GRE disabled=no new-routing-mark=ToADSL1 passthrough=yes

But this does not work for PPTP. Does anybody have suggestions what I can change to policy route all PPTP connections via a particular interface. At the moment I'm doing it based on destination adress which is obviously not "correct".

nicopretorius · Sat Apr 11, 2009 9:21 am

See the following topic which acknowledge and explains the disconnects:

http://forum.mikrotik.com/viewtopic.php ... th#p151317

mehrzud · Fri Jun 17, 2011 9:30 pm

Does anyone have any idea about this post?
http://forum.mikrotik.com/viewtopic.php?f=2&t=52654