Problem with http-downloads via loadbalanced lines

canram · Tue Mar 21, 2006 11:55 pm

Hello,

I´m using Router OS 2.9.17 as PPPoE Server with transparent Web-Proxy. All outgoing request to port 80 with source address of the Web-Proxy is mangled with a routing-mark "webtraffic". Behind this PPPoE-Server there are two ADSL-Lines.

Via Policy-Routing, I loadbalance alle Webtraffic to both ADSL-Lines. Everything seems to work very fine. But If I want to download files via http (> 5 MB), the transfer gets aborted after a short time.

If I do no loadbalancing, the transfer succeeds without any errors. Is this a bug in policy-routing or am I doing something wrong?

Thanks.
canram_de

changeip · Wed Mar 22, 2006 5:03 am

What are your timeout settings on connection tracking? What does your load balancing config look like? you say policy routing - are you forcing traffic to a specific line or other, or using ECMP ?

Sam

canram · Wed Mar 22, 2006 8:55 am

Here are my tracking-configs:

----------------------------------------------
ip firewall connection tracking> print
enabled: yes
tcp-syn-sent-timeout: 5s
tcp-syn-received-timeout: 5s
tcp-established-timeout: 1d
tcp-fin-wait-timeout: 10s
tcp-close-wait-timeout: 10s
tcp-last-ack-timeout: 10s
tcp-time-wait-timeout: 10s
tcp-close-timeout: 10s
udp-timeout: 10s
udp-stream-timeout: 3m
icmp-timeout: 10s
generic-timeout: 10m
tcp-syncookie: no
max-entries: 532800
total-entries: 9869
---------------------------------------------

I´m forcing all Proxy-Outgoing Traffic to two different gatways.
You´re right. I´m using ECMP.

Here´s my route-config.

-----------------
101 A S dst-address=0.0.0.0/0 gateway=10.22.0.1,10.22.0.2 check-gateway=arp interface=ether1 backbone gateway-state=reachable scope=255 target-scope=10 routing-mark=Web-Traffic
----------------

Are those informatio helpfull for you? If you need other informations, please let me know.

Thank so far.
canram

nazemg · Wed Mar 22, 2006 4:33 pm

large download breaking for me too, tracking settings are:

enabled: yes
tcp-syn-sent-timeout: 10s
tcp-syn-received-timeout: 10s
tcp-established-timeout: 1d
tcp-fin-wait-timeout: 20s
tcp-close-wait-timeout: 20s
tcp-last-ack-timeout: 20s
tcp-time-wait-timeout: 20s
tcp-close-timeout: 20s
udp-timeout: 20s
udp-stream-timeout: 3m
icmp-timeout: 20s
generic-timeout: 10m
tcp-syncookie: no
max-entries: 552240
total-entries: 3259

mag · Fri Mar 24, 2006 10:41 am

I was discussing ECMP with some collegues and we are quite unsure if it would be reliable. Problems are known with HTTP and Instant Messenger-applications. One idea was if ECMP could be get to work on src-address only, these problems should disappear. i hadn't the time yet to test anything, but if someone has an idea...

There are a lot of ECMP-related threads in the forum, but i can't see any with a clear statement from MT, though.

Some "bonding" of cheap xDSL-lines becomes common now and a few competing vendors do offer it with their routers already.

cmit · Fri Mar 24, 2006 1:14 pm

Hi Matthias,

I actually do have an idea for some simple solution that is distributing users/traffic (only) according to source address...
If I find the time to put this up as a test installation I would post some more information.

Best regards,
Christian Meis

changeip · Fri Mar 24, 2006 7:27 pm

If your not using NAT then ECMP works perfectly. We have 2 100mb pipes being ECMP'd for outbound traffic and never have issues. I think the problem is that ECMP can't match on a source if all the sources are the same, ie a natted address. I think you can fix the issue by using policy routing and 'same' nat action, i think ...

Sam

spire2z · Fri Mar 24, 2006 8:46 pm

I would agree with change ip.

I spent ages trying to get that to work like you described using NAT I never succeded! I now use policy routing as next best option.

I wish somebody from MT would commect on this issue though as I never got to the bottom of it?

canram · Sat Mar 25, 2006 12:43 pm

Yes you´re right, but how should I use policy routing, if my source-machine is a webproxy?

spire2z · Sat Mar 25, 2006 1:04 pm

Well I guess you could use mangle rules to do that. I just run my proxy down one line and the rest of traffic down another. This works better anyway if both lines are from different providers.

mag · Sat Mar 25, 2006 2:09 pm

I actually do have an idea for some simple solution that is distributing users/traffic (only) according to source address...

sounds good, i'm looking forward to it ;-)

Yes you´re right, but how should I use policy routing, if my source-machine is a webproxy?

but not the web-proxy on the MT itself, i'd guess?

canram · Sun Mar 26, 2006 2:35 pm

Sure. The Webproxy on the Mikrotik-Box.

eugenevdm · Sat Apr 01, 2006 7:00 pm

I spent about three months to get ECMP working with 8 DSL lines (NATted on the DSL modems) and MT as a web proxy. In the end, after many weeks of research and frustration and loosing quite a few customers, I abandoned the idea.

ECMP just simply doesn't seem to work properly for this application. I had it sort of working by source-routing certain protocols such as IM and SSH and so on but the ECMP on port 80 just kept on breaking which was a bit of a show stopper. There was no specific pattern or time when it would break but it always broke.

I even posted many times on this board specifying my configuration and hoping for a decent reply from Mikrotik. But nothing happened. Recently I attempted doing this differently but using mangle on the input chain and then trying to source route on the output chain...but this apparently this only works with connection marks in version 2.8 and you will see in version 2.9 that policy routing does not regonise connections marks but only routing marks. I still have to downgrade my router to test this theory.

Here is the recent post I made regarding this:
http://forum.mikrotik.com/viewtopic.php?t=7472

Here are some old posts:
http://forum.mikrotik.com/viewtopic.php ... light=ecmp

My advice is to stop using ECMP and rather use policy routing before you start loosing customers. And forget about using a proxy server unless you decide to only proxy one of your lines.

I think the Mikrotik guys just got sick of this issue and they aren't responding any more. Perhaps someone should bring this up at that conference their having in Dallas because there is a presentation on load balacing. Many people are actually trying to do this using multiple cheap DSL lines and bonding is not widely available yet.

iNET2000 · Tue Apr 04, 2006 10:01 am

Well, if I knew for sure that MT had a solution to load balance two different lines from two different providers, I'd gladly go to Dallas to find out how. Of course, I have to suspect that if those features actualy worked, they'd answer any of the dozens of questions on here as to how.

If you search for 'load balancing' or 'ecmp', there are many dozens of people who have spent many dozens of hours each, and as far as I know, NO ONE has it working the way we'd all like it to work.

changeip · Tue Apr 04, 2006 10:33 am

Well, if I knew for sure that MT had a solution to load balance two different lines from two different providers, I'd gladly go to Dallas to find out how.

ECMP will load balance outbound traffic. If you use NAT and webproxy then you might have extra configuration. You might need policy routing. Policy routing is not as easy as it appears to be, you need routes for the local networks even. Also, multiple providers with different ip space? You have to deal with marking packets that you want to go back out the same interface they came in on.

there are many dozens of people who have spent many dozens of hours each, and as far as I know, NO ONE has it working the way we'd all like it to work. :(

We have it working just fine. We do not use web proxy however. If you have a specific problem or getting stuck post about it (with specifics).

Sam

eugenevdm · Tue Apr 04, 2006 10:49 am

You are right the Mikrotik staff does not respond to questions about ECMP and load balancing. There is probably a reason for this but who knows.

changeip: Isn't the point to use a proxy?

Anyway I spent another couple of hours trying to get load balancing working using source routing and a transparent proxy. Of course I cannot do this using 2.9 because the policy routing does not recognise connection marks. So I used another 2.8 router:

1. dst-nat port 80 calls from certain source addresses to redirect to local proxy
2. mangle connection marks.
3. use policy routing to direct connection marks.

Does not work. Unfortunately.

iNET2000 · Tue Apr 04, 2006 9:36 pm

Yes - they say there is ECMP routing as an option, and the manual says that this is a good solution because it's a ''per session'' round-robin balancing, so it does not suffer from the problems of per-packet balancing

It worked for us in 2.7, but after a year or so, our Cache drive was getting errors that resulted in having to clear & rebuild the cache once a week or so. We were advised to upgrade to 2.8.x and from that point, we can't get ECMP routing working with Transparent proxy any longer.

ekkas · Tue Apr 04, 2006 11:24 pm

I've also bashed my head trying to get LoadBalancing right, after attempts with my Mikrotik & Wingate proxy servers, I just purchased a DLink RV016 router. Can handle 7 Internet lines & works well, just plug and pray... (Excuse the mention of other brands here, but I'm just trying to help my 'buddies'

)

Some issues with some (especially banking) websites that doesn't like the idea of one session using more than one IP, but I route through one line only when we encounter that. So it's not perfect either, but maybe MT guys can download the source from DLink website and get an idea to implement this easily..???
I also think it's the way (loadbalancing) many of us are going, trying to supply high-bandwidth at low cost...

eugenevdm · Tue Apr 04, 2006 11:40 pm

I'll be sure to check out the DLink router because I really need to have an alternative that can load balance over multiple lines and do proxy.

I think what must have happened with Mikrotik is they had an underlying change in the architecture of their product which caused this to stop working. So instead of facing up to their customers and telling them it's too hard to fix they are keeping silent. Damn shame. So 90s.

ekkas · Tue Apr 04, 2006 11:54 pm

Ok good luck. I see you're from Stellenbosch. I'm from Jeffreys Bay.

)
The router isn't 'available' in RSA, so you'll have to import it or get a friend to bring it in... Gatkant van die wêreld... :-0

Ekkas

ekkas · Wed Apr 05, 2006 12:25 am

Ok good luck. I see you're from Stellenbosch. I'm from Jeffreys Bay.

)
The router isn't 'available' in RSA, so you'll have to import it or get a friend to bring it in... Gatkant van die wêreld... :-0

Ekkas

JimHeck · Wed Apr 05, 2006 9:47 am

Hi

I am one of the others who have been struggling with load balancing for months, but without success. I did get the policy routing working on ver 2.8, but on 2.9 it was difficult, and I am not sure if it was working really well. (We had 2 ADSL Lines).

At the MUM in Prague someone suggested "Eddies Box" for bonding multiple adsl line. See http://www.freestuffjunction.co.uk/bondedcd.shtml.

I took the plunge a couple of weeks ago, and bingo, it works perfectly, and you get a Linux firewall thrown in (as well as a load of other stuff) and its cheap. My box is running two adsl lines, and has been up for 10 days so far with zero probs. Its easy to set up (if you read the documentation). I think the only thing to watch out for is that your adsl provider supports genuine MLPPP (RFC 1990) as per Eddies page. There are some in UK, but you have to look around.

The nice thing is, that to the MT box, the multiple adsl lines are just one big fat ethernet connection

I'm a great enthausiast of MT boxes, and dont like to mention another project on this forum, but the two boxes together have changed my life !!!

rgds
Jim Heck

Eugene · Wed Apr 05, 2006 3:02 pm

Well, maybe it's time for someone to post description of the problem together with configuration snippets instead of just complaining that it doesn't work?
NAT+transparent proxy+ECMP works for large downloads for me

Eugene

JimHeck · Wed Apr 05, 2006 4:56 pm

Eugene,

I didnt mean to say that it cannot be made to work, only that I could not get it to work on my box. I did have a session with MT support a couple of months ago (not on this forum) and sent them about 8 versions of rif files over a period of 2-3 months, but still no luck. I think that they and I just ran out of steam!! (energy).

I am not complaining. As I said, I am a great fan of MT. I think one of the advantages of the solution that I have hit on is that not only does it balance load TCP sessions, but also UDP. I dont think the MT load balancing over non MLPPP links does this. Am I correct in thinking this.

rgds
Jim Heck

Eugene · Wed Apr 05, 2006 7:01 pm

You can post your configuration to enable us to help you.

JimHeck · Thu Apr 06, 2006 11:57 am

Hi Eugene,

Many thanks for your offer, but unfirtunately I have now lost the configuration details, and now that I have the "BandingBox" solution have no need to get this functionality of MT working.

Again, many thanks for your offer of assistance, its really appreciated

rgds
Jim Heck

eugenevdm · Thu Apr 06, 2006 1:21 pm

I will rebuild a configuration from scratch to show you an example. It will take me approximately one week to complete.

Eugenevdm

dannyboy · Mon Apr 10, 2006 8:46 pm

http://www.edimax.com

Works for me!! No problems at all

eugenevdm · Tue Apr 11, 2006 12:05 am

Edimax does not work for:

1) http://www.absadirect.co.za secure banking site

it keeps on prompting for you pin

2) MSN, ICQ (maybe)

Some banking sites do not like to have change source address changed ala ECMP.

if you do not require these sites you are ok, else you will have to keep on looking or source routing the problem areas

boardman · Wed Apr 12, 2006 7:20 pm

I would agree with change ip.

I spent ages trying to get that to work like you described using NAT I never succeded! I now use policy routing as next best option.

I wish somebody from MT would commect on this issue though as I never got to the bottom of it?

Can you post an example of your Policy routing configuration please.

Thks

eugenevdm · Wed Apr 12, 2006 10:42 pm

Here is one example that does not work because downloads break:

http://forum.mikrotik.com/viewtopic.php?t=3302

Here is another example of unanswered questions regarding traffic originating locally:

http://forum.mikrotik.com/viewtopic.php?t=6009

freebird · Wed Apr 19, 2006 1:31 pm

Any news here ??

I would like to know if anyone has a working configuration for 2 (+ X) DSL PPPoE dialins and load-balancing.

Any comments / working configs from Mikrotik ??

seandsl
--

jo2jo · Tue May 30, 2006 7:41 am

*bump*

Tue May 30, 2006 10:49 am

bump what? nobody posted the configuration for Eugene. so why complain if you have nothing configured?

jo2jo · Tue May 30, 2006 10:19 pm

well then he needs to do that..or something this is from what i can see the BIGGEST feature needed but has been ignored by mikrotik.

thats why i did the bump..to bring it up again.

Hellbound · Sat Jun 03, 2006 9:06 am

We've tried to get ecmp working since 2.8 and its like a dead feature in RouterOS , I'll be amazed if anyone can make it work.

we followed the manual step by step but it was negative.

pekr · Sun Jun 04, 2006 12:12 pm

Normis,

I am on some 8 or so MLs/forums for various kind of products, I do know many computing communities. MT's one, is the better one, friendly.

And I also note the fact, that users get help mostly from other users. But, I am known as a person not fearing voice my own opinion

So, I have to object to your post, I am sorry to do so. It does not help anybody and sounds kind of strange to me. Of course someone could eventually post some configs, but WHAT is the point?! You are product authors and you can see, that ppl get some trouble using your product. You claim you have some solution working, but you let ppl struggling for hours to get similar things to work.

Being your sales director, I would smash tech support for such kind of reply as yours. It does not help anyone, really. What is the point? So you can see ppl have some difficulcy, you can see, that ppl are starting to use different solution/boxes for it, which could distract someone from using your product in overall.

With few folks, we are discussing such topic on czech forum too. Well, it can be us, users, and probably it IS us, users, uncapable to get such things working. But then why to waste our time, and not produce some wiki-entry with some more examples when you claim you have it working? That would be much more constructive imo and would take some 10 - 15 minutes to produce?

Now sorry if you feel offended, please don't. I am sometimes kind of stright with my opinion. But I am also a business owner, and I like ppl being focused on solving potential problems. We are one of MT-only boxes satisfied customers, but as you can see from posts here, some ppl struggle to get some things working, and you decide the best way to help them, as this issue is here for quite some time, and docs are not apparently satisfactory enough in this particular case ....

Petr

Mon Jun 05, 2006 10:42 am

The Wiki and the docs do contain examples on this issue. But many times the docs and wiki is not enough, in this case attending a training, coming to the MUM or even hiring a certified consultant for a specific task will be very helpful to understand certain issues. RouterOS is a huge system with a manual that already is more than 600 pages + wiki + forum + support emails and people still find it too little. So I recommend the above - personal experience with someone to explain in person.

jo2jo · Tue Jun 06, 2006 7:02 am

OR

you could make load balencing more user friendly and sell ALOT more routers with route OS on them..... I used my own work around by breaking up traffic but load balencing was what initally brought me to this platform.

Although i did not get what i wanted, i discovered how great RouteOS is and have begun to learn as much as i can about it. If someone would have clearly said that load balencing is difficult on routeOS and requires training or a 1 on 1 lessons, than i probablly would still be sticking with power hungry unix boxes and never discovred this great OS....

EDIT: to clearify, the reason i thought LBing was not a problem on routeOS was due to the example in the wiki/manual....

joe

Eugene · Thu Jun 08, 2006 5:53 pm

Have you seen this: http://wiki.mikrotik.com/wiki/Improved_ ... e_Gateways ?

Eugene

eugenevdm · Thu Sep 07, 2006 8:31 pm

Eugene,

The example on the wiki although usefull for people who can't get ECMP working properly will do nothing for people who use a transparent proxy. Do you have a good load balancing example that uses Mikrotik's internal proxy server?

kind regards,
eugenevdm

jo2jo · Sat Oct 07, 2006 3:42 am

actually i think eugne is right,

http://wiki.mikrotik.com/wiki/Improved_ ... e_Gateways

is the answer to all these problems, i have not tested it but i think it will work reading through it...the key here was the use of:

/ ip firewall mangle

add chain=prerouting in-interface=Local connection-state=new nth=1,1,0 \
action=mark-connection new-connection-mark=odd passthrough=yes comment="" \
disabled=no

the connection state=New is the key

i will test when i can and report back.

tks

jo2jo · Sat Oct 07, 2006 6:34 pm

--THREAD SOLVED AND SOLUTION TESTED --
--THREAD SOLVED AND SOLUTION TESTED --

ok i tested it (SEE WIKI LINK ABOVE) and it works GREAT!! exactly as advertised / needed...again i think the missing key in everyones attempts before this (and there were ALOT of ppl trying to get this right) was the new-connection filter on the mangle rule

i didnt have any instant messengers around but i tested it with ftp, a few banking and investing secure sites...large http downloads...all ok..i'm getting 2 x 12 mbit cable modems and balencing all my users p2p traffic over that, and keepine very thing else on my fast dsl line

anyway...this should be made a stiki post. Oh and i only had 2 connections but i dont see why you cant expand this for "infinite" connections...amazing! when 8 months ago i paid 1500$ for a 3port load balencer that did not even work right.

joe

eugenevdm · Sun Oct 08, 2006 12:25 pm

jo2jo,

Before you get too happy please remember that traffic originating from the local proxy server goes via the output chain so still this example does nothing for people who use a Mikrotik as a transparent proxy.

thanks,
eugenevdm

iNET2000 · Mon Oct 23, 2006 8:12 am

The Wiki and the docs do contain examples on this issue. But many times the docs and wiki is not enough, in this case attending a training, coming to the MUM or even hiring a certified consultant for a specific task will be very helpful to understand certain issues.

Hi. I haven't been around here much lately. For me, I'd GLADLY hire a consultant or attend training, BUT can someone from MikroTik confirm that the current versions of MT work with both Transparent Caching Proxy and with Multiple Gateways at the same time?

It worked perfectly for us in version 2.7, but our cache drive occasional was having errors and having to have it's index rebuilt. The MT suggested solutions was to upgrade to 2.8, which did fix that issue, but Load Balancing with Transparent Proxy has never worked since.

Can anyone confirm that the current instructions do in fact allow proper load balancing with transparent proxy?

Thank you.

eugenevdm · Mon Oct 23, 2006 11:57 am

I don't think anyone from Mikrotik is going to answer you, so I'll give it my best shot:

Yes it's possible to load balance and transparently proxy port 80 using ECMP.
No it's not reliable because large downloads break.

Mon Oct 23, 2006 11:59 am

why? only if you do multithreaded downloads afaik

eugenevdm · Mon Oct 23, 2006 12:02 pm

All downloads break unless you use a download manager. It usually breaks after 2 MB and 6 MB, but this varies. It never goes much above 6 MB though.

iNET2000 · Tue Oct 24, 2006 5:12 am

I don't think anyone from Mikrotik is going to answer you...

Why would that be? Again, being new here, is that a general Mikrotik doesn't answer questions thing, or just a ''when asking about transparent proxy / load balancing'' thing?

eugenevdm · Tue Oct 24, 2006 5:48 am

No generally Mikrotik tries to answer questions, especially new questions. But old difficult questions just linger.

iNET2000 · Wed Oct 25, 2006 6:04 am

OK, thanx. This problem would fall into the old/difficult category then.

That's OK then - I just wanted to know if any solution had come up. It used to work perfectly for us in 2.7, but has never since then.

OK, now I know. Thanx.

eugenevdm · Wed Oct 25, 2006 7:51 am

Well I would still suggest you try it. I have heard some isolated reports where is works. It could really depend on your network configuration. For example, I have heard this works:

Internet <-> Public IP Router 1 <-> MT with transparent proxy, NAT and ECMP <-> Clients
Internet <-> Public IP Router 2 <->

but this definitely does not work for me, i.e. broken downloads:

Internet <-> NAT Router 1 <-> MT with transparent proxy and ECMP <-> Cilents
Internet <-> NAT Router 2 <->

Problem with http-downloads via loadbalanced lines

Problem with http-downloads via loadbalanced lines

ECMP download large files breaking

Alternative Solution ??

Working?

So many problems? So little time? look at this!

Who is online