Hello!

We have bonded 2 ADSL lines of the same speed (around 5 mbits) and same latency (around 7ms) using EoIP Tunnels.
The ADSL lines are set to MTU 1492, both EoIP tunnels are set to MTU 1500 and the bonding is set to MTU 1500 as well.
Tests (i.e. www.speed.io) resulting in around 10 mbits but http and ftp downloads are as slow as one line (its going up and down if watching Rx of the eoip tunnels: each tunnels got 1800k - 2.7 mbits).

I also tried to add a QoS from http://forum.mikrotik.com/viewtopic.php?f=9&t=33313 to the bonding interface but also no change (i think the QoS isn't really working).

Anyone got an idea what could be wrong? If need more configuration infos, please let me know. I mostly followed the configuration of http://wiki.mikrotik.com/wiki/Manual:Bonding_Examples

Regards,
Juergen

Hi,

You will have problems with TCP Downloads, because bonding interfaces with balance-rr mode will make packets arrive out of order.
So, if the server sends packets numbered 1,2,3,4 they could arrive in order 1,2,4,3

Therefore, the TCP algorithm will assume that packet 3 is lost, because it receives packet 4 too early.
That will trigger a TCP pipeline stall, and retransmit of Packets 3 and 4.

This will effectively kill your throughput if you are using TCP.

I would suggest, that you are using balance-xor with hashing algorithm layer 3 and 4. That way, you will still need
two concurrent TCP connections to use the full bandwidth, but you will have less problems.

Hello hedele,

thank you for your recommandation.
I thought i can only bundle two lines with balance-rr and balance-xor is just something like pcc load balancing.
I will try balance-xor with hashing layer 3 and 4.

Do you think some QoS/queue tree is important for that or not needed at all?

Best regards,
Juergen

Ah ok, it is really like PCC it just uses the full bandwidth more efficiently and i don't have problems with different ip addresses as i have on passive ftp and such.

Why the bonding doesn't have some intelligent logic which waits for the correct packet order before process it with the routing? You will say, because it's tcp. But bonding feature could have its own protocoll splitting and merging packets, don't it?

Any other tipps on increase performance with balance-rr on tcp connection to get the whole throughput of both dsl lines?

Juergen

Hi,

No, extra QoS or queue-tree should not be needed. Just do whatever you normally do in terms of QoS.
As far as I know, there is only one way of reliably bonding two links together - using ML PPPoE.

If your ISP cannot enable ML-PPPoE for your two circuits, you will have to create an EoIP over each
PPPoE connection, then do a ML-PPPoE connection over both EoIPs, and create another EoIP over that ML-PPPoE connection.
However, Mikrotik PPPoE Server implementation is not ML-PPPoE capable, so you would have to use
some Cisco Router as termination point for the ML-PPPoE connection. All that is a lot of work, so a 300 MHz Routerboard will
already struggle doing 3 PPPoE connections and 3 EoIPs in parallel.

Just imagine the workflow for one packet:
Packet ingress - EoIP Encapsulation - Packet splitting and 2x PPPoE Encapsulation - 2x EoIP Encapsulation - 2x PPPoE Encapsulation - 2x Packet egress
That really bogs down the CPU in the Routerboard. a RB 433 can handle less than 20 Megabits of traffic in that kind of configuration (with no additional firewalling or queueing).

I do agree though, that balance-rr could definitely use an update, so that it supports packet reordering.

I already tried to set up mlpppoe over the eoip tunnels but looks like pppoe is not getting thru eoip tunnels.
If i just add a simple pppoe server on the other side of the eoip tunnel and try to pppoe-scan or add a pppoe client, no packet will be received on the other router. Just looks like if packets is dropped for some reasons. I already googled for "pppoe over eoip" but found no working solution, so i stopped trying to get mlpppoe working

Ok, pppoe is working now over eoip. Looks like that was a firewall issue.
So i will try to set up some linux mlpppoe server solution to test if that works better than bonding.

Hopefully the mikrotik people will implement mlpppoe soon

Yeah give it a try, as ML-PPPoE does segment and packet reordering. It should give the expected results.

I googled since yesterday but cannot find any linux pppoe server which supports mlpppoe. The available Linux/MLPPP which is currently in alpha state also seems to be just a client for bonding DSL lines on a linux PC but not a server.
Does someone know a (stable) MLPPPOE server which can be installed on a linux server?

Well... that's exactly why I always mentioned cisco router and also why there is no ML-PPPoE Server in Mikrotik RouterOS.
I think there is a very rudimentary MLPPP server for BSD based systems, but I am not sure.

Unfortunately i got no chance to place a cisco at the datacenter

Hi together,

I am investigating in the same issue / solution you guys are looking for.

I just want to tell you what I found so far. There are 2 products on the market at the moment that do something called like

Multi-Wan Bonding

Both for both products, you need to place a central hub in to a datacenter and a router were the WAN lines are terminating, basically at your home or office where the bonded bandwith is needed.

1. Viprinet: http://www.viprinet.com

2. Peplink Site-to-Site VPN with Drop-in mode: http://www.peplink.com

Both of them work with a modified TCP stack and a VPN, but both solutions are in the upper price segment > $ 5000.

BTW: Just to make that clear I am not in any relation with the mentioned companies, this is just the result of my personal opinion because I m investigating in how to bond the following scenario:

1 x WiMax: 10/2 Mbit/sec
4 x ADSL: 2/0.3 Mbit/sec

The key point to lower the cost here is to share the central hub in the datacenter and reuse it. So if somebody in EU is interested to share a hub close to DE-CIX in Frankfurt talk to me.

--
Kind Regards
Uwe Guenther
uwe@cscc.de

I guess it's probably easier (and cheaper) to house a Cisco 18xx series Router and a Routerboard somewhere (if all lines are the same speed)... and it should only cost a fraction of that.
The Routerboard terminates the EoIP necessary, and bridges them to the Cisco. The Cisco 1800 series can work as LNS to terminate MLPPP.

This should be possible to realise with less than $500 invest. A Cisco 1800 Series router should handle about 40 Megabits of ML-PPPoE Traffic.

On bonding several different WAN connections... well that might be a little tougher

Thanks for our answer,

BTW: If I use EoIP why do I need the ML-PPPD on the other end instead of simply bridging my 5 different broadband EoIP lines? Goes there something wrong with the Ethernet bridge I create on top of the 5 EoIP devices?

I am just curious..

a bridge with 5 EoIPs on it will behave the same as a bridge with 5 ethernet links - you will get a loop.
alternatively, if you are using STP, only one link will be active.

So creating a PPPoE Client on one bridge interface will not help, you will need ML-PPPoE.

However, ML-PPPoE can only multiply the slowest link capacity.
So, in your case, you would get 5x ADSL speed, instead of 4xADSL and the WiMax speed added.

So which solution would work for me then. Looks like I am screwed and I have to go to implement the vibrinet thing or rent it from meconet.de http://www.meconet.de/de/loesungen/host ... index.html

But their sales representative told me I hav to calculate in additional 15-17 % package overhead.