Hello, I have been using MikroTik for around 2 years, and am familiar with a lot of the features.

I have been load balancing 5 x 6mbps DSL lines at my office and it works OK, however I now have access to a 1000 mbps internet connection at a datacenter 100 miles away, and I would like to utilize the data center Internet connection as my offices "WAN".

Location A (Office) 5 DSL Lines

Location B (Data center) 1000 mbps fiber connection

Is it possible to setup a site-to-site VPN bonding application, with Mikrotiks at each end, and then utilize Location B's WAN for Location A? I.E. be able to download at the maximum capacity of all DSL lines, in this case 30 mbps?

I have found some appliances that claim to do this but they are all over $8,000 each and require two. Can Mikrotik do this?

Hi,

You won't get more bandwidth using Location B's WAN, because you will be limited by your current DSL's bandwidth. The only reason I can think of doing this is if you need some latency sensitive traffic and "DSL to Location B latency" + "Location B to destination server latency" is lower (or better quality, ie less jittering) than "DSL to destination server latency".

On the other hand, you may try bonding EoIP tunnels and setup IPSec over them. Take care of checking each DSL status manually (scripts) to take them out of the bonding or you'll get packet loss if any goes down and it's EoIP tunnel remains active at the bonding.

bye

i am pretty sure if you set up 2 eoip tunnels and bond them together both on the remote side and the datacenter side you will see more bandwidth

i am setting up a test lab now to do this

Please post results over here - I'm in two minds about this as technically:

IF each line is 10mb download 10mb upload - the max combined total is 20/20 using a datacenter cannot increase this limit but however can potentially prove the flow rate as all source/destinations will be to one location at which point should be able to fetch the data from the remote side faster - and then creating a queue/buffering system to hold it.

but however can potentially prove the flow rate as all source/destinations will be to one location at which point should be able to fetch the data from the remote side faster - and then creating a queue/buffering system to hold it.

What? I've read that sentence several times and it doesn't make any sense to me.

Well I could be completely wrong but...
Lets say I'm downloading a file of 100mb across two 10mb lines (20mb total) at best I'll be getting 2mb/sec download taking 50seconds. Now try and take it to real life - where there is overheads and packet checks this would add another 5 seconds making a total of 55seconds.

Now if we introduce a server with 100mb line at the end;
100mb at best will get 10mb/sec taking 10seconds - add in 10% for overheads = 11seconds. This file now sits queued on the server;
Since the client is connecting to the server to download from the file is ready and queued (i.e. less hops/ping/overheads but more CPU) we can remove the overheads saving 5 seconds as there is constant network link between the locations.

Thats the way I'd see it anyway; if it works out to be like that who knows.

But something is for sure:
1) More expense
2) 1 point of failure
3) Will the end user care about 5 seconds?

There is as much overhead downloading the file from the data center as there is from a remote location. There's no such thing as a "file ready and queued" unless you bring WAN accelerators into play, which will cost you upwards of $50,000 (which would be a steal - decent solutions are six figures and up). Just because the data center has a 100Mbps uplink doesn't mean it's fetching the file for you at that speed if it's just a routing hop in the middle every goes through - it's just a conduit for the 20Mbps limited pipe at one end. And it certainly doesn't make any sense to cache files that large that you're very unlikely to ever serve again.

What is this proposed "server" doing at the data center? If it's just a router with RouterOS on it it doesn't "queue files" for clients. It's just a router.

On a side note, if I may suggest it, you may want to be more careful with units and capitalization. There's a difference between Mbps, MB/s, and MB - and most of it is made non-ambiguous by capitalization. I found your example very hard to read due to this.

That would make sense in a Remote Desktop/Citrix environment: you have the Citrix farm at the datacenter and connect to it from the DSL's lines. If you download someting in the remote session it will use the datacenter's bandwidth and traffic flowing by the dsl's would be only Citrix/Remote Desktop.

Besides that, I don't see the point of "using the datacenter bandwidth", as I stated before.

@tricer: I'm waiting for you to post your results of the EoIP tunnel tests. I tried it some time ago, but couldn't devote enough time to it. Thanks!!

I think the Background idear is to Bundle/Bonding Lines
Where ISP dont Dell Bonded Services...
Like MLPpp over DSL.
In Gernany MLPPP is expensiv, A-DSL Bonding
are very Interesting for me to, Datacenter
with FiberConbection to locate a Second Mikrotik
Like RB1200 oder RB2011 if avaible is no Problem.

Hope anybody find a solution!
I will Test at weekend,
RB1100 with 4x DSL 6k
To
RB1100 with 100MBit Fiber Connection

Mistry7

Hi,

Which are the expected results you want to accomplish with that setup?

I think the Background idear is to Bundle/Bonding Lines
Where ISP dont Dell Bonded Services...
Like MLPpp over DSL.
In Gernany MLPPP is expensiv, A-DSL Bonding
are very Interesting for me to, Datacenter
with FiberConbection to locate a Second Mikrotik
Like RB1200 oder RB2011 if avaible is no Problem.

Hope anybody find a solution!
I will Test at weekend,
RB1100 with 4x DSL 6k
To
RB1100 with 100MBit Fiber Connection

Mistry7

Point to point links are just as expensive, and you need point to point links to do your own actual bonding. Without bonding you're still just running PCC or some other load balancing scheme and get the exact same results.

It is easy to do. I ahve this running - 3 links.

* You need separate IP addresses on the data cetner side for every VPN link.
* On the office, generate one VPN link or every physical link. ROute all traffic do one physical link via IP based routing (i.e. target IP 1 = dsl 1, taret IP 2 = dsl 2 etc.)
* USe NTH routing to splpit packets to the data center.

On the data center, use Nth routing marks to split downstream traffic to the other side into the 5 VPN's.

Works like a charm here. I bundle:
* 2x 6/0.5
* 1 x 6/6

to a total of 18/7

I put QQOS on top and route ALL (!) traffic through the VPN because one thing you dont want is one linke clogged (no dynamic adjjustments).

It would be aaLOT nicer and MORE stable (here all goes down with one link) if Mikrotik would get around delivering a real PPTP server - PPTP has dynamic link management.

Bonding non-like links (or even spreading packets in the same connection across non-like links) is usually an absolutely terrible idea. It leads to a lot of out-of-sequence TCP segments being delivered, which will cause ACKs to be delayed and TCP windows being negotiated down, while also probably leading to a lot of retransmits. Depending on the application layer checking for traffic that uses UDP it could fare even worse. Whether you're "fake bonding" by putting one big VPN tunnel across multiple links or using something like MLPPP is irrelevant - it's the links not being equal in latency that makes this a bad idea. You have no control over the physical path, so you shouldn't treat the links as like unless they are terminated at the exact same point - at which point you can bond them up to that point, but no further.

But I guess I have no dog in this fight. Can this be done? Yes. But in my opinion it's pointless and counter productive as it will degrade the quality of the service provided across the link. Just because you can doesn't mean you should, or that it makes sense.

Did you get the results of this test?

Sorry, very out of Time at the Moment, but il be sure Test this between Chrismas an the new year...

Hi,

So first Test i done between Xmas an the New Year.
EOIP Bonding works for DSL Bonding.

Howto:

VPN over every ADSL-Line to Datacenter (different IP´s and Interface in Datacenter)
EOIP over every VPN Connection
Bonding Interface over the EOIP Connections

Todo:

Routing Static Subnet e.g. /25 over the Bonding interface (not doable in Test, no address space in Test avaible)

But i get real Bonded Line, with 13Mbit Down / 3,8Mbit Upstream (Bonded 2 x Annex J 7000/2000 ADSL Lines)

I buyed RB/1200 today and send it to Datacenter for Colocation.
All 10 Interfaces connected to real Gigabit Internet, enough ip adresses.
More test Follow.

mistry7

Nice!

Try to use that for jitter sensitive traffic like VoIP or Citrix/Remote Desktop. Also, try to emulate problems with the DSL lines. My main concerns about that setup are packet reordering and retransmisions and resiliency to DSL problems.

In my tests taking down one of the DSL's didn't took it out from the bonding and as a result I got a 50% packet loss. I don't remember right now if I used EoIP over VPN or direct EoIP. Also, I used DSL's of different providers and packet reordering became a problem too.

Thanks for your feedback!

@leonset


the loose of one Line is a problem at the moment,
Failover Test are not absolved no, but will follow.

But Bonding will work good for me!

mistry7

@leonset

Packet reordering is a problem you need same routing on the DSL Interfaces!
I testet VOIP over the bonded Interface with perfect voice quality.

I tested out needed MTU and set TCP MSS, on Upload i reach 3,8 MBit ( my links have 4 MBit) und Download 13,8 (Links have 14)
i think it is a very good result.


mistry7

I can confirm that using EoIP and bonding across ADSL links is an 'ok' way to get further bandwidth if you have a datacenter nearby to bond back to.

Preferably the links you're bonding would be from the same ADSL ISP or at least terminate to the same exchange (putting the latency over each link as close as possible to being the same).
You may as mentioned encounter some issues with out of order packets however provided you aren't saturating your links this is definitely a workable solution for aggregating some same-speed slower links.