Hi everyone,

I want load balancing with aggregation from two ISP with fail-over feature. As ISP1 has 30 mb and ISP2 has 12 mb,and near future I have another ISP3 with 45MB. I want output of those 87 mb. My network scenario is like this: as I attached the same.

I am new to mikrotik as well as ISP. I googling for the same and found it is possible that what I willing to do, load-balancing in mikrotik routerboard.

Note: ISP 1 is just connected with me Via Static IP 103.28.x.y and ISP 2 is via 121.54.x.y, I want Mikrotik Router LAN should be in static IPs as my MSC(Bandwidth Manager) is connected, that only needed for remote access of MSC. I need Load-balancing before MSC.

Please help? As those questions are panic me.
I have some questions in mind as:

1) load-balancing can be done via BGP or PCC or Bonding or round robin or using LISP method etc. Which one is best if my customer are connected to MSC (Bandwidth Manager) via PPPOE dialup with DHCP IPs.

2) As if BGP is used then what should I need to do configuration in Mikrotik. As someone told I need to purchase IP pool, AS number and allow routes to all ISPs. Then there is another confusion:
a) What type of IP pool I need for this e.g Provider Independent or normal one (as ISP1 and ISP2 gave to me). What
cost it will have, how much IPs I need for this network.
b) How much cost an AS number will have?
c) What configuration is required at my end on Mikrotik RB for BGP.
d) What type of routes need to allow on my ISPs and what configuration they will do for the same.

3) For PCC or round robin or Bonding ,Mikrotik Wiki has lot off answers. But I think those are only applicable if I used Local Lan IP at MSC WAN port and done its load-balancing with My all 3 ISPs static IPs.
a) Is it possible to use Local IP at Mikrotik outer end facing to MSC (Bandwidth Manager)
b) If this is not local IP then which ISP static IP I need to?
c) which one of them is best, kindly share what configuration I need to do in Mikrotik.
d) Should my all ISP need to change configuration if I used PCC,or Round robin or Bonding type Load Balancing.

4) For more scalability and reliability purpose what should I demand from My 3rd ISP as they going to terminate WAN link3.


One and Last question,
5) How in near future I will give Bandwidth to another ISP who is going to resell my Bandwidth to his customers. What configuration should I do at my end.

You cannot get a single aggregated stream of (isp1+isp2+1sp3) - especially not with three different ISPs. This is simply not doable unless you use some external box (hosted somewhere else on the Internet) that has at least twice as much available bandwidth, and do some creative tunneling to it from your 3 WAN addresses. Any public LAN-side addresses at your site would need to be provided from the space of whoever is hosting the remote server and sent to your site via the tunnel.

---

BGP will accomplish IP portability, and fault tolerance between all 3 ISPs seamlessly, but you will need a block of IP addresses at least /24 in size, or larger. If you can get one ISP to grant you this much space, then you can get an AS number from your regional registry (RIPE, ARIN, AFRNIC, LACNIC, or APNIC) - check their website for fees and qualifications for an ASN. You will probably not be able to qualify for your own provider-independent allocation anymore, since IPv4 space is pretty much gone (unless you're in the AFRNIC region, I think)... so one of your ISPs will almost certainly have to supply it for you, and then you'll need to get a letter of authorization proving you have permission to advertise it in BGP with your other ISPs - finally, you'd need to make sure that you never cancel service with whichever ISP whose IP space you're using, because you'd lose the addresses if you were to cancel with them.

BGP will not give load balancing - just "load sharing" - meaning that if some particular source of traffic chooses to route to you via ultimately ISP2, and the ISP2 link is full, it won't re-route additional traffic via ISP3... connections from that particular source (and any others arriving via ISP2 will simply have to deal with congestion) - you can tweak your inbound traffic a bit by adjusting AS-prepending and/or communities, but it is in no way granular, or bandwidth-utilization-aware.

---

PCC load balancing, on the other hand can let your network as a whole approach 100% load, but any given stream will be limited at maximum to whichever connection it is being currently routed through. Furthermore, you are correct in that you cannot use a public LAN address from ISP1 and choose to forward it / receive traffic for it on either of the other ISPs. You must completely use NAT.

---

If you're going to be an ISP and provide bandwidth not only to customers but to other ISPs, then BGP is pretty much your only real choice. I suggest that you start setting up a lab and getting some hands-on experience with it before you go production. You really won't be able to make a single 87M of bandwidth, though.

I hope this post has helped you understand what your choices entail, and what the pros/cons are for each method.
Good luck with your network.

You cannot get a single aggregated stream of (isp1+isp2+1sp3) - especially not with three different ISPs. This is simply not doable unless you use some external box (hosted somewhere else on the Internet) that has at least twice as much available bandwidth, and do some creative tunneling to it from your 3 WAN addresses. Any public LAN-side addresses at your site would need to be provided from the space of whoever is hosting the remote server and sent to your site via the tunnel.

---

BGP will accomplish IP portability, and fault tolerance between all 3 ISPs seamlessly, but you will need a block of IP addresses at least /24 in size, or larger. If you can get one ISP to grant you this much space, then you can get an AS number from your regional registry (RIPE, ARIN, AFRNIC, LACNIC, or APNIC) - check their website for fees and qualifications for an ASN. You will probably not be able to qualify for your own provider-independent allocation anymore, since IPv4 space is pretty much gone (unless you're in the AFRNIC region, I think)... so one of your ISPs will almost certainly have to supply it for you, and then you'll need to get a letter of authorization proving you have permission to advertise it in BGP with your other ISPs - finally, you'd need to make sure that you never cancel service with whichever ISP whose IP space you're using, because you'd lose the addresses if you were to cancel with them.

BGP will not give load balancing - just "load sharing" - meaning that if some particular source of traffic chooses to route to you via ultimately ISP2, and the ISP2 link is full, it won't re-route additional traffic via ISP3... connections from that particular source (and any others arriving via ISP2 will simply have to deal with congestion) - you can tweak your inbound traffic a bit by adjusting AS-prepending and/or communities, but it is in no way granular, or bandwidth-utilization-aware.

---

PCC load balancing, on the other hand can let your network as a whole approach 100% load, but any given stream will be limited at maximum to whichever connection it is being currently routed through. Furthermore, you are correct in that you cannot use a public LAN address from ISP1 and choose to forward it / receive traffic for it on either of the other ISPs. You must completely use NAT.

---

If you're going to be an ISP and provide bandwidth not only to customers but to other ISPs, then BGP is pretty much your only real choice. I suggest that you start setting up a lab and getting some hands-on experience with it before you go production. You really won't be able to make a single 87M of bandwidth, though.

I hope this post has helped you understand what your choices entail, and what the pros/cons are for each method.
Good luck with your network.

Thanks for kind information sir, That is very helpful for me.
Can you explain what type of external box your mentioning, any examples may you have please share? What method it is name? There is any method that Can bind 3 of ISP Bandwidth. I am really willing to do this.

So BGP can't make Mikrotik Lan side bandwidth 87 MB with load sharing. Can PCC will able to do that without need of public ip pool of /24 or with the /8 Public IP pool.

So BGP can't make Mikrotik Lan side bandwidth 87 MB with load sharing. Can PCC will able to do that without need of public ip pool of /24 or with the /8 Public IP pool.

No, even PCC cannot do that. All PCC does is balance connections so that no one link is carrying all of the data for your local networks. However, once a connection is set to use a particular WAN interface, then all packets for that connection are routed on the same interface, so no one connection can ever go faster than the link that it was connected on. Note that any connection that gets mapped to your 7Mbps connection will not be able to go faster than 7Mbps - assuming it's the only connection on that link at that time. If other connections are mapped to it at the same time, then they're going to be sharing a 7M pipe for that one particular connection.

Furthermore, PCC load balancing via multiple ISPs requires that you NAT everything to the IP address of the wan interface selected, so that the reply packets will come back to that same interface.

If you're planning to be an ISP to other ISPs, then you should consider using BGP and getting faster connections instead of trying to do things as cheaply as possible, because it will come back to haunt you later if customers get problems with your service.

So BGP can't make Mikrotik Lan side bandwidth 87 MB with load sharing. Can PCC will able to do that without need of public ip pool of /24 or with the /8 Public IP pool.

No, even PCC cannot do that. All PCC does is balance connections so that no one link is carrying all of the data for your local networks. However, once a connection is set to use a particular WAN interface, then all packets for that connection are routed on the same interface, so no one connection can ever go faster than the link that it was connected on. Note that any connection that gets mapped to your 7Mbps connection will not be able to go faster than 7Mbps - assuming it's the only connection on that link at that time. If other connections are mapped to it at the same time, then they're going to be sharing a 7M pipe for that one particular connection.

Furthermore, PCC load balancing via multiple ISPs requires that you NAT everything to the IP address of the wan interface selected, so that the reply packets will come back to that same interface.

If you're planning to be an ISP to other ISPs, then you should consider using BGP and getting faster connections instead of trying to do things as cheaply as possible, because it will come back to haunt you later if customers get problems with your service.

No sir, right know I am only plan to terminate downtime for customers, so PCC will be possible. ISP1 or ISP2 don't need to allow routes outside of my network, is it right? As I see another post this man is able to get total bandwith. I think I going to right direction and use right now PCC then after 2-3 months I will buy ASN with Pools.

Here is the MAN:
http://forum.mikrotik.com/viewtopic.php?t=31125

I think this will be right, or may I should use mangle!

Is there is any one can help me with my requirement? kindly share configuration need to done on my end.

No sir, right know I am only plan to terminate downtime for customers, so PCC will be possible. ISP1 or ISP2 don't need to allow routes outside of my network, is it right? As I see another post this man is able to get total bandwith. I think I going to right direction and use right now PCC then after 2-3 months I will buy ASN with Pools.

That's definitely a way to go about it - it will definitely keep the costs lower while you get started.
Just remember that your customers' equipment will need to use private IP addresses and you'll be doing NAT on each of your ISP links.

Here is the MAN:
viewtopic.php?t=31125

I think this will be right, or may I should use mangle!

The balancing methods commonly used are PCC and Nth matching. The post you linked to doesn't say that the guy was able to download a single stream at full ISP1+ISP2 speed - just that the links were balancing. That's what I want to explain to you - because you need to understand what your choices mean.

Balancing is not the same thing as bonding. You are thinking about it as if it were "bonding" - which requires that all links go to the same ISP.

If you balance between 3 different-speed links, then your network as a whole is going to be able to carry traffic approaching 87M (realistically you'll never see 87M in use - things would have to happen perfectly and in the real world, this isn't usually the case). However, nobody whose connection is going across ISP1 can go faster than ISP1. Of course you realize that if the customer is downloading two things at the same time, then one download could be coming from ISP1 and the next from ISP2, so in total, the user could have ISP1+ISP2 bandwidth available to them - they're just never going to be able to click on a link, and when it downloads, have a speed of 87M.

My other exception to this that I mentioned earlier (having a server located somewhere else that you can make 3 tunnels to it and bond it) well, this is technically possible, but realistically, it is expensive because you'd have to buy a connection somewhere that was at least 174Mbps in order to give full 87up/87down capacity to your tunnels- and of course you'd be paying for the 3 ISP links plus this ~180M link somewhere else.... It doesn't make financial sense to do that...

Thanks sir for your support!

My other exception to this that I mentioned earlier (having a server located somewhere else that you can make 3 tunnels to it and bond it) well, this is technically possible, but realistically, it is expensive because you'd have to buy a connection somewhere that was at least 174Mbps in order to give full 87up/87down capacity to your tunnels- and of course you'd be paying for the 3 ISP links plus this ~180M link somewhere else.... It doesn't make financial sense to do that...

Can you give this method real name and configuration senerio as an example?. As you know I am just the beginner starting out an ISP.