Hi There

I have setup where the Customer router (Router Boards) is connected to a MPLS ELAN (VPLS) cloud provided by a third party. We, the ISP are connected redundantly to that cloud with two Cisco Routers. I now set up two BGP
sessions from the Customer Routers to the ISP Routers for redundancy and announce the Prefix the Customer gets assigned and send a default route to them from the Cisco routers.

I now like to do load balancing over these two default gateways. I know that MikroTik has the standpoint that BGP can not do ECMP by design, but all other vendors I know can do it (Cisco, Juniper, Brocade, Fortinet etc.)

As the two BGP sessions do not end on the same router I can not use the loopback method mentioned in the BGP ECMP manual here: http://wiki.mikrotik.com/wiki/Manual:BG ... interfaces
But they reference a Cisco page to explain that other vendors do it the same way and that this is only the right way... fortunately on this same page Cisco explains how BGP multipath works! http://www.cisco.com/en/US/tech/tk365/t ... 45bf.shtml (Load Sharing When Dual-Homed to One Internet Service Provider (ISP) Through a Single Local Router)

Will MikroTik reconsider their standpoint about multipath BGP or do I have to switch back to Cisco for the customer routers?

Regards

Matthias

It is a hack and we also can do it.
http://wiki.mikrotik.com/wiki/Manual:BG ... interfaces

Please explain how this is different to what I already wrote. Loopbacks are not possible. There are two (2) BGP sessions to two (2) different routers. So the destination can not be the same.

Or am I wrong ?

Regards

Matthias

Read that cisco article and mikrotik article, they both uses the same principle and loopback address.

Have you red the section: Load Sharing When Dual-Homed to One Internet Service Provider (ISP) Through a Single Local Router


This is what I like to accomplish. The other examples are clear and is common practice, I agree...

This is exactly the same setup as Load Sharing with the Loopback Address as a BGP Neighbor except that remote end has two routers.
Setup on mikrotik router would be the same, both remote peers establishes BGP session to loopback address.

Remote peers are not the problem. I want two default routes on the Mikrotik to the remote peers. That does not change anything if I make a session to remote peer loopbacks.

So to make it clear you have two BGP R1 and R2 running iBGP between then. They bothhave peering to ISPs router (R3).
And you want to have ECMP routes on R1 and R2?

shell code

R1------------\
 |             \
ibgp           ISP R3
 |             /
R2------------/

No, the other way around:
R1 is Mikrotik
ISP_R1 and R2 are Cisco

Then the same setup as described in the manual
R1 has loopback address and both ISP routers should establish connection to loopback. Exact same config is also in cisco article.

Yes, exactly ... But with the little command "maximum-paths 2" in the bgp config!

It just limits ecmp gateways to two. Mikrotik does not limit amount of gateways.

So then the missing A on route 1 does not mean anything ?
If I make traceroutes from a public ip behind the mikrotik (no nat) to several locations, it always takes the path over route 0.

And now there is silence ...
Are you now convinced that there should be a "maximum-path" feature like in cisco ios to accomplish this? Or am I just to boring to be eligible for a rely.

maybe, post at least your BGP config first?..

Shure, here ist the relevan config of the mikrotik router.
peerA and peerB are Cisco Routers, which only announce a default route.
I'm happy for any inputs

Ok, sorry for that. Yes you are right your intended setup is currently not possible on MikroTik routers. BGP will install only one gateway.

Thanks for your confirmation. At least I know now that I'm not doing something wrong. Are there plans to support this? Would be a very nice and appreciated feature.

I am really disappointed to see that this is not supported. We have 4 CCR1036-8G-2S+EMs in production that I need to do this to coming up.

What can we do to get this feature implemented? We have been migrating away from Cisco to Mikrotik, but missing features like this are really hurting us.

Please add suport for many (2-3) ways for gateway for BGP

very interested

MT guys ,, can you please update the post if you are going to add this feature or not ?

Any updates on this topic ?
Is BGP ECMP routing on Mikrotik's roadmap ?

Regards,

Jon

Would like to know about this as well - Maybe this is a v7 feature?

It would be really useful.

BGP multipathing can be used, for example, to turn a router into a load balancer for a cluster of servers (web, DNS, etc) and it's a technique used in many large installations.

This article describes it very well.
http://bits.shutterstock.com/2014/05/22 ... -software/

This depends on a multipath BGP implementation that can do a flow or hash based load balancing. In a configuration like this several servers would announce the same IP address via BGP and the routers would distribute connections among them. Flow based balancing (or some kind of IP address hashing) is needed so that all the packets belonging to a TCP connection are sent to the same servers.

This technique is very cheap to implement (unless you have to use expensive routers for it) and in my opinion it's much more reliable and scalable than the typical load balancer "magic boxes".

So, do it, please!

Any updates on this topic ?
Is BGP ECMP routing on Mikrotik's roadmap ?

Regards,

Any updates on this topic ?
Is BGP ECMP routing on Mikrotik's roadmap ?

I would guess that it is all in the RouterOS v7 super duper routing code!

More than four (4) years since my initial post, and still there is no proper ECMP support ...

More than four (4) years since my initial post, and still there is no proper ECMP support ...

Welcome to Mikrotik... I guess they should remove the "Router" from "RouterOS", as it seems there had been no progress on the routing engine for years now. Instead hundreds of fixes for the wifi and access point stuff are appearing in every new ROS release. Mikrotik nowadays is a great access point and WISP supplier, but not a company I'd buy another edge/core router or switch from. Seems their focus and market orientation has changed after releasing the CCR-series. Sad story, as there would have been potential.

I've been wanting to see this as well, but i'd rather have recursive routing in IPv6 for BGP fixed first.

I've been wanting to see this as well, but i'd rather have recursive routing in IPv6 for BGP fixed first.

maybe in ROS7 years after 10 I’m very impressed with the support of mikrotik for 2019, it is planned to fully transfer the kernel to devices from other manufacturers.

I solved this for me with Routing Filters under Set Next-Hop-in with multiple Gateway-Addresses. This works for me even for BGP.

Please observe, that in CISCO the max-path n does 2 things: (1) Installation of best path and (n-1) looser pathes (2) Forming an ECMP Route from the multiple routes to the same destination prefix. The first part is not needed in MikroTik, because MikroTik installs always all BGP Routes in the Routing Table, of couse not neccessarily activ. Only the second part is required and could be done manually, by routing filters or by scripts.

Second thing to observe is, you will get never ever on any router 2 routes active pointing to the same destination prefix.

Second thing to observe is, you will get never ever on any router 2 routes active pointing to the same destination prefix.

But that is the idea of ECMP.... have 2 or more routes for the same prefix pointing to a different path, and the router has some mechanism to use them alternately for outgoing traffic.

Sorry, for my confusing wording. I think we mean the same.

ECMP Route is for me 1 route with multiple gateways, it appears as 1 destination prefix in the table. Of course, that are 2 or more routes , but considered as 1 with respect to administrative distance.

With 2 active routes I ment 2 seperate entries in the table, and each one is marked as active, as the OP in his last post was asking, why 1 default route route is active and the second is not..
My comment is, that the second will never get active.

No issue. We mean the same.

Regards

Came to that point today and i am disappointed that mikrotik developers are not able to give us some information if that feature is on a todo list or not.

Please add my +1 vote for ECMP, too.

as a workaround, ECMP works perfectly with OSPF instead of BGP, assuming your setup allows for it.
alternatively, the tip provided by heribertos using routing filters forcing the "Next-Hop-in" to your desired multipath gateways might work, though you'll lose the fully dynamic aspect of BGP by essentially enforcing static ECMP routes.

with that said, i'd really like to have ECMP supported in MPLS LDP. it's really weird that it works fine with OSPF but when you run LDP on top of it it just picks whatever gateway is listed first on the route, this is terrible for load-balancing VPLS.

I solved this for me with Routing Filters under Set Next-Hop-in with multiple Gateway-Addresses. This works for me even for BGP.

Thanks for this, I could workaround this issue and now working!

How have i to setup "Set In Nexthop" on Actions-Tab as @heribertos suggested?
What value have to be in there?

with that said, i'd really like to have ECMP supported in MPLS LDP. it's really weird that it works fine with OSPF but when you run LDP on top of it it just picks whatever gateway is listed first on the route, this is terrible for load-balancing VPLS.

It is even worse. RouterOS "BGP" (I am using quotes because it is not RFC 4271 compliant) picks not just the first gateway for equal cost routes, but even for UNEQUAL cost routes, since it does not care whatsoever for IGP costs to reach the BGP next hop gateway. This not only violates RFC 4271 but essentially also means that RouterOS routes packets in an BGP or MPLS cloud essentially in a random fashion. Yes, packets reach the target, but to go from New York to Boston, RouterOS first sends you first to Los Angeles, then Seattle, detour through Chicago, and then to Boston, since its "BGP" has no understanding of the IGP topology in an AS.

With any other BGP implementation on the market (even free Open Source solutions), you get true recursive lookups of the BGP next hops within the IGP routing table including their costs, as it is described in RFC 4271. And yes, with most you even get ECMP.

You are absolutely right. That's why I do not use mikrotik for larger scale BGP deployments. They should use something prooven as routing suite, like frr or bird underneath.

It is even worse. RouterOS "BGP" (I am using quotes because it is not RFC 4271 compliant) picks not just the first gateway for equal cost routes, but even for UNEQUAL cost routes, since it does not care whatsoever for IGP costs to reach the BGP next hop gateway. This not only violates RFC 4271 but essentially also means that RouterOS routes packets in an BGP or MPLS cloud essentially in a random fashion. Yes, packets reach the target, but to go from New York to Boston, RouterOS first sends you first to Los Angeles, then Seattle, detour through Chicago, and then to Boston, since its "BGP" has no understanding of the IGP topology in an AS.

That became clear to me pretty quickly when I started using RouterOS BGP, and I quickly decided to use a different AS number for every site.
(of course such decisions are only possible in a closed network not BGP-connected to internet)

The sad thing about this is that since the last 10 years, we see a creeping trend for "BGP everywhere". It is no longer just a WAN protocol for nation-wide ISPs and occasional Internet Exchange peerings, it first creeped into LANs and campus networks through MPLS and now we see whole data centre networks being constructed purely on BGP (i.e. RFC7938). That way you can scrap the underlying OSPF/ISIS completely and run on one protocol - BGP everywhere - from your spine to your leaf switches to your Internet peerings. MikroTik has catapulted itself out of that race with its current implementation. BGP ECMP is an absolute must in the year 2020 (actually it already was in 2010).

Similar to what that RFC proposes, I quickly switched to EBGP-only to avoid a lot of strange routing issues that I encountered at first when I had used the same AS for a couple of sites in the same region so there was a mix of IBGP/EBGP.
I now use IBGP only between routers at the same site that are directly linked.
Of course still no ECMP but at least no unpredictable routing.

@normis Is there any hope on ever getting ECMP implemented for BGP?! Considering it's the standard it should already be in ros.

ugh i guess there is no way to tag people on this forum

It's on the roadmap for protocol support in the v7 status page

https://help.mikrotik.com/docs/display/ ... col+Status

Thanks. Not very hopeful on it being any time soon since v7 has been in the works for years now :'(

It's on the roadmap for protocol support in the v7 status page
https://help.mikrotik.com/docs/display/ ... col+Status

I just need OSPF route conversion from v6 and then I'm golden.

Thanks. Not very hopeful on it being any time soon since v7 has been in the works for years now :'(

Mikrotik spent years putting the groundwork in place, building the framework for the new routing engine to ensure it would scale and be easy to maintain. They also hired a bunch more developers.

You might be surprised at how quickly features are added from this point forward...

Thanks. Not very hopeful on it being any time soon since v7 has been in the works for years now :'(

Mikrotik spent years putting the groundwork in place, building the framework for the new routing engine to ensure it would scale and be easy to maintain. They also hired a bunch more developers.

You might be surprised at how quickly features are added from this point forward...

Its a big software dev problem I come across it all the time - 80% of the work is background "do-nothing" code (to a customer perspective) so the 20% of the frontend is super quick, zippy, easy to add and remove things, make it modular etc. Once you get over that hump it becomes so much quicker.

@mrz

coming back to your first post

has anything changed since then?
https://wiki.mikrotik.com/wiki/Manual:B ... interfaces

I'm playing with multihop=yes and the target-scope that im receiving is 30 by default without any filter rule.

Thanks. Not very hopeful on it being any time soon since v7 has been in the works for years now :'(

Mikrotik spent years putting the groundwork in place, building the framework for the new routing engine to ensure it would scale and be easy to maintain. They also hired a bunch more developers.

You might be surprised at how quickly features are added from this point forward...

Well that would be real nice if this could move to a stable release real soon. ROS has just felt stagnant for the last couple years. Hardware moving forward while the os at stand still.

Thanks. Not very hopeful on it being any time soon since v7 has been in the works for years now :'(

Mikrotik spent years putting the groundwork in place, building the framework for the new routing engine to ensure it would scale and be easy to maintain. They also hired a bunch more developers.

You might be surprised at how quickly features are added from this point forward...

Well that would be real nice if this could move to a stable release real soon. ROS has just felt stagnant for the last couple years. Hardware moving forward while the os at stand still.

You must not have been paying much attention!

RouterOS v6 has seen an immense number of new features and bugfixes in the last 2 years. And on top of that RouterOS v7 has entered beta with an entirely new "ground up" routing stack, Wireguard support, Multi-Chassis LAG and other long-requested features.

Thanks. Not very hopeful on it being any time soon since v7 has been in the works for years now :'(

Mikrotik spent years putting the groundwork in place, building the framework for the new routing engine to ensure it would scale and be easy to maintain. They also hired a bunch more developers.

You might be surprised at how quickly features are added from this point forward...

Well that would be real nice if this could move to a stable release real soon. ROS has just felt stagnant for the last couple years. Hardware moving forward while the os at stand still.

In fairness, their routers are dirt cheap even by consumer router standards and offer massive value *even without* BGP ECMP. I'd like to see it, too. But their devices aren't exactly priced to support a rapid pace of development while remaining stable, either.

More than four (4) years since my initial post, and still there is no proper ECMP support ...

Well, is August 2021, still not. I found your post trying to solve the exactly same issue

same for me please

7.2rc3 is out now - and no ECMP implementation so far.

7.11.2 out, september 2023, still no ecmp support

7.13 Stable out now, NO ECMP Support

7.14.2 stable and still no bgp ECMP.

This is such a huge gap in Mikrotik's feature set. Ubiquity have had BGP multipathing in EdgeOS for a long time now. To not even have a timeline or response for years on this thread suggests it is not a priority despite this topic having among the most views on this forum. This omission will be a deal-breaker for our continued use of RouterOS.

Don't forget BGP UCMP, i.e. weighted ECMP with link bandwidth awareness.

I've been wondering if there's some sort of confusion at MikroTik about what exactly we're asking for.

Maybe someone at MT thinks they need to implement BGP ADD_PATH before working on this? Some of the comments I've seen over the years from them seem to point that way.

I bet most of us would be very happy with a single ECMP route per BGP peer, with no ADD_PATH required.

There is no confusion, for OP setup ADD PATH is necessary.

I'm sorry but I still don't understand why. My read was that the OP had two BGP sessions with a default coming from each, and wanted the ECMP route over those. So no BGP session is carrying more than one route per prefix. What did I miss, where's the need for ADD PATH?

but you can already have ECMP in any other case, each instance session can install its own best route and form ecmp.

Okay, confusion confirmed, although I can't rule out the confusion being in my head I don't understand what this has to do with ADD PATH.

The OP (like me) wants to have the defaults received from two BGP peers automatically result in ECMP routing. This is what's happening in all the other vendor gear I use (Cisco, Juniper, Bird on Linux, Dell OS10, etc), without configuring ADD PATH.

I think you're saying that there's a workaround: just have dedicated BGP instances for each peer, each of the defaults will look like the best path within that instance and get installed. Great. But that's not ECMP. There's nothing looking at the route costs to determine that they're equal. Every route from every peer is deemed equal, so as soon as you're talking about anything other than defaults, it falls down.

The way I'd expect ECMP to work without ADD PATH is that if you get far enough in the BGP route selection algorithm (basically at the point where you're comparing router IDs) both/all of the routes get inserted in the FIB (the main routing table in the non-ASIC case). That's what everyone else is doing. It doesn't have to change any BGP messages, at all, so I don't understand this functionality being gated behind ADD PATH. A single best path is still selected and advertised to peers. It's just that if there are routes with the same cost as the best path, they get inserted as well. That's it!

That only works (reliably) when you are facing the simple situation of two BGP peers that send you a locally originated route (e.g. a default route) over 1 hop.
When the network is a little more complicated (several routers with several links between them, i.e. a partial mesh) such simple hacks may develop into a routing loop.

Can you help me figure out a napkin scenario where it actually does cause a loop? I tried to think up one but couldn’t, at least not with EBGP and sane tie-breaking rules (like those I described).

The way you enable this on Junos (just add "multipath;" on the bgp level) and IOS (set "maximum-paths" to more than 1) really seems like standard operating procedure rather than "a hack" and is documented without much of a warning sign. So it's weird to me that this is ok for them, and others, and unacceptable here. They do support add path, though, so there's that.

I'm far from an expert at this so I hope I'll be educated. I'm getting somewhat worried about what I'm overlooking since I'm current running my Junos default free zone routers with multipath enabled, without add path, on a complex (well, not totally trivial) network, pretty much oblivious to any problems.

I agree that for BGP to install ECMP routes, add path is not necessary, but since both ADD PATH and ECMP would need changes in best path selection code, those features should be linked and implemented at the same time.

But that's not ECMP. There's nothing looking at the route costs to determine that they're equal.

In contrary it actually is ECMP. If you try you will see that v7 adds multiple routes with + flag to the same destination and all of those routes are active, meaning that these routes are ECMP routes. And if you want to exclude for example one peer from adding ECMP route change the distance with routing filters.

Can you help me figure out a napkin scenario where it actually does cause a loop? I tried to think up one but couldn’t, at least not with EBGP and sane tie-breaking rules (like those I described).

The issue is that in a more complicated network, a route that you get from a peer may ultimately be coming from/via yourself.
Normally BGP would avoid that but when you install routes beyond what BGP considers the best path, such paths may form.

but you can already have ECMP in any other case, each instance session can install its own best route and form ecmp.

I'm not sure what this means. Have employees at MikroTik never deployed BGP multipathing on Cisco, Juniper or even straight FRR on Ubuntu? There is no add-path involved. That is misinformation.
https://docs.frrouting.org/en/latest/bg ... -bandwidth

What does "each instance session" mean?

If I have two peers:
peer1: 1.1.1.1
peer2: 2.2.2.2

Am I supposed to have two BGP sessions with each or what?

To my knowledge none of the MT employees have said that add path is required to install ECMP routes.
And if you are so familiar with Cisco, Juniper and FRR and have at least once deployed BGP on the RouterOS you should know what BGP instance is.
There is a lot of info all over the internet on what BGP instance is, hint, try typing "bgp multi instance" in the google search box.

To my knowledge none of the MT employees have said that add path is required to install ECMP routes.
And if you are so familiar with Cisco, Juniper and FRR and have at least once deployed BGP on the RouterOS you should know what BGP instance is.
There is a lot of info all over the internet on what BGP instance is, hint, try typing "bgp multi instance" in the google search box.

You mrz are the one who said we need add path based on your public messages on this thread, stop making shit up.

In Cisco BGP multi-instance doesn't mean much, at best this is what it means:
https://community.cisco.com/t5/routing/ ... 05#M342994

On Juniper BGP multi-instance doesn't mean much beyond this:
https://www.reddit.com/r/Juniper/commen ... nos_mx204/

In both vendors, an instance can refer to an AFI (same ASN on both sides, same IP, but different AFI) or simply local-as spoofing/breaking the contiguous AS of a router.

Stop making shit up, this has absolutely NOTHING to do with BGP multipathing, and I've deployed BGP multipathing on Cisco, Juniper and Huawei.

Here's Cisco BGP multipathing, NOTHING to do with multi-instance:
https://vxplanet.com/2019/01/03/bgp-loa ... balancing/

Stop propagating fake information.

I take @mrz's comment to be that are NOT related other than if their changing BGP code, they want to do both at same time. e.g.

I agree that for BGP to install ECMP routes, add path is not necessary,
but since both ADD PATH and ECMP would need changes in best path selection code, those features should be linked and implemented at the same time.

I fully agree with @markonen and @DarkNate

One BGP-Instance, 2 BGP Peers, if both peers send 0.0.0.0/0 or any other prefix with 2-Hop or 1-Hop reachable so the costs are exactly the same, it has to be imported as ECMP-Route.

Having two BGP-Instances is not the same. Because if 0.0.0.0/0 is announced once via 2-Hop and the another with 4-Hops, both routes get installed as ECMP-Route because both instances have seen there routes only for there instance as best route. This will result in an unwanted routing.

BGP Additional Pathes is not nessecary for ECMP Routing, because with ADD-PATH only informations will be sent to the neighbor that normally filtered out by best path selection.

Also if a router have 2 Pathes and forms an ECMP, a neighbour have not to know wich two pathes are selected. Only send the first path on an ECMP-Route as long ADD-PATH configuration is not supported or not configured. ADD-PATH have to be configured for RX/TX path but is not neccessary.