Hi All
We have installed MT as our main router for using on multihomed connections with handoff from 2 providers. Both providers provide us with multiple PUBLIC class Cs.
We now want to do BGP failover so all our ips work if one provider goes down. We are in the US and I understand we need to get an AS # from Arin but the documentation MT is very little for BGP .
Is it as simple as configuring as per the document with multiple Class Cs from both providers.
Mikrotik is an excellent piece of software and works well for us far with 50mbps throughput on each provider (100mbps total)
Please help us out.
Thanks
Re: bgp
Ok, this "It served only on e prived" was realy hard to read. So the correct form: It served only one provider.Did you have 2 providers and IPs from both (multihomed)? How did they configure on their end. The documentation makes us put only 1 bgp peer how do you define multiple (because i am assuming each provider will be a peer)
Re: bgp
it would be even more interesting, if someone is doing this by using two (or more) mt systems and VRRP.anybody worked with mikrotik and multihoming bgp through 2 providers for seamless failover ?
I am sure people have used it for keeping it up and running.
regards.
matthias
Re: bgp
Maybe I'm missing something, but we're using it with dual providers and I simply created two peers on the Routing->BGP selection via winbox. We're taking full tables from each peer, which does cause a bit of a problem when we loose one because removing the routes through the lost peer appears to take awhile, but it does eventually figure things out.anybody worked with mikrotik and multihoming bgp through 2 providers for seamless failover ?
Make sure neither peer is sending you a default route unless you like seeing all your outbound traffic go through only one of them instead of through the interface that is the shortest path.
Re: bgp
it would be even more interesting, if someone is doing this by using two (or more) mt systems and VRRP.
regards.
matthias
We have to MTs and want to do this eventually but to keep things simple its better to do it on one and then implement VRRP and check how things are working
Re: bgp
Maybe I'm missing something, but we're using it with dual providers and I simply created two peers on the Routing->BGP selection via winbox. We're taking full tables from each peer, which does cause a bit of a problem when we loose one because removing the routes through the lost peer appears to take awhile, but it does eventually figure things out.
Make sure neither peer is sending you a default route unless you like seeing all your outbound traffic go through only one of them instead of through the interface that is the shortest path.
Great Thanks for the info . I am going to try this out. I actually want to control the outbound for each network. BTW whats your router config cpu/memory etc. since your taking full tables from each peer ?
Thanks again . Merry Christmas
The router in question is running a 1.7Ghz CPU and has 512Meg of ram. It normally averages under 10% utilization and roughly 400Meg free, so the box is probably way more than is actually needed.
One note though, while this box does have firewall rules, the connection table tracking has been turned off. If that was turned on I'd expect CPU usage to be a lot higher.
One note though, while this box does have firewall rules, the connection table tracking has been turned off. If that was turned on I'd expect CPU usage to be a lot higher.
Re: bgp
I need to clarify this statement, I didn't word it very well.Make sure neither peer is sending you a default route unless you like seeing all your outbound traffic go through only one of them instead of through the interface that is the shortest path.anybody worked with mikrotik and multihoming bgp through 2 providers for seamless failover ?
If you're taking full tables from your peer, the default route is going to go unused for the most part because you've probably got a more specific route in your tables from a peer. Still, I wouldn't have those peers send you a default, I'd set it locally as a static route if I felt the need for it to exist.
Another problem you could well run into is that a peer might send you routes that they've artificially weighted to favor them. What you'll see is that all of your outbound traffic favors that peer unless the destination is so close that one of the other peers becomes a better path. That's the case I'm fighting at the moment, I believe. Unfortunatly, MikroTik doesn't appear to have a way to view the raw routing tables instead of just the current best route, and it certainly doesn't have a way to alter the weights of incoming routes the way some other routers have. This leaves me with no way to verify my idea that one peer is messing with the weights of their routes, and no way to undo that tampering if I could verify it.
This means that even with multiple peers, our outbound traffic is almost exclusivly through just one peer. If that peer goes down, the traffic does flip to the secondary as it should, but only after the routes from the first peer have slowly gotten removed from the routing table.
Re: bgp
this means your wasting bw on the other peer by not using it? . Are your peers actually weighting -- are you really landing up using just one ? . It also means do you have to have static routing enabled and some sort of failover script changing the static routing where if you loose a connection you would want to change the static route which i think would be advertised by our bgp router to the internet ???????.This means that even with multiple peers, our outbound traffic is almost exclusivly through just one peer. If that peer goes down, the traffic does flip to the secondary as it should, but only after the routes from the first peer have slowly gotten removed from the routing table.
Just wondering how this would be done . Also wondering are we the only 2 people here concerned with multihomed bgp for seamless failover?
Re: bgp
The second peer is very underutilized, yes. My guess is that it's the weighting causing it, and the fact I can't verify that because MT provides no way to see the raw routing table is one of my complaints with them.Just wondering how this would be done . Also wondering are we the only 2 people here concerned with multihomed bgp for seamless failover?
There is no static route to change, we use BGP to advertise our networks by placing them in the list of networks it knows about and turning off all the other forms of announcements it can make. When a peer goes down, those announcements stop and our traffic flows through the remaining peer as the original routes timeout on the network backbone.
There might not be a lot of people using BGP on their border router or subscribed to these forums to get a good sample, but I know I've ran into enough problems with it that I'd prefer I didn't have to use it. The problems I have with trying to force the speed/duplex settings of an ethernet interface and the complete lack of error counters just make things worse on top of the limitations on how you can configure and monitor BGP.
One other thing you're going to want to watch out for is accidently re-announcing routes from one peer through the other. When I added our networks to the bgp config to announce them, I quickly discovered that those were in addition to all the routes I was learning via bgp through each peer. I wasn't a leaf node, I was an interchange point between the two peers.
My solution was to add an outbound prefix list to limit what I announced. I created a new prefix list with a default "reject" rule, added my networks and installed it. Despite that default "reject", I was still announcing everything and I believe that's a bug in the MT. To stop it from announcing anything except my specific networks I had to add an explicit "reject everthing else" as the last rule of the prefix list.
My solution was to add an outbound prefix list to limit what I announced. I created a new prefix list with a default "reject" rule, added my networks and installed it. Despite that default "reject", I was still announcing everything and I believe that's a bug in the MT. To stop it from announcing anything except my specific networks I had to add an explicit "reject everthing else" as the last rule of the prefix list.
I'm setting up something very similar to this, but I've got one annoying problem I haven't yet figured out.
I'm taking a full BGP feed (all 150,000 routes in the global table) from my upstream. (It was easy enough to set it up this way, as the router has plenty of RAM and CPU to spare.) When you do an /ip route print, or look at the routes list in Winbox, it shows you those routes. ALL of them. Takes Winbox about half an hour to load the routes window.
I'm using the same router to handle our internal stuff (about 50 static routes), and trying to find those few routes mixed up in the giant routing table is, um, a pain in the behind. (I just keep a printout of our static routes handy, just in case...)
Is there some hidden option to get RouterOS to *not* show me dynamic BGP routes? That would make my life so very much easier. (I could also do this by getting a second device that just speaks BGP, and creating a /30 between it and the internal router, but that'll cost a few dollars that I'd rather not spend if I can avoid it.)
I'm taking a full BGP feed (all 150,000 routes in the global table) from my upstream. (It was easy enough to set it up this way, as the router has plenty of RAM and CPU to spare.) When you do an /ip route print, or look at the routes list in Winbox, it shows you those routes. ALL of them. Takes Winbox about half an hour to load the routes window.
I'm using the same router to handle our internal stuff (about 50 static routes), and trying to find those few routes mixed up in the giant routing table is, um, a pain in the behind. (I just keep a printout of our static routes handy, just in case...)
Is there some hidden option to get RouterOS to *not* show me dynamic BGP routes? That would make my life so very much easier. (I could also do this by getting a second device that just speaks BGP, and creating a /30 between it and the internal router, but that'll cost a few dollars that I'd rather not spend if I can avoid it.)
Nope, and that's another problem report I sent in last summer. Even if you close that window in winbox, look at the yellow activity dot in the upper right corner, it's still solid yellow. Closing the box early doesn't stop the download process, it simply means it hasn't got a place to be displayed.
If you open up a terminal window though and display the route table from there, you can page through it one page at a time.
Displaying routes in winbox REALLY needs a setup screen to select what you want to look at. When I have to do this on my setup, I get lucky in that the routes are loading from the high numbered IP range downward, so I've got a few minutes to edit my static routes and then exit winbox to stop the download from eating all the memory in my workstation.
If you open up a terminal window though and display the route table from there, you can page through it one page at a time.
Displaying routes in winbox REALLY needs a setup screen to select what you want to look at. When I have to do this on my setup, I get lucky in that the routes are loading from the high numbered IP range downward, so I've got a few minutes to edit my static routes and then exit winbox to stop the download from eating all the memory in my workstation.
oh, good.
If you're having this problem, that implies you've used Mikrotik's BGP implementation to handle a full feed in a production environment, and that it didn't die. That's reassuring, as today was the first time I fired it up.
As a fix for this "problem" doesn't appear to be forthcoming, I'll probably just get a second rackmount PC to do the job. (A RouterBoard with enough memory could probably handle it, but a RB230 plus enough memory to be sure it'll work ends up costing about as much as a mid-range 1U, something in the 1.5GHz range. For the $100 difference, might as well get something that I *know* will work for the next few years.)
If you're having this problem, that implies you've used Mikrotik's BGP implementation to handle a full feed in a production environment, and that it didn't die. That's reassuring, as today was the first time I fired it up.
As a fix for this "problem" doesn't appear to be forthcoming, I'll probably just get a second rackmount PC to do the job. (A RouterBoard with enough memory could probably handle it, but a RB230 plus enough memory to be sure it'll work ends up costing about as much as a mid-range 1U, something in the 1.5GHz range. For the $100 difference, might as well get something that I *know* will work for the next few years.)
advantage of full feeed?
Is there any advantage of taking full feeds from both providers????
I have to decide whether my peers give me
1. Full routes will include the entire routing table minus the default.
2. Partial routes will include only localy originated routes
3. customer routes(no transit).
4. Default will be only 0.0.0.0/0
What should i choose
This is an option given by one of the peers . They also want us to join radb.net . Is it really so essential after paying arin to get the AS number ?
We have 1.1.1.x/24 +other /24s from one provider A [public ips]
and 2.2.2.x/24 from the other B [public ips]
I am also using policy routes at the moment to send specific 1.1.1.x ipaddress (not all of the range) data of A out through other B .
Wonder what will happen to this . BGP info is very scanty in MT
I have to decide whether my peers give me
1. Full routes will include the entire routing table minus the default.
2. Partial routes will include only localy originated routes
3. customer routes(no transit).
4. Default will be only 0.0.0.0/0
What should i choose
This is an option given by one of the peers . They also want us to join radb.net . Is it really so essential after paying arin to get the AS number ?
We have 1.1.1.x/24 +other /24s from one provider A [public ips]
and 2.2.2.x/24 from the other B [public ips]
I am also using policy routes at the moment to send specific 1.1.1.x ipaddress (not all of the range) data of A out through other B .
Wonder what will happen to this . BGP info is very scanty in MT
If you don't have an AS number, you're wasting time with BGP because the point of using it is to advertise your own routes. If your peers are doing the annoucing for you, then if you go down they still announce and traffic to you gets thrown away, you get blackholed on the network. If you announce, then when a peer goes down all the traffic starts coming in through the good peer because your announcements through the down one have stopped.
If you're going to take a BGP feed, you might as well take the full table. Smaller tables are a hack to try and let people with old routers that don't have enough memory to handle the full feed, still run BGP.
If you're going to take a BGP feed, you might as well take the full table. Smaller tables are a hack to try and let people with old routers that don't have enough memory to handle the full feed, still run BGP.
bgp
We already got our AS number from ARIN. What i wanted to know is to take full feeds from 2 providers . Its not a memory issue but as i see on this thread that its a management issue. I think there are few peole using bgp on mikrotik . Thats probably the only thing keeping it away from deployment on large scale. On the whole MT is gr8 peace of software can you please address my other questions.
Thanks
Thanks
Ok, I didn't read your last message clearly, I thought you were asking if you should get an AS number. Whether you join some other organization also is up to you, ask your peer why they think you need to join one.
You might as well take the full feed instead of a truncated one, it allows you to make better choices for which peer to use for any given outbound packet as long as one of the peers isn't artificially weighting their announcements. If you're doing policy routing though, that probably overrides BGP's choice.
You might as well take the full feed instead of a truncated one, it allows you to make better choices for which peer to use for any given outbound packet as long as one of the peers isn't artificially weighting their announcements. If you're doing policy routing though, that probably overrides BGP's choice.
I can't think of any good reason not to take the full BGP feeds...
Also, if you're big enough to get an ASN, you may also wish to consider getting your own direct allocation of IP space from ARIN, instead of depending on your upstreams for IP space. This one bit my company a couple years ago; when Cable and Wireless' American branch declared bankruptcy, closed up shop, and sold the remnants to Savvis, my company was using a couple thousand of CW's IP addresses, and they wanted them back. Emergency IP renumbering isn't fun.
Getting full BGP feeds (and, for that matter, getting a direct IP allocation, if you qualify) basically gives you more control over your network. While Mikrotik's BGP implementation doesn't support all the really juicy options (like MEDs, communities, confederations, or even simple traffic weighting), it might someday. Or you might someday put in a Cisco instead of your present router, which will allow you to do all kinds of wacky traffic shaping, fancy BGP load balancing, and so on.
In the meantime, it doesn't cause any real problems, is easy to do, takes minimal resources (you'll need at least 256MB of RAM to run Mikrotik and juggle two full BGP feeds, and I'd recommend 512, but either way RAM is fairly inexpensive), and just might be handy and useful someday.
radb charges way too much for no real benefit. Track down altdb.
Also, if you're big enough to get an ASN, you may also wish to consider getting your own direct allocation of IP space from ARIN, instead of depending on your upstreams for IP space. This one bit my company a couple years ago; when Cable and Wireless' American branch declared bankruptcy, closed up shop, and sold the remnants to Savvis, my company was using a couple thousand of CW's IP addresses, and they wanted them back. Emergency IP renumbering isn't fun.
Getting full BGP feeds (and, for that matter, getting a direct IP allocation, if you qualify) basically gives you more control over your network. While Mikrotik's BGP implementation doesn't support all the really juicy options (like MEDs, communities, confederations, or even simple traffic weighting), it might someday. Or you might someday put in a Cisco instead of your present router, which will allow you to do all kinds of wacky traffic shaping, fancy BGP load balancing, and so on.
In the meantime, it doesn't cause any real problems, is easy to do, takes minimal resources (you'll need at least 256MB of RAM to run Mikrotik and juggle two full BGP feeds, and I'd recommend 512, but either way RAM is fairly inexpensive), and just might be handy and useful someday.
radb charges way too much for no real benefit. Track down altdb.
-
-
Cameron Earnshaw
Frequent Visitor
- Posts: 90
- Joined:
- Contact:
I'm getting ready to add a second T1 from the same provider as the 1st. I now have a Cisco 1700 series router acting as the gateway and the MT behind it doing DHCP, firewall and web proxy duties for the LAN. The Cisco has a 2nd T1 interface. Do I need to ask my provider to bond the T1s or is it just a matter of connecting the 2nd T1 in the other port and reconfiguring the router? Would the MT with a Cyclades card be better suited to this than the Cisco?
(I think we're getting a bit off-topic for a BGP thread, but...)
Depends on the upstream provider. They're in a better position to give you a definite answer on this.
Some of them will require you to set up BGP just to do load-balancing. Some can do it with static routes. In either case, the extra hop between your Cisco and your RouterOS system adds very little latency (usually under 1ms), and it works. I'm generally not a fan of "messing with stuff that works".
Depends on the upstream provider. They're in a better position to give you a definite answer on this.
Some of them will require you to set up BGP just to do load-balancing. Some can do it with static routes. In either case, the extra hop between your Cisco and your RouterOS system adds very little latency (usually under 1ms), and it works. I'm generally not a fan of "messing with stuff that works".
prefix list
Happy new year !!
The prefix list looks very important considering the fact that our router will get flooded with traffic as an traffic Xchangepoint.
I am wondering how do i setup the prefixlists
I am doing this from winbox.
Should I create 2 prefix lists
1) rejectprefixlists (default action reject)
2) acceptprefixlist (default action accept)
Then I goto prefix rules
Under rejectprefixlists since i dont want our networks to be rejected should i add our connected networks under the rules as accept ?
What is the prefix list length????
Since the default rule is reject all others should get rejected.
BGP Setup -- redistribute connected
Then when i define the both the peers
the prefixlist in should be acceptprefixlist (this will accept all routes )
the prefixlist out should be rejectprefixlist(this will advertise only our connected networks). also what is the benefit of route reflect, it looks like it can do something significant for the interXchange point of view.
Am i correct in this kind of action ????
BTW I have 3 routing tables (main, isp1, isp2), with a static route isp1 and isp2 to their gateway as default routes which is the directly connected (/30) interface to us . We then do policy routing to send a particular ip out through our preferred source(isp).
We are using a 2.8Ghz box with 1gb ram so i think we are adequately covered for 2 full feeds.
MT should have better documentation for bgp.
The prefix list looks very important considering the fact that our router will get flooded with traffic as an traffic Xchangepoint.
I am wondering how do i setup the prefixlists
I am doing this from winbox.
Should I create 2 prefix lists
1) rejectprefixlists (default action reject)
2) acceptprefixlist (default action accept)
Then I goto prefix rules
Under rejectprefixlists since i dont want our networks to be rejected should i add our connected networks under the rules as accept ?
What is the prefix list length????
Since the default rule is reject all others should get rejected.
BGP Setup -- redistribute connected
Then when i define the both the peers
the prefixlist in should be acceptprefixlist (this will accept all routes )
the prefixlist out should be rejectprefixlist(this will advertise only our connected networks). also what is the benefit of route reflect, it looks like it can do something significant for the interXchange point of view.
Am i correct in this kind of action ????
BTW I have 3 routing tables (main, isp1, isp2), with a static route isp1 and isp2 to their gateway as default routes which is the directly connected (/30) interface to us . We then do policy routing to send a particular ip out through our preferred source(isp).
We are using a 2.8Ghz box with 1gb ram so i think we are adequately covered for 2 full feeds.
MT should have better documentation for bgp.
Update
UPDATE!!
Just got full feed from one of our providers. Had to do some default routes and move the default route to the "main" routing table at the bottom of the route rules otherwise all the routing out was happening through the single bgp session. Waiting to setup the other bgp session with our other provider and do some testing.
Lets see what happens ..
Just got full feed from one of our providers. Had to do some default routes and move the default route to the "main" routing table at the bottom of the route rules otherwise all the routing out was happening through the single bgp session. Waiting to setup the other bgp session with our other provider and do some testing.
Lets see what happens ..
ACTIVE
Full feed from 2 providers ACTIVE....
hardly any cpu load (nothing at all)
BGP works fine we had all our incoming coming in from one provider as soon as bgp started propagting properly. after the other provider started up we had all incoming come in normally through both again..
Now we have policy routes to route outbound tarffic via providers due to price issue per provider. I wanted to know how in case of any failure on the bgp we can switch off those policy routes , can we do it via a script or something else?
hardly any cpu load (nothing at all)
BGP works fine we had all our incoming coming in from one provider as soon as bgp started propagting properly. after the other provider started up we had all incoming come in normally through both again..
Now we have policy routes to route outbound tarffic via providers due to price issue per provider. I wanted to know how in case of any failure on the bgp we can switch off those policy routes , can we do it via a script or something else?
One of the things to watch for will be the load on the router when one of those sessions drops for some reason. I graph my cpu average with mrtg and it hits 100% for quite awhile as the router works to clean up the table after loosing a peer, and during that time the routes for outbound traffic are going to be wrong for any of them that were through the missing peer.
If I'm around when it happens, I find it faster to reboot the router rather than wait for it to clean itself up.
If I'm around when it happens, I find it faster to reboot the router rather than wait for it to clean itself up.
We have recently enabled BGP on two MT routers. One has 512mb RAM and the other has 1GB RAM. Both are 3.2GHz P4 using Intel server motherboards with PCI-X Intel fiber and copper GB interfaces.
We are running OSPF between the two as well as EBGP from each router to its upstream provider and IBGP between the two. The routers are announcing properly and when the link between the two is broken they adjust routes quickly. I have not seen any unusual CPU utilization at any time while BGP is running.
Each router is able to pull 150k+ routes quickly, however it takes WINBOX 15-20minutes to load all the routes. This is a major pain and MikroTik needs a fix that allows a view filter or some other mechanism to bypass the wait before you are able to check any of your internal routes.
I am very new to BGP and have the help of a seasoned Cisco tech. Overall the Cisco tech seems OK with the MikroTik OS, but is quick to point out many, many BGP shortcomings with MikroTik. We believe MikroTik will need to bring their BGP support up to speed with the rest of the router industry before much of the BGP benefits can be realized.
Best,
Brad
We are running OSPF between the two as well as EBGP from each router to its upstream provider and IBGP between the two. The routers are announcing properly and when the link between the two is broken they adjust routes quickly. I have not seen any unusual CPU utilization at any time while BGP is running.
Each router is able to pull 150k+ routes quickly, however it takes WINBOX 15-20minutes to load all the routes. This is a major pain and MikroTik needs a fix that allows a view filter or some other mechanism to bypass the wait before you are able to check any of your internal routes.
I am very new to BGP and have the help of a seasoned Cisco tech. Overall the Cisco tech seems OK with the MikroTik OS, but is quick to point out many, many BGP shortcomings with MikroTik. We believe MikroTik will need to bring their BGP support up to speed with the rest of the router industry before much of the BGP benefits can be realized.
Best,
Brad
hi
Well I tested the failover and when we switch off one of the connections to a provider. We loose ALL connectivity to the router and the providers for about a minute. Why would this be happening. Its only after a minute that it recovers back on the other line. During this test we desiabled ALL policy routes to ensure it was a pure routing switch over.
Any ideas and recommendations ?
Any ideas and recommendations ?
Who is online
Users browsing this forum: EnglishInfix, raphaps and 27 guests