I'm about to put a RB1000 into production shortly. I'll probably be running ROS v3.21 (beta) on it, as it will be used as a HTTP proxy forwarder to a parent proxy, and I need to able to forward HTTPS that is fixed in v3.21.

This router will also be eBGP peering with two peers and receive a few hundred prefixes from each peer. One peering directly over an ethernet if and the other over an IPIP over IPsec tunnel.

This has worked flawlessly in my lab environment, but now that I'll put this into production, I'd just like to ask the community, what is the recommended ROS version to run BGP on that is most stable?

Stock routing package or routing-test? Or perhaps a backlevel? If a backlevel is required to get a stable router, I'll have to get two of them, one to do the routing and one to use as a proxy forwarder.

ROS 3.18 has MD5 issues with cisco and juniper router's peerings
ROS 3.20 has some problems with routes that disappear when a new peer is established
ROS 3.20+ Routing Test looks good, i'm using it and works fine.. but sometimes it hangs for a while when acquiring the full internet routing table. And looks like it doesn't use all the multi-cpu of a quad-core x86. Might work better in a RB1000

RB1000 has the same issue with the cli locking up when querying the route table. Ironically it always works fast when you query specific prefixes (eg. /ip route print where dst-address=n.n.n.n/xx).
It doesn't seem to be loosing traffic when it locks up, so it's not harmful in any way.

ROS 3.20 has some problems with routes that disappear when a new peer is established
ROS 3.20+ Routing Test looks good, i'm using it and works fine.. but sometimes it hangs for a while when acquiring the full internet routing table. And looks like it doesn't use all the multi-cpu of a quad-core x86. Might work better in a RB1000

Ouh, dissaperaring routes doesn't look good. Have you tried 3.21(beta), or heard anything about it getting fixed there?

Otherwise, it seems like 3.21 + routing-test should be my best bet right now.

I don't have any link to 3.21 yet

3.21 is not released yet. It's only in beta. Search for "3.21" and "https" here in the forum and you will find links to the beta. But keep in mind that it is a beta and not for production use.

It would still be nice to know if it fixes any BGP bugs though.

As long I use ROS (since 2.8.x), I do not use its BGP implementation.

Our network is big (more than 800 private AS'es now) and reveals easily any problem or bug. Most stable is probably cisco and quagga's stable versions.

Many people here around, say that mikrotik is not bad with BGP. We (in AWMN) have test it extremely, trying to avoid the use of quagga with extra router (which makes it more difficult to administer). Today I cannot trust mikrotik's BGP implementation. I would not even try it.

I use mikrotik with success for, firewall, wireless, hotspots, vlan and special interface manipulation, but not for routing.

mojiro, good feedback.

I'm evaluating platforms for bgp. We have been using zebra/quagga for many years to provide bgp multihoming. Also using ospfd.

I'm questioning it right now because of some problems I'm seeing. I'm interested in Medianet's comments about quagga "losing routes". I'm seeing some unreachable destinations (eg. ns1.google.com!) when I have 2 bgp peers installed, but if I remove either of the 2 bgp sessions then it works fine. Is this similar to the behaviour others have seen?

I was considering Mikrotik bgp, but maybe not now. I was kind of assuming that Mikrotik's bgp is really just quagga in disguise or something like that.

Cisco would be an expensive experiment, so if I have other options then I'd rather explore them all first.

Mojiro: Have you reported your findings to MT? We use a rather small configuration of Mikrotiks BGP routing, and would love to utilize it more, but if indeed there are these bugs you describe, I would really appreciate if you took the time to report them to Mikrotik, so they can be fixed.

@scottt, Eising: don't use it it has so many bugs and is so unflexible, i would not recommend it for anything more than two peers with just the smallest acls or filters you can think of as mojiro said, you can't use it in anything bigger than that.

get a juniper 2320 if a cisco is too expensive for you. rockstable, all features of junos is also in this box and great for traffic <350mbit/s. and it costs around 1600 euro streetprice including two ge-interfaces. and you get loads of free support in irc or forums, as many people are actually using it with small and big bgp-setups.

@scottt, Eising: don't use it it has so many bugs and is so unflexible

If you have found so many bugs maybe you can share your knowledge with Mikrotik support? And why Mikrotiks BGP is not flexible?

i personally haven't found that much bugs, simply because many things cannot be configured (as they are non-existant) and can't generate errors because of that

but there were many times of session-disconnects, no communication after the hand-shake, no propagation of routes. most times a restart fixed everything, but in a production environment you cannot restart a core-router once a week just because it stopped working

i tried to debug this a few times with mt-support, and many others did the same as you can read in quite a few threads on this forum, but mostly the only solution was a restart or an update which created other problems not only in the bgp-stack.

so for me this experiment is done, i won't try it until there have been some really really really promising case studies with traffic more than 300mbit/s and more than two peers. as openbgpd is a fine and stable software-router and juniper makes really very cheap routers there is no need to use mt desperately in every part of our network.

don't get me wrong, i like the software and most of the routerboards are fine, but there are parts where mt is good at (wireless, pppoe) and there are parts where mt (in my opinion) should have left the market for another company, as a bad protocol-implementation is worse than no implementation, especially in a thing like bgp, which the big mass of their customers is not needing, but is a backbone technology, so they cannot depend on their customers to beta-test it as they do with other technologies.
i can test a stupid-l3-switch in any part of my network without any problems, but a bgp router can only be tested in a realistic environment with working peers and live traffic, so i need to replace a maybe working bgp-setup to test a new product. that will happen only rarely i think

@scottt, Eising: don't use it it has so many bugs and is so unflexible

If you have found so many bugs maybe you can share your knowledge with Mikrotik support? And why Mikrotiks BGP is not flexible?

Yes, please! I really want to run my BGP setup on Mikrotik, so please, report every bug you find, so it can be fixed!

http://marlow925.blogspot.com/2008/10/m ... 6-bgp.html


ipv6 is one of the really b0rken parts, so start with that.

thadem,

What version did you test on ? IPv6 BGP was a problem in a few versions but seems to work now in my setup. I am using 3.17 and 3.19 now and there are some lingering bugs in 3.x BGP. BGP on 2.9.51 is very stable with 100k - 200k routes.

Can anyone really say that any other brand of BGP handles 500k+ routes without a problem? I don't even think crisco handles it very well.

Sam

i talked to marlow yesterday (thats the guy who wrote this blog-entry) and he said, the issues he had are fixed in the last versions.
nonetheless he is not using it anymore, as there is too less support in the rest of the system (dual stack or single stack) for v6 and he also had stability/performance issues.

@changeip: junos is really good nowadays, i would presume it can handle this without any glitches.

i tested with some 3.x version, but cannot really remember which specific one, was somehting in the middle of 3.x

Probably right now I cannot remember all the problems we had in past with mikrotik's BGP. In general, the problems had to do with the peers, not with the number of routes.

ex. One peer -> no problems, Second peer -> problems arise...

My router BGP v3.20 restart and freeze randomly from 2 till 8 days. Technical support could not help me. I switched off SNMP, changed all network cards on Intel-Pro, changed motherboards, memory and the processor - is useless. I exclude hardware incompatibility since the same equipment without problems works pptp-server with the big loading - more than 1000 simultaneous connections. Problems really do not exist at 1 BGP connection as soon as you connect the second - casual reboots and freezing. I very much would wish to hear people at which without problems works BGP FULL VIEW more than on 2 links.

Use v3.10.
I have very big problem whith 3.20. It is unstabel version!

3.20 stably works more month without reboots in quality pptp a server.
Tell please - v3.10 for you works stably and without problems BGP as a router with two and more full BGP links?
MultiCPU use?

Interesting discussion, could people mention more about what they are running on (i386 or RB1000 etc) when they have faults?

We are running ROS 3.11 on a RB1000 with 300k+ routes and several peers (8) and current uptime of 221 days.
Im just planning to upgrade to 3.20 so keep the discussion going

Cheers

Justin

to JwTPN:
You really accept Full BGP?
Full Bgp = +- 250k routes *3 peer = 750к routes in route table.

firewall, Queues, dynamic IP access list, use on the same router?
multicpu enable?

Thanks

You can upgrade the RAM of the RB1000, so multiple full tables shouldn't be a problem at all...

to JwTPN:
You really accept Full BGP?
Full Bgp = +- 250k routes *3 peer = 750к routes in route table.

firewall, Queues, dynamic IP access list, use on the same router?
multicpu enable?

Thanks

the main routes are 270k+ of international from our upstream (national Telco), other mostly small peering with other ISPs and National traffic...
We did have two upstream providers in there a little while ago and it multihomed well, cant remember details as i wasnt doing that at the time. but had a lot more routes then.

Otherwise, mangling international up/down and national up/down (some layer7) and the coresponding queues to shape everything.
(its obviously our edge router to the outside world)
AVG CPU usage 5% over the last week and 21% max usage
handles around 100gb of traffic per day.
Stock routerboard 1000 with 512mb of RAM

Maybe i should leave it alone and not upgrade it then eh?
are other people seeing the same sort of loads and results?

Cheers
Justin

AVG CPU usage 5% over the last week and 21% max usage
handles around 100gb of traffic per day.
Stock routerboard 1000 with 512mb of RAM
Justin

That is very impressive! I'm not pushing nearly as much traffic as you are, and have similar good results.

Are you doing any traffic engineering/filtering?

AVG CPU usage 5% over the last week and 21% max usage
handles around 100gb of traffic per day.
Stock routerboard 1000 with 512mb of RAM
Justin

That is very impressive! I'm not pushing nearly as much traffic as you are, and have similar good results.

Are you doing any traffic engineering/filtering?

i will admit its taken a couple of goes it getting the mangle/filters setup just right to keep the CPU load down, i have it setup a little different to a lot of examples i see here.

It is doing some basic filtering to stop ssh,ftp bruteforce etc..
otherwise just marking ALL packets going through the unit some like p2p, skype etc with layer7 filters.
And using queue trees afterwards.

That answer your question?

Well, I was actually refering to BGP traffic engineering, ie. prepending, as-path filtering etc.
You do any of that?

Well, I was actually refering to BGP traffic engineering, ie. prepending, as-path filtering etc.
You do any of that?

haha yeah sorry, its late here my brains not working
no not really sorry

I'm running 3.20 with the standard routing package on an x86 network appliance. We have 2 BGP peers with full routes and everything seems to be running fine. We do about 800 GB per day of traffic on this router and the CPU load is around 10%.

My BGP peers on v3.20 are not stable at all. The whole thing crashes it fails to pick up the updates most of the times. The funny thing is that I have only 3 peers and I had to remove one because the mikrotik couldn't handle them. The hardware is x86 box with more than enough RAM and processing power. When I asked the mikrotik support they said wait for the new version.
Also preppend doesn't work. I tried and tried no luck.

Can anyone suggest a more stable version???

Thanks

feb/03/2009 19:16:46 system,error,critical router was rebooted without proper shutdown
feb/04/2009 01:41:53 system,error,critical router was rebooted without proper shutdown
feb/14/2009 00:57:24 system,error,critical router was rebooted without proper shutdown
feb/14/2009 05:22:59 system,error,critical router was rebooted without proper shutdown
feb/18/2009 17:52:26 system,error,critical router was rebooted without proper shutdown
mar/02/2009 15:37:54 system,error,critical router was rebooted without proper shutdown
mar/04/2009 18:30:38 system,error,critical router was rebooted without proper shutdown
mar/07/2009 21:07:02 system,error,critical router was rebooted without proper shutdown

Mikrotik 3.20, x86, Intel Core, Intel Pro PciE server Gigabit adapter, routing-test package, multicpu enable (if disable - mikrotik boot from sata drive unlicensed) two full-bgp peer, 20 prefix-filters (community, prepend added), 30 simple queues, Firewall (enable-disable clients by static ip), no autosupout file generated after reboot/freezing

Prepend work, no works "Resend" after reconfiguration filters.

@thadem
With regard to IPv6 BGP I think that the issues illustrated in the blog are referred to the usage of filters using winbox or prefix-lists. I am using IPv6 BGP on RB1000/3.20 routing-test and it works just fine, also the routing filters, configured by command line. It is easy, at present, to break up IPv6 things using winbox.

@mojiro
I've an RB1000 3.20/routing-test with about twenty BGP sessions (the biggest with about 8k routes) and I have no issues at all. But on another RB1000 3.20/routing-test with two full table peers I have problems when I have a big change in one of the peers (for example one up/down). It seems that when the CPU raises to 100% due to the unloading and reloading of the FRT the software has problems to keep on other tasks (maybe, but I am not sure, keeping up OSPF sessions). I hope that since 3.21 the handling of the routing table will be optimized, I think that the problems are just there, I had no problems with other BGP functionalities.

@Helios
I'm using three RB1000/3.20 routing-test with IPv4 and IPv6 sessions (two full routes) and I had never a "spontaneous" reboot.

@Perit
Are you running IBGP in any of those circumstances?

I have been trying to mix V2.9.51 Routing-test, V3.20 Routing, and an upstream providers Cisco together with-out success using IBGP.

Some of the issues I have experienced are:
I have to close out winbox because it will become unresponsive after enabling a BGP peer.
The V3 will occasionally have issues with MD5
Sometimes I will have to do a BGP resend on the v2.9 after turning up the v3 to get the prepends working the way I would expect.

I was hoping to ease into the transition of V3, but it sounds like I may need to upgrade both routers to V3 routing-test before I start seeing consistent good results?

-Louis

@letni
To tell the truth what I referred as "CPU hang" was exactly what you're talking about. Two RB1000, one per upstream provider, and an IBGP session between them (and other iBGP sessions). The session between them seems to be the most problematic (and most cpu intensive), but as far as I know because the bgp cpu hog (and consequent winbox high delay or freezing) makes OSPF go down and loose local connectivity between the two (consider that they're both on the same Lan segment, but I use IPs on bridges to recreate Loopbacks, and that sometimes one side of the session remains up and other side goes down).

At present i'm trying to reduce prefixes number (to see it anything changes) and to investigate details of configuration to understand if that is my fault (maybe route recursion with high scopes when getting routes from external peers).

Apart this problem I've not anything else to blame, except CPU going 100% everytime you touch a routing-filter (could be useful to commit at the end of the changes of the whole route-filter and not rule by rule).

Considering other ibgp sessions i've, only the symmetric one seems to be problematic, others to routers linking peers and from which I get routes but to which I do not announce back the same routes from other peers seems to have no glitch at all.

I'm sure next versions will put order in this, but I did not test 3.21, yet.

Did you find any improvement?

I am dealing with a strange problem.

Before: PowerRouter 732 v3.13 <- BGP/OSPF over LAN gig ethernet -> Cisco 7204VXR NPEG1 -> DS3 (with ethernet convertor) -> another Cisco 7204VXR NPEG1

After: PowerRouter <- BGP/OSPF over DS3 -> Cisco 7204VXR

3 days, no issues

Now every other day short drops OSPF, BGP drop and come back. I have put static routes in to solve the issue at the moment.

Cannot find a reason why the PowerRouter to the Cisco on the LAN was rock solid for 200+days straight but now I am having drops...figured it would be ok because all I did was take out a single Cisco that was basically just in the way and no longer needed...nothing else changed. Ciscos running same IOS.

I am thinking it could be the DS3 -> ethernet converter. When I moved the cable from the Cisco gige port the adapter NIC LED stayed on, but I have some logging going on to monitor the ethernet interface as well.

I know it's not an RB1000, but similar strange issues and wondering about BGP stability (hence subject)

Popping this up as one of my RB1000 with BGP/OSPF yesterday ceased service after running
a few weeks without problem. CPU was at 100% 2 vlans did not work correctly and
BGP does not work (connections up but no incoming traffic). I was lucky to reach it for a reboot.
A reboot and then a BGP-resend cleared things up. Files show that storage it is fully used before
booting. After reboot 40MB free (I do not use proxy).

Version is 3.16 routing-test.
Anyone sucessfull running a newer version?

Stefan

Popping this up as one of my RB1000 with BGP/OSPF yesterday ceased service after running
a few weeks without problem. CPU was at 100% 2 vlans did not work correctly and
BGP does not work (connections up but no incoming traffic). I was lucky to reach it for a reboot.
A reboot and then a BGP-resend cleared things up. Files show that storage it is fully used before
booting. After reboot 40MB free (I do not use proxy).

Version is 3.16 routing-test.
Anyone successful running a newer version?

Stefan

We have noticed that, 3.x routerboards reach 100% when simple queues are being used. Do you have simple queues?

Popping this up as one of my RB1000 with BGP/OSPF yesterday ceased service after running
a few weeks without problem. CPU was at 100% 2 vlans did not work correctly and
BGP does not work (connections up but no incoming traffic). I was lucky to reach it for a reboot.
A reboot and then a BGP-resend cleared things up. Files show that storage it is fully used before
booting. After reboot 40MB free (I do not use proxy).

Version is 3.16 routing-test.
Anyone successful running a newer version?

Stefan

We have noticed that, 3.x routerboards reach 100% when simple queues are being used. Do you have simple queues?

No. There's no queue at all.