Hi All

was someone already able to test the BGP performance of the CCRs using full internet routing tables?
I wonder how well it could handle route flaps on a 1 Gig Internet connection.

- Mat

Im canging a juniper j6350 to a ccr on Monday (if I recive it).
I will have a global bgp routing table to it and we'll se then.

Thanks for your reply!
I'm very interrested to see your results. What do you plan to monitor? I would look at the different CPU Cores to see if the load gets balanced proprerly and how fast the routing table gets copied to the CCR.

If work permits, I will test it by myself within the next two weeks. We've got a Brocade CER (65 Mpps) at the Data Center which has some free ports and could be used to "stress test" the CCR.

I'm very curious if the CCR can handle that or if it will almost die like others would @ 1 Gig.

- Mat

I will test that it is working.

No, but seriously, I have had some trouble using the vrf part of 6.x ros, and im not quite sure if the routing is good enough to be used in production yet. I will try to stress test, (maybe to my j6350 and/or my zte zxr10 8902) for eBGP and to my 1100AH2X for iBGP, combinded with the global routing table from my 1gig ip transit. I put it in production, so its depends on how it impact customer.

I will test that it is working.

Sure.
I already tested BGP on the CCR using a RB1100AHx2 so I can say it works in general.

I won't put it in production now. I will just set it up as an iBGP peer to see how it performs when a powerful router sends the full table. So I can plan how it can be used in our Network. We won't use it as a border router anyway, but we've got some GigE links between our POPs and because of that we need it to be able to handle this load. We used Cisco routers in the past that could handle the traffic well but as soon as routes begin to flap, they almost died. That's why I'm a bit cautious now. From my tests, I know the CCR can handle multiple Gigs very well but what about BGP...? That's also why I will have a look at the ping times while the routing table gets (re)built on the CCR. A router that generates a delay of 100ms+ or even worse begins to drop packets, while it updates its routing table, is just useless to us.

You see, I want to go more into detail. But I would appreciate your results.

- Mat

I just found a video on youtube on which someone sent 4 full feeds to the CCR. It looks like the load doesn't get balanced over multiple cores. This is what I suspected. It's too bad that we can't use the full potential of this CPU.

This tell to less to conclude annything other than mikrotik is working to get a 100% systeem running. its still to be consider as a beta test system.

This tell to less to conclude annything other than mikrotik is working to get a 100% systeem running. its still to be consider as a beta test system.

I'm not 100% sure what you meant, but we do not complain about low performance we're just testing. We know it's in beta. I was a bit disappointed to see that the CCR isn't really faster than the RB1100AH (at this point). I'm sure it will get better...

It looks like the load doesn't get balanced over multiple cores. This is what I suspected. It's too bad that we can't use the full potential of this CPU.

Mikrotik are furiously working away on a new routing engine that will make use of massively multicore architectures like the Tilera processors in the CCR.

No ETA from them yet, but hopefully soon

I've got some test results:

- It takes about 30 sec. to load 400k prefixes (one core of the CPU @ 100%)
- MD5 passwords don't work
- As soon as I get more than about 200k prefixes, the "Routes" windows doesnt show all filter options and routes

" MD5 passwords don't work"
With latest RouterOS version MD5 should work

"As soon as I get more than about 200k prefixes, the "Routes" windows doesnt show all filter options and routes"
It is made so that bandwidth is not flooded with enormous amount of winbox data to display route updates.

" MD5 passwords don't work"
With latest RouterOS version MD5 should work

I'm running RC10 built on 15 Feb. and it definitely doesn't work in this version. I've got a Cisco and a Brocade router connected to this CCR and both session don't work with MD5.
I will update to the most recent version of RouterOS and test again.
Btw. I've got a 1100AH running on 5.22 which peers also with this two routers mentioned above and it runs MD5 with no problem.

" MD5 passwords don't work"
"As soon as I get more than about 200k prefixes, the "Routes" windows doesnt show all filter options and routes"
It is made so that bandwidth is not flooded with enormous amount of winbox data to display route updates.

I thought it's because of that. But at some point I wasn't able to filter on routing protocol, etc. Only Dst. Address was visible. At the moment it works. Well, like I wrote above, I will update to the most recent version and keep an eye on this.

Thanks mrz for your reply. It's good to know that we get "heard".

-----------------------------------------------------
Edit:

I just saw that I'm running the latest Version of RouterOS.
You are right mrz it's not a Bug it's just how it works when you've got a lot of routes. It needs getting used to.
The problem is you need to know the subnet (and size of it) you search for. For example: You can't search for "192.168.0.1" if you don't know that this IP is in "192.168.0.0/24". On Ciscos I can run "show ip route 192.168.0.1" or "show ip bgp 192.168.0.1" and I get the right prefix. Or am I just using it wrong?

Hey Samsung

were you able to test BGP?
My CCR is currently running 3 Sessions to upstream routers and 1 Session to a downstream router. Uptime is almost a week and I've seen no problems so far.

- Mat

Hi Mat,

thanks for sharing!

Your BGP Sessions are up with Fulltables? I plan to use Mikrotik in my Edge with 3x Uplinks, but i miss 10 GbE Support over SFP+.

still the vrf issue. so its not possible for me to start full test.

I have a global ebgp routing table, and one internal ibgp and see no problem to that.

Hi Mat,

thanks for sharing!

Your BGP Sessions are up with Fulltables? I plan to use Mikrotik in my Edge with 3x Uplinks, but i miss 10 GbE Support over SFP+.

Hi raz

I'm running one full table, one part table and one table from the local IX (~50k). The downstream router gets the "mix" of all tables.
But I don't see any problems to run 3 full tables. It's just a matter of ram and at the moment I've got ~3.25G free of the 4G installed in the CCR, so there's room to grow. If not, we could upgrade the ram.

I also think the CCR needs some more carrier grade features like I mentioned in another thread, but it's a platform with a lot of potential.

The only negative point I see so far is, that the CCR can only use one core to process BGP. Compared to a 1100AH or 1100AHx2 (which can also only use one core), it's not much faster if it's any faster at all. Sure, the positive side effect is that the packets running trough the CCR don't get delayed, which is what a carrier would expect from such devices, but it would be nice if the CCR could use multiple cores for BGP to speed up this process. If it could use a user configurable amount of cores would be awsome of course.

- Mat

Hi Mat,

thanks for your Answer! If im buy the Mikrotik i will start with 16 GB RAM, to be safe. And RAM is currently cheap.

With the Prefixes, im sure Mikrotik make here in future a Update to optimize this Feature, maybe we should send an Feature Request to their Support?

Regards,

– raz

16 GB should absolutely be sufficent.

Hi Mat,

thanks for sharing!

Your BGP Sessions are up with Fulltables? I plan to use Mikrotik in my Edge with 3x Uplinks, but i miss 10 GbE Support over SFP+.

Hi raz

I'm running one full table, one part table and one table from the local IX (~50k). The downstream router gets the "mix" of all tables.
But I don't see any problems to run 3 full tables. It's just a matter of ram and at the moment I've got ~3.25G free of the 4G installed in the CCR, so there's room to grow. If not, we could upgrade the ram.

I also think the CCR needs some more carrier grade features like I mentioned in another thread, but it's a platform with a lot of potential.

The only negative point I see so far is, that the CCR can only use one core to process BGP. Compared to a 1100AH or 1100AHx2 (which can also only use one core), it's not much faster if it's any faster at all. Sure, the positive side effect is that the packets running trough the CCR don't get delayed, which is what a carrier would expect from such devices, but it would be nice if the CCR could use multiple cores for BGP to speed up this process. If it could use a user configurable amount of cores would be awsome of course.

- Mat

But, if there was more corses to do the bgp. would it be speeded up? Whats really nice, is that each port, don't share internal recourses like the 1xxx series. 1Gig strigt into the core (kernel).

But, if there was more corses to do the bgp. would it be speeded up? Whats really nice, is that each port, don't share internal recourses like the 1xxx series. 1Gig strigt into the core (kernel).

Sure would it speed up the building of the routing table. Especially when more peers with full routing tables are connected to the CCR.

Hello

we tested a CCR1036 with RC10, RC11, RC12; with two peers (100 Mbps each) and two full tables, in production

it runs well, BUT it has a maximum uptime of 9d; after that, all interfaces disappear, and it has to be power cycled

when it runs, CPU usage is 2% on average

for comparation, a RB1100AXH2 in the same position has a 40% CPU average

we do hope that MT will soon find and remove all bugs

after that, the hardware platform has surely a big potential

so, at the moment we can still use the RB1100AXH2

but by end of the year we will upgrade to three peers, each 1Gbps, so I will really need the CCR1036 to run flawlessy (otherwise we'll have to use a J*** or C*** as BGP router)

I can't confirm your problems. Our BGP Sessions are up since several weeks. But our Routers are in the Lab and don't have to push a lot of traffic around.

- Mat

Hi, could you please upgrade to RC13 and report here if it works in your environment stable, as it seems working for you until now; I upgraded to RC13 but don't want to risk again putting it in production; in our case, RC10 RC11 and RC12 are not working more than 9 days, when doing BGP in real world with high traffic

our other production CCR, doing NAT, not doing BGP, with RC10, works flawlessy, uptime 49 days and counting

Okay, I upgraded to the most recent version (RC14) and will report back if it runs longer than 9 days.

Btw. the CCR I just updated ran ~2 weeks on RC13 without problems.

Hmm i'm testing RC13 from today night (2 full bgp tables + 0.5 gbps traffic)
after half a day i have around 10k - 20k tx drops o vlan interfaces.
Are there any solution for this packet drops??
I have also noticed that i can upgrade firmware from 3.03 to 3.05 (can this fix anything?)
Where did You get RC14?

bootloader has only minimal to no impact on higher features of RouterOS.

Where did You get RC14?

http://www.mikrotik.com/download/share/ ... .0rc14.npk

Is there any changelog to RC14?

I am running two BGP route reflectors which are fed with two full views and igp. They have ~90 peers each. Currently if I disable one of the two full view, the topology change takes about 20 minutes to be completely propagated. BGP convergence is still very slow, and it uses only one core, which is almost always loaded at 100%.

No uptime problems so far. 6.0rc12 worked for more than one month. I upgraded to 6.0 two days ago and it seems stable. I hope that BGP routing table management will be optimized in the near future.

Hello guys!

I just wonder if it's possible to run 2xBGP Full View with 100 Mbit each on RB1100AHx2? I have routing and NAT. For current 100M load cpu utilization is about 30%. Does full view increase CPU utilization and make impact for latency?

I need time until CCR became suitable for BGP production and don't want to feed cisco

You can run two full views on the RB1100AHx2 but you will most likely see higher latency during convergence.

Hello guys!

I just wonder if it's possible to run 2xBGP Full View with 100 Mbit each on RB1100AHx2? I have routing and NAT. For current 100M load cpu utilization is about 30%. Does full view increase CPU utilization and make impact for latency?

I need time until CCR became suitable for BGP production and don't want to feed cisco

Why cant you use ccr for bgp? unless you need the vrf function, a "simple" ebgp is no deal for the ccr.

Hello guys!

I just wonder if it's possible to run 2xBGP Full View with 100 Mbit each on RB1100AHx2? I have routing and NAT. For current 100M load cpu utilization is about 30%. Does full view increase CPU utilization and make impact for latency?

I need time until CCR became suitable for BGP production and don't want to feed cisco

Yesterday I replaced one of my BGP routers with CCR. So far everything runs smoothly, there are 7 peers, 3 full routing tables. I took the risk, because of full redundancy for this device, so if you have such possibility, I say try CCR now. Only problem I observed is with reboot, I wrote about it in CCR topic.

@szastan

Now we are some days later. What is your experience with CCR/BGP?

So far everything works as intended, made during this time few reboots at the beginning, at the moment CCR uptime is almost 11 days. Below are some screenshots:
3 IPv4 and 3 IPv6 full tables (one eBGP, two iBGP), 3 BGP clients, and 3 IPv4 + IPv6 OSPF peers. With this configurations everything works stable, any changes made to routing filters are applied as intended.

If you need any more details, feel free to ask.

Do you also experience one core almost always at 100% load? How much does it take a route state-change to be applied? On my CCR (3 IPv4 full views and 3 IPv6 full views) it takes almost 20 minutes.

Do you also experience one core almost always at 100% load? How much does it take a route state-change to be applied? On my CCR (3 IPv4 full views and 3 IPv6 full views) it takes almost 20 minutes.

We see this too. It's because BGP runs on one core only.

Do you also experience one core almost always at 100% load? How much does it take a route state-change to be applied? On my CCR (3 IPv4 full views and 3 IPv6 full views) it takes almost 20 minutes.

Yes, in fact above CPU graph is showing cores reaching 80% in a daily view, here they are averaged. Full route state-change depends on type of router on the other side of link. In my case there are 3 Quagga's, and this takes 1-3 minutes, it's pretty quick. But with Cisco routers used by my IP Transit provider it can take up to 5 minutes per link, sometimes quicker. So it's pretty close to what you experience, comparing it to Quagga, it's pretty much the same.

szastan, thank you for reply!

Does one core full utilization makes influence on traffic flow? What is the latency to nexthop via your bgp router? Is one core have 100% utilization every time or for some times?

P.S. My RB1100AHx2 with two full BGP peers has convergence time about 3 minutes, do one core of AHx2 pretty close to one core of CCR.

szastan, thank you for reply!

no problem

Does one core full utilization makes influence on traffic flow?

No, this is not a problem since traffic is ~equally divided between cores

What is the latency to nexthop via your bgp router?

It depends as some of nexthops are in same city, some are in totally different places. I guess what you ask is, is CCR causing higher latency than previous x86 system? The answer is: no.

Is one core have 100% utilization every time or for some times?

This 100% usage is jumping between cores and yes, all the time one core is 100% loaded

I can confirm szaszans observations.

Hello

I have the same performance problem. Two BGP connection and a very long time rebuilding routing configuration. One of the core loaded to 100 percent. There is a some solution? currently I'm using version 6.4 of the software.

Hello

I have the same performance problem. Two BGP connection and a very long time rebuilding routing configuration. One of the core loaded to 100 percent. There is a some solution? currently I'm using version 6.4 of the software.

Hello

I have the same performance problem. Two BGP connection and a very long time rebuilding routing configuration. One of the core loaded to 100 percent. There is a some solution? currently I'm using version 6.4 of the software.

Unfortunately, yes. In my case (config above didn't change, only traffic went up to >500 Mbit / interface, ~3 Gbit full router passthrough) all routes are learned in about 5-10 minutes, and that occured on v6.3. Now on 6.7 once my router stuck on learning BGP routes for about an hour. No solution I'm aware of though.

Sorry to jump in last minute...
I am using 1000's and 1100AHx2's ..(just added in a CCr1036)
Is bgp still only being done on one core? Has Mikrotik addressed this?

If only one core can do the bgp process... the other 35 are not as usefull
also

Is there benchmarkes on the CCr cores? 1.2ghz ?

If I can Build my own 4ghz quadcore it would perform much better.

Any thoughts
Thanx

Sorry to jump in last minute...
I am using 1000's and 1100AHx2's ..(just added in a CCr1036)
Is bgp still only being done on one core? Has Mikrotik addressed this?

Yes

If only one core can do the bgp process... the other 35 are not as usefull
also

Other 35 can do packet processing.

Is there benchmarkes on the CCr cores? 1.2ghz ?

1.2 like 1100AHx2

If I can Build my own 4ghz quadcore it would perform much better.

Of course. But be aware of stability ROS ox X86. I have very bad experiences with ROS runing on several different configurations. Mikrotik is always a lottery. You can win once something really good for cheap price, but then missed everything.

I have very good experience with 2-years work of RB1100AHx2 but when I've upgraded to X86 then CCR it's became like nightmare.