sharing experiences with RouterOS/DS3/BGP

ldvaden · Sat Sep 23, 2006 9:53 pm

Is anyone willing to share their experiences with RouterOS/DS3/BGP?

After 11+ years, I'm about fed up with Cisco's policies like limiting access to <http://www.cisco.com/pcgi-bin/Support/B ... bugtool.pl>.

rgds/ldv

Stryker777 · Sun Sep 24, 2006 7:30 am

Ram, Ram, and more Ram. Other than that, I have seen no major issues. I have read some posts by people with problems but never experienced problems myself. Watch your upgrades though and always have a backup unit when doing upgrades.

ps, a nice processor will help too. More than anything though, Ram

ldvaden · Sun Sep 24, 2006 4:21 pm

ps, a nice processor will help too. More than anything though, Ram

RAM is what is causing us to look away from the Cisco 7206s in our stable because of the 256 MB limitation of the NPE-300 (262Mhz). CEF crashes, resulting in a necessary reload if and only if you want to continue with production. Current IOS for the Cisco 7206 doesn't list that as a resolved caveat (IMHO).

We like the Sun Fire X2100 (Opteron, 2.6 GHz) at $745 if we could get favorable references on a DS3 card for the PCI bus.

There's a lot of things MikroTik could do better, but Cisco 7206s burping/crashing under attack is an ugly sight and the smell is worse.

What DS3 card do you use?

rgds/ldv

Stryker777 · Mon Sep 25, 2006 12:40 am

The only one I have ever used is the SBE card.

ldvaden · Mon Sep 25, 2006 2:15 am

The only one I have ever used is the SBE card.

Thanks for the lead.

rgds/ldv

BrianHiggins · Mon Sep 25, 2006 2:22 am

just a thought, use the 7206 to terminate the DS3 and hand it off to the MT box via ethernet. that eliminates any potential driver issues or problems with the DS3 card, and the 7206 will be plenty suifficiant to perform that task once you remove BGP from it...

btw, what's your free memory level on your 7206? I've got a 7204, NPE300, 256mb, full routes from 2 providers, ~50mb free memory, but every 60 seconds my ping times jump sky high while BGP updates... I'm quite fed up with it.

please post any updates on what you choose to do.

ldvaden · Mon Sep 25, 2006 3:39 am

just a thought, use the 7206 to terminate the DS3 and hand it off to the MT box via ethernet. that eliminates any potential driver issues or problems with the DS3 card, and the 7206 will be plenty suifficiant to perform that task once you remove BGP from it...

btw, what's your free memory level on your 7206? I've got a 7204, NPE300, 256mb, full routes from 2 providers, ~50mb free memory, but every 60 seconds my ping times jump sky high while BGP updates... I'm quite fed up with it.

please post any updates on what you choose to do.

ITX has been under DOS attack for about 10 days for a couple of sites we host in the community's interest, one having to do with spam and one having to do with fixes for M$ exploits (see recent DDJ article).

The symptom is that CEF crashes after exhausting memory (down as low as 411.91 kB, processor utilization higher than 85%.

Level 3 claims not to have a traceback facility so they have chosen not help with the spoofed addresses. As a consequence, I can no longer recommend Level 3 as an upstream.

We run 3 BGP sessions, two to upstreams and one to Akamai (in house), OSPF (customer announcements) on a 7206 NPE-300 w/256 MB (max). The other box, a 7206 with NPE-400 and 256 MB is not causing grief at this time.

We are happy to hear about any real world solutions; I like your idea about using the 7206 as a front end to the MT to keep the DS3 terminated on the 7206. Recent bugs just fixed in ROS's BGP code concern me somewhat. I believe there was one about multiple BGP neighbors. BGP is hard to implement correctly.

rgds/ldv

UniKyrn · Mon Sep 25, 2006 4:02 am

Be aware that you'll lose a lot of the ability to configure your BGP sessions if you move to ROS. The last place I worked, our border router was a ROS box after the ancient Cisco box we had couldn't handle the full routing table any more. We were dual homed, full tables from each provider. Something as simple as shutting off one session for testing? You can't, you have to remove the config for that provider and you better write down the settings before you do it. Want to block a default route from one of your providers? Never could figure that one out, the Prefix Lists don't work as expected. Want to change the weights of routes from a provider? Forget it, not possible.

If you're in a vanilla environment, nothing special, ROS is an option, but I couldn't recommend it otherwise.

Beccara · Mon Sep 25, 2006 4:08 am

Be aware that you'll lose a lot of the ability to configure your BGP sessions if you move to ROS. The last place I worked, our border router was a ROS box after the ancient Cisco box we had couldn't handle the full routing table any more. We were dual homed, full tables from each provider. Something as simple as shutting off one session for testing? You can't, you have to remove the config for that provider and you better write down the settings before you do it. Want to block a default route from one of your providers? Never could figure that one out, the Prefix Lists don't work as expected. Want to change the weights of routes from a provider? Forget it, not possible.

If you're in a vanilla environment, nothing special, ROS is an option, but I couldn't recommend it otherwise.

Routing -> Filters work in my experence, dual homed full route loads?
routing bgp peer disable XX - Disables a bgp per and drops any loaded routes from that peer

The main thing is to set the in and out filter name in hte peer details, then when using routing -> filter you can apply the right filter to the right peer.

Only issues with ROS BGP is stablity of it, set next-hop was an issue in .28 and .29 but fixed in .30

UniKyrn · Mon Sep 25, 2006 4:26 am

You can't disable a peer using WinBox, only delete them. I assume you're talking about a disable option that's only available from the command prompt.

Beccara · Mon Sep 25, 2006 4:34 am

You cant?

UniKyrn · Mon Sep 25, 2006 4:49 am

Not in 2.8, and we stopped doing automatic upgrades after the 2.8 fiasco where they didn't regression test changes to BGP and broke routing for anybody who upgraded. That was about 2.8.24 or 2.8.26 I believe. Taking your border router down for an upgrade is enough trouble for an ISP, taking it down and not having it be usable afterwards teaches you to never trust that vendor again.

Beccara · Mon Sep 25, 2006 4:53 am

Not in 2.8, and we stopped doing automatic upgrades after the 2.8 fiasco where they didn't regression test changes to BGP and broke routing for anybody who upgraded. That was about 2.8.24 or 2.8.26 I believe. Taking your border router down for an upgrade is enough trouble for an ISP, taking it down and not having it be usable afterwards teaches you to never trust that vendor again.

It's called a test lab - Set one up and test using dual homed bgp feeds before looking at your production egde router - which might need backup.

You can go around saying dont use BGP because it's rubbish when your using software 1gen old!

UniKyrn · Mon Sep 25, 2006 5:07 am

How many small ISP's do you know of that have a test lab with a duplicate border router sitting there for the occasional test? And for the record, I'm not saying don't use BGP, I'm warning people that MT's implementation is not feature rich, nor do they have a good track record for regression testing their own product before releasing it.

Your first post in this thread included your own example of that in 2.9.

Stryker777 · Mon Sep 25, 2006 5:09 am

Exactly. I started with 2.9.27 and have had good experiences. Like I said, have a backup one to run every time you do an upgrade. Testing is always best.

Mon Sep 25, 2006 10:02 am

How many small ISP's do you know of that have a test lab

all?

I'm warning people that MT's implementation is not feature rich,

how can you say that, if you have not even used it for more than a year? since 2.8 almost everything has changed

changeip · Mon Sep 25, 2006 7:38 pm

mt bgp is almost working very well. Outbound filters do not work properly in all cases, I just ran into this today.

iBGP is still a little squirly. 512mb of ram is good for full route tables. A beefy CPU is required if you want minimal problems.

We are using .27 on 2 border routers for months now. We are not receiving all routes because they lockup when exchanging routes between themselves.

Having a dev environment for a bgp setup is required, you cannot expect it to work reliably under load until you've tested in dev. Chicken egg problem though, how can you get a dev bgp feed to test with : )

Sam

UniKyrn · Mon Sep 25, 2006 8:27 pm

Not in 2.8, and we stopped doing automatic upgrades after the 2.8 fiasco where they didn't regression test changes to BGP and broke routing for anybody who upgraded. That was about 2.8.24 or 2.8.26 I believe. Taking your border router down for an upgrade is enough trouble for an ISP, taking it down and not having it be usable afterwards teaches you to never trust that vendor again.
It's called a test lab - Set one up and test using dual homed bgp feeds before looking at your production egde router - which might need backup.

You can go around saying dont use BGP because it's rubbish when your using software 1gen old!

Also, I assume you're using the routing-test package in 2.9, because those enable/disable boxes are not present in the default BGP package.

UniKyrn · Mon Sep 25, 2006 8:43 pm

How many small ISP's do you know of that have a test lab
all?

no

I'm warning people that MT's implementation is not feature rich,
how can you say that, if you have not even used it for more than a year? since 2.8 almost everything has changed

And we'd know this how? The 2.9 PDF manual doesn't even include a chapter for BGP any more, you removed it after 2.8. And don't tell me about the routing-test package, I'm talking about the default routing package.

MikroTik violates the trust of it's customers every time it releases a version that hasn't been regression tested well enough to prevent introducing bugs in previously working packages. If you're going to compete with other companies for the network edge market, you better start hiring in your own test lab.

Beccara · Tue Sep 26, 2006 12:41 am

Have you ACTUALLY ever delt with cisco or juniper with their releases? your luckey if they even load it on a router before releasing it.

Test, Test, Test - If you dont have a lab get one, if you cant get one get out of the ISP business.

UniKyrn · Tue Sep 26, 2006 3:05 am

Have you ACTUALLY ever delt with cisco or juniper with their releases? your luckey if they even load it on a router before releasing it.

I worked for Alcatel for three years and we weren't allowed to release anything without the QA group doing a complete regression test, no exceptions. If we changed anything at all once that test began, they had to start over. If Cicso and Juniper don't do the same thing, that's their problem, but I know how it's done when it's done right.

tpsretard · Tue Sep 26, 2006 3:39 am

AMEN BROTHA

we have a cisco 5300 running 12.1t because of some problem or the other, we can not upgrade it, our 2 7206vxr routers with NPE 400's they are back down in 12.2 and the 2 GSR's man we not going there..

I dont have anyware the problems with Mikrotiks as i do with them, some times i wonder why i test, and then think if dont test this upgrade it will be the one that makes the network go tits up...

just my 2cents worth...

Have you ACTUALLY ever delt with cisco or juniper with their releases? your luckey if they even load it on a router before releasing it.

Test, Test, Test - If you dont have a lab get one, if you cant get one get out of the ISP business.

ldvaden · Tue Sep 26, 2006 4:25 am

please post any updates on what you choose to do.

Two steps were taken early today:

1. Upgrade from IOS 12.0(31)S to IOS 12.0(32)S4.

ACHTUNG! the upgrade replaced configured parameters on interface cards with its own choices (things like duplex and clock source). IIRC, that's the first time we've seen that in a decade or at least no mention was made.

2. Actions based on

"Achieve Optimal Routing and Reduce BGP Memory Consumption"
<http://www.cisco.com/warp/public/459/41.shtml>.

We've now levelled off near historical memory and cpu utilizations on the 7206VXR boxen and the interfaces have stopped reporting false errors while under attack.

Now, I need to go to the workshop and find the biggest piece of wood!

rgds/ldv

BrianHiggins · Tue Sep 26, 2006 8:34 am

Now, I need to go to the workshop and find the biggest piece of wood!

save a piece for me... ever since we added 3 new /24 address spaces to our network (advertized through BGP) my ping times look like this...

btw, this is one of my upstream providers, I'm directly connected to them via 65 miles of fiber, the pings here cover about 1min and 15sec of time...

Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=247ms TTL=61
Reply from 206.53.239.195: bytes=32 time=81ms TTL=61
Reply from 206.53.239.195: bytes=32 time=136ms TTL=61
Reply from 206.53.239.195: bytes=32 time=255ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=72ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=2ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=127ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=54ms TTL=61
Reply from 206.53.239.195: bytes=32 time=114ms TTL=61
Reply from 206.53.239.195: bytes=32 time=161ms TTL=61
Reply from 206.53.239.195: bytes=32 time=264ms TTL=61
Reply from 206.53.239.195: bytes=32 time=3ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=4ms TTL=61
Reply from 206.53.239.195: bytes=32 time=6ms TTL=61

when I see 2 or 3 builds in a row that have no stability complaints about BGP, ye old 7204vxr beheamoth will become my hot mounted standby router, and ROS will become primary...

sharing experiences with RouterOS/DS3/BGP

sharing experiences with RouterOS/DS3/BGP

Who is online