Thought i'd throw this out there to get a case study of OSPF backbone networks that have continually grown and not segregated into multiple area's or multiple instances
I'd primarily like to hear from people who have routers that include wireless links, slow/unreliable links, long daisy chained segments etc and hear about their convergence time and how the network handles link flapping, failover and reconvergence

We currently have multiple Area0 networks (with additional areas but they don't matter in this case) that are geographically separated but have grown to overlap each other and use each other for transit and uplink/core failover. This is accomplished by using multiple Area0 instances and redistributing between them with route tags to differentiate which region it came from and prevent loops when redistributing back the other way. This keeps those Area0 networks logically separated, allowing summarization and and stopping LSA's from some far edge corner of the network from going 20 hops to some other router that doesn't give a flying hoot.

They're kept separate primarily to reduce convergence time, especially with flapping links. However i'm wondering how necessary this is? I don't like it because its a bit messy, it's not by-the-book design at all and its very complicated for outside eyes to look at the network and make sense of it
I'm looking at making a major redesign of the network to get rid of this situation and have a single Area0 with as many other routers as possible off into their own areas to keep convergence times as low as possible with no impact when an edge router has a flapping link. The process of doing this would be greatly eased if I just threw everything into Area0 during the redesign, but i'm wondering about the real world impact on the network if I did that. And maybe its all overthought and I should just throw everything into Area0 and forget about it

There are publications that suggest no more than X number of routers, but often these are for campus or enterprise style networks with copper and fibre everywhere. Wireless is a whole different ball game and most of our backbone is built on wireless. We have a number of 24ghz and 60ghz links running OSPF+BFD that go down during rain with 5ghz failover. These are routed links so every time this happens LSA's are generated. If the routers don't re-converge quickly because they are waiting for all routers to agree on the network topology then it could result in connections being dropped before the failover kicks in. Reconverge needs to happen within a couple seconds at most, ideally much less than that

What's your experience with larger Area0 networks? If you have i.e. 20-200 routers that are spread out all the place, do you know how long it takes to reconverge? Do you end up with unpopulated routes and broken connectivity when there's a flapping network somewhere on the edge thats causing all the routers to wait because the OSPF tables remain incomplete?

I'm also interested in other's experience with large single area OSPF networks. We have four independent OSPF area zero networks that were geographically isolated at one time but now overlap. One of these areas includes an acquired Siklu-radio-based network with extra capacity but few alternate paths. This network substantially overlaps with two other networks that have less capacity but redundant paths for all major buildings. Our easiest solution for adding capacity in the existing networks and adding redundancy in the Siklu network would be to merge these three networks which currently have:
1. 276 routers and 2385 active routes
2. 202 routers and 1338 active routes
3. 142 routers and 846 active routes

Each network has a mix of routers, mostly MikroTik but also a few Ubiquiti EdgeRouters. The MikroTik routers range from a few CCR1036s down to a large number of RB750UPs near the edges with RB2011 and RB4011 routers in between.

The current networks are stable, even when one or two routers are flapping (due for example to a marginal radio link). On the other hand, our current success may or may not be helped by a kludge we introduced in 2012 to solve route re-computation delays during radio-induced flapping . In 2012, we saw RB750UPs near the edges that would loose all their OSPF routes for a couple of seconds while recomputing new routes. We fixed that with a script that we have included ever since in our baseline for all our MikroTik routers. This script runs each day at 6am and, if various sanity checks are met, makes static route copies (with higher distance) of each OSPF route. Then, should OSPF routes vanish for a second or two, there are mostly correct static routes that forward traffic during the re-computation interval. I have not verified if this script is still required, however, with lightweight MT routers and the firmware in existence in 2012, this script solved a real problem.

So my question is: does anyone have a MT-based OSPF Area 0 network equal or larger in size to what my combined network would be? I.e. a network with >600 routers and >4500 active routes?

Hello guys!! I think that I'm the one who can help you, currently my network is like 350 routers which were deployed under the same backbone area, that network is actually moving like 30gbps of traffic in a Mikrotik only architecture (at the routing level), the entire BGP/OSPF network is actually moving like 6000 prefixes between CCR1009; CCR1036; CCR1072.

I found two interesting problems; OSPF worked really great, but a wise advice, move your network to a multi-area design using ABR and summarizing networks as much as possible, it allows to reduce the LSA amount at each router, those Mikrotik CCR tile processors are optimized to forward packets as much as possible, so the control plane is not much reliable, that's why the best solution is to split your network in some routers depending of the function of those (BGP; RouteReflectors; Transit; GRE concentrators).

Besides that, those Mikrotik works great, the convergence by default in OSPF (hello/dead timers at 10secs & 40secs) is practically imperceptible for my customers; the ECMP load-balancing works great too, it's my best solution to deploy a 20gig ring between two cities in the country, or even to aggregate some wireless links (using AirFiber 5xHD) to add them as a single port to increase the total throughput of that node.

Besides that, those Mikrotik works great, the convergence by default in OSPF (hello/dead timers at 10secs & 40secs) is practically imperceptible for my customers; the ECMP load-balancing works great too, it's my best solution to deploy a 20gig ring between two cities in the country, or even to aggregate some wireless links (using AirFiber 5xHD) to add them as a single port to increase the total throughput of that node.

I don't your situation this is quite applicable to us then. How many of those links are wireless and how often do they go down?
The vast majority of our network is wireless not wired. We have a few links that are on 24/60/80ghz that do go down when it rains because we are running them beyond normal distance, but they provide much higher bandwidth 99% of the time and then failover to 5ghz backup. So it's quite common for links to go down but this is by design (the problem comes when the primary just barely hangs on and refuses to die causing a route flap, but this isn't often). They are routed links and I use BFD for sub second reconvergence. We primarily terminate customers with PPPoE and this doesn't cause the session to drop. But if we just used OSPF default timers with no BFD they'd have an extended outage

How often do you reboot routers?
I know that routers when first peering with neighbors will not install any routes in their table until they receive the entire link state database. I had a fundamental misunderstanding and assumed that was also the case when OSPF is already running and stable but a link drops and the entire area would have to be updated. That isn't the case, the message is flooded but recalculation is done on that router as soon as its aware that something changed it doesn't have to wait for all other routers
Nevertheless if you reboot a router and you have a large area it could take a while, even with ~350 routers you are not seeing large delays in OSPF routes when you that router first comes online and establishes neighbors?

1. 276 routers and 2385 active routes
2. 202 routers and 1338 active routes
3. 142 routers and 846 active routes

The current networks are stable, even when one or two routers are flapping (due for example to a marginal radio link). On the other hand, our current success may or may not be helped by a kludge we introduced in 2012 to solve route re-computation delays during radio-induced flapping . In 2012, we saw RB750UPs near the edges that would loose all their OSPF routes for a couple of seconds while recomputing new routes. We fixed that with a script that we have included ever since in our baseline for all our MikroTik routers. This script runs each day at 6am and, if various sanity checks are met, makes static route copies (with higher distance) of each OSPF route. Then, should OSPF routes vanish for a second or two, there are mostly correct static routes that forward traffic during the re-computation interval. I have not verified if this script is still required, however, with lightweight MT routers and the firmware in existence in 2012, this script solved a real problem.

So my question is: does anyone have a MT-based OSPF Area 0 network equal or larger in size to what my combined network would be? I.e. a network with >600 routers and >4500 active routes?

Those are all large OSPF networks. Most of that is wireless I take it? Quite surprised its working with just the 1 area. I've segregated routers off into other areas way sooner than that
Not sure about the kludge, it skews the results somewhat
How does it go with failover though? As above we have links that are assumed to go down in rain, but pushing 1600mbit/s most of the time beats 80mbit/s all of the time, especially since it fails over anyway. But this generates LSA's that flood the entire domain, and it can be several at short time frames, can go up/down 10 times a minute with BFD until it finally dies

ECMP load-balancing works great too, it's my best solution to deploy a 20gig ring between two cities in the country, or even to aggregate some wireless links (using AirFiber 5xHD) to add them as a single port to increase the total throughput of that node.

do you use any mangle rule while using ecmp ?

1. 276 routers and 2385 active routes
2. 202 routers and 1338 active routes
3. 142 routers and 846 active routes

The current networks are stable, even when one or two routers are flapping (due for example to a marginal radio link). On the other hand, our current success may or may not be helped by a kludge we introduced in 2012 to solve route re-computation delays during radio-induced flapping . In 2012, we saw RB750UPs near the edges that would loose all their OSPF routes for a couple of seconds while recomputing new routes. We fixed that with a script that we have included ever since in our baseline for all our MikroTik routers. This script runs each day at 6am and, if various sanity checks are met, makes static route copies (with higher distance) of each OSPF route. Then, should OSPF routes vanish for a second or two, there are mostly correct static routes that forward traffic during the re-computation interval. I have not verified if this script is still required, however, with lightweight MT routers and the firmware in existence in 2012, this script solved a real problem.

So my question is: does anyone have a MT-based OSPF Area 0 network equal or larger in size to what my combined network would be? I.e. a network with >600 routers and >4500 active routes?

Those are all large OSPF networks. Most of that is wireless I take it? Quite surprised its working with just the 1 area. I've segregated routers off into other areas way sooner than that
Not sure about the kludge, it skews the results somewhat.
How does it go with failover though? As above we have links that are assumed to go down in rain, but pushing 1600mbit/s most of the time beats 80mbit/s all of the time, especially since it fails over anyway. But this generates LSA's that flood the entire domain, and it can be several at short time frames, can go up/down 10 times a minute with BFD until it finally dies

Yes, all three networks are almost entirely wireless and yes, we have radios subject to rain fade (Ubiquiti AF24, Siklu EH and Ignitenet ML60). We try to keep links short but we definitely have multiple links that go down during heavy rain. We haven't seen problems with the resulting flood of LSAs but I haven't done enough diagnostics to know if that because I'm missing something, it just works or our script kludge is covering for problems. I'm sorry I don't have more information.

ECMP load-balancing works great too, it's my best solution to deploy a 20gig ring between two cities in the country, or even to aggregate some wireless links (using AirFiber 5xHD) to add them as a single port to increase the total throughput of that node.

do you use any mangle rule while using ecmp ?

No I don't have any mangle rule at my network (excepting when I need to configure GRE tunnels in the middle so I change the TCP-MSS via mangle rules).

Hello Brough, answering your questions:

Actually, we're handling like 200 wireless links (Airfiber 5XHD / AirFiber24X / Netmetal) deployed around the country, and found an interesting problem in RouterOS, that problem was caused by a L2MTU mismatch (I don't have the reply from the Mikrotik suport team yet, but I've regenerated the scenario in a connected device); I'll post a forum entry later today to discuss this issue, it's like when the router A is configured to allow Jumbo Frames (L2MTU = 10000) & Router B is configured by default (L2MTU = 1580) and there are some LSA floods with a convergence in the middle that router crash, so it'll need to be rebooted (watchdog timer that I've set up there), and that only occurs when that combination appears, if the Router B is configured to allow Jumbo Frames the convergence works great and the router didn't crash.

How often do you reboot routers?

That scenario was occurring like 1 -2 times in a week at the beginning; today it happens like 1 - 2 times in a month (after isolating everything with multi-areas) that's when I realized that the L2MTU is the only thing that has a mismatch between affected routers & the other ones.

Nevertheless if you reboot a router and you have a large area it could take a while, even with ~350 routers you are not seeing large delays in OSPF routes when you that router first comes online and establishes neighbors?

Not at all, If I reboot a router it comes up immediately (90% of times), the other 10%, I need to reboot the OSPF instance, because the router is not able to form adjacencies with the rest of neighbors.

At only 7 sites in and 250 routes, we are already looking for a new solution before we grow out of control.

There are a few options considering. Unfortunately OSPF will always need to be part of it, but thinking of moving OSPF to Loopback propagation only, and MPLS for customer routes. This can have advantage of traffic engineering and VPLS on top. The other solution we have thought of is a central route server, all sites BGP back to the route server and it reflects out to all sites. Still yet to consider the best path.

Due to IPv4 exhaustion (And being a business ISP, so clients want IPv4 to their edge) we cant really use summary routes, as it is too wasteful to have unused IP's at certain sites, so we are full of /32 routes. The network is also very much a mesh, so its hard to push it into defined OSPF areas, we are always trying to create a loop around back into the network (for redundancy pathways).

In my opinion you should consider migrate to BGP with ospf. With good planning it's not painful and it's not necessary to get clients without service.


Enviado desde mi Mi A2 mediante Tapatalk

do you use any mangle rule while using ecmp ?

No I don't have any mangle rule at my network (excepting when I need to configure GRE tunnels in the middle so I change the TCP-MSS via mangle rules).

thanks for your answer sri2007,
Im using ecmp but faced some problems. for example roter that with ecmp have no internet. also dont you need any mangle for mark packet for if packets comes from ether1 to go over ether1 ?

No prob amt!

Actually I don't have issues related to ECMP, I think that OSPF do the load balancing per connection, and I don't care if the traffic is symmetric (same interface in/out), in my tests the only inconvenience is when my customers need to to a classic bandwidth test using speediest.net, so the connection is only forwarded to one interface, so he thinks that the service is not completed, however when he starts to move traffic between my links everything works great.

Btw, I choose the election of do not block any traffic at the forward chain (also not blocking the invalid connections either).

At only 7 sites in and 250 routes, we are already looking for a new solution before we grow out of control.

The concept of Area0, no area-to-area communication (must go through area0) and all area's must connect to 0, no ability to summarize except at ABR's is just awful for WISP design where the networks can grow in all kinds of directions, very variable bandwidth and reliability, often a big possibility of split area 0, not always possible to have even a single redundant path (body corp may only prohibit a single radio on a building for instance). It's really hard and often impossible to design a 'proper' OSPF network like its supposed to be. OSPF is designed for enterprise networks with a lot of control, not WISPs
It's my opinion that EIGRP would be the best protocol for many MikroTik deployments. It solves all the above problems, you can grow your network in any direction with daisy chained AS's, it's arguably faster and more reliable than OSPF as it keeps backup routes in memory, it can summarize anywhere on any router, you don't need to start with an Area0 and then worry about the merging later you can just start each site on i.e. AS100/200/300/400/500 and merge them later with zero issue

Every time I think about it I can't find a single good reason for using OSPF over EIGRP with the only exception being "ospf works with any vendor". If MikroTik supported EIGRP then that wouldn't be a problem for me, i'd run MikroTik and Cisco everywhere, and redistribute between OSPF when not possible i.e from a merger. At least that way can keep multiple Area0 OSPF networks from merges and just tag routes

At only 7 sites in and 250 routes, we are already looking for a new solution before we grow out of control.

The concept of Area0, no area-to-area communication (must go through area0) and all area's must connect to 0, no ability to summarize except at ABR's is just awful for WISP design where the networks can grow in all kinds of directions, very variable bandwidth and reliability, often a big possibility of split area 0, not always possible to have even a single redundant path (body corp may only prohibit a single radio on a building for instance). It's really hard and often impossible to design a 'proper' OSPF network like its supposed to be. OSPF is designed for enterprise networks with a lot of control, not WISPs
It's my opinion that EIGRP would be the best protocol for many MikroTik deployments. It solves all the above problems, you can grow your network in any direction with daisy chained AS's, it's arguably faster and more reliable than OSPF as it keeps backup routes in memory, it can summarize anywhere on any router, you don't need to start with an Area0 and then worry about the merging later you can just start each site on i.e. AS100/200/300/400/500 and merge them later with zero issue

Every time I think about it I can't find a single good reason for using OSPF over EIGRP with the only exception being "ospf works with any vendor". If MikroTik supported EIGRP then that wouldn't be a problem for me, i'd run MikroTik and Cisco everywhere, and redistribute between OSPF when not possible i.e from a merger. At least that way can keep multiple Area0 OSPF networks from merges and just tag routes

Hello millenium7, if your WISP grows enough you'll see that an OSPF structured design combined with BGP is a great advantage, I've seen so many networks that grows in many directions became really unstable (I used to work with IPArchitechs and I've seen networks in lots of countries at the world) EIGRP may be a good option however, if we need to choose an election to a new protocol I'll prefer IS-IS instead of EIGRP, that may be a most stable and if you want to check that there is a forum entry discussing IS-IS.

The speed of convergence using OSPF and EIGRP, can be modified by changing some timers, also I'm not a big fan of redistributing things between protocols.

millenium7, if your WISP grows enough you'll see that an OSPF structured design combined with BGP is a great advantage, I've seen so many networks that grows in many directions became really unstable

You can't always structure it the way OSPF wants you to due to its inherently restrictive enterprise oriented rules

For instance maybe you're growing 2 separate area's, they start off as separate area's joined to Area0, but eventually they grow enough to start to overlap each other, and you pick up customers that have businesses in each, they're neighbors only 1 or 2 routers away with bandwidth-a-plenty between them and they want site to site connectivity. It's physically quite a simple thing, just have traffic go from area to area, but OSPF's design says "well, too bad, can't do it, screw you buddy" and everything has to go all the way down to area0 - murphy's law says through low bandwidth flakey links - then back up through the other area. And if your area0 connection goes down the site-to-site connectivity dies, and 1 area can't use the other as failover transit (without virtual links, at which point you're already breaking those nice perfect OSPF recommended designs). Alternatively you have to expand your area0 all the way through 1 or both area's. Both of these options are bad and don't make any reasonable sense

I'm sorry, but that design just flat out sucks and is total rubbish for the WISP industry. The real world doesn't work that way (hence my guess why there are so many networks running nothing but area0, because its just too restrictive in many environments, not due to laziness)
OSPF is fine for campus deployments, or when you have nothing but data center presence and can get 10gbit cross connects to everything. For WISPs, you just can't magically make it happen. You're bound by geographical and political restrictions. This is why EIGRP fits the bill so perfectly, you can fix all of these problems without having to constantly redesign your whole network, throw up links where you just don't need/want them or rely on additional protocols, tunnels etc to bodge it

millenium7, if your WISP grows enough you'll see that an OSPF structured design combined with BGP is a great advantage, I've seen so many networks that grows in many directions became really unstable

You can't always structure it the way OSPF wants you to due to its inherently restrictive enterprise oriented rules

For instance maybe you're growing 2 separate area's, they start off as separate area's joined to Area0, but eventually they grow enough to start to overlap each other, and you pick up customers that have businesses in each, they're neighbors only 1 or 2 routers away with bandwidth-a-plenty between them and they want site to site connectivity. It's physically quite a simple thing, just have traffic go from area to area, but OSPF's design says "well, too bad, can't do it, screw you buddy" and everything has to go all the way down to area0 - murphy's law says through low bandwidth flakey links - then back up through the other area. And if your area0 connection goes down the site-to-site connectivity dies, and 1 area can't use the other as failover transit (without virtual links, at which point you're already breaking those nice perfect OSPF recommended designs). Alternatively you have to expand your area0 all the way through 1 or both area's. Both of these options are bad and don't make any reasonable sense

I'm sorry, but that design just flat out sucks and is total rubbish for the WISP industry. The real world doesn't work that way (hence my guess why there are so many networks running nothing but area0, because its just too restrictive in many environments, not due to laziness)
OSPF is fine for campus deployments, or when you have nothing but data center presence and can get 10gbit cross connects to everything. For WISPs, you just can't magically make it happen. You're bound by geographical and political restrictions. This is why EIGRP fits the bill so perfectly, you can fix all of these problems without having to constantly redesign your whole network, throw up links where you just don't need/want them or rely on additional protocols, tunnels etc to bodge it

Well I think that everything can be done in a networking environment; I you have that scenario I'll prefer BGP to the customer instead of OSPF; actually I've never seen EIGRP as an IGP in a big network, the most used protocols are OSPF or IS-IS; I've played with EIGRP only between CPEs & main routers.

I know that at the beginning is hard to get 10gbps for WISPs, we took like 2 years to develop like 500km of fiber to deploy 10gig everywhere, and we did it considering a really hard country (geographically we're at Ecuador), and yes there are some political restrictions but those can be solved with some agreements between both parts.

<...>, if your WISP grows enough you'll see that an OSPF structured design combined with BGP is a great advantage, I've seen so many networks that grows in many directions became really unstable (I used to work with IPArchitechs and I've seen networks in lots of countries at the world) EIGRP may be a good option however, if we need to choose an election to a new protocol I'll prefer IS-IS instead of EIGRP, that may be a most stable and if you want to check that there is a forum entry discussing IS-IS.
<...>

EIGRP looks interesting but it is Cisco proprietary and unlikely to ever be standardized, so it's irrelevant for me (and seemingly in this forum).
IS-IS also looks interesting, it is an open standard and, if MicroTik could be persuaded to provide support, I'd definitely investigate further.
Meanwhile, I'm stuck with OSPF and BGP, possibly modified by script-based kludges to over come specific problems.

Am I correct that the largest OSPF Area 0 network that anyone has run is sri2007's network of 350 routers and 6000 routes running on CCR routers only?
And the largest OSPF network running on a mix of MT routers (including original RB750UPs, RB2011s, etc and CCRs) is my largest Area 0 with 276 routers and 2385 routes?

EIGRP was Cisco proprietary for a long time, it no longer is. There are a couple other vendors that use it
Many extensions to EIGRP are still Cisco proprietary but the core functionality and the bits that matter are there for anyone to implement

There is no technical reason why EIGRP couldn't be a widespread protocol used throughout the world. It scales just as well, it's just as fast if not faster, it's quite a lot simpler and easier to understand, it's much more flexible, it's arguably more reliable and faster to reconverge due to keeping backup routes in memory. But because it remained proprietary for a long time there's a huge problem if you had a Cisco network and then introduced another router it couldn't participate, hence I suspect the #1 reason why OSPF took off so much cause nobody ever wanted to be put in that position. There's technically no reason why vendors can't implement EIGRP themselves, they don't because not many people want to use it, they don't want to use it because not many vendors implement it..... chicken and egg scenario

; I you have that scenario I'll prefer BGP to the customer instead of OSPF;

Regarding BGP, it's not an option in the Mikrotik world if you want MPLS based services to the customer (we almost exclusively use PPPoE and carry it over VPLS tunnels) as Mikrotik doesn't support BGP LU