Hi,
Can anyone offer some simple suggestions for routing the attached scenario. At present I have a fully routed network and in the next few days I am about to close the "ring" by completing the last hop from 1 village across to another.
From then on, each village (AP) will have a choice of routes, should any route break then the traffic would go the other way.
But I want to go one stage further and split the load between each half of the loop, (diversity) and also if 1 half breaks, then the traffic continues to flow through the other surviving half (redundancy).
It looks simple when it comes to splitting traffic between 2 wireless links as I have suggested to someone else in a similar scenario.
But is there a routing policy out there that would make both these tasks easy??
Simon
-
-
wirelesswaves
Member
- Posts: 311
- Joined:
Routing a Ring Network with Diversity & Redundancy
You do not have the required permissions to view the files attached to this post.
Re: Routing a Ring Network with Diversity & Redundancy
Use OSPF and configure proper path costs, and each router should calculate proper routes for itself.
If the ring breaks, OSPF will kick in again and everything will still work.
Dynamic routing is the way to go.
If the ring breaks, OSPF will kick in again and everything will still work.
Dynamic routing is the way to go.
-
-
wirelesswaves
Member
- Posts: 311
- Joined:
Re: Routing a Ring Network with Diversity & Redundancy
Hi Tom
Thanks for the suggestion, but am I missing something, I was looking for my bread buttered on both top and bottom.
Will your suggestion also split the load ? ie say 20Mb in one direction and 20Mb in other.
Simon
Thanks for the suggestion, but am I missing something, I was looking for my bread buttered on both top and bottom.
Will your suggestion also split the load ? ie say 20Mb in one direction and 20Mb in other.
Simon
Re: Routing a Ring Network with Diversity & Redundancy
Sending traffic 2 ways in a ring topology to one destination is problematic. Packets can arrive out of order since conditions on one side of the ring can be different then on the other side. Also, how are you going to split the traffic? 50% one way, 50% the opposite way? What if one of the paths is already at full load?
Sub-packet load balancing should not be used, packet based load balancing doesnt make sense either (ECMP), so you would need to send traffic through different paths around the ring based on PCC (PCC src-address:dst-address probably best)
Implementing all that on top of dynamic routing (OSPF) ... I dont think its possible. But OSPF will properly split the traffic depending on which path is the closest for different destinations.
You could implement load balancing with failover based on PCC as well; but that would all have to be done manually on each node in the ring. It would provide failover, routing and load-balancing, but it still doesnt solve the issue of how to know how much traffic to send one direction in the ring, and how much the other direction.
You could go crazy and do it all using MPLS TE tunnels, but I dont know enough about MPLS and TE on this scale to help.
Sub-packet load balancing should not be used, packet based load balancing doesnt make sense either (ECMP), so you would need to send traffic through different paths around the ring based on PCC (PCC src-address:dst-address probably best)
Implementing all that on top of dynamic routing (OSPF) ... I dont think its possible. But OSPF will properly split the traffic depending on which path is the closest for different destinations.
You could implement load balancing with failover based on PCC as well; but that would all have to be done manually on each node in the ring. It would provide failover, routing and load-balancing, but it still doesnt solve the issue of how to know how much traffic to send one direction in the ring, and how much the other direction.
You could go crazy and do it all using MPLS TE tunnels, but I dont know enough about MPLS and TE on this scale to help.
-
-
wirelesswaves
Member
- Posts: 311
- Joined:
Re: Routing a Ring Network with Diversity & Redundancy
Hi Tom, I agree with most of what you say..
The current backbone is adequate for my input. However I am sure that having extra capacity available by halving the load could do no harm.
It doesnt have to be exactly 50/50, I would happily settle for 60/40..
It just seems such a waste to have a functioning circuit sitting there redundant.
Making a "failover" is so simple, with possibly a dozen methods.
I was thinking of alternate "new connection marks" in mangle to be used to make "route marks" and then using a script to perhaps modify those marks according to the surviving path.
Maybe I take a look at our PCC suggestion, I hadnt thought of that... I am sure someone on this forum must be using a ring type network with both diversity and redundancy.
The current backbone is adequate for my input. However I am sure that having extra capacity available by halving the load could do no harm.
It doesnt have to be exactly 50/50, I would happily settle for 60/40..
It just seems such a waste to have a functioning circuit sitting there redundant.
Making a "failover" is so simple, with possibly a dozen methods.
I was thinking of alternate "new connection marks" in mangle to be used to make "route marks" and then using a script to perhaps modify those marks according to the surviving path.
Maybe I take a look at our PCC suggestion, I hadnt thought of that... I am sure someone on this forum must be using a ring type network with both diversity and redundancy.
Re: Routing a Ring Network with Diversity & Redundancy
Actually, look at the presentation in my sig, it explains something very similiar.
Still, even dividing traffic 60/40, what if the part of the ring where 40% of the traffic goes is already at full load? The problem with scripting and mangling it localy is that you simply dont know about the conditions of the other links in the network to make an informed decision in the script as to where to send what part of traffic.
Thats why I suggested MPLS TE, because that would solve that.
Still, even dividing traffic 60/40, what if the part of the ring where 40% of the traffic goes is already at full load? The problem with scripting and mangling it localy is that you simply dont know about the conditions of the other links in the network to make an informed decision in the script as to where to send what part of traffic.
Thats why I suggested MPLS TE, because that would solve that.
Last edited by tomaskir on Sat Dec 22, 2012 2:04 pm, edited 1 time in total.
-
-
wirelesswaves
Member
- Posts: 311
- Joined:
Re: Routing a Ring Network with Diversity & Redundancy
Hi Tom.
Thanks, didn't see the link you posted until I re-read you posts.
I had a quick look at MPLS this morning, and in honesty I think its going to be beyond me.. Even if I do crack it, there is another thorn in my side: One of the segments of the ring is made up from a pair of Nanbridge M and I'm not prepared to throw weeks of studying into Ubiquity equipment.
So at the end of the day, unless there is further input, I might have to just resort to a failback scenario only. Shame.
Simon
Thanks, didn't see the link you posted until I re-read you posts.
I had a quick look at MPLS this morning, and in honesty I think its going to be beyond me.. Even if I do crack it, there is another thorn in my side: One of the segments of the ring is made up from a pair of Nanbridge M and I'm not prepared to throw weeks of studying into Ubiquity equipment.
So at the end of the day, unless there is further input, I might have to just resort to a failback scenario only. Shame.
Simon
Re: Routing a Ring Network with Diversity & Redundancy
If its a fully routed ring and no NAT is used, use OSPF. It will still provide partial load distribution, since the traffic is always going to take the shortest route around the ring.
When multiple sessions to multiple destinations in the ring are open, they can go different paths, depending on what their distance is to the origin of the connection.
OSPF will also deal with all redundancy and routing issues for you automatically. I think that is the best solution if you dont want to do MPLS TE.
When multiple sessions to multiple destinations in the ring are open, they can go different paths, depending on what their distance is to the origin of the connection.
OSPF will also deal with all redundancy and routing issues for you automatically. I think that is the best solution if you dont want to do MPLS TE.
Re: Routing a Ring Network with Diversity & Redundancy
I can even imagine setup with static routing only. Therefore you can use masquerading (if needed) and loadbalancing with PCC at village AP stations without complications. I imagine it this way:
-prepare two L2 semi-rings (bridged vlans or bridged eoip tunnels or switched vlans or mpls solutions - simply any kind of L2 connectivity). Connect every village AP with RB1100AH double way. One way in first L2 connection clockwise. Second way in second L2 connection anticlockwise,
-connect every semi-ring with RB1100AH only one way. First semi-ring from north (clockwise), the second semi-ring from south (anticlockwise),
-prepare two /29 subnets. One subnet for every L2 semi-ring. Add one ip for every AP in every semi-ring. Add one ip for RB1100AH in every semi-ring.
Suppose these are suggested subnets for semi-rings:
10.0.0.0/29
10.0.0.8/29
Suppose these are subnets used behind village AP's:
192.168.0.0/24
192.168.1.0/24
192.168.2.0/24
192.168.3.0/24
Suppose L2 rings are connected to AP's and RB1100AH via bridge interfaces:
bridge1
bridge2
Suppose you put addresses like this:
RB1100AH:
1 AP:
2 AP:
And so on...
Now you can use simple trick with static routing (but still adapting second gateway in case of connection problems):
1 AP, 2 AP, 3 AP, 4 AP:
RB1100AH:
You can play with distance parameter on every AP and on RB1100.
You can use two way static routing (loadbalancing) like this:
So a little bit advanced STATIC routing is sometimes better solution than dynamic routing. I do think so. 
The only missing functionality with this kind of setup is BFD protocol. You can use BFD with dynamic routing only. The question is to Mikrotik. Is it possible to implement BFD for static routes?
For instance this way:
Maybe only for PtP links... but still for static routing?
-prepare two L2 semi-rings (bridged vlans or bridged eoip tunnels or switched vlans or mpls solutions - simply any kind of L2 connectivity). Connect every village AP with RB1100AH double way. One way in first L2 connection clockwise. Second way in second L2 connection anticlockwise,
-connect every semi-ring with RB1100AH only one way. First semi-ring from north (clockwise), the second semi-ring from south (anticlockwise),
-prepare two /29 subnets. One subnet for every L2 semi-ring. Add one ip for every AP in every semi-ring. Add one ip for RB1100AH in every semi-ring.
Suppose these are suggested subnets for semi-rings:
10.0.0.0/29
10.0.0.8/29
Suppose these are subnets used behind village AP's:
192.168.0.0/24
192.168.1.0/24
192.168.2.0/24
192.168.3.0/24
Suppose L2 rings are connected to AP's and RB1100AH via bridge interfaces:
bridge1
bridge2
Suppose you put addresses like this:
RB1100AH:
Code: Select all
/ip address
add address=10.0.0.6/29 interface=bridge1
add address=10.0.0.14/29 interface=bridge2
Code: Select all
/ip address
add address=10.0.0.1/29 interface=bridge1
add address=10.0.0.9/29 interface=bridge2
Code: Select all
/ip address
add address=10.0.0.2/29 interface=bridge1
add address=10.0.0.10/29 interface=bridge2
Now you can use simple trick with static routing (but still adapting second gateway in case of connection problems):
1 AP, 2 AP, 3 AP, 4 AP:
Code: Select all
/ip route
add gateway=10.0.0.6 check-gateway=arp distance=1
add gateway=10.0.0.14 check-gateway=arp distance=2
Code: Select all
/ip route
add dst-address=192.168.0.0/24 gateway=10.0.0.1 check-gateway=arp distance=1
add dst-address=192.168.1.0/24 gateway=10.0.0.2 check-gateway=arp distance=1
add dst-address=192.168.2.0/24 gateway=10.0.0.3 check-gateway=arp distance=2
add dst-address=192.168.3.0/24 gateway=10.0.0.4 check-gateway=arp distance=2
add dst-address=192.168.0.0/24 gateway=10.0.0.9 check-gateway=arp distance=2
add dst-address=192.168.1.0/24 gateway=10.0.0.10 check-gateway=arp distance=2
add dst-address=192.168.2.0/24 gateway=10.0.0.11 check-gateway=arp distance=1
add dst-address=192.168.3.0/24 gateway=10.0.0.12 check-gateway=arp distance=1
You can use two way static routing (loadbalancing) like this:
Code: Select all
/ip route
add dst-address=192.168.0.0/24 gateway=10.0.0.1,10.0.0.9 check-gateway=arp distance=1
The only missing functionality with this kind of setup is BFD protocol. You can use BFD with dynamic routing only. The question is to Mikrotik. Is it possible to implement BFD for static routes?
For instance this way:
Code: Select all
/ip route
add gateway=1.2.3.4 check-gateway=bfd
Re: Routing a Ring Network with Diversity & Redundancy
Using it with "gateway=10.0.0.1,10.0.0.9" is ECMP (Equal Cost Multi Path). I already mentioned it in the posts above, also describing why its a bad idea.
Imagine that your ring looks like this:
Now imagine something behind node 1 is trying to communicate with something behind node 2. Half traffic will go 1 -> 2. Other half traffic will got 1->4->6-> etc ->3->2. Obviously this is bad. Packets will arrive out of sequence. You dont know if links 6->7 is already at full capacity, etc. I personally think ECMP would break more then help anything.
Also, with OSPF, redundancy and automatic routing changes are handled much better then with any static config. You can indeed use BFD with OSPF if you need fast timers but no BFD with static routes.
Imagine that your ring looks like this:
Code: Select all
1---2---3
| |
4 5
| |
6---7---8
Also, with OSPF, redundancy and automatic routing changes are handled much better then with any static config. You can indeed use BFD with OSPF if you need fast timers but no BFD with static routes.
Last edited by tomaskir on Tue Dec 25, 2012 5:55 pm, edited 1 time in total.
Re: Routing a Ring Network with Diversity & Redundancy
I only show one of possible solutions. I do not say: static way (my way) is better than dynamic. I use described static routing in few places and it works without any problems. In biggest case I route this way about 1.5 Gbps of traffic from few thousands of customers.
It is possible to add additional static routes to counteract routing between segments through RB1100. But I omit that for simplicity.
And again: OSPF solution is very good for discussed case! I don't deny that. OK?
I only show possible alternative.
I use described static routing in few places and it works without any problems. In biggest case I route this way about 1.5 Gbps of traffic from few thousands of customers.
If you mean: "bad idea because of unordered packets flow..." it is simply not true. In ECMP packets are sorted per connection and in one connection they use only one gateway. So no chance to unordered packets.Using it with "gateway=10.0.0.1,10.0.0.9" is called ECMP (Equal Cost Multi Path). I already mentioned it in the posts above, also describing why its a bad idea.
Again not true. No packets out of sequence in this solution. It is, of course, possible than you will have asymetric routing. But not unordered packets. Asymetric routing this case means: packets in up direction use not the same way than packets in down direction. But this situation is not as wrong as you suggest. For instance in global BGP routing you have many cases when routing is everyday asymetric.Now imagine something behind node 1 is trying to communicate with something behind node 2. Half traffic will go 1 -> 2. Other half traffic will got 1->4->6-> etc ->3->2. Obviously this is bad. Packets will arrive out of sequence.
It is possible to add additional static routes to counteract routing between segments through RB1100. But I omit that for simplicity.
And again: OSPF solution is very good for discussed case! I don't deny that. OK?
I only show possible alternative.
Last edited by femur on Tue Dec 25, 2012 6:01 pm, edited 1 time in total.
Re: Routing a Ring Network with Diversity & Redundancy
You are right. For some reason I remembered ECMP is packet-based. Did some reasearch, and you are entirelly right, its src-dst address based (route decisions are cached.)
http://wiki.mikrotik.com/wiki/Manual:IP ... .29_routes
How to detect network outages farther then one hop tho if all routing is static tho? Also, since this is wireless and all traffic from all nodes is constantly going both ways around the ring, dont you think it would actually cause worse performance then if the traffic always took the shortest route? (considering a scenario that all nodes are transfering data)
Im not saying OSPF is the best answer either, just offering my opinions
http://wiki.mikrotik.com/wiki/Manual:IP ... .29_routes
How to detect network outages farther then one hop tho if all routing is static tho? Also, since this is wireless and all traffic from all nodes is constantly going both ways around the ring, dont you think it would actually cause worse performance then if the traffic always took the shortest route? (considering a scenario that all nodes are transfering data)
Im not saying OSPF is the best answer either, just offering my opinions
Re: Routing a Ring Network with Diversity & Redundancy
As you can see at attached picture there is no more hops than one. In layer2 you see gateway directly. So there is no need to look farther then one hop. And in this situation router automatically detects outages via arp or via ping (check-gateway=arp or check-gateway=ping). In my network I use check-gateway=arp. Routers need about 8-10 seconds to detect outage. When link returns after outage, there is no such brake.How to detect network outages farther then one hop
In simplest case, when you do not use ECMP routing, there is no such problem. For instance:Also, since this is wireless and all traffic from all nodes is constantly going both ways around the ring, dont you think it would actually cause worse performance then if the traffic always took the shortest route?
-traffic from and to AP1 and AP2 you can route via bridge1
-traffic from and to AP3 and AP4 you can route via bridge2
Code: Select all
# this is at AP1 and AP2
/ip route
add gateway=10.0.0.6 check-gateway=arp distance=1
add gateway=10.0.0.14 check-gateway=arp distance=2Code: Select all
# this is at AP3 and AP4
/ip route
add gateway=10.0.0.6 check-gateway=arp distance=2
add gateway=10.0.0.14 check-gateway=arp distance=1Code: Select all
# this is at RB1100
/ip route
add dst-address=192.168.0.0/24 gateway=10.0.0.1 check-gateway=arp distance=1
add dst-address=192.168.1.0/24 gateway=10.0.0.2 check-gateway=arp distance=1
add dst-address=192.168.2.0/24 gateway=10.0.0.3 check-gateway=arp distance=2
add dst-address=192.168.3.0/24 gateway=10.0.0.4 check-gateway=arp distance=2
add dst-address=192.168.0.0/24 gateway=10.0.0.9 check-gateway=arp distance=2
add dst-address=192.168.1.0/24 gateway=10.0.0.10 check-gateway=arp distance=2
add dst-address=192.168.2.0/24 gateway=10.0.0.11 check-gateway=arp distance=1
add dst-address=192.168.3.0/24 gateway=10.0.0.12 check-gateway=arp distance=1
There is no need to use ECMP. This is only possibility if you want or need to loadbalance traffic more precise than suggested erlier. It is not necessary.
And finaly, for precision... this is STATIC config. You can test, observe and manually configure which link should use every AP node. It adapts to link outages, but does not automatically adapt to link congestions.
You do not have the required permissions to view the files attached to this post.
-
-
wirelesswaves
Member
- Posts: 311
- Joined:
Re: Routing a Ring Network with Diversity & Redundancy
Femur: Quite brilliant feedback thanks..
Apologies it took so long to respond, for some reason my "topic notifications" seem to be radomly sent to trash by those kind people at hotmail!!
In the meantime, I did consult Butch (GOD) Evans and he suggested a look at OSPF and in the meantime I had thought the thread had died a death.
Anyhow love the idea, and its going to fit nicely into the plan, since there is a rotten apple in the barrel, and I decided (foolishly) to give Ubiquity nanostations a try recently (fed up of waiting for MT to deliver something with more gain than the sextant)
As it turns out the Nanostation is mechanically good but randomly drops out (when paired with a MT). But the right hand side of the circuit is finally closed (next week) with a pair of nanobridges! (give it one last try!) and hopefully they will be a little more reliable than when they are mixed with MT.
But the biggest issue so far is that the ubnt radio products have limited routing capabilities. So the pair that form one of the links will have to be at Layer2 anyway with WDS.
Thanks for all the input.
Apologies it took so long to respond, for some reason my "topic notifications" seem to be radomly sent to trash by those kind people at hotmail!!
In the meantime, I did consult Butch (GOD) Evans and he suggested a look at OSPF and in the meantime I had thought the thread had died a death.
Anyhow love the idea, and its going to fit nicely into the plan, since there is a rotten apple in the barrel, and I decided (foolishly) to give Ubiquity nanostations a try recently (fed up of waiting for MT to deliver something with more gain than the sextant)
As it turns out the Nanostation is mechanically good but randomly drops out (when paired with a MT). But the right hand side of the circuit is finally closed (next week) with a pair of nanobridges! (give it one last try!) and hopefully they will be a little more reliable than when they are mixed with MT.
But the biggest issue so far is that the ubnt radio products have limited routing capabilities. So the pair that form one of the links will have to be at Layer2 anyway with WDS.
Thanks for all the input.
Re: Routing a Ring Network with Diversity & Redundancy
Just use l2 protocol for ring purposes on switch platform that supported it like EAPS on extreme, ERPP on others or something like that.Hi,
Can anyone offer some simple suggestions for routing the attached scenario. At present I have a fully routed network and in the next few days I am about to close the "ring" by completing the last hop from 1 village across to another.
From then on, each village (AP) will have a choice of routes, should any route break then the traffic would go the other way.
But I want to go one stage further and split the load between each half of the loop, (diversity) and also if 1 half breaks, then the traffic continues to flow through the other surviving half (redundancy).
It looks simple when it comes to splitting traffic between 2 wireless links as I have suggested to someone else in a similar scenario.
But is there a routing policy out there that would make both these tasks easy??
Simon
Thx
//Robert
Re: Routing a Ring Network with Diversity & Redundancy
I know this post is old, but did you find a solution for your issue?
Who is online
Users browsing this forum: No registered users and 2 guests