Hi,

I would like some suggestions how to solve my routing problem. Here it goes....

I have a router, connected to ISP1 and to my wireless backbone. There is also a EOIP + OSPF tunnel through internet back to the backbone (backup line). Everything works smoothly. Sometimes just happens that our clients are trying to access an IP that is in the same class as our public IP on ISP1 (we have a default route on a /16 network). Normally this wouldn't be a problem if those clients were from private address range, but they have public addresses and have to be routed into our backbone first and then through other ISPs out back to that /16 class.

What would be preferred way to solve this and keep our dynamic routing?


Thanks

A network diagram and actual routing tables will help us to help

Hi,

Here is the diagram. I actually know were the problem is, but I don't know how to solve it. On R3 (port ISP3), there is a public network /16 (for backup in case of wireless links goes down). I can actually reproduce this in our lab. The problem is the connected route on public port of R3, with distance 1. If the client with our public IP is going for internet, it is routed correctly through our network up to R1 and out. If the same client is going for the IP in the same network as R3 public port then R3 wants to route it on ISP3 public port, this I have to prevent. Our public IPs can go out only through R1 and R2.

I exported just the relevant routes from R3.
/ip route print
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 A S 0.0.0.0/0 172.31.253.250 1
77 ADC 10.11.11.0/30 10.11.11.1 ether4 0
78 ADC 10.11.11.4/30 10.11.11.5 ether5 0
89 ADo 172.16.6.0/24 10.11.11.2 110
479 ADC 172.31.253.0/24 172.31.253.253 ether1 0
480 Do 172.31.253.0/24 10.11.11.6 110

The static route number 0 has distance of 1, but it is also reachable at route number 480 with distance 110. I need to accomplish that route 480 is the preferred route. I could change that default 0.0.0.0/0 to point just to our R1 EOIP endpoint (for OSPF backup) but it will not solve this specific problem.

R3 config details:
/ip address print
Flags: X - disabled, I - invalid, D - dynamic
# ADDRESS NETWORK INTERFACE
0 10.11.11.1/30 10.11.11.0 ether4
1 172.31.253.253/24 172.31.253.0 ether1 (public IP - lab test)
2 10.11.11.5/30 10.11.11.4 ether5

traceroute to 8.8.8.8 (8.8.8., 30 hops max, 60 byte packets
1 172.16.6.250 (172.16.6.250) 0.274 ms 0.303 ms 0.362 ms
2 10.11.11.1 (10.11.11.1) 0.978 ms 0.990 ms 1.001 ms
3 10.1.5.251 (10.1.5.251) 1.039 ms 1.052 ms 1.065 ms
4. internet

traceroute to 172.31.253.200 (172.31.253.200), 30 hops max, 60 byte packets
1 172.16.6.250 (172.16.6.250) 0.263 ms 0.308 ms 0.366 ms
2 10.11.11.1 (10.11.11.1) 0.982 ms 0.993 ms 1.003 ms
3 172.31.253.200 (172.31.253.200) 1.277 ms 1.317 ms 1.398 ms

My goal is to route 172.31.253.0/24 through 10.1.5.251.


Thanks.

Question / Suggestion:- Are you not able to run BGP with ISP3 ? that would allow for a much more consistent network setup ....

setup eBGP with one peer on each of the routerboards.
run iBGP between the three routers (a mesh)
run ospf between the three, and your internal network.

This way you can control the traffic better...

Question / Suggestion:- Are you not able to run BGP with ISP3 ? that would allow for a much more consistent network setup ....

setup eBGP with one peer on each of the routerboards.
run iBGP between the three routers (a mesh)
run ospf between the three, and your internal network.

This way you can control the traffic better...

ISP2 and ISP3 is the same provider on different locations.

ISP2 and ISP3 is the same provider on different locations.

That doesn't matter. There's an entire metric in BGP, "med" which is dedicated to routing decisions between two ASNs with multiple links between them. In fact, some providers allow a private advertisement of prefixes longer than /24 and you can leverage that for some really nice traffic engineering with that ISP.

For example, you could possibly advertise the customer's /32 directly to "ISP3" and the ISP would use this link to reach that specific customer. Traffic arriving for that /32 via ISP2 would still work normally as well....

ISP2 and ISP3 is the same provider on different locations.

That doesn't matter. There's an entire metric in BGP, "med" which is dedicated to routing decisions between two ASNs with multiple links between them. In fact, some providers allow a private advertisement of prefixes longer than /24 and you can leverage that for some really nice traffic engineering with that ISP.

For example, you could possibly advertise the customer's /32 directly to "ISP3" and the ISP would use this link to reach that specific customer. Traffic arriving for that /32 via ISP2 would still work normally as well....

I've checked with the provider, it could be done as you say. But there is a 30% increase of the price. At this time I am not willing to do this for 2 clients. Any other options would you recommend?

I was searching the forum and came to a conclusion (I think was your post), upgrade to MPLS and build VPLS tunnels for all public IPs from every POP up to the core.

Ok - I've looked at your problem statement again - the thread was starting to veer away from the original problem.

As I understand it:
You have public IPs from R1, and use some of them on customers behind R3. If the link from R1 to R3 goes down, you want the EoIP tunnel between R1 and R3 to take over the same job. You want the internet link at R3 to only be there for this tunnel, and not direct IP....

I don't see why it should be an issue if the IP goes to the Internet via the connection at R3 and the replies come in via R1 --> R3 tunnel... unless the ISP is blocking packets with that source IP range. (Since it's the same ISP, they should be willing to allow such traffic and it should NOT cost even one penny more.)
Asymmetric routing is a normal fact of life on the Internet.

If they won't allow this, then here's your fix:

Make sure the EoIP tunnel uses the actual isp1 interface IP of R1 and isp3 interface IP of R3. (not your own public IP addresses) say x.x.x.x = isp1 interface on R1, and y.y.y.y = isp3 interface on R3.

On R1, create a static route y.y.y.y/32 gw=isp1
On R3, create a static route x.x.x.x/32 gw=isp3
On R3, do not create any default GW route as a static route. Let OSPF determine that.

Enable OSPF across the EoIP tunnel, and make sure the cost of the EoIP interface is significantly higher than the internal links' costs so that it's never used except when your own link is down. Make sure of this in both R1 and in R3.

Done!

When your own internal link is up, OSPF will route the traffic across your own links as normal. When that link is down, OSPF will route across the tunnel for inbound traffic towards the customer's public IPs, AND it will route through the tunnel towards the internet. (default route will be learned in OSPF via the tunnel)


MPLS/VPLS won't have any benefit for this problem you're trying to solve.

Ok - I've looked at your problem statement again - the thread was starting to veer away from the original problem.

As I understand it:
You have public IPs from R1, and use some of them on customers behind R3. If the link from R1 to R3 goes down, you want the EoIP tunnel between R1 and R3 to take over the same job. You want the internet link at R3 to only be there for this tunnel, and not direct IP....

I don't see why it should be an issue if the IP goes to the Internet via the connection at R3 and the replies come in via R1 --> R3 tunnel... unless the ISP is blocking packets with that source IP range. (Since it's the same ISP, they should be willing to allow such traffic and it should NOT cost even one penny more.)
Asymmetric routing is a normal fact of life on the Internet.

If they won't allow this, then here's your fix:

Make sure the EoIP tunnel uses the actual isp1 interface IP of R1 and isp3 interface IP of R3. (not your own public IP addresses) say x.x.x.x = isp1 interface on R1, and y.y.y.y = isp3 interface on R3.

On R1, create a static route y.y.y.y/32 gw=isp1
On R3, create a static route x.x.x.x/32 gw=isp3
On R3, do not create any default GW route as a static route. Let OSPF determine that.

Enable OSPF across the EoIP tunnel, and make sure the cost of the EoIP interface is significantly higher than the internal links' costs so that it's never used except when your own link is down. Make sure of this in both R1 and in R3.

Done!

When your own internal link is up, OSPF will route the traffic across your own links as normal. When that link is down, OSPF will route across the tunnel for inbound traffic towards the customer's public IPs, AND it will route through the tunnel towards the internet. (default route will be learned in OSPF via the tunnel)


MPLS/VPLS won't have any benefit for this problem you're trying to solve.

OK, now it is configured per your instructions and I am bumping into same problem I explained at the start. Both links wireless and fiber (through R3 are doing fine), OSPF is working. The problem is if I try to connect to server, with the IP in the same class as my R3 ISP IP.

R3 config and routing table:
/ip address print
Flags: X - disabled, I - invalid, D - dynamic
# ADDRESS NETWORK INTERFACE
7 xx.yy.246.221/16 xx.yy.0.0 vlan8

/ip route print
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADo 0.0.0.0/0 10.1.5.250 110
84 ADC xx.yy.0.0/16 xx.yy.246.221 vlan8 0
85 A S xx.yy.25.82/32 xx.yy.0.1 1

Now, my client is trying to reach xx.yy.254.2 (same class as my R3 ISP IP) this one is a connected route with distance 0 and it is trying to go out on my vlan8 interface (ISP3) not through EOIP via ISP3 nor through 10.1.5.250 (R1). I hope I've explained myself well.

Mitja

Why aren't more specific routes coming through the tunnel?
The ISP3 interface is in the same /16 as your range - that's quite a large netmask for that interface - that looks odd to me.
Make sure that's not mis-configured. Even if the ISP is having you use a /16 there - then as long as you can get your main routers to advertise the more specifc prefixes of your main network via the tunnel (not just a default route) then the more specific route will win.

[flash=][/flash]

Why aren't more specific routes coming through the tunnel?
The ISP3 interface is in the same /16 as your range - that's quite a large netmask for that interface - that looks odd to me.
Make sure that's not mis-configured. Even if the ISP is having you use a /16 there - then as long as you can get your main routers to advertise the more specifc prefixes of your main network via the tunnel (not just a default route) then the more specific route will win.

Could you elaborate a bit more what specific routes should I be looking for?

I double checked, ISP3 interface is in /16 range and the settings are correct. Now we found there is not just one webpage (IP) that is problematic, there is also a local Google cache in the same range, that's even worse.

You are saying that I should advertise more specific routes aka that /16 range in our backbone to force OSPF to change preferred route from R3 via tunnel instead of the connected route. Just to double check, OSPF can win against connected route if specific routes are injected into backbone even if connected routes have distance=1?

Thanks

PS: Sorry for my late answers, but I can do this tests just at night when there is little or no traffic.

Could you elaborate a bit more what specific routes should I be looking for?

Remember that the first rule of routing is that the most specific route wins.
Surely you aren't using a /16 netmask everywhere.
I'm assuming the ISP has allocated a /24 or /23 or some similar prefix length to you for your use in BGP routing / numbering your own network segments internal to your network. Even if they fall inside of the ISP's /16, if a /24 route will take priority over a /16.

So if the ISP3 interface is 172.16.4.198/16 and you have your own networks from the ISP on R1 and R2 - say 172.16.92.0/23 - then you should have a route in OSPF for 172.16.92.0/23. This more precise prefix will cause your traffic to route via the tunnel and not the ISP. Without this more specific prefix, your routing table says this:

/16 = isp3
/0 = tunnel to self

If a target IP falls inside of the /16 then that route will be chosen.
That's why you're having this problem.
Make sure your own prefixes also appear on tunnel to self so that the tunnel and not the ISP gets chosen for those destinations.

Could you elaborate a bit more what specific routes should I be looking for?

Remember that the first rule of routing is that the most specific route wins.
Surely you aren't using a /16 netmask everywhere.
I'm assuming the ISP has allocated a /24 or /23 or some similar prefix length to you for your use in BGP routing / numbering your own network segments internal to your network. Even if they fall inside of the ISP's /16, if a /24 route will take priority over a /16.

So if the ISP3 interface is 172.16.4.198/16 and you have your own networks from the ISP on R1 and R2 - say 172.16.92.0/23 - then you should have a route in OSPF for 172.16.92.0/23. This more precise prefix will cause your traffic to route via the tunnel and not the ISP. Without this more specific prefix, your routing table says this:

/16 = isp3
/0 = tunnel to self

If a target IP falls inside of the /16 then that route will be chosen.
That's why you're having this problem.
Make sure your own prefixes also appear on tunnel to self so that the tunnel and not the ISP gets chosen for those destinations.

We are getting close now...I think you missed some of my first data I've posted and you'll have a whole picture now:
Our BGP network is /22 in size. Currently only one segment say 172.16.92.0/23 is for end users that requires public IPs. R1 and R2 have a VRRP interface with IP 172.16.92.1/32 from this segment and act as a default gw for clients of this segment.
Clients with public IPs are getting their settings from local DHCP relay on every pop. Their settings looks like this:
IP: 172.16.92.100/23, GW:172.16.92.1. After delivering the IP, DHCP server injects a static route 172.16.92.100/32 into pop's interface where the IP is connected. And finally with the use of proxy-arp the client is able to reach their gateway at VRRP interfaces and back.

Yes, it is a very confusing and old configuration, that's why I opted to upgrade to MPLS, where I could just separate a POP into 2 VLANs (public and private) and tunnel both up to the core routers.

Clients with public IPs are getting their settings from local DHCP relay on every pop. Their settings looks like this:
IP: 172.16.92.100/23, GW:172.16.92.1. After delivering the IP, DHCP server injects a static route 172.16.92.100/32 into pop's interface where the IP is connected. And finally with the use of proxy-arp the client is able to reach their gateway at VRRP interfaces and back.

Yes, it is a very confusing and old configuration, that's why I opted to upgrade to MPLS, where I could just separate a POP into 2 VLANs (public and private) and tunnel both up to the core routers.

I know EXACTLY what you're doing - I set this up with Cisco routers at a previous job. I dubbed it 'secret sauce'

But that doesn't change anything I said. The most specific route wins. It doesn't matter that I said /24 in my example - /22 is still more specific than /16 - and /32 is most definitely more specific.....

(Let's make sure I still have your problem correct in my mind - you're seeing packets FROM R3 and downstream from R3 going to R1 and R2 via the ISP3 link and not via the tunnel, whenever the normal link from R1 to R3 is down.)

Assuming that's right, and given what you've shared from your routing table, I would expect this behavior.

Let's look at the routes you posted:
0 ADo 0.0.0.0/0 10.1.5.250 110
84 ADC xx.yy.0.0/16 xx.yy.246.221 vlan8 0
85 A S xx.yy.25.82/32 xx.yy.0.1 1

84 is the automatic "directly connected" route which is there as a consequence of having x.y.246.221/16 as the interface IP. (apparently the ISP us using 'secret sauce' as well)
85 is your nail-up route for the remote tunnel endpoint on R1.... right?
The default route is obviously via the tunnel.

So - by these three routes, packets for your hosts connected to R1 and R2 will go via ISP3 and not the tunnel.
Why?
Your addresses are a subset of xx.yy.0.0/16
Unless there are /32 routes for your hosts off of R1 and R2, why in the world would R3 use the tunnel?
At the very least, your router should be receiving a /22 for your particular subset - say xx.yy.92.0/22 - via the tunnel if you want the tunnel to be used.

I see that the default route via the tunnel is learned via OSPF.
Surely the /32 routes for hosts in the R1/R2 area (I don't say backbone because I'm not referring to OSPF backbone area and I don't want that to become a source of confusion) are also coming in across the tunnel... Your routing table is at least 85 entries long!

I recommend the following:
- create a static /22 black hole route for your own prefix (to prevent wasted throughput by sending packets around looking for hosts that you haven't assigned yet) on all of your routers.
- Make sure all assigned hosts / subnets of your space are appearing properly in OSPF
- Make sure these include routes via the tunnel - but that the tunnel has significantly higher link cost (at both endpoints) than the normal direct R1/R3 link.
- Use the ISP-provided IP addresses as the EoIP tunnel endpoints and their associated /32 static routes, not IP addresses from your own /22 prefix. (I think this is already the case, but the IP obfuscation makes it a tad uncertain to me so I decided to list this detail also)