eBGP and iBGP config with OSPF for internal

WISPa · Fri Dec 18, 2015 12:59 pm

We currently have 6 x CCR 1009 running RouterOS v6.33.3 all of which are configured with OSPF interconnected routing. There are 2 devices (edge1 and edge2) at our edge with provider peers, 2 devices in the middle performing QoS functions, and 2 devices for access (PPPoE aggregaters). OSPF is correctly providing connectivity even as various connections are removed from the network, as you would expect.

On our edge devices, we have configured addresses on bridge loopback interfaces for iBGP and on physical interfaces for our upstream peers with providers.

Our upstream peers are established and functioning fine (this is also to MikroTiks with RouterOS v6.28).

iBGP just will not establish between edge1 and edge2 no matter how we try. The configs for eBGP and iBGP are identical, except for the addresses, route-reflect and AS numbers. The loopback addresses have reliable TCP connectivity to each other and no firewall rules exist to prevent tcp port 179.

With bgp and debug logging enabled, the entries we see in the logs on both peers are:

TCP connection established
  RemoteAddress=x.x.x.x
Entering OpenSent State
  RemoteAddress=x.x.x.x
Sent OPEN message
  Remote Address x.x.x.x
  Length=45
Connection terminated
  RemoteAddress=x.x.x.x

Our configs for the 2 edge devices are:
Edge2:

[x@R6-Edge2] > routing bgp instance print            
Flags: * - default, X - disabled 
 0 *X name="default" as=65530 router-id=0.0.0.0 redistribute-connected=no redistribute-static=no redistribute-rip=no redistribute-ospf=no redistribute-other-bgp=no 
      out-filter="" client-to-client-reflection=yes ignore-as-path-len=no routing-table="" 

 1    name="bgp-external" as=201xxx router-id=1xx.9x.75.250 redistribute-connected=yes redistribute-static=no redistribute-rip=no redistribute-ospf=no 
      redistribute-other-bgp=no out-filter=to-xx client-to-client-reflection=no ignore-as-path-len=no routing-table="" 

 2    name="bgp-internal" as=65531 router-id=1xx.8x.155.246 redistribute-connected=yes redistribute-static=no redistribute-rip=no redistribute-ospf=no 
      redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table="" 
      
[x@R6-Edge2] > routing bgp peer print status 
Flags: X - disabled, E - established 
 0 E name="peer-xx" instance=bgp-external remote-address=1xx.9x.75.249 remote-as=49xxx tcp-md5-key="" nexthop-choice=default multihop=no route-reflect=no hold-time=3m 
     ttl=default in-filter="" out-filter="" address-families=ip default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no remote-id=1xx.9x.75.249 
     local-address=1xx.9x.75.250 uptime=38m42s prefix-count=4 updates-sent=3 updates-received=5 withdrawn-sent=2 withdrawn-received=1 remote-hold-time=3m used-hold-time=3m 
     used-keepalive-time=1m refresh-capability=yes as4-capability=yes state=established 

 1   name="peer-r5-edge1" instance=bgp-internal remote-address=1xx.8x.155.245 remote-as=65531 tcp-md5-key="" nexthop-choice=default multihop=no route-reflect=yes 
     hold-time=3m ttl=default in-filter="" out-filter="" address-families=ip default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no 
     state=opensent

Edge1:

[x@Edge1] > routing bgp instance print 
Flags: * - default, X - disabled 
 0 *X name="default" as=65530 router-id=0.0.0.0 redistribute-connected=no redistribute-static=no redistribute-rip=no redistribute-ospf=no redistribute-other-bgp=no 
      out-filter="" client-to-client-reflection=yes ignore-as-path-len=no routing-table="" 

 1    name="bgp-internal" as=65531 router-id=1xx.8x.155.245 redistribute-connected=yes redistribute-static=no redistribute-rip=no redistribute-ospf=no 
      redistribute-other-bgp=no out-filter="" client-to-client-reflection=no ignore-as-path-len=no routing-table="" 

 2    name="bgp-external" as=201xxx router-id=1xx.9x.78.226 redistribute-connected=no redistribute-static=no redistribute-rip=no redistribute-ospf=no 
      redistribute-other-bgp=no out-filter=to-xx client-to-client-reflection=no ignore-as-path-len=no routing-table="" 
      

[x@Edge1] > routing bgp peer print status 
Flags: X - disabled, E - established 
 0 X name="bgp-r1-ppp1" instance=bgp-internal remote-address=10.240.100.1 remote-as=65432 tcp-md5-key="" nexthop-choice=default multihop=no route-reflect=no hold-time=3m 
     ttl=default in-filter="" out-filter="" address-families=ip default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no state=active 

 1 E name="peer-xx" instance=bgp-external remote-address=1xx.9x.78.225 remote-as=49xxx tcp-md5-key="" nexthop-choice=default multihop=no route-reflect=no hold-time=3m 
     ttl=default in-filter="" out-filter="" address-families=ip default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no remote-id=1xx.9x.78.225 
     local-address=1xx.9x.78.226 uptime=42m37s prefix-count=4 updates-sent=3 updates-received=5 withdrawn-sent=2 withdrawn-received=1 remote-hold-time=3m 
     used-hold-time=3m used-keepalive-time=1m refresh-capability=yes as4-capability=yes state=established 

 2   name="peer-r6-edge2" instance=bgp-internal remote-address=1xx.8x.155.246 remote-as=65531 tcp-md5-key="" nexthop-choice=default multihop=no route-reflect=yes 
     hold-time=3m ttl=default in-filter="" out-filter="" address-families=ip default-originate=never remove-private-as=no as-override=no passive=no use-bfd=no 
     state=opensent

Can anyone point us in the right direction to establish the iBGP peer between edge1 and edge2? We are experiencing the same problem with other internal router BGP peers but I expect the issue to be the same for those and resolved by sorting out just the one.

Any help would be appreciated. Thanks

Fri Dec 18, 2015 3:38 pm

Don't use a different instance for iBGP. A different instance will have a completely separate table of routes.

WISPa · Fri Dec 18, 2015 5:14 pm

Don't use a different instance for iBGP. A different instance will have a completely separate table of routes.

Thanks for that. I've removed the separate instance and configured both peers to use the same instance.

The bgp session still fails to establish with the same log entries however. If I move the bgp addresses from the loopback bridge to a physical interface then the session establishes correctly. Best practice is to use loopback for bgp tough, so I'd much rather find a solution.

As soon as I move it back to the loopback bridge it fails again but provides a little more info - saying that the peer is not directly connected and multihop is not enabled. So obviously I enabled multihop only to find the exact same opensent state and Connection terminated log entry.

Any other ideas?

Fri Dec 18, 2015 5:22 pm

The bgp session still fails to establish with the same log entries however. If I move the bgp addresses from the loopback bridge to a physical interface then the session establishes correctly. Best practice is to use loopback for bgp tough, so I'd much rather find a solution.

The loopback address is DEFINITELY what you want - interface addresses can cause issues in iBGP.
You need to set the update-source to be the IP of the loopback interface on the iBGP peers.

(and of course make sure you're using the main instance's AS number)

Sorry - I meant to mention that in the previous reply, but I distracted myself with the instance thing.

WISPa · Fri Dec 18, 2015 5:33 pm

The bgp session still fails to establish with the same log entries however. If I move the bgp addresses from the loopback bridge to a physical interface then the session establishes correctly. Best practice is to use loopback for bgp tough, so I'd much rather find a solution.
The loopback address is DEFINITELY what you want - interface addresses can cause issues in iBGP.
You need to set the update-source to be the IP of the loopback interface on the iBGP peers.

(and of course make sure you're using the main instance's AS number)

Sorry - I meant to mention that in the previous reply, but I distracted myself with the instance thing.

Awesome - update source was indeed the thing I was missing. And yes, I'm using the main public AS number on public eBGP

This is my first flirt with iBGP over OSPF and I'm loving it so far. Thanks for the heads up on update source. Much appreciated.

Fri Dec 18, 2015 6:04 pm

This is my first flirt with iBGP over OSPF and I'm loving it so far. Thanks for the heads up on update source. Much appreciated.

No problem. That's one of those details that just becomes a part of the mental checklist for setting up iBGP.

You should use your public AS in the iBGP session as well (I don't know if you really are because all of your examples were sanitized).

WISPa · Fri Dec 18, 2015 6:07 pm

This is my first flirt with iBGP over OSPF and I'm loving it so far. Thanks for the heads up on update source. Much appreciated.
No problem. That's one of those details that just becomes a part of the mental checklist for setting up iBGP.

You should use your public AS in the iBGP session as well (I don't know if you really are because all of your examples were sanitized).

Yes I am in between the 2 edge devices. Then on the 4 x internal devices I run a private AS which then peer with the public AS.

Thanks again

dudleyrees · Mon Jul 23, 2018 7:15 pm

"You need to set the update-source to be the IP of the loopback interface on the iBGP peers."
THANK YOU!! Still great advice three years on..

alex_rhys-hurn · Mon Aug 13, 2018 9:50 pm

"You need to set the update-source to be the IP of the loopback interface on the iBGP peers."

Yup. Its 4 years, on and the advice is as good as ever!

This one saved me.

Karma

vincent123 · Mon Aug 13, 2018 9:56 pm

Saved the day.

We were able to complete the ISP design

https://au.int/sites/default/files/docu ... design.pdf

Karma

eBGP and iBGP config with OSPF for internal [SOLVED]

eBGP and iBGP config with OSPF for internal

Re: eBGP and iBGP config with OSPF for internal

Re: eBGP and iBGP config with OSPF for internal

Re: eBGP and iBGP config with OSPF for internal [SOLVED]

[SOLVED] Re: eBGP and iBGP config with OSPF for internal

Re: [SOLVED] Re: eBGP and iBGP config with OSPF for internal

Re: [SOLVED] Re: eBGP and iBGP config with OSPF for internal

Re: eBGP and iBGP config with OSPF for internal

Re: eBGP and iBGP config with OSPF for internal

Re: eBGP and iBGP config with OSPF for internal

Who is online