I have setup a bgp monitoring tool which sends me alerts by email.
I constantly receive alerts which I cant make any sense of.
The BGP we have in place was configured by our former net admin.

We have a /20 allotted to us by Arin however we have opted to use 8 /24's at each of our edge routers(2 edge routers)
We have what we call the north network and south network.
we'll say our /20 is 1.1.1.1/20

We have 2 upstream providers both are ATT but setup in a way to give us redundancy should one of them fail.
At each of of our edge routers we are peered with upstream on wan interface
On inside interfaces we have a /29 private range setup with both our edge routers using this range to communicate ibgp.

the edge router for north end of network has 1.1.1.1/24, 2.2.2.2/24, 3.3.3.3/24, 4.4.4.4/24, 5.5.5.5/24, 6.6.6.6/24, 7.7.7.7/24, 8.8.8.8/24 6.6.6.6/27 on the inside interface. I know the /27 is redundant but this is how he set it up to give a business customer a /27 for themselves.
We also have 172.16.16.3/29 on inside interface which is remote peer address for the south


the edge router for the south has 9.9.9.9/24, 10.10.10.10/24, 11.11.11.11/24, 12.12.12.12/24, 13.13.13.13/24, 14.14.14.14/24, 15.15.15.15/24, 16.16.16.16/24
We have 172.16.16.2/29 on inside interface which is remote peer address for mikrotik on north network.

Our configuration doesn't seem to cause customer problems, however last week when our upstream provider on the south network went down instead of the south traffic going over our ibgp link to the north network upstream provider
it did nothing.
I have copied and pasted our configurations into notepad but will paste those in a reply post to avoid this post becoming a page long.
We also have some private ip ranges on both edge routers inside interfaces for our management subnets.

I read about something called route flapping and I have a feeling this may be going on or issues with our routing filters, which only have one entry apiece.
Please calling all BGP experts to advise me how to find my former net admin's configuration mistakes.

Below is configs for both edge routers

North Network Edge Router North BGP instances

0 ;;; IBGP
name="default" as=33317 router-id=172.16.16.3 redistribute-connected=yes
redistribute-static=yes redistribute-rip=no redistribute-ospf=no
redistribute-other-bgp=yes out-filter="" client-to-client-reflection=yes
ignore-as-path-len=no

1 X ;;; EBGP <<<<Disabled
name="ATT" as=33317 router-id=123.123.123.10 redistribute-connected=yes
redistribute-static=yes redistribute-rip=no redistribute-ospf=no
redistribute-other-bgp=no out-filter="" client-to-client-reflection=yes
ignore-as-path-len=no

2 ;;; EBGP
name="ATT-NEW" as=33317 router-id=123.123.123.10
redistribute-connected=yes redistribute-static=yes redistribute-rip=no
redistribute-ospf=no redistribute-other-bgp=no out-filter=""
client-to-client-reflection=yes ignore-as-path-len=no

LZ North BGP Peers

0 name="Jorie" instance=default remote-address=172.16.16.2 remote-as=33317
tcp-md5-key="" nexthop-choice=default multihop=no route-reflect=no
hold-time=3m ttl=255 in-filter="" out-filter="" address-families=ip
update-source=(unknown)

1 X name="ATT" instance=ATT remote-address=123.123.123.9 remote-as=7018 <<<Disabled
tcp-md5-key="" nexthop-choice=default multihop=no route-reflect=no
hold-time=1m ttl=255 in-filter="" out-filter="" address-families=ip
update-source=(unknown)

2 name="ATT" instance=ATT-NEW remote-address=123.123.123.9 remote-as=7018
tcp-md5-key="" nexthop-choice=default multihop=no route-reflect=no
hold-time=1m ttl=255 in-filter="" out-filter="" address-families=ip

LZ North BGP Networks

# NETWORK SYNCHRONIZE
0 A 1.1.1.0/24 yes
1 A 2.2.2.0/24 yes
2 A 3.3.3.0/24 yes
3 A 4.4.4.0/24 yes
4 A 5.5.5.0/24 yes
5 A 6.6.6.0/24 yes
6 A 7.7.7.0/24 yes
7 A 8.8.8.0/24 yes
8 X 9.9.9.0/24 no
9 X 10.10.10.0/24 no
10 X 11.11.11.0/24 no
11 X 12.12.12.0/24 no
12 X 13.13.13.0/24 no
13 X 14.14.14.0/24 no
14 X 15.15.15.0/24 no
15 X 16.16.16.0/24 no

LZ routes

0 X S ;;; Default OUT LZ
0.0.0.0/0 xxx.xxx.xxx.77 (upstream providers inside interface) 1
1 ADb 0.0.0.0/0 reachable xxx.xxx.xxx.77 20 eth1_WAN
2 Db 0.0.0.0/0 reachable 172.16.16.2 200 eth2_LAN
3 ADC 192.0.0.0/8 192.46.0.25 0 eth2_LAN
4 Db 192.0.0.0/8 reachable 172.16.16.2 200 eth2_LAN
5 ADb xxx.xxx.xxx.56/30 (south network gateway) reachable 172.16.16.2 200 eth2_LAN
6 ADC xxx.xxx.xxx.76/30 xxx.xxx.xxx.78 0 eth1_WAN
7 ADC 1.1.1.0/24 1.1.1.1 0 eth2_LAN
8 ADC 2.2.2.0/24 2.2.2.1 0 eth2_LAN
9 ADC 3.3.3.0/24 3.3.3.1 0 eth2_LAN
10 ADC 4.4.4.0/24 4.4.4.1 0 eth2_LAN
11 ADC 5.5.5.0/24 5.5.5.1 0 eth2_LAN
12 ADC 5.5.5.32/27 5.5.5.33 0 eth2_LAN
13 ADC 6.6.6.0/24 6.6.6.1 0 eth2_LAN
14 ADC 7.7.7.0/24 7.7.7.1 0 eth2_LAN
15 ADC 8.8.8.0/24 8.8.8.1 0 eth2_LAN
16 X S ;;; Failover route
9.9.9.0/21 xxx.xxx.xxx.77 1
17 X S ;;; force traffic out then back in
9.9.9.0/24 xxx.xxx.xxx.77 1
18 ADb 9.9.9.0/24 reachable 172.16.16.2 200 eth2_LAN
19 X S ;;; force traffic destined for south out and back in
10.10.10.0/24 xxx.xxx.xxx.77 1
20 ADb 10.10.10.0/24 reachable 172.16.16.2 200 eth2_LAN
21 ADb 10.10.10.52/32 reachable 172.16.16.2 200 eth2_LAN
22 X S ;;; force traffic destined for south out then back in
11.11.11.0/24 xxx.xxx.xxx.77 1
23 ADb 11.11.11.0/24 reachable 172.16.16.2 200 eth2_LAN
24 X S ;;; force traffic destined for south out and back in
12.12.12.0/24 xxx.xxx.xxx.77 1
25 ADb 12.12.12.0/24 reachable 172.16.16.2 200 eth2_LAN
26 X S ;;; force traffic destined for south out and back in
13.13.13.0/24 xxx.xxx.xxx.77 1
27 ADb 13.13.13.0/24 reachable 172.16.16.2 200 eth2_LAN
28 ADb 14.14.14.0/24 reachable 172.16.16.2 200 eth2_LAN
29 X S ;;; force traffic destined for south out and back in
14.14.14.0/24 xxx.xxx.xxx.77 1
30 X S ;;; force traffic destined for south out and back in
15.15.15.0/24 xxx.xxx.xxx.77 1
31 ADb 15.15.15.0/24 reachable 172.16.16.2 200 eth2_LAN
32 X S ;;; force traffic destined for south out and back in
16.16.16.0/24 xxx.xxx.xxx.77 1
33 ADb 16.16.16.0/24 reachable 172.16.16.2 200 eth2_LAN
34 ADC 172.16.16.0/29 172.16.16.3 0 eth2_LAN
35 Db 172.16.16.0/29 reachable 172.16.16.2 200 eth2_LAN
36 ADb 172.16.17.0/24 reachable 172.16.16.2 200 eth2_LAN

South Network Edge Router

Jorie BGP instances

0 ;;; EBGP
name="default" as=33317 router-id=xxx.xxx.xxx.58 redistribute-connected=yes redistribute-static=yes
redistribute-rip=no redistribute-ospf=no redistribute-other-bgp=no out-filter=""
client-to-client-reflection=yes ignore-as-path-len=no

1 ;;; IBGP
name="foresite" as=33317 router-id=172.16.16.2 redistribute-connected=yes redistribute-static=yes
redistribute-rip=no redistribute-ospf=no redistribute-other-bgp=yes out-filter=""
client-to-client-reflection=yes ignore-as-path-len=no
Jorie BGP Peers

0 name="ATT" instance=default remote-address=xxx.xxx.xxx.57 remote-as=7018 tcp-md5-key=""
nexthop-choice=default multihop=no route-reflect=no hold-time=1m ttl=255 in-filter="" out-filter=""
address-families=ip update-source=(unknown)

1 name="LZ" instance=foresite remote-address=172.16.16.3 remote-as=33317 tcp-md5-key=""
nexthop-choice=default multihop=no route-reflect=no hold-time=3m ttl=255 in-filter="" out-filter=""
address-families=ip update-source=(unknown)
Jorie BGP Networks

0 A 9.9.9.0/24 yes
1 A 17.17.17.0/24 yes
2 A 11.11.11.0/24 yes
3 A 12.12.12.0/24 yes
4 A 13.13.13.0/24 yes
5 A 14.14.14.0/24 yes
6 A 15.15.15.0/24 yes
7 A 16.16.16.0/24 yes
8 X 1.1.1.0/24 yes
9 X 2.2.2.0/24 yes
10 X 3.3.3.0/24 yes
11 X 4.4.4.0/24 yes
12 X 5.5.5.0/24 yes
13 X 6.6.6.0/24 yes
14 X 7.7.7.0/24 yes
15 X 8.8.8.0/24 yes

Jorie Routes

0 X S ;;; test
0.0.0.0/0 xxx.xxx.xxx.57 1
1 ADb 0.0.0.0/0 reachable xxx.xxx.xxx.57 20 ether1_WAN
2 Db 0.0.0.0/0 reachable 172.16.16.3 200 ether2_LAN
3 ADC 192.0.0.0/8 192.0.0.25 0 ether2_LAN <<<<management subnet


4 Db 192.0.0.0/8 reachable 172.16.16.3 200 ether2_LAN
5 ADC xxx.xxx.xxx.56/30 xxx.xxx.xxx.58 0 ether1_WAN
6 ADb xxx.xxx.xxx.76/30 reachable 172.16.16.3 200 ether2_LAN
7 ADb 1.1.1.0/24 reachable 172.16.16.3 200 ether2_LAN
8 X S ;;; Failover Route
1.1.1.0/24 xxx.xxx.xxx.57 20
9 ADb 2.2.2.0/24 reachable 172.16.16.3 200 ether2_LAN
10 ADb 3.3.3.0/24 reachable 172.16.16.3 200 ether2_LAN
11 ADb 4.4.4.0/24 reachable 172.16.16.3 200 ether2_LAN
12 ADb 5.5.5.0/24 reachable 172.16.16.3 200 ether2_LAN
13 ADb 5.5.5.32/27 reachable 172.16.16.3 200 ether2_LAN
14 ADb 6.6.6.0/24 reachable 172.16.16.3 200 ether2_LAN
15 ADb 7.7.7.0/24 reachable 172.16.16.3 200 ether2_LAN
16 ADb 8.8.8.0/24 reachable 172.16.16.3 200 ether2_LAN
17 ADC 9.9.9.0/24 9.9.9.1 0 ether2_LAN
18 ADC 10.10.10.0/24 10.10.10.1 0 ether2_LAN
19 A S ;;; static route
10.10.10.52/32 reachable 10.10.10.58 1 ether2_LAN
20 ADC 11.11.11.0/24 11.11.11.1 0 ether2_LAN
21 ADC 12.12.12.0/24 12.12.12.1 0 ether2_LAN
22 ADC 13.13.13.0/24 13.13.13.1 0 ether2_LAN
23 ADC 14.14.14.0/24 14.14.14.1 0 ether2_LAN
24 ADC 15.15.15.0/24 15.15.15.1 0 ether2_LAN
25 ADC 16.16.16.0/24 16.16.16.1 0 ether2_LAN
26 ADC 172.16.16.0/29 172.16.16.2 0 ether2_LAN
27 Db 172.16.16.0/29 reachable 172.16.16.3 200 ether2_LAN
28 ADC 172.16.17.0/24 172.16.17.2

Hi,

I'm curious, why are you running separate BGP instances for your iBGP and eBGP peers? You can terminate both on the same BGP instance.

What exactly did the traffic do when the South router's eBGP peer dropped? I see you're sending default routes between the two routers, so your traffic should have re-routed properly when that eBGP connection dropped.

This was all configured by our former net admin, I'm not sure as to why he did it this way, maybe only way he could figure out.
I personally don't understand how we could run the ibgp and ebgp on same instance given it appears you can only have one remote id, but again I can be completely wrong here This is my first exposure really to bgp so I'm trying to learn from his config which apparently isn't right;;

Peoples traffic was still to go through the down upstream gateway last week.

I can help you understand some BGP fundamentals if you want. Feel free to give me a call at the number below.

I can help you understand some BGP fundamentals if you want. Feel free to give me a call at the number below.

"you pay only for phone call", eh ?

skype/icq would be much more handy...

p.s. many BGP instances (i.e. one per peer) is a common mistake. might this be influenced by some strange vendors hardware that usually gets configured exactly that way (one instace=one peer)?