IPv6 recursive nexthops via iBGP

liteforce · Sat Jun 05, 2010 9:38 am

Hi,

We recently added a new RouterOS device to our network and as we use iBGP and eBGP, this was the first instance where we have had to use multihop iBGP (all devices were directly connected to each other at this point; this new device is only reachable via routed IPv4/IPv6).

All devices are running RouterOS v4.10.

The problem that I'm seeing is that IPv4 multihop iBGP works absolutely fine as long as the 'target-scope' for a route is set in the incoming filter for that session; routes appear in the routing table as I would expect them to:

'a.a.a.a recursive via b.b.b.b ether1'

Before I figured out that 'target-scope' was the issue, I was seeing a.a.a.a unreachable' instead - nevertheless, IPv4 multihop iBGP is working fine now so I am a happy bunny.

Unfortunately, IPv6 multihop iBGP has got me well and truly stumped.

The same tweak for 'target-scope' on the incoming filter for the v6-only iBGP session does not have the same effect; the filter is definitely working as I am seeing the changes in the 'Attributes' tab in Winbox for all of the routes learnt via that session.

Is anyone out there in RouterOS land willing to share any tips/suggestions/example config that might help me solve this ?

I am happy to post config details if this will help.

Thanks in advance!

Regards,
Terry Froy
Spilsby Internet Solutions

changeip · Sun Jun 06, 2010 11:34 pm

try setting the next hop or gateway using routing filters to override whats coming in.

Mon Jun 07, 2010 8:24 am

please print one of the IPv6 unresolved routes.

liteforce · Mon Jun 07, 2010 9:36 am

try setting the next hop or gateway using routing filters to override whats coming in.

That worked - once I had sorted out some quick VRRP to ensure that the nexthop wasn't a specific routers' IPv6 address; no point in doing BGP if it can't failover to an alternative route!

please print one of the IPv6 unresolved routes.

Reverting back to previous config; I get the following:

[admin@tcw-gw1.man.spilsby.net.uk] > /ipv6 route print where dst-address is 2a01:568:3000::/64
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, o - ospf, b - bgp, U - unreachable 
 #      DST-ADDRESS              GATEWAY                  DISTANCE
[admin@tcw-gw1.man.spilsby.net.uk] > /ipv6 route print where dst-address is "2a01:568:3000::/64"
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, o - ospf, b - bgp, U - unreachable 
 #      DST-ADDRESS              GATEWAY                  DISTANCE
[admin@tcw-gw1.man.spilsby.net.uk] >

Going by the above, the route doesn't actually appear via the console but it does appear within Winbox (fields and values copied below):

Dst. Address: 2a01:568:3000::/64
Gateway: 2a01:568:fff:f003:20c:42ff:fe43:739c (unreachable)
Check Gateway: (empty)
Type: unicast

Distance: 200
Scope: 40
Target Scope: 255 (this is being set via incoming route filter like its' IPv4 cousin)

Received From: hex-ipv6-int-gw1

BGP Local Pref.: 100
BGP Origin: EGP (now this confuses me; both routers and their single BGP instances each specify the same AS number - 43950 - so why this is being considered as EGP, I don't know - this should be igp, right ?)

Both routers can ping each other and I can confirm there are currently no traffic filters between them; these only get put in place after issues like this are fixed otherwise you end up diagnosing problems that may not actually exist, etc, etc.

Enabling 'Check Gateway' with method 'ping' doesn't seem to provide any other useful information either

Although, just to assure you that I'm not going mad:

[admin@tcw-gw1.man.spilsby.net.uk] > /ping 2a01:568:fff:f003:20c:42ff:fe43:739c
2a01:568:fff:f003:20c:42ff:fe43:739c 64 byte ping: ttl=63 time=8 ms
2a01:568:fff:f003:20c:42ff:fe43:739c 64 byte ping: ttl=63 time=8 ms
2a01:568:fff:f003:20c:42ff:fe43:739c 64 byte ping: ttl=63 time=8 ms
2a01:568:fff:f003:20c:42ff:fe43:739c 64 byte ping: ttl=63 time=8 ms
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 8/8.0/8 ms
[admin@tcw-gw1.man.spilsby.net.uk] > /tool traceroute 2a01:568:fff:f003:20c:42ff:fe43:739c
     ADDRESS                                    STATUS
   1 2a01:568:fff:f001:20c:42ff:fe20:7ee4 1ms 1ms 1ms 
   2 2a01:568:fff:f003:20c:42ff:fe43:739c 9ms 9ms 9ms

... and peer config:

[admin@tcw-gw1.man.spilsby.net.uk] > /routing bgp peer export
<snip countless other IPv4/IPv6 peers>
add address-families=ipv6 as-override=no comment="" default-originate=never \
    disabled=no hold-time=3m in-filter=spilsby-tcw-ipv6-multihop-ibgp-in \
    instance=default interface=ether1 multihop=yes name=hex-ipv6-int-gw1 \
    nexthop-choice=force-self out-filter="" passive=no remote-address=\
    2a01:568:fff:f003:20c:42ff:fe43:739c remote-as=43950 remove-private-as=no \
    route-reflect=no tcp-md5-key="" ttl=default use-bfd=no

Config on the remote router is identical (except remote peer address of course!) and a full iBGP mesh is configured between all routers on this AS (all of which are RouterOS 4.10 devices - either RB1000 or RB1100).

Regards,
Terry Froy
Spilsby Internet Solutions

liteforce · Mon Jun 07, 2010 12:59 pm

Once I had sorted out some quick VRRP to ensure that the nexthop wasn't a specific routers' IPv6 address; no point in doing BGP if it can't failover to an alternative route!

Replying to my own post...

VRRP wasn't actually necessary once I realized that a combination of multiple IPv6 nexthops could be set and 'Check Gateway' --> 'ping' would handle any failover automatically.

Quite impressed that it actually worked

Regards,
Terry Froy
Spilsby Internet Solutions[/quote]

timoid · Thu Jan 20, 2011 2:53 pm

Anyone manage to solve this?

Looks like theres no communication between OSPFv3 and BGP so BGP isnt seeing the interior route?

[admin@hostname] /ipv6 route> print detail where distance=200
Flags: X - disabled, A - active, D - dynamic,
C - connect, S - static, r - rip, o - ospf, b - bgp, U - unreachable
0 Db dst-address=2000:2000::/32 gateway=2403:9000::1:0:0:1000:3
gateway-status=2403:9000::1:0:0:1000:3 unreachable distance=200
scope=40 target-scope=30 bgp-as-path="AS_PATH" bgp-local-pref=1000
bgp-med=0 bgp-origin=igp received-from=IPv6-ACCESS01

1 Db dst-address=2000:2000::/32 gateway=2403:9000::1:0:0:1000:3
gateway-status=2403:9000::1:0:0:1000:3 unreachable distance=200
scope=40 target-scope=30 bgp-as-path="AS_PATH" bgp-local-pref=1000
bgp-med=0 bgp-origin=igp received-from=IPv6-ACCESS02

21 ADo dst-address=2403:9000::1:0:0:1000:0/124 gateway=fe80::222:90ff:fe0c:25f1%vlan0621
gateway-status=fe80::222:90ff:fe0c:25f1%vlan0621 reachable distance=110 scope=20 target-scope=10
ospf-metric=10

Wed Jan 26, 2011 2:48 pm

This is a known bug.
Recursive lookup is not working if gateway is link local address. To make BGP routes work you need static route with global address as gateway.

brointhemix · Wed Oct 05, 2011 11:22 pm

Do you plan to fix it or introduce an official workaround for it in an upcoming release?

sten · Thu Nov 03, 2011 11:24 am

You can't announce anything with link local addresses in them, they are after all, link local.
Use non link-locals for routing (even for gateway).

brointhemix · Wed Nov 23, 2011 4:20 pm

Hello sten,

Can you provide me with an example as to how to do that? Try as I might, I cannot force my MTs to accept a global unique next hop over a link-local one...

marcof · Thu Jul 05, 2012 3:40 pm

Hi,

Is there any update on this? I just faced this bug, in 5.18! This is really basic routing stuff, working on every other system I know. What's the problem that RouterOS cannot handle this? OSPFv3 nexthops are always link-local so there is no workaround at all I guess.

It's best practice to use loopback addresses for iBGP, routing the loopback addresses with OSPFv3.

when will this bug be fixed? This is actually a total no-go for MikroTik as BGP Routers (at least if you have more than 1 router).

thanks
marco

igwana · Sat Jul 21, 2012 9:37 pm

Hi,

I have the same problem (running 5.17)

OSPFv3 routes have link-local nexthops (as per RFC) and are not used to recursively resolve nexthops of iBGP routes!

Route filters (using set-in-nexthop-ipv6 in ospf-in chain) are a workaround only if you have no more than one OSPF neighbor because the filter can't determine from which neighbor the route came from and set the nexthop accordingly.

Can we hope this is fixed soon?

The iBGP route with the unresolved nexhop:

35  Db  dst-address=2001:db8::/32 gateway=fc00::2 
        gateway-status=fc00::2 unreachable distance=200 scope=40 
        target-scope=30 bgp-as-path="65000" bgp-local-pref=100 bgp-origin=igp 
        received-from=ibgp_router2

The route from OSPFv3 that should be used to resolve the nexthop but is not:

32 ADo  dst-address=fc00::2/128 gateway=fe80::a61:702%sit4-vr1 
        gateway-status=fe80::a61:702%sit4-vr1 reachable distance=110 scope=20 
        target-scope=10 ospf-metric=10

marcof · Tue Jun 04, 2013 11:04 am

Hi,

Months later, routeros 6.0 is released and this problem is still not solved. Did anyone find a good workaround besides adding static routes? static routes is not really a workaround because static routes simply do not work in a redundant network layout!

I would really appriciate to know when this problem is fixed. It's actually a total no-go for BGP dual-stacked backbones with MikroTIK.

thanks
Marco

paoloaga · Tue Jun 11, 2013 1:16 pm

I also am having big troubles with RouterOS and iBGP with IPv6 due to this issue. Please Mikrotik, consider fixing it!

marcof · Wed Jun 12, 2013 9:27 am

I also am having big troubles with RouterOS and iBGP with IPv6 due to this issue. Please Mikrotik, consider fixing it!

You may also write an e-Mail to MT Support about this issue.

It's really sad that such basic stuff does not work. MikroTik needs to change thinking about IPv6, it's nothing new, it's there, years ago and every ISP has or is impelmenting it, now. MikroTik can only be a full alternative to cisco, etc. in the ISP branch when it has at least the simple and basic routing availible and working (and yes, ipv6 is simple).
Please fix it.

thanks

paoloaga · Sat Jun 15, 2013 12:44 pm

I also am having big troubles with RouterOS and iBGP with IPv6 due to this issue. Please Mikrotik, consider fixing it!
You may also write an e-Mail to MT Support about this issue.

I already mailed them a couple of months ago, when I replaced some cisco backbone infrastructure with CCR routers (~ 25 CCR). They answered that it will be fixed in the future.

Actually I am having really big troubles. I started to deploy IPv6 to customers in 2008, and now I have a network with all backbone links and router loopbacks distributed with OSPF and customers routes into BGP. The problem is that it's impossible to use IBGP with IPv6. I can't even redistribute BGP routes into OSPFv3 because OSPFv3 lacks of route-filtering. Actually I had to put a huge number of static routes (!!!!!!) in order to make the iBGP route reachable, and this sucks because static routes are indeed STATIC and it's a nightmare to do maintenance on it.

Probably Mikrotik is underevaluating the importance of this issue. I just hope that they will fix it soon.

zimage · Tue Sep 24, 2013 6:05 am

The problem still exists in RouterOS 6.3. Upgrading to 6.4 later in the week.

zimage · Sat Oct 05, 2013 12:15 am

Still occurs with 6.5rc1. I emailed support@ and was told that I simply need to tweak my scope and target-scope values. Not sure what I'm supposed to do in that regard.

Mon Oct 07, 2013 10:48 am

Still occurs with 6.5rc1. I emailed support@ and was told that I simply need to tweak my scope and target-scope values. Not sure what I'm supposed to do in that regard.

You do not have problem discussed in this topic. Recursive lookup doesn't work only when nexthop is link-local. You had global nexthops.

zimage · Sun Oct 13, 2013 7:42 am

I might be interpreting things wrong, but it looks like the next hops are link local.

 0  Db  dst-address=2001:470:1f10:9a7::/64 gateway=2001:470:c334::b
        gateway-status=2001:470:c334::b unreachable distance=200 scope=40 target-scope=30
        bgp-local-pref=100 bgp-origin=igp received-from=fhl-r3

 1 ADC  dst-address=2001:470:c334::9/128 gateway=loopback gateway-status=loopback reachable
        distance=0 scope=10

 2 ADo  dst-address=2001:470:c334::a/128 gateway=fe80::20c:42ff:feff:5c7d%ether5
        gateway-status=fe80::20c:42ff:feff:5c7d%ether5 reachable distance=110 scope=20
        target-scope=10 ospf-metric=20

 3 ADo  dst-address=2001:470:c334::b/128 gateway=fe80::d6ca:6dff:fe35:d345%ether1
        gateway-status=fe80::d6ca:6dff:fe35:d345%ether1 reachable distance=110 scope=20
        target-scope=10 ospf-metric=20

 4 ADo  dst-address=2001:470:c334::13/128 gateway=fe80::20c:42ff:feff:5c7d%ether5
        gateway-status=fe80::20c:42ff:feff:5c7d%ether5 reachable distance=110 scope=20
        target-scope=10 ospf-metric=30

 5 ADC  dst-address=2001:470:c334:1::/64 gateway=ether1 gateway-status=ether1 reachable distance=0
        scope=10

 6 ADC  dst-address=2001:470:c334:4::/64 gateway=ether5 gateway-status=ether5 reachable distance=0
        scope=10

 7 ADo  dst-address=2001:470:c334:6::/64 gateway=fe80::20c:42ff:feff:5c7d%ether5
        gateway-status=fe80::20c:42ff:feff:5c7d%ether5 reachable distance=110 scope=20
        target-scope=10 ospf-metric=20

 8 ADo  dst-address=2001:470:c334:7::/64 gateway=fe80::20c:42ff:feff:5c7d%ether5
        gateway-status=fe80::20c:42ff:feff:5c7d%ether5 reachable distance=110 scope=20
        target-scope=10 ospf-metric=20

 9 ADo  dst-address=2001:470:c334:b::/64 gateway=fe80::d6ca:6dff:fe73:25ce%ether2
        gateway-status=fe80::d6ca:6dff:fe73:25ce%ether2 reachable distance=110 scope=20
        target-scope=10 ospf-metric=20

10 ADC  dst-address=2001:470:c334:c::/64 gateway=ether2 gateway-status=ether2 reachable distance=0
        scope=10

line 0 is received via iBGP. It's gateway is set to the router's loopback address, 2001:470:c334::b. Line 3 then contains the routing entry for how to get to the router loopback. The gateway is fe80::d6ca:6dff:fe35:d345%ether1, a link-local address. These two routing entries seem a lot like those in igwana's post

xxiii · Wed Nov 06, 2013 10:48 pm

I'd just like to add my plea that this get fixed, soon.

I just recently emailed support about this, being unaware of this forum thread, and got the (apparently common) "known issue, will be fixed in the future" response.

I need to know if this is going to be near future, or far future, as this is a show-stopper for us, and the age of this thread has me rather worried.

zimage · Tue Nov 12, 2013 12:43 am

Same here. We'd like to have full IPv6 support in 2014. We might have to abandon our mikrotik gear and buy something else.

Sun Jan 19, 2014 11:58 pm

Same here. We'd like to have full IPv6 support in 2014. We might have to abandon our mikrotik gear and buy something else.

Any idea if this bug has been resolved ?

brotherdust · Sun Jan 26, 2014 7:55 am

I was working on a lab this morning attempting to implement this very scenario and ran into the same problem. I would like to see the fix for this issue implemented as this thread is now over three years old.

The workaround of using incoming filters to manually set a link-local hop is not very scalable at all and breaks a number of built-in redundancy features should otherwise be available in this type of setup.

Please, can we get a firm commit from MK staff to resolve this?

Thanks!

xxiii · Fri Mar 21, 2014 6:10 pm

This is becoming an increasingly serious problem.

asenci · Tue Apr 15, 2014 6:00 pm

This is becoming an increasingly serious problem.

And here we are... Deploying IPv6 using static routes... Who would have thought?!

AndrewTFry · Thu Apr 24, 2014 1:40 am

I'm having the exact same issues. What I don't understand is, why if the router has a valid route to the Loopback of its neighbor can it not use that route to figure out a path to the neighbor for a BGP route to that Loopback? This really needs to be fixed. For all intents and purposes the router knows how to reach the loopback of its neighbor. You can prove that by pinging. Why can't it see that the BGP route is reachable by that address? Fix this quickly please. As others have mentioned OSPFv3 shouldn't even be a feature if it's not going to play nice with BGP.

paoloaga · Fri Apr 25, 2014 7:39 pm

I'm running out of IPv4 addresses.

Hope this will get fixed soon.

SwissWISP · Tue May 13, 2014 12:43 pm

I'm running out of IPv4 addresses. Hope this will get fixed soon.

+1

Tue May 13, 2014 1:58 pm

+1

ETA please Mikrotik

Sent from my SM-G900F using Tapatalk

Tue May 13, 2014 2:34 pm

This feature most likely will be implemented in version 7.

Tue May 13, 2014 2:52 pm

This feature most likely will be implemented in version 7.

Good to hear Maris.

xxiii · Wed May 14, 2014 9:37 pm

Will the (related) OSPFv3 /128/LA-Bit bug also be fixed?

Thu May 15, 2014 10:09 am

Will the (related) OSPFv3 /128/LA-Bit bug also be fixed?

Will be fixed also in v7.

Thu May 15, 2014 11:14 am

Will the (related) OSPFv3 /128/LA-Bit bug also be fixed?
Will be fixed also in v7.

Obvious question..

When can we expect v7 ?

StubArea51 · Sun May 18, 2014 10:22 pm

Great question!

rickrude · Thu May 22, 2014 4:38 am

Can someone explain the LA-bit bug briefly? Is it related to ospfv3 not importing LSAs with the LA-bit set?? i.e. /128 loopbacks from other vendors?

If so, what are you doing in the meantime? /127s or redistributing connected?

cheers!

tomaskir · Thu May 22, 2014 10:59 am

And here we are... Deploying IPv6 using static routes... Who would have thought?!

thanks for the laught

xxiii · Sat May 24, 2014 2:10 am

Can someone explain the LA-bit bug briefly? Is it related to ospfv3 not importing LSAs with the LA-bit set?? i.e. /128 loopbacks from other vendors?

If so, what are you doing in the meantime? /127s or redistributing connected?

Yes. sadly, static routes, sometimes multiple ones with check-gateway with differing metrics to hopefully capture the most likely failure scenarios. But alas, when there is a hiccup somewhere, most of our routeros devices in the affected part of the topology just drop off the network (on IPv6) until the situation is resolved (alternatively someone could go manually update the static routes).

If you have ciscos you can force the loopback link type to point-to-point and get them to advertise /64's (assuming you used /64's). We have another vendor though that that workaround isn't available, and they insist (as per the RFC requirements) on advertising their loopbacks as /128's with the LA bit set. I've thought about seeing if I could get them to also advertise the companion /64 (perhaps with a redistribute or policy), but I'm hoping this will be fixed before I have to resort to that (or making sure there are no RouterOS devices in the relevant pathways when we get IPv6 connectivity requests).

You could also possibly see if you could strip off the LA-bit on the originating devices.

lyma · Mon Jul 14, 2014 8:59 pm

Will the (related) OSPFv3 /128/LA-Bit bug also be fixed?
Will be fixed also in v7.
Obvious question.. When can we expect v7 ?

Great question... ping timeout!

Tue Jul 15, 2014 6:52 pm

When its ready

Wed Jul 16, 2014 3:20 am

When its ready

haha classic Mikrotik

paoloaga · Wed Dec 03, 2014 1:30 am

Is there any news about a fix for V6.x or about the new V7? My network is screaming in pain!

cegner · Wed Dec 03, 2014 4:40 pm

Is there any workaround besides configuring static routes?

lyma · Wed Dec 24, 2014 12:55 am

Is there any news about a fix for V6.x or about the new V7? My network is screaming in pain!

Try another solution. Cisco, Extreme, Juniper, Datacom... No ETA here.

(and no more IPv4 address too)

paoloaga · Mon Aug 03, 2015 5:10 pm

We are seriously short of IPv4 addresses, and IPv6 support on RouterOS is still marginal.

Take a look at the global situation:
http://www.potaroo.net/tools/ipv4/

We need to start making plans to migrate everything we can on IPv6, and we need to have an idea on when IPv6 will start to be seriously taken in consideration into RouterOS.

If the answer is "wait for v7", then please let us know an estimate pessimistic timeframe (a date far enough in the future, that we can expect will be met even if development goes wrong and takes more time) when Mikrotik plans to release it.

MT, I know that it might sound bothering, but try to understand our (ISP) problems.

paoloaga · Thu Nov 05, 2015 8:17 pm

StubArea51 · Thu Nov 05, 2015 9:12 pm

Would like to see a timeline on this as well since our need to design and deploy IPv6 networks has gone up in the last 6 months.

paoloaga · Thu Mar 03, 2016 11:50 am

I am having a lot of troubles with IPv6 routing because of this.

With 2 upstream (full-view) and ~ 60 peers at the N.A.P., I am not able to optimize IPv6 traffic. IPv6 is relevant today and I am delaying the deployment because of this issue.

Am I the only one that needs recursive nexthop resolution so much?

I hope this is not considered a "necro-post", just a reminder.

Thu Mar 03, 2016 12:36 pm

@paoloaga

You are not the only one!

This and the OSPFv3 /128 LSA bug are preventing us from rolling IPV6 out across our wider network. We have been waiting for these to be fixed for years now...

StubArea51 · Thu Mar 03, 2016 3:50 pm

@paoloaga and @nz_monkey

Same boat here.....we've gotten by with peering on the subnet instead of on a /128 loopback until it's fixed. We have a ton of customer IPv6 migrations we are working on that need this since the IPv4 topology is recursion based with BGP and OSPF.

RouterOS v7 #thelongwait

Thu Mar 03, 2016 10:44 pm

RouterOS v7 #thelongwait

I keep expecting to see this every time there's a new MUM....

There's so much that's been promised in ROSv7.

StubArea51 · Thu Mar 03, 2016 10:49 pm

RouterOS v7 #thelongwait
I keep expecting to see this every time there's a new MUM....

There's so much that's been promised in ROSv7.

Maybe we'll get lucky and it will be Dallas in a month and a half

Sat Mar 05, 2016 12:20 am

It is no trivial task developing a new routing engine, adding major new features and testing all the corner cases, all the while maintaining the existing code base and back porting drivers so you can release new hardware.

I'm sure Mikrotik are just as desperate as we are to get v7 out, so they can focus on one code base and a newer kernel.

Dallas does seem to be the likely place we will see v7 announced [WHITE SMILING FACE]

paoloaga · Sun Mar 06, 2016 10:38 am

It is no trivial task developing a new routing engine, adding major new features and testing all the corner cases, all the while maintaining the existing code base and back porting drivers so you can release new hardware.

What we are asking is not an exotic feature, it's basic IPv6 functionality...

Sun Mar 06, 2016 4:51 pm

It is no trivial task developing a new routing engine, adding major new features and testing all the corner cases, all the while maintaining the existing code base and back porting drivers so you can release new hardware.
What we are asking is not an exotic feature, it's basic IPv6 functionality...

Monkey's statement was directed at the comments on here hoping that ROSv7 should come out soon, and how long it's been said by Mikrotik that "such-and-such a feature will be in next version of ROS" What he was referring to is the fact that they're apparently re-writing the routing engine for ROSv7....

I've found several "misbehaviors" of ROS in both OSPF and BGP that are about this same level - they don't stop the system from being a viable platform for routing, but a few common "best practice" designs are broken due to these oddities. (The one which irritates me most is how a backup OSPF default-gw router won't drop its announcement if the original default-gw is also an OSPF route.)

Thu Mar 10, 2016 2:20 pm

v7 Teaser alert, recursive ipv6 works

[admin@tested] > /routing/route/print 
Flags: D - dynamic, X - disabled, I - inactive, A - active, 
C - connect, S - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, Y - synthetic, T - terminal, 
> - best, ' - candidate, U - updated, F - filtered 
        DST-ADDRESS        GATEWAY            DISTANCE SCOPE TARGET-SCOPE IMMEDIATE-GW                          
 AS >'   0.0.0.0/0          10.5.130.1                1    30           10 10.5.130.1%ether1                     
DAC >'   10.5.130.0/24      ether1                    0    10              ether1                                
 AS >'   ;;; router-test
        2001:2001::/64     fe80::29:dfff:f...        1    15           10 fe80::29:dfff:fe2b:14d0%ether2        
 AS >'   ;;; router-test
        2001:2002::/64     2001:2001::1              1    30           15 fe80::29:dfff:fe2b:14d0%ether2

hedele · Thu Mar 10, 2016 2:27 pm

v7 Teaser alert, recursive ipv6 works
Code: Select all
[admin@tested] > /routing/route/print 

ooops.. looks like CLI will change a lot?

Chupaka · Thu Mar 10, 2016 3:57 pm

C - connect, S - static, r - rip, b - bgp, o - ospf

and the very first v7 bug report

a typo: should be 'connected', I think

patrick7 · Thu Mar 10, 2016 5:53 pm

@mrz More informations about v7 please

Even it would be a fully broken beta, I'd be happy to see the news

Fri Mar 11, 2016 12:34 am

v7 Teaser alert, recursive ipv6 works

[admin@tested] > /routing/route/print 
Flags: D - dynamic, X - disabled, I - inactive, A - active, 
C - connect, S - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, Y - synthetic, T - terminal, 
> - best, ' - candidate, U - updated, F - filtered 
        DST-ADDRESS        GATEWAY            DISTANCE SCOPE TARGET-SCOPE IMMEDIATE-GW                          
 AS >'   0.0.0.0/0          10.5.130.1                1    30           10 10.5.130.1%ether1                     
DAC >'   10.5.130.0/24      ether1                    0    10              ether1                                
 AS >'   ;;; router-test
        2001:2001::/64     fe80::29:dfff:f...        1    15           10 fe80::29:dfff:fe2b:14d0%ether2        
 AS >'   ;;; router-test
        2001:2002::/64     2001:2001::1              1    30           15 fe80::29:dfff:fe2b:14d0%ether2

What I see, I like

Particularly the "> - best, ' - candidate, U - updated, F - filtered " feature. This will save us a lot of troubleshooting time.

paoloaga · Fri Mar 11, 2016 5:49 pm

Wonderful news. Hope we can play with it very soon.

Now there is another question that bugs my mind: if the CLI syntax is new, I will have to refactor all the automated software (and it is complex!): Is it possible to access to an early documentation or have an early version to install on a lab box so we can start working on the management software and have it ready for when v7 will be released???

Fri Mar 11, 2016 5:57 pm

Wonderful news. Hope we can play with it very soon.

Now there is another question that bugs my mind: if the CLI syntax is new, I will have to refactor all the automated software (and it is complex!): Is it possible to access to an early documentation or have an early version to install on a lab box so we can start working on the management software and have it ready for when v7 will be released???

I noticed that too - if the command syntax requires slashes now, that's a huge change that will require everyone to modify their scripts and so forth. Hopefully they're going to go through a deprecation period where older syntax is still supported but gives deprecation warnings if you use it in an interactive shell, or something like that. I don't personally use scripts much, but many do. I wonder how that's going to unfold.

Chupaka · Fri Mar 11, 2016 10:02 pm

I can't see any reason to use slashes instead of spaces, moreover slash has different positions on different keyboards — I don't believe they will change current command style...

Sat Mar 12, 2016 2:17 am

Mikrotik have used this different CLI syntax in previous early test releases for new systems.

I don't think what is shown above will be in the final release.

StubArea51 · Tue Mar 15, 2016 3:22 pm

Mikrotik have used this different CLI syntax in previous early test releases for new systems.

I don't think what is shown above will be in the final release.

Well that's good because my brain can't digest yet another CLI syntax without imploding. On a daily basis, I'm already juggling Cisco [IOS, NX-OS, ASA and Wireless Controller], Juniper, HP, Dell, ZyXEL, AdTran, IBM, Lenovo, etc...as well as MikroTik.

#nonewcli

Fri Mar 18, 2016 1:58 am

Old menu without slashes will be available, so scripts will not break.

Fri Mar 18, 2016 7:30 am

Old menu without slashes will be available, so scripts will not break.

Thanks for the confirmation Maris.

Chupaka · Fri Mar 18, 2016 11:21 am

Just available? So, new style will stay here in release version? %)

hknet · Sun Aug 07, 2016 6:30 pm

after reading this thread I'm still trying to make sense, so I conclude:

OSPFv3 and Loopback-bridge-interfaces with /128 IPv6 addresses assigned in RouterOS will only be shown reachable if one sets an admin-mac to the bridge (named eg Loopback0).
well, that's not 100% intuitive, but I guess that's something I can live with.

to receive those Looback IPs (/128) via OSPFv3 one has to remove the LA bit somehow (simple workaround suggested by MikroTik is to simply import those Loopback IPs as external routes), well this is also not perfectly straight forward, but one can work with this.

but on the iBGP case I read something about setting static routes on the RouterOS side to make the connection to Loopback IPs of other iBGP speakers...
well, tried that, but I'm with no luck in getting any routes received from a routereflector installed as active.

currently tested with RouterOS v6.36

any advice greatly appreciated

savage · Thu May 25, 2017 12:59 am

7 years later, and still not fixed

Thank you MT. You pretty much put the final nail in the coffin as far as using Mikrotik goes. I (like many others), can not continue to wait 'indefinitely' until the mythical v7 finally appears.

savage · Thu May 25, 2017 1:01 am

OSPFv3 and Loopback-bridge-interfaces with /128 IPv6 addresses assigned in RouterOS will only be shown reachable if one sets an admin-mac to the bridge (named eg Loopback0).
well, that's not 100% intuitive, but I guess that's something I can live with.

What you perhaps don't know, and can't live with... If Loopback0 (in your example) is in a VRF for IPv4, IPv6 won't work completely. The connected route never becomes active, and OSPF never injects the route - ANOTHER MT bug...

guilhermeramires · Tue Aug 29, 2017 4:18 pm

Use ULAs instead of LL. No need to use filters or static this way.

Tue Aug 29, 2017 4:55 pm

And how exactly does one cause OSPFv3 to use ULAs as the next hop address instead of LL addresses?
If it takes work-arounds to use ULAs, you may as well use the global-unique (public) addresses instead of having a third layer on there.

hknet · Tue Aug 29, 2017 5:39 pm

Hi
saw this thread reactivated

The only workaround we have seen so far for iBGP IPv6 routes to get active is to add a static ipv6 route for the loopback IP for the next-hop delivered through the route-reflectors.
(again tested with RouterOS 6.40.2)

If there is any workaround, I'd be glad to hear about it.

Dear MikroTik it would be nice to know how static routes work in this case and how eg. ospfv3 routes fail or maybe there is the possibility for a script run every 30 seconds or so to get the /128 routes (probably loopbacks) out of the ospfv3 table and into static routes and cleaning up those routes that were lost since the last run. I'm not savvy enough in terms of MikroTik scripting to come up with a script that does that.

thx
hk

JimmyNyholm · Sun Sep 03, 2017 1:31 pm

Hi. RR's need not to be in data path (most often aren't) so please consider your own setup before fiddeling with above statement.

@Mikrotik please fix IPV6 it is 2017 after all and Ipv4 is getting more and more expensive.

hknet · Sun Sep 03, 2017 3:03 pm

Hi. RR's need not to be in data path (most often aren't) so please consider your own setup before fiddeling with above statement.

ahem, the nexthop delivered by RRs was not implying the nexthop in fact is the RR, in fact the nexthop is usually the IP set by "next-hop self" (or similar) by BGP-routers talking to the RRs, but as long as those next-hop IPs are delivered via ospf (which is pretty common imho) routes are not getting active, therefore static routes to those IPs set by BGP-routers as "next hop" are the only workaround I'm aware of, but I'd be happy to hear any other solution.

Regards
hk

JimmyNyholm · Sun Sep 03, 2017 8:39 pm

Hi. RR's need not to be in data path (most often aren't) so please consider your own setup before fiddeling with above statement.
ahem, the nexthop delivered by RRs was not implying the nexthop in fact is the RR, in fact the nexthop is usually the IP set by "next-hop self" (or similar) by BGP-routers talking to the RRs, but as long as those next-hop IPs are delivered via ospf (which is pretty common imho) routes are not getting active, therefore static routes to those IPs set by BGP-routers as "next hop" are the only workaround I'm aware of, but I'd be happy to hear any other solution.

Regards
hk

"Nexthop self" in an bgp environment is an abomination. I agree that it has some good corner cases but a good setup IGP for distributing nexthops (ie: ospf with loopbacks) and a non nexthop modifying ebgp and ibgp. Depending on number of routers you have fullmesh ibgp is fast responsive but explodes in management if you are constantly growing. This is where clustered RR's is a must to keep peering sessions manageable. It's in this space I just claimed that a RR is most of the time not in the packets Datapath and it need/should not to be it should only RR your network.

Having Mikrotiks they can not be pure reflecting as they only reflect active routes. This is a limitation on the MT but then again the good IGP takes care of this even here. The request for being true RR mode has been asked here before and we are told to expect that in the unicorn version 7.
I think that we speak of the same thing but may from different perspective.

As for IPV6 my statement is even more clear from my perspective. MT Please fix IPV6.
Yours J

hknet · Sun Sep 03, 2017 9:17 pm

Well

"Nexthop self" in an bgp environment is an abomination.

It should not be needed - agreed, but real world teaches us:
One looses an external interface and therefore the nexthop is removed from the IGP.
In a small network this converges fast and causes virtually no service disruption.
Think about lots of BGP-peers and lots (fulltable times N) of routes all handled with policies and communities and so on.
IGP (eg. ospf) still converges within seconds and removes the route to the previously valid next-hop.
Now all BGP-routes that learned these routes have to reevaluate their situation on where to go next. If you loose say 500k routes from this single event this can take a moment, packets are still forwarded to the now unreachable next-hop and get lost.
Here a forward to a still existing loopback-IP is a much less disruptive solution: packets are still forwarded to the loopback, but the router owning it knows its interface is down and can decide where to go next with those packets as it has probably already marked all routes for removal that were formerly learned from this external peer. You may see some funny packetflows, but usually don't loose as much packets.

Cheers,
hk

wsftech · Thu Sep 07, 2017 9:40 am

Hi
The only workaround we have seen so far for iBGP IPv6 routes to get active is to add a static ipv6 route for the loopback IP for the next-hop delivered through the route-reflectors.
(again tested with RouterOS 6.40.2)
hk

Let me recap, and please correct me if I'm wrong.

1. Under no circumstances does ROS6 recursively resolve BGP IPv6 routes using link-local addresses; or OSPF-populated loopback IPs using either link-local, ULA, or static ('public') IPv6 addresses
2. ROS can resolve IPv6 routes that recurse to *static* routes with appropriate scope/target scope values.
3. It is impossible to use a modern network design of OSPF-populated loopbacks and multihop BGP peering for IPv6 routes.
4. We are stuck in the dark ages of static addressing.

I've tested this on 6.38 and 6.39, and I can only assume from this thread that <6.41 doesn't fix this either.

I assume that one could simulate an OSPF-like failover behavior by using multiple static routes, ping gateway, and distance, but yuck!

Is the only alternative for, say, IGP, to use OSPF only?

Thu Sep 07, 2017 5:43 pm

Pretty much correct.

For those following along at home, here is a summary of the issue:
OSPFv3 installs the link-local address of its neighbor as the next-hop address when it places routes into the routing table.
eBGP will use the peer's address as the next-hop. Note that this is directly-attached eBGP peering (which is the standard methodology)
iBGP preserves the next hop address of the prefixes shared.

example:
Two EBGP neighbors link up over a /126 connection (2001:db8:ffff::0/124 with the external peer being ::1
The local EBGP router learns 2001:db8:cafe::/48 via 2001:db8:ffff::1 and installs this into the local routing table. Everything is a-okay so far.
There is some arbitrary topology within the local ASN.
Somewhere inside the ASN is an iBGP peer of the EBGP border router.

The BGP strategy is to use the IGP (OSPF in this case) to determine the best next hop for reaching the egress point chosen by BGP.
So in the RIB (live routing table), the next hop will be the OSPF-determined best next hop to reach the peering address.
Since OSPFv3 uses link-local, this is where the problem comes from.

example:
So the iBGP peer has some set of paths to reach the eBGP router above.
The eBGP router will also be advertising its peering link into the local IGP (ospf) so that other routers will know how to reach this peering point.
So 2001:db8:ffff::/0 will be in the OSPF database.
the iBGP router will have a route with dst=2001:db8:ffff::/0 gateway=fe80::abcd -- where fe80::abcd%ether1 is the link-local IPv6 address of the OSPF neighbor through which the best path leads.
The iBGP router learns prefix 2001:db8:cafe::/48 from the eBGP router. The next hop for this prefix will be 2001:db8:ffff::1
The iBGP router must now perform a recursive lookup on 2001:db8:ffff::1 in order to set the next-hop for 2001:db8:cafe::/48

In ROS, this is where the problem happens because the next hop is going to be fe80::abcd%ether1. The iBGP recursive lookup bug prevents this route from being installed as an active route in the routing table. The work-around is to use a static route to reach 2001:db8:ffff::/124 where the next hop address is not a link-local address.

e.g. if the router with link-local address fe80::abcd%eth1 is also using 2001:db8:beef::1 as a global public address on the link to iBGP router, then on iBGP router would need a static route as follows:
dst=2001:db8:ffff::/124 gateway=2001:db8:beef::1

This is obviously quite problematic.
Apparently the root cause is something deep enough in the routing engine code that Mikrotik has stated this will only be fixed in RouterOS v7

In a nutshell, it is impossible to deploy a standard OSPF+iBGP ASN for IPv6 using Mikrotik. If your network does not use iBGP, then you can use IPv6 in your core w/o worrying about this issue.

You may be able to make a work-around with GRE tunnels directly between all bgp-speaking routers and some next-hop-self tweaks, but this seems very undesirable to me as well.

wsftech · Fri Sep 08, 2017 2:58 am

ZeroByte, thanks for the detailed comments.

After further testing, I have reached the same conclusions you have. I would rather not tunnel — although I did briefly contemplate a full-mesh MPLS/VPLS for directly peering each IPv6 router. (Traffic queue management would be just one problem with that design.)

I settled on sticking with OSPFv3, as that at least works, and can use link-local, so not actually very painful to implement. In addition I found out today that my upstream provider (a publicly traded company) also uses OSPFv3 as IGP, and they're a Juniper shop… go figure.

For eBGP, no problem is encountered, as next-hops are rewritten anyway upon further route advertisement.

<sigh>

aplvmn · Sun Sep 24, 2017 7:53 pm

I just started deploying IPv6 and found this issue, going over this post made me feel disappointed, I think I have spent money for nothing on this boxes, I have found so many bugs with other technologies in MK (Radius, PPP, and an endless list), I guess that is why is so cheap compared to Cisco or Juniper.

SplitHorizon · Thu Oct 05, 2017 10:46 am

Just marking the register on this issue.. it has caused me enough grief..

patrick7 · Thu Oct 05, 2017 12:28 pm

I think there will be no fix in the near future.

zimage · Thu Oct 05, 2017 6:05 pm

I gave up on mikrotik when we moved to a dual stack network because of this bug. You can find new Juniper SRX routers pretty cheaply if you look hard. Don’t pay more than 25% of the list cost, though.

savage · Thu Oct 05, 2017 7:09 pm

I gave up on mikrotik when we moved to a dual stack network because of this bug. You can find new Juniper SRX routers pretty cheaply if you look hard. Don’t pay more than 25% of the list cost, though.

I'm in the same boat. Can't use MT in my core / borders. MT is definitely not aware of the actual seriousness of this issue, especially on larger networks.

There's actually PLENTY of IPv6 broken, especially on the PPP side (filters, address-lists, IP assignments, the list goes on). BGP not working, is a HUGE issue.

JimmyNyholm · Fri Aug 24, 2018 8:46 pm

Passing Into Late 2018 And still this is big issue when @Mikrotik WHEN will recursive routing work in routeros. Installed V6 routes that have reachables nexthops (recursivly that is) will never be active due to something broken. FIX NOW. IPV4 days are over and we must deploy ipv6.

StubArea51 · Wed Aug 29, 2018 6:25 pm

We could really use an update on this MikroTik.

We are seeing IPv6 adoption move at a much faster pace in 2018 and are having to modify the routing architecture or use other brand routers for our clients to solve this problem.

This has been an issue for a long time but we could really use a fix

patrick7 · Wed Aug 29, 2018 6:55 pm

@IPANetEngineer If it would be important for them, they would have fixed this issue years ago.
Just proceed with FRRouting

It's better anyways.

StubArea51 · Wed Aug 29, 2018 7:48 pm

@IPANetEngineer If it would be important for them, they would have fixed this issue years ago.
Just proceed with FRRouting It's better anyways.

Depends on your use case. I like FRR and talk to a number of the developers at FRR on a regular basis. However, it's still software that's go to go on bare metal or a VM and there are a number of operational challenges with that if you're doing something outside of BGP Full table or RR.

Would really just like to see this fixed.

wsftech · Wed Aug 29, 2018 7:52 pm

It's not possible to use Mikrotik in a real production-quality dual-stack service provider network without working BGP.

I'd hate to see the CCR series, for example, be left out in the cold, if we have to wait for an entirely new generation of hardware to get ROS7. Not to spread FUD, but I don't have much to go on.

sep · Thu Sep 20, 2018 11:25 pm

is there any updates on this incredibly critical part of the puzzle ? is there no other option that just avoiding using mikrotik for routing ?
there is nothing we deployed without ipv6 nowadays. But plenty that we deploy as ipv6 only. and CCR would be a excellent fit is so many cases.

but this is a brick wall show stopper for mikrotik.

kind regards
Ronny Aasen

bbs2web · Sat Sep 22, 2018 11:07 pm

Getting decent IPv4 performance without packet loss on CCR routers has required us to use raw firewall rules to stop connection tracking on forwarded traffic and building a distributed core network where P routers talk only IPv4 OSPF and PE routers exchange routes via BGP route reflectors so that they reference the remote router's loopback IP. This way traffic ingressing at a PE router will be switched to the destination router via MPLS.

IPv6 OSPFv3 is reported as being the culprit but without IPv6 MPLS support we would be back at P routers having to route each packet. Setting up a BGP signaled VPLS using split horizon bridging allows one to set the bridge protocol mode to none, whilst avoiding bridging loops. We then assigned IPv6 /128 loopback IPs and assigned the same IP with a /64 subnet to the VPLS bridge interfaces.

Is my thinking in this regard wrong or is the IPv6 routing problem on RouterOS actually a recursive routing issue at heart?

Topology:

Summary:

IPv4 OSPF with MPLS using loopback IPs, where only the default gateway is redistributed on R1

RR1 has two IPv4 BGP sessions to R1 and R5. RR1 reflects routes and both R1 and R5 set the nexthop as their loopback interface IPs

BGP peering sessions are running with IPv4, IPv6 and L2VPN address families

Everything establishes without errors and I can ping from the test PC connected to R5 to 8.8.8.8 using IPv4 (MPLS switched through R4, R3 and R2)

I can ping IPv6 gateways and routes appear to install correctly but I can not ping remote IPv6 addresses

R1:

R5:

Configuration:
R1:

/interface bridge
add name=bridge-ipv6-mpls protocol-mode=none
add name=lo
/interface ethernet
set [ find default-name=ether2 ] mtu=2026
/routing bgp instance
set default redistribute-connected=yes redistribute-static=yes router-id=10.0.0.1
/routing ospf instance
set [ find default=yes ] distribute-default=if-installed-as-type-1 router-id=10.0.0.1
/interface vpls bgp-vpls
add bridge=bridge-ipv6-mpls bridge-horizon=1 export-route-targets=1:1 import-route-targets=1:1 name=bgp-vpls1 route-distinguisher=1:1
/ip address
add address=10.0.0.1 interface=lo
add address=172.16.254.5/30 interface=ether2
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=ether1
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
/ipv6 address
add address=2001:db8::1/128 advertise=no interface=lo
add address=2001:db8:1000::1/48 advertise=no interface=ether1
add address=2001:db8::1 advertise=no interface=bridge-ipv6-mpls
/mpls interface
set [ find default=yes ] mpls-mtu=1992
/mpls ldp
set distribute-for-default-route=yes enabled=yes lsr-id=10.0.0.1 transport-address=10.0.0.1
/mpls ldp interface
add hello-interval=1s hold-time=10s interface=ether2
/mpls local-bindings
add label=impl-null
/routing bgp peer
add address-families=ip,ipv6,l2vpn in-filter=bgp-in multihop=yes name=RR1 nexthop-choice=force-self remote-address=10.0.0.6 remote-as=65530 ttl=default update-source=lo
/routing ospf interface
add authentication=md5 authentication-key=PHi02OnUPN1gzENt dead-interval=10s hello-interval=1s interface=ether2 network-type=point-to-point
/routing ospf network
add area=backbone network=10.0.0.1/32
add area=backbone network=172.16.254.4/30
/system identity
set name=R1

R2:

/interface bridge
add name=lo
/interface ethernet
set [ find default-name=ether1 ] mtu=2026
set [ find default-name=ether2 ] mtu=2026
/routing ospf instance
set [ find default=yes ] router-id=10.0.0.2
/ip address
add address=10.0.0.2 interface=lo
add address=172.16.254.9/30 interface=ether1
add address=172.16.254.6/30 interface=ether2
/ip dns
set servers=1.1.1.1,1.0.0.1
/mpls interface
set [ find default=yes ] mpls-mtu=1992
/mpls ldp
set distribute-for-default-route=yes enabled=yes lsr-id=10.0.0.2 transport-address=10.0.0.2
/mpls ldp interface
add hello-interval=1s hold-time=10s interface=ether1
add hello-interval=1s hold-time=10s interface=ether2
/routing ospf interface
add authentication=md5 authentication-key=PHi02OnUPN1gzENt dead-interval=10s hello-interval=1s interface=ether2 network-type=point-to-point
add authentication=md5 authentication-key=3r5MTWTfAD4dDMeH dead-interval=10s hello-interval=1s interface=ether1 network-type=point-to-point
/routing ospf network
add area=backbone network=10.0.0.2/32
add area=backbone network=172.16.254.4/30
add area=backbone network=172.16.254.8/30
/system identity
set name=R2

R3:

/interface bridge
add name=lo
/interface ethernet
set [ find default-name=ether1 ] mtu=2026
set [ find default-name=ether2 ] mtu=2026
set [ find default-name=ether3 ] mtu=2026
/routing ospf instance
set [ find default=yes ] router-id=10.0.0.3
/ip address
add address=10.0.0.3 interface=lo
add address=172.16.254.13/30 interface=ether2
add address=172.16.254.10/30 interface=ether1
add address=172.16.254.21/30 interface=ether3
/ip dns
set servers=1.1.1.1,1.0.0.1
/mpls interface
set [ find default=yes ] mpls-mtu=1992
/mpls ldp
set distribute-for-default-route=yes enabled=yes lsr-id=10.0.0.3 transport-address=10.0.0.3
/mpls ldp interface
add hello-interval=1s hold-time=10s interface=ether1
add hello-interval=1s hold-time=10s interface=ether2
add hello-interval=1s hold-time=10s interface=ether3
/routing ospf interface
add authentication=md5 authentication-key=3r5MTWTfAD4dDMeH dead-interval=10s hello-interval=1s interface=ether1 network-type=point-to-point
add authentication=md5 authentication-key=ZxOnGXB2ZLsCr0dJ dead-interval=10s hello-interval=1s interface=ether2 network-type=point-to-point
add authentication=md5 authentication-key=5bnwqgfAomwbU9B3 dead-interval=10s hello-interval=1s interface=ether3 network-type=point-to-point
/routing ospf network
add area=backbone network=10.0.0.3/32
add area=backbone network=172.16.254.12/30
add area=backbone network=172.16.254.8/30
add area=backbone network=172.16.254.20/30
/system identity
set name=R3

R4:

/interface bridge
add name=lo
/interface ethernet
set [ find default-name=ether1 ] mtu=2026
set [ find default-name=ether2 ] mtu=2026
/routing ospf instance
set [ find default=yes ] router-id=10.0.0.4
/ip address
add address=10.0.0.4 interface=lo
add address=172.16.254.17/30 interface=ether2
add address=172.16.254.14/30 interface=ether1
/ip dns
set servers=1.1.1.1,1.0.0.1
/mpls interface
set [ find default=yes ] mpls-mtu=1992
/mpls ldp
set distribute-for-default-route=yes enabled=yes lsr-id=10.0.0.4 transport-address=10.0.0.4
/mpls ldp interface
add hello-interval=1s hold-time=10s interface=ether1
add hello-interval=1s hold-time=10s interface=ether2
/routing ospf interface
add authentication=md5 authentication-key=ZxOnGXB2ZLsCr0dJ dead-interval=10s hello-interval=1s interface=ether1 network-type=point-to-point
add authentication=md5 authentication-key=A54RUkZukh3lDRBb dead-interval=10s hello-interval=1s interface=ether2 network-type=point-to-point
/routing ospf network
add area=backbone network=10.0.0.4/32
add area=backbone network=172.16.254.16/30
add area=backbone network=172.16.254.12/30
/system identity
set name=R4

R5:

/interface bridge
add name=bridge-ipv6-mpls protocol-mode=none
add name=lo
/interface ethernet
set [ find default-name=ether2 ] mtu=2026
/ip pool
add name=DHCP ranges=192.168.248.50-192.168.248.199
/ip dhcp-server
add address-pool=DHCP disabled=no interface=ether1 lease-time=12h name=ether1
/routing bgp instance
set default redistribute-connected=yes redistribute-static=yes router-id=10.0.0.5
/routing ospf instance
set [ find default=yes ] router-id=10.0.0.5
/interface vpls bgp-vpls
add bridge=bridge-ipv6-mpls bridge-horizon=1 export-route-targets=1:1 import-route-targets=1:1 name=bgp-vpls1 route-distinguisher=1:1 site-id=5
/ip address
add address=10.0.0.5 interface=lo
add address=172.16.254.18/30 interface=ether2
add address=192.168.248.1/24 interface=ether1
/ip dhcp-server network
add address=192.168.248.0/24 gateway=192.168.248.1
/ip dns
set servers=1.1.1.1,1.0.0.1
/ipv6 address
add address=2001:db8::5/128 advertise=no interface=lo
add address=2001:db8:5000::1/48 advertise=no interface=ether1
add address=2001:db8::5 advertise=no interface=bridge-ipv6-mpls
/mpls interface
set [ find default=yes ] mpls-mtu=1992
/mpls ldp
set distribute-for-default-route=yes enabled=yes lsr-id=10.0.0.5 transport-address=10.0.0.5
/mpls ldp interface
add hello-interval=1s hold-time=10s interface=ether2
/routing bgp peer
add address-families=ip,ipv6,l2vpn in-filter=bgp-in multihop=yes name=RR1 nexthop-choice=force-self remote-address=10.0.0.6 remote-as=65530 ttl=default update-source=lo
/routing ospf interface
add authentication=md5 authentication-key=A54RUkZukh3lDRBb dead-interval=10s hello-interval=1s interface=ether2 network-type=point-to-point
/routing ospf network
add area=backbone network=10.0.0.5/32
add area=backbone network=172.16.254.16/30
/system identity
set name=R5

RR1:

/interface bridge
add name=bridge-ipv6-mpls protocol-mode=none
add name=lo
/interface ethernet
set [ find default-name=ether1 ] mtu=2026
/routing bgp instance
set default redistribute-connected=yes redistribute-static=yes router-id=10.0.0.6
/routing ospf instance
set [ find default=yes ] router-id=10.0.0.6
/interface vpls bgp-vpls
add bridge=bridge-ipv6-mpls bridge-horizon=1 export-route-targets=1:1 import-route-targets=1:1 name=bgp-vpls1 route-distinguisher=1:1 site-id=6
/ip address
add address=10.0.0.6 interface=lo
add address=172.16.254.22/30 interface=ether1
/ip dns
set servers=1.1.1.1,1.0.0.1
/ipv6 address
add address=2001:db8::6/128 advertise=no interface=lo
add address=2001:db8::6 advertise=no interface=bridge-ipv6-mpls
/mpls interface
set [ find default=yes ] mpls-mtu=1992
/mpls ldp
set distribute-for-default-route=yes enabled=yes lsr-id=10.0.0.6 transport-address=10.0.0.6
/mpls ldp interface
add hello-interval=1s hold-time=10s interface=ether1
/routing bgp peer
add address-families=ip,ipv6,l2vpn in-filter=bgp-in multihop=yes name=R1 remote-address=10.0.0.1 remote-as=65530 route-reflect=yes ttl=default
add address-families=ip,ipv6,l2vpn in-filter=bgp-in multihop=yes name=R5 remote-address=10.0.0.5 remote-as=65530 route-reflect=yes ttl=default
/routing ospf interface
add authentication=md5 authentication-key=5bnwqgfAomwbU9B3 dead-interval=10s hello-interval=1s interface=ether1 network-type=point-to-point
/routing ospf network
add area=backbone network=172.16.254.20/30
add area=backbone network=10.0.0.6/32
/system identity
set name=RR1

mducharme · Sun Sep 23, 2018 12:07 am

We then assigned IPv6 /128 loopback IPs and assigned the same IP with a /64 subnet to the VPLS bridge interfaces.

I'm guessing that might be the cause of your issue. Use different IPs with a /64 subnet for the VPLS bridge interfaces than you are using for the loopback. I expect the ping above would then work.

bbs2web · Sun Sep 23, 2018 6:59 am

The point is to get IPv6 ingressing at a PE switched across P routers using MPLS. You also missed the fact that I can ping R5's IPv6 loopback from R1 and vice versa, so the gateways are reachable.

We then assigned IPv6 /128 loopback IPs and assigned the same IP with a /64 subnet to the VPLS bridge interfaces.
I'm guessing that might be the cause of your issue. Use different IPs with a /64 subnet for the VPLS bridge interfaces than you are using for the loopback.

mducharme · Sun Sep 23, 2018 7:06 am

The point is to get IPv6 ingressing at a PE switched across P routers using MPLS. You also missed the fact that I can ping R5's IPv6 loopback from R1 and vice versa, so the gateways are reachable.

I did not miss this, I can see quite well that the gateways are reachable. We offer IPv6 over VPLS tunnels and it works quite well. But given the routing tables you have shown, there is no other reason I can see (short of a bug) why the ping would fail. I expect that the duplication of IPs from the loopback to the VPLS bridge interface is not impacting the direct ping, but may be preventing the recursive routing from functioning properly. For instance, your routing tables show that the nexthop for 2001:db8::1 is 2001:db8::1, the nexthop for 2001:db8::5 is 2001:db8::5, and the nexthop for 2001:db8::6 is 2001:db8::6. This would seem to me to create a loop that can not be recovered from, hence not being reachable when it comes to recursive routing, but reachable when it comes to a direct ping response.

bbs2web · Sun Sep 23, 2018 8:28 am

IPv6 appears extremely unreliable in the GNS3 virtual lab I put together. The following initially only worked in one direction (R1 -> R5) until I restarted R5, after which it worked in both.

Added IPv6 prefix filter to the route reflector (RR1):

/routing filter add chain=bgp-in address-family=ipv6 prefix=2001:db8::/64 prefix-length=64-128 action=discard
/routing filter add chain=bgp-out address-family=ipv6 prefix=2001:db8::/64 prefix-length=64-128 action=discard
/routing bgp peer set [ find ] in-filter=bgp-in out-filter=bgp-out

R1:

R5:

mducharme · Sun Sep 23, 2018 8:39 am

IPv6 appears extremely unreliable in the GNS3 virtual lab I put together. The following initially only worked in one direction (R1 -> R5) until I restarted R5, after which it worked in both.

Again, what I see here as a common link is that you are assigning the same IPv6 address to the loopback interface as a /128 as you are to the VPLS interface (/64). I strongly suspect that this is the cause of your issues.

I have found that RouterOS does not always treat interfaces properly with IPv6. For instance, if two OSPFv3 neighbors share the same link local address (even though their interface IDs are different), the OSPFv3 process will get confused and not allow one of the two neighbors to become an OSPFv3 neighbor.

bbs2web · Sun Sep 23, 2018 9:37 am

My intention with this lab exercise was to find a solution to efficiently switch IPv6 packets between provider edge (PE) routers R1 and R5, through an IPv4 MPLS core, using iBGP. RouterOS 6.43.2 can not be used to recursively resolve IPv6 iBGP nexthop using OSPFv3 and running OSPFv3 without IPv6 MPLS in the core is not optimal anyway and would unnecessarily duplicate administrative work. The lab essentially attempts to emulates Cisco 6PE where IPv6 PE routers communicate with each other over an IPv4 MPLS core.

Routing lookups should reference the best match so 2001:db8::5 from R1 matches the 2001:db8::/64 route referencing the BGP signaled VPLS bridge interface and transmit it out there. RouterOS is Linux based so all IPs are present in table 255 without a netmask, which is hard coded as the first lookup rule when processing sent or received packets. One can run 'ip rule list; ip route show table 255' on any Linux system to observe this behaviour (the same reason why RouterOS can not fully isolate VRFs).

The environment does now appear to be working consistently, the problem was due to IPv6 addresses associated with the VPLS bridge interfaces having been set as 'advertise=no'. 2nd amendment to the lab environment:
R1 + R5 + RR1:

/ipv6 address set [ find interface=bridge-ipv6-mpls ] advertise=yes

Edit: This was working randomly, see further comment.

You mention having numerous production networks doing just this, I have no production experience and would appreciate it if you would be so kind as to share your knowledge in this regard.

mducharme · Mon Sep 24, 2018 10:34 am

The environment does now appear to be working consistently, the problem was due to IPv6 addresses associated with the VPLS bridge interfaces having been set as 'advertise=no'. 2nd amendment to the lab environment:
R1 + R5 + RR1:
Code: Select all
/ipv6 address set [ find interface=bridge-ipv6-mpls ] advertise=yes

Strange - advertise=yes shouldn't make a difference, you should not need router advertisements for this to work. They must be somehow working around an underlying issue.

You mention having numerous production networks doing just this, I have no production experience and would appreciate it if you would be so kind as to share your knowledge in this regard.

I didn't quite say that. I said we offer IPv6 over VPLS tunnels, but there is no BGP involved in the way we provision services - we bridge the customer at one end to a VPLS tunnel, and the other end of the tunnel has their gateway IP bound. We do L2VPN, not L3VPN.

bbs2web · Tue Sep 25, 2018 1:44 am

As you state the advertise option is not needed and was most probably only effecting a change by it flapping the IPv6 address when applying the change. Problem resurfaces if the layer 2 VPLS tunnels re-establish and automatically get removed and added to the bridge, thereby changing its MAC address. I assume a deeper bug relating to the link local address not updating properly to track the interface's MAC so I simply set a static MAC address on the 'bridge-ipv6-mpls' interfaces to avoid the quirk.

Recap:

Configure infrastructure as per post 94 (viewtopic.php?f=14&t=42268&start=50#p688104)

Apply filtering on route reflectors as per post 98 (viewtopic.php?f=14&t=42268&start=50#p688155)

Set static MAC addresses on 'bridge-ipv6-mpls', as detailed below

R1:

/interface bridge set [ find name=bridge-ipv6-mpls ] auto-mac=no admin-mac=00:14:F2:00:00:01

R5:

/interface bridge set [ find name=bridge-ipv6-mpls ] auto-mac=no admin-mac=00:14:F2:00:00:05

RR1:

/interface bridge set [ find name=bridge-ipv6-mpls ] auto-mac=no admin-mac=00:14:F2:00:00:06

The environment does now appear to be working consistently, the problem was due to IPv6 addresses associated with the VPLS bridge interfaces having been set as 'advertise=no'. 2nd amendment to the lab environment:
R1 + R5 + RR1:
Code: Select all
/ipv6 address set [ find interface=bridge-ipv6-mpls ] advertise=yes
Strange - advertise=yes shouldn't make a difference, you should not need router advertisements for this to work. They must be somehow working around an underlying issue.

mducharme · Tue Sep 25, 2018 1:52 am

As you state the advertise option is not needed and was most probably only effecting a change by it flapping the IPv6 address when applying the change. Problem resurfaces if the layer 2 VPLS tunnels re-establish and automatically get removed and added to the bridge, thereby changing its MAC address. I assume a deeper bug relating to the link local address not updating properly to track the interface's MAC so I simply set a static MAC address on the 'bridge-ipv6-mpls' interfaces to avoid the quirk.

Yes, I have seen issues like this as well. In cases where a port leaves a bridge with auto-mac and rejoins, the link local shown in IPv6->Addresses doesn't always match the link-local that the router responds on, which leads to all kinds of other issues. Setting an admin MAC is the safest workaround.

MasterJames · Wed Apr 03, 2019 3:29 pm

I think this is still not right in 6.44.1 the date is 2019:04:03 feels like a really bad April fools joke?!

Maybe this problem I am having is unrelated but similar.

viewtopic.php?f=2&t=147216

I think this thread is ten years old!? this can't still be waiting for version 7 can it?

liteforce · Wed Apr 03, 2019 6:02 pm

Original poster here

I am both shocked and amazed that this issue is still present in current day RouterOS v6.44.1 even though I reported it back in the days of RouterOS v4.x.

Regards,
Terry Froy
Spilsby Internet Solutions

hknet · Wed Apr 03, 2019 8:59 pm

#funfact a little over three years ago we were teased about the working solution with ROSv7 and we were shown a printout on how it behaves, but that's about it I guess...

actually I guess that's one main reason we use MTs for layer2-setups and as CPEs, the core is and probably will be junipers and ciscos for the foreseeable future

timoid · Wed Apr 03, 2019 11:25 pm

MT need to rebrand: "Routing the legacy world"

bbs2web · Thu Apr 11, 2019 1:03 am

We've been running IPv6 since September last year without issues. Multiprotocol IPv4 BGP sessions set next hop as the router's loopback IPv4 or IPv6 addresses and IPv6 is MPLS switched between routers, avoiding route lookups at each hop. Core routers don't run BGP, only IPv4 with MPLS so reconvergence is quick.

Recap:

Configure infrastructure as per post 94 (viewtopic.php?f=14&t=42268&start=50#p688104)

Apply filtering on route reflectors as per post 98 (viewtopic.php?f=14&t=42268&start=50#p688155)

Set static MAC addresses on 'bridge-ipv6-mpls', as per post 102 (viewtopic.php?f=14&t=42268&start=50#p688490)

paoloaga · Mon Jul 29, 2019 1:49 pm

Maybe it could help in a few small cases...

viewtopic.php?f=9&t=150616&p=742085#p742085

doof · Tue Aug 17, 2021 10:38 am

This bug still seems to be present in 6.48.3.

Tue Aug 17, 2021 10:40 am

This feature is added in ROS v7, v6 will stay as it is.

Who is online