DHCPv6 client cannot recover after outage

Kentzo · Wed Jul 12, 2023 2:52 am

Recently my ISP did some work and as a result upstream route changed. However, after the connectivity was restored, RouterOS's DHCPv6 client failed to obtain new address and prefix.

> /ipv6/dhcp-client/print detail
Flags: D - dynamic; X - disabled, I - invalid 
 0    interface=ether1-gateway status=searching... duid="..." dhcp-server-v6=fe80::201:5cff:fe6e:1246 
      request=address,prefix add-default-route=no use-peer-dns=no dhcp-options="" pool-name="global" pool-prefix-length=64 
      prefix-hint=::/48 script="" dhcp-options=""

With a packet sniffer I can see that RouterOS tries to solicit a server and that upstream sends an advertisement in reply:

No.    Time         Source     Destination  Protocol  Length  Info
31449  1445.232227  fe80::...  ff02::1:2    DHCPv6    198     Solicit XID: 0x6c6fd5 CID: ... 

DHCPv6
    Message type: Solicit (1)
    Transaction ID: 0x6c6fd5
    Client Identifier
        Option: Client Identifier (1)
        Length: 10
        DUID: ...
        DUID Type: link-layer address (3)
        Hardware type: Ethernet (1)
        Link-layer address: ...
    Identity Association for Non-temporary Address
        Option: Identity Association for Non-temporary Address (3)
        Length: 12
        IAID: 00000001
        T1: 1800
        T2: 2880
    Elapsed time
        Option: Elapsed time (8)
        Length: 2
        Elapsed time: 311640ms
    Rapid Commit
        Option: Rapid Commit (14)
        Length: 0
    Identity Association for Prefix Delegation
        Option: Identity Association for Prefix Delegation (25)
        Length: 41
        IAID: 00000001
        T1: 1800
        T2: 2880
        IA Prefix
            Option: IA Prefix (26)
            Length: 25
            Preferred lifetime: 2880
            Valid lifetime: 3600
            Prefix length: 48
            Prefix address: :: (::)




No.    Time         Source                 Destination  Protocol  Length  Info
31450  1445.260668  2001:558:4000:114::10  fe80::...    DHCPv6    234     Advertise XID: 0x6c6fd5 CID: ... IAA: 2001:558:6045:cc:81fa:a299:4c3e:f1b7 

DHCPv6
    Message type: Advertise (2)
    Transaction ID: 0x6c6fd5
    Client Identifier
        Option: Client Identifier (1)
        Length: 10
        DUID: ...
        DUID Type: link-layer address (3)
        Hardware type: Ethernet (1)
        Link-layer address: ...
    Server Identifier
        Option: Server Identifier (2)
        Length: 14
        DUID: 0001000115cad37f842b2bfc70d5
        DUID Type: link-layer address plus time (1)
        Hardware type: Ethernet (1)
        DUID Time: Aug  2, 2011 08:25:51.000000000 PDT
        Link-layer address: 84:2b:2b:fc:70:d5
    Identity Association for Non-temporary Address
        Option: Identity Association for Non-temporary Address (3)
        Length: 40
        IAID: 00000001
        T1: 1800
        T2: 2880
        IA Address
            Option: IA Address (5)
            Length: 24
            IPv6 address: 2001:558:...
            Preferred lifetime: 3600
            Valid lifetime: 3600
    Identity Association for Prefix Delegation
        Option: Identity Association for Prefix Delegation (25)
        Length: 41
        IAID: 00000001
        T1: 1800
        T2: 2880
        IA Prefix
            Option: IA Prefix (26)
            Length: 25
            Preferred lifetime: 3600
            Valid lifetime: 3600
            Prefix length: 60
            Prefix address: 2601:645:...

However, RouterOS's log shows the solicitation only but not the reply:

16:14:32 dhcp,debug,packet send ether1-gateway -> ff02::1:2%8 
16:14:32 dhcp,debug,packet type: solicit 
16:14:32 dhcp,debug,packet transaction-id: 6c6fd5 
16:14:32 dhcp,debug,packet  -> clientid:   00030001 e48d8cea 7f9c 
16:14:32 dhcp,debug,packet  -> ia_na:  
16:14:32 dhcp,debug,packet    t1: 1800 
16:14:32 dhcp,debug,packet    t2: 2880 
16:14:32 dhcp,debug,packet    id: 0x1 
16:14:32 dhcp,debug,packet  -> elapsed_time: 250 
16:14:32 dhcp,debug,packet  -> rapid_commit: [empty] 
16:14:32 dhcp,debug,packet  -> ia_pd:  
16:14:32 dhcp,debug,packet    t1: 1800 
16:14:32 dhcp,debug,packet    t2: 2880 
16:14:32 dhcp,debug,packet    id: 0x1 
16:14:32 dhcp,debug,packet   -> ia_prefix:  
16:14:32 dhcp,debug,packet     prefix: ::/48 
16:14:32 dhcp,debug,packet     valid time: 3600 
16:14:32 dhcp,debug,packet     pref. time: 2880

Note that DHCPv6 client clings to fe80::201:5cff:fe6e:1246 which was indeed an address of the upstream router previously. Also note that the advertisement is sent using GUA instead of link-local.

Could it be that RouterOS ignores the reply because of a different scope? Is this a valid behavior?

Kentzo · Wed Jul 12, 2023 3:18 am

Hmm, apparently it didn't work because of the following rules from Building Advanced Firewall:

add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/16

bandit1200 · Wed Jul 12, 2023 6:42 pm

It might be a problem similar to one we have been seeing on Xfinity.

They seem to have changed the location of the upstream DHCPv6 PD server to a global address -- if you modify that rule to allow global (remove the local address restriction) it should be able to get the PD.

If you are also using "add default route" in your client, it will probably break that functionality. (the PD will still work).
The consensus is that the MT function to add default route is a kludge, and just uses the location of the server as the gateway -- which works fine if both the DHCPv6 server and the gateway are on the same local host.

One work-around is to disable the "add default route" and add it another way.

This works for me: Turn on "accept router announcements" in IPV6 settings. This will add the default route automatically (give it a few minutes, and look). Unfortunately, the route provided may include a large MTU setting -- apparently Xfinity is proud of being able to handle jumbo packets

. Since your wan port on MT is likely set to MTU = 1500, you will start getting lots of log warnings from radvd that the MTU is too big.

If your MT can handle jumbo packets, you can change the MTU on your wan interface to match, and the warnings will stop.
If your MT cannot handle jumbo packets, you can try the following:

Create a static route to the gateway by copying the automatic one (and leave the MTU setting blank), then turn off "accept router advertisements".

That should work until Xfinity changes something again.

Another option is to just add a route for ::/0 to your WAN interface. When you do this, ND kicks in as packets appear on your WAN and each will "discover" where to send it (the gateway will answer as the path to global addresses.) I think this is a bit of a kludge as each new address will have to discover where to go, and your neighbors list will fill up with global addresses. The only upside, i guess, is that if Xfinity changes the location of the gateway, it will still find it. I prefer the first option.

I'd be interested if there is another way to do this, and if folks think that Xfinity is wrong (by adding the MTU to the route) or if MT is wrong in how it determines the default route in DHCPv6 client, or if MT is wrong in having radvd give MTU warnings. Not sure who to blame

Kentzo · Wed Jul 12, 2023 8:11 pm

Disabling the rule (or rather altering it by removing src address matcher) resolves the issue.

I didn't use the "add-default-route" attribute before, nor it seems to be necessary now.

I started seeing incorrect MTU with the value of 9192 being sent in RAs as well. Which is plain wrong, as MTU advertised there must be applicable to all segments. My cable modem surely cannot ever handle it.

However it doesn't seem to cause RouterOS (7.9.2) to reject RA overall as I didn't need to add static routes to get it working. I do have /ipv6/nd configured to run on the WAN interface (ra-lifetime=none ra-preference=low advertise-mac-address=yes advertise-dns=no managed-address-configuration=no other-configuration=no) and the appropriate dynamic route is added automatically.

As a side note: I created a conversation on Xfinity discussion forum. Please leave a message there confirming that you also observe incorrect MTU.

bandit1200 · Wed Jul 12, 2023 10:07 pm

I'm glad it worked. Seems like we are fighting the same battle.

So I'm curious your configuration -- you say you are running ND on the WAN interface. That means your wan is Advertising itself as a router -- i don't believe that's where your default route is coming from.

I suspect you are getting the default route because you enabled "accept router advertisements" in IPv6 settings. That's about inbound RA. Is that what you have configured? Or has this been off the whole time?

The default route is coming to you inbound from Xfinity's RA. Its Xfinity that is declaring that high MTU. The radvd daemon that is running in the mikrotik is the thing that's complaining about it to the log. (just clarifying)

Am I getting something wrong?

I thought it was best practice NOT to advertise out on the WAN, but only to advertise to your internal segments.

(also, you mentioned your cable modem can't pass jumbo packets, my modem doesn't indicate anywhere in its specs. Why do you say there is no way? Not that it matters much, but i'm curious).

Anyway, learning new stuff all the time.

I'll pop over to that xfinity forum and leave a note. Perhaps we can gang up on them and force a coherent answer.

Kentzo · Wed Jul 12, 2023 11:07 pm

So I'm curious your configuration -- you say you are running ND on the WAN interface. That means your wan is Advertising itself as a router -- i don't believe that's where your default route is coming from.

I thought it was best practice NOT to advertise out on the WAN, but only to advertise to your internal segments.

My router advertising itself on WAN was a concern. But sniffing did not confirm that. Somehow RouterOS doesn't do this. In addition, I set ra-lifetime=0 (ra-lifetime=none, RouterOS replaces 0 for none) which excludes the router from the Default Router List selection on receivers if ever such advertisement takes place. I set ra-preference=low just in case, although I don't think it does anything because the router is not in the Default Router List.

RAs are not the only ICMPv6 messages related to Neighbor Discovery. This is the reason for running ND on WAN. In particular it's important for RouterOS to learn via neighbor solicitation that upstream gateway is a Router as the appropriate flag will be set in the advertisement sent in reply.

I wish RouterOS was more specific and allowed control over whether I want to run RAs in addition to other Neighbor Discovery facilities.

I suspect you are getting the default route because you enabled "accept router advertisements" in IPv6 settings. That's about inbound RA. Is that what you have configured? Or has this been off the whole time?

Correct, I rely on RAs from the upstream for IPv6 configuration. These RAs set flags that require me set up a DHCPv6 client for address and prefix.

The default route is coming to you inbound from Xfinity's RA. Its Xfinity that is declaring that high MTU. The radvd daemon that is running in the mikrotik is the thing that's complaining about it to the log. (just clarifying)

Correct, this is the behavior that I observe.

(also, you mentioned your cable modem can't pass jumbo packets, my modem doesn't indicate anywhere in its specs. Why do you say there is no way? Not that it matters much, but i'm curious).

My Arris SB6141 (of which Comcast is fully aware of) uses DOCSIS 3.0 which supports MTU of up to 1500 per various sources I Googled.

biomesh · Wed Jul 12, 2023 11:40 pm

Here is a simplified and sanitized version of my ipv6 config that I have been using with comcast for close to 10 years (basically when it was added to ROS).

/ipv6 address
add disabled=no from-pool=comcast_ipv6 interface=internal
/ipv6 dhcp-client
add add-default-route=yes interface=wan pool-name=comcast_ipv6 prefix-hint=::/60 request=address,prefix use-peer-dns=no
/ipv6 firewall filter
add action=accept chain=input protocol=icmpv6
add action=accept chain=input connection-state=established,related
add action=accept chain=input dst-port=546 in-interface=wan protocol=udp src-port=547
add action=drop chain=input connection-state=invalid
add action=drop chain=input in-interface=wan
add action=drop chain=forward hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward protocol=icmpv6
add action=accept chain=forward connection-state=established,related
add action=accept chain=forward connection-state=new in-interface=!wan
add action=drop chain=forward connection-state=invalid
add action=drop chain=forward in-interface=wan
/ipv6 nd
set [ find default=yes ] advertise-dns=no disabled=yes
add interface=internal other-configuration=no ra-delay=5s ra-interval=5s-30s 
/ipv6 nd prefix default
set preferred-lifetime=45m valid-lifetime=1h30m

Kentzo · Thu Jul 13, 2023 12:01 am

Close to mine except I rely on RAs instead of DHCPv6 Client default routes.

bandit1200 · Thu Jul 13, 2023 12:59 am

Thanks for the clarification.

I think you can safely turn off the ND entry for the WAN completely -- Its just for the RA stuff as far as I can tell. Neighbor Solicitation still works just fine on that interface with it off. I've seen this in my setup. I suspect if an interface allows IPV6, it must do the neighbor solicitation, so no way to shut it off, i think.

I agree with you, MT naming it ND makes it a bit confusing, sounding like its turning off all the related things when its just about outbound RA. You seem to be a packet sniffing expert, perhaps you can confirm this.

As an example, my second option above, where you don't need a default route at all to the xfinity gateway, just to your wan interface, works just fine without an entry in the ND Interfaces table for that interface. The packets get to the WAN interface and the neighbor solicitation process does its stuff and finds the gateway just fine without a route, fills up the Neighbors table, though. Etc etc. OK, getting off field.

I'll be curious to see what we learn from Xfinity about that big MTU.

@biomesh

Xfinity seems to be rolling out network config changes that might affect you. @kentzo it was recent. For me, it was a while back. When i look at your config I think its possible the "add default route" in the dhcpv6 client may break when you get upgraded.... If it does, see my first post, it may help.

Kentzo · Thu Jul 13, 2023 1:32 am

[deleted]

Kentzo · Thu Jul 13, 2023 2:18 am

I agree with you, MT naming it ND makes it a bit confusing, sounding like its turning off all the related things when its just about outbound RA. You seem to be a packet sniffing expert, perhaps you can confirm this.

That's not the behavior that I just observed: with disabled /ipv6/nd on the WAN interface I don't get the default route.

With /ipv6/nd disabled on ether1-gateway:

DAc   dst-address=<WAN GUA> routing-table=main gateway=ether1-gateway immediate-gw=ether1-gateway distance=0 scope=10

With /ipv6/nd enabled on ether1-gateway:

DAg   dst-address=::/0 routing-table=main gateway=fe80::21c:73ff:fe00:99%ether1-gateway immediate-gw=fe80::21c:73ff:fe00:99%ether1-gateway distance=1 scope=30 target-scope=10 

DAc   dst-address=<WAN GUA> routing-table=main gateway=ether1-gateway immediate-gw=ether1-gateway distance=0 scope=10

(where fe80::21c:73ff:fe00:99 is LLA of the upstream router).

In other words, it appears that /ipv6/nd needs to be enabled on ether1-gateway for RouterOS to act as a proper SLAAC host.

bandit1200 · Thu Jul 13, 2023 3:57 am

Thanks for doing that

I wasn't able to replicate that initially, but i think that was because i was changing the settings without rebooting and things were getting confused for some reason. So i just did a series of tests rebooting after each config change, and now I see what you do.

I guess its fair to say, in order for the WAN to properly configure its default route automatically via SLAAC, (1) ipv6/ND needs to be ON for the wan, and (2) "accept router advertisements" needs to be "yes" in ipv6 settings?

So while we wait to hear from Xfinity on the MTU issue, I'll stick to the static route method. That way the log wont fill with MTU warnings. (rather than suppress the warnings, which i may forget to undo).

I really appreciate your going through all this with me. Still lots to learn.

-

bandit1200 · Fri Jul 14, 2023 4:47 pm

A bit of a post mortem on the default firewall rule for DHCPv6...

According to RFC 8415, locating the DCHPv6 server on global scope is perfectly valid. When this is done, it is expected that a DHCP "relay agent" be on the local link to forward the initial client request.

The initial request from the client is sent to a multicast group called "All_DHCP_Relay_Agents_and_Servers". If the server is off link, a local relay agent forwards the request, and the reply comes from the off link (globally addressed) server.

I think this confirms that the default rule in MTs IPv6 firewall is in some sense broken, since it filters the replies to ll scope. I think the suggestion would be for MT to consider removing the ll restriction from that default rule. (Is there a valid argument against this?)

Aside: The RFC discusses situations where there may be multiple DHCPv6 servers available, and that the client would receive multiple replies in that case. The client would then choose one to follow up with the request for configuration. They also explain the "rapid commit" parameter that can be set on the MT client -- setting rapid commit assumes there is just one server, and sends both the inital request for a server and the request for configuration during the first message, rather than waiting for a response for requesting the configuration information. (ie, req-resp rather than req-resp-req-resp..) I didn't see this explanation for rapid commit in the MT documentation.

Thanks again @Kentzo for the insights.

Kentzo · Fri Jul 14, 2023 7:49 pm

RFC is not specific whether GUA <-> LLA is allowed. In my opinion a reply must be sent using the same or smaller scope. Since you use LLA source and on-link multicast, I expect the on-link relay agent to substitute source address for its own.

Currently this alteration of the firewall rule due to Comcast config opens up your router to spoofing by anyone on the net. You’re at the mercy of the upstream switch you are connected to to filter rogue DHCPv6 replies.

bandit1200 · Fri Jul 14, 2023 10:38 pm

I'll buy that. The RFC isn't specific on that -- it doesn't describe the relay process in any kind of detail.

I just did a little research on DHCPv6 relays, and the vendors I looked at (juniper/cisco) describe the relays handing bi-directional communication with the server, not just the first discovery. Which is where you were going.

So here's a head scratcher, you might sniff and see more detail, but the packets that come back from the DHCPv6 server are addressed TO the WAN LLA, but are FROM the server's GUA..... How is that possible?

Kentzo · Fri Jul 14, 2023 10:52 pm

How is that possible?

It reached right interface with right destination address over UDP. Reply is not necessary. That's how

bandit1200 · Fri Jul 14, 2023 11:06 pm

OK, I can't contain my self.

Theory: ICPv6 relay is handling all the communications, not just the initial client solicitation. Except that the replies are spoofed as being from the server's location. (So it is all local scoped communication which is what you said it should be.)

I mean, you can't get a packet addressed to a LL from another off link. Its simply not permitted as being unrouteable. So it must be coming from the relay with a spoofed global FROM address.

So why are they doing this?

Perhaps so future communication (renewals, etc) can be initiated by the client directly with the server without going through the relay?

I'm changing my firewall rule back to local scope filter, but this time for the destination, not the source. I think that solves the security issue you were worried about, doesn't it?

Kentzo · Sat Jul 15, 2023 12:16 am

I don't see in RFCs whether a given IPv6 address scope pair is allowed. There are restrictions about routability. However, consider having both LLA and GUA on the same interface. Another on-link host might use its LLA to reach your GUA or GUA to reach your LLA.

My opinion is that alike or a smaller scope should be used after all other restrictions are considered, but I have no RFC to back it. Except this IETF thread.

I'm changing my firewall rule back to local scope filter, but this time for the destination, not the source. I think that solves the security issue you were worried about, doesn't it?

As long as the upstream router behaves, it does. But it also should already drop DHCP packets since they do run their on DHCP there.

kiler129 · Thu Dec 21, 2023 2:28 pm

For people following from Google here - see viewtopic.php?p=1043563#p1043563

Strangely a DHCP relay like this seems to be valid, and it delivers addresses over LLA still.... just with GUA source.

DHCPv6 client cannot recover after outage

DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Re: DHCPv6 client cannot recover after outage

Who is online