Some websites unavailable on IPv6

vasco · Sun Nov 29, 2020 3:31 pm

Hello,

I am setting up a new home network with RB2011 as the main router. It's a small network with up to 10 devices (Linux desktops, Android phones, etc.) The ISP is a local VDSL provider, RB2011 is connected to the VDSL modem on eth1, and internet connectivity is via pppoe-client. When everything was set up I enabled IPv6 package in RB2011 and added this configuration:

/interface pppoe-client
add add-default-route=yes disabled=no interface=ether1 max-mru=1480 max-mtu=1480 name=pppoe-vdsl password=adsl service-name=internet user=adsl
/ipv6 address
add address=::1 from-pool=IP6-pool interface=bridge
/ipv6 dhcp-client
add add-default-route=yes interface=pppoe-vdsl pool-name=IP6-pool request=prefix
/ipv6 nd
set [ find default=yes ] interface=bridge

Now the main router and all devices in LAN network have IPv6 addresses and can connect to any IPv6 server. However, there is one major problem: some websites are unavailable on IPv6 when using HTTPS. For example, https://mikrotik.com is one of these websites:

$ curl -v https://mikrotik.com
* Rebuilt URL to: https://mikrotik.com/
*   Trying 2a02:610:7501:1000::2...
* Connected to mikrotik.com (2a02:610:7501:1000::2) port 443 (#0)
* Operation timed out after 0 milliseconds with 0 out of 0 bytes received
* Closing connection 0
curl: (28) Operation timed out after 0 milliseconds with 0 out of 0 bytes received

When I try to access this website via IPv4, no problem at all:

$ curl -v https://mikrotik.com
* Rebuilt URL to: https://mikrotik.com/
*   Trying 159.148.147.196...
* Connected to mikrotik.com (159.148.147.196) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate: mikrotik.com
* Server certificate: DigiCert SHA2 Extended Validation Server CA
* Server certificate: DigiCert High Assurance EV Root CA
> GET / HTTP/1.1
> Host: mikrotik.com
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
.....

I don't have any problem with mikrotik.com over IPv6 with HTTP and ping to mikrotik.com over IPv6 is working as well. Mikrotik.com is not the only website I am not able to access via IPv6. Another website with this behavior is https://ipv6.test-ipv6.com/

This behavior is the same on Macbook connected to my network via WiFi or on the Ubuntu desktop connected directly to the eth2 on RB2011. Even Android phones have the same problem with https://mikrotik.com - that's why I think the problem is in the main router.

What is happening here? Why some HTTPS websites on IPv6 works and others don't? Do you have any ideas about what is wrong with my RouterOS setup or what should I change? Thank you for any ideas or comments. I am working on this issue for several hours now without any luck or progress.

Also, here are my IPv6 firewall rules (just basic rules):

/ipv6 firewall address-list
add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
add address=::1/128 comment="defconf: lo" list=bad_ipv6
add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
add address=100::/64 comment="defconf: discard only " list=bad_ipv6
add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
add address=::224.0.0.0/100 comment="defconf: other" list=bad_ipv6
add address=::127.0.0.0/104 comment="defconf: other" list=bad_ipv6
add address=::/104 comment="defconf: other" list=bad_ipv6
add address=::255.0.0.0/104 comment="defconf: other" list=bad_ipv6
/ipv6 firewall filter
add action=drop chain=input comment="defconf: rfc4890 drop ll if hop-limit!=255" dst-address=fe80::/10 hop-limit=not-equal:255 protocol=icmpv6
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/16
add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN

Znevna · Sun Nov 29, 2020 4:29 pm

Looks like broken PMTUD for whichever of the many reasons that can cause that.
I see that you have MTU 1480 on you WAN interface, try setting MTU 1480 in IPv6/ND too, so that your clients get that instead of 1500.
Maybe not the best way to deal with this but it is one way.
I have it set to 1492 because I'm also on a PPPoE connection and I had issues with other websites which required me to hit refresh after PMTUD did its thing.

StubArea51 · Sun Nov 29, 2020 4:35 pm

Set your IPv6 MTU to 1280 and see if that solves the issue as there are places on the internet that are still 1280 for IPv6. If that resolves it, you can slowly raise it until things break again to understand what your effective MTU is.

https://blog.cloudflare.com/increasing-ipv6-mtu/

vasco · Sun Nov 29, 2020 5:16 pm

Thank you Znevna and IPANetEngineer! The issue was indeed caused by MTU. I tried several values from 1280 to 1480 and all of them worked. Now my settings are:

$ /ipv6 nd set 0 mtu=1480
$ /ipv6 nd print
Flags: X - disabled, I - invalid, * - default
 0  * interface=bridge ra-interval=3m20s-10m ra-delay=3s mtu=1480 reachable-time=unspecified retransmit-interval=unspecified ra-lifetime=30m hop-limit=unspecified
      advertise-mac-address=yes advertise-dns=yes managed-address-configuration=no other-configuration=no

Anything greater than 1480 will broke things again. So I leave the value to 1480 (which is the same as my PPPoE MTU). The IPv6 internet is working now and I need to read more about MTU, PMTUD, and IPv6.

fflo · Wed Dec 02, 2020 9:44 am

Hi vasco,

Hello,
[...]
What is happening here? Why some HTTPS websites on IPv6 works and others don't? Do you have any ideas about what is wrong with my RouterOS setup or what should I change? Thank you for any ideas or comments. I am working on this issue for several hours now without any luck or progress.
[...]

did you setup a IPv6 firewall mangle rule for your PPPoE uplink device?

/ipv6 firewall mangle
add action=change-mss chain=forward comment=\
    "update PMTU for PPPoE via WAN" new-mss=clamp-to-pmtu \
    out-interface=pppoe-out1 packet-size=1421-65535 passthrough=yes protocol=\
    tcp tcp-flags=syn

hint: out-interface=pppoe-vdsl in your example.

Using this rule you should not have to mess with reduced MTU values on your LAN side.

Znevna · Wed Dec 02, 2020 2:44 pm

Any advantage / disadvantage messing with mss instead of using lower mtu for IPv6 on LAN in cases like this?
PS: contrary to what IPANetEngineer said above, it worked for some websites because some websites still force a lower, fixed MTU for IPv6, like 1280, disregarding the mss from the syn sent by the client.

vasco · Thu Dec 03, 2020 12:48 pm

Won't the "change-mss" mangle rule add latency and increase the CPU load on my router?

DarkNate · Thu Jan 07, 2021 11:11 am

Set your IPv6 MTU to 1280 and see if that solves the issue as there are places on the internet that are still 1280 for IPv6. If that resolves it, you can slowly raise it until things break again to understand what your effective MTU is.

https://blog.cloudflare.com/increasing-ipv6-mtu/

ICMPv6 is there to ensure automatic MTU negotiation. The above is a poor resolution and would affect bandwidth throughput.

https://www.cisco.com/c/en/us/td/docs/i ... C7C0DCDA29

Znevna · Thu Jan 07, 2021 11:59 am

Again, not even here he didn't went with 1280. Read: viewtopic.php?f=2&t=169757#p831468
The marked "solution" which stinks was not applied.

DarkNate · Thu Jan 07, 2021 12:24 pm

Again, not even here he didn't went with 1280. Read: viewtopic.php?f=2&t=169757#p831468
The marked "solution" which stinks was not applied.

It totally stinks and that's great. I hate these broken MTU promoters. My primary uplink provider still caps MTU at 1460 on their so-called "next-gen" fibre infrastructure.

Some people never graduated 1500 ethernet MTU basics.

pe1chl · Thu Jan 07, 2021 2:03 pm

The main problem is that Path MTU Discovery sucks. Both because the ICMP messages are often deleted by bad firewall admins, and also because such a mechanism always has some lifetime and after this lifetime it again tries with 1500 byte packets and has to scale back again. Often this lifetime is very short.
So in practice it only works correctly when you can have 1500 byte MTU. A totally arbitrary value that has become "the standard".

StubArea51 · Thu Jan 07, 2021 6:28 pm

It totally stinks and that's great. I hate these broken MTU promoters. My primary uplink provider still caps MTU at 1460 on their so-called "next-gen" fibre infrastructure.

Some people never graduated 1500 ethernet MTU basics.

First day on the Internet with MTU? :-)

His effective MTU was 20 bytes less than 1500 from his provider and PMTUD wasn't working properly. You have to adjust and test MTU when it's not working on defaults to understand the problem. Having designed and built hundreds of ISPs and MPLS networks that require complex MTU math, I stand by my statement. MSS adjust is less than ideal and is always considered to be a "band-aid" in a provider network. When the MTU cannot reach 1500 due to transport encapsulation like PPPoE, setting the IP subnet to the maximum value is always the most efficient solution for the router and for performance.

If you have a better way given the conditions of reduced MTU and broken PMTUD, i'm all ears.

DarkNate · Thu Jan 07, 2021 7:25 pm

It totally stinks and that's great. I hate these broken MTU promoters. My primary uplink provider still caps MTU at 1460 on their so-called "next-gen" fibre infrastructure.

Some people never graduated 1500 ethernet MTU basics.
If you have a better way given the conditions of reduced MTU and broken PMTUD, i'm all ears.

Exactly, PMTUD is unreliable because moronic admins block it and MSS doesn't fix UDP traffic, does it now?

Solution (two in one, one for broken MTU and the other for an RFC4638 compliant ISP): viewtopic.php?f=2&t=171390&p=838106#p838089

For fun, another member tried to find out where the 12bytes went as this is exclusive to MikroTik, he's sent a bug report to MikroTik: viewtopic.php?f=2&t=171390&p=838106#p838114

seridohost · Thu Sep 07, 2023 3:42 pm

[SOLUTION]:
viewtopic.php?p=1024003#p1024003

Some websites unavailable on IPv6 [SOLVED]

Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6 [SOLVED]

Re: Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6

Re: Some websites unavailable on IPv6

Who is online