Dear forum,

today I am banging my head against the wall because I don't even know where to start debugging the issue.

The issue: When wanting to test a dummy-Telegram Bot, I get a lot of timeouts. This issue seems to be related to IPv6 because if I manually curl and force IPv4, I get an answer instantly.

A few details: I can curl via IPv6 just fine Google or other services. So IPv6 connectivity itself works. When looking at my router config then you will see that I have setup NAT for IPv6 and only am announcing ULAs inside my networks. This is due to me not knowing better how to give my k8s nodes and LB static IPs and at the same time have the cluster reachable from the outside.

What is suspicious to me is the TCP Dup ACK and Retransmissions that I see in Wireshark when dumping the curl attempt via the Mikrotik Sniffer.

The setup:

• DrayTek Vigor 167
• Mikrotik hAP ax^3
• Mikrotik CSS326-24G-2S-RM

Attached is my router configuration and the screenshot from the Mikrotik Sniffer attempt.

At this stage, a "know your tools" issue obfuscates the actual one, so you have to fix that first.

When sniffing on the Mikrotik itself, you did not filter by interface name, so each packet is captured at three interfaces (my guess without studying your config is that these are the physical LAN interface, the interface of the bridge the LAN port is a member of, and the VLAN interface attached to that bridge one). Unfortunately, Mikrotik keeps storing capture files in .pcap format so the interface names are not included in the packet metadata, and Wireshark TCP dissector treats each occurrence of the packet as an individual one, and would do so even if the interface ID was present in the metadata - it is also older than the .pcapng format, and even if it wasn't, ignoring interface ID when dissecting TCP is not strictly wrong.

You can use editcap to "deduplicate" the .pcap file, but doing so may cause information loss because if the actual issue is an MTU one and the Mikrotik itself is somehow involved in it, deduplication may obfuscate the fact that the Mikrotik has received some packet but not forwarded it.

Since you use private IPv6 addresses, it is possible that there is a NAT on the Mikrotik where you sniff, so a limitation of the sniffing to the VLAN interface and the WAN interface could be enough to make Wireshark show it as two independent TCP sessions.

From your post I understand that I need to perform the following steps:

• Perform a sniff on the VLAN interface (this is the before firewall view)
• Perform a sniff on the pppoe interface (this should now be the after firewall view)
• Compare the the two to see if packets are lost and to see if there are actually duplicates

With that I should have then a non duplicated view of the traffic.

Thank for the pointer that I am missing the interface information. Would that information be present when I decide to stream the data to Wireshark? I wasn't able to find an answer in the Mikrotik Wiki.

You can sniff on the VLAN interface and on the PPPoE interface simultaneously as lists can be used in the filter (as in interface=vlanX,pppoe-out1); if there is NAT, the "local side" addresses will differ so Wireshark will show to separate TCP sessions.

If you configure the sniffer itself to stream traffic, it will use TZSP encapsulation for the streaming, and TZSP also doesn't contain the interface ID. But if you use mangle rules to stream traffic, you can use more complex conditions to choose packets to be streamed, and you can use different destination port for each interface, so the information about the source interface will be conveyed to Wireshark that way.

There is also the CALEA encapsulation but I have never bothered to try whether Wireshark can dissect it out of the box, i.e. without having to write a dissector of your own.

Okay I am back on my desk and have taken three captures:

1. IPv6 capture on the VLAN interface for Telegram
2. IPv4 capture on the VLAN interface for Telegram
3. IPv6 capture on the VLAN interface for Google

For both IPv4 for Telegram and IPv6 for Google I can see the Ack for the Client Hello and I see a following TLSv1.3 Server Hello. For my IPv6 Telegram capture I am seeing a "TCP Previous segment not captured" message (see attached screenshot).

Just for good measure I tried the curl again and it worked. The capture showed no traffic VLAN traffic though. When checking my local computer it seems that I received a globally routable IPv6 address (which is incorrect). After removing the globally valid address I noticed that the curl didn't work anymore. So I guess something is indeed incorrect in my IPv6 NAT setup. I moved the router IP to my bridge now (as it was in the VLAN before). I will restart my computer and see if I get a global IPv6 now.

Edit: Restart of the router and PC now achieve the desired result of only ULA on my local PC interface.

I now have a pcap where one of my curl commands is working and multiple other ones that don't work. Same firewall config, same computer.

Edit: Comparing non-working pcaps between the pppoe interface and the VLAN one doesn't reveal any differences except the IP. At least to my untrained eye.

Okay so judging by the sequence number of the first "TCP Previous segment not captured" and comparing that to the working curl command pcap it seems that the packet with the "Server Hello" is missing. My understanding is that the MTU of my PPPoE connection is 1492 which means that the Server Hello cannot be accepted in a single packet.

I have executed http://icmpcheckv6.popcount.org/ and the second test is failing. I did this after reading viewtopic.php?t=195044

Found the issue! My local PC has an MTU of 1500 set (by the RA). Setting this manually on the Linux box to 1492 makes the curl succeed every time. This was a hard one for myself.

Edit: Yeah I restarted both the router and PC after I set in the RA the MTU of 1492 and now asking Telegram for my Bot works every time via IPv6. The reason why the MTU wasn't set to 1492 is that the ULA addresses don't inherit the MTU from the PPPoE connection like the GUA ones do.

ON Windows.

First, press Windows Key + X to access the quick menu and launch Windows Terminal (Admin).
Now run the following command to list down all the network adapters:
Now note down the Interface name for which you want to change the MTU.

Next, to change the MTU value of your preferred interface, run the following command:
Over here, replace <subinterface name> with the interface name.
Also, you change the mtu=1492 value to something else per your requirements.

Chances are that it is not your fault. I have a flawless IPv6 setup, but Telegram (API.telegram.org) causes troubles. This is the only address I have blocked in my firewall for IPv6 to force IPv4. Not sure though if it is Telegram itself or anything on my provider's side that causes this...

Setting this manually on the Linux box to 1492 makes the curl succeed every time.

As @eworm has stated, this should not be necessary if everything worked the way it should. There is that thing called Path MTU Discovery (PMTUD) that allows the sender of a packet to get notified that at some place on the path towards the destination the packet did not fit to the MTU and that it should retry with a smaller amount of payload data. If this does not work, it usually means someone somewhere blocks ICMP traffic without distinguishing between ECHO packets (ping) and other stuff.

Setting this manually on the Linux box to 1492 makes the curl succeed every time.

As @eworm has stated, this should not be necessary if everything worked the way it should. There is that thing called Path MTU Discovery (PMTUD) that allows the sender of a packet to get notified that at some place on the path towards the destination the packet did not fit to the MTU and that it should retry with a smaller amount of payload data. If this does not work, it usually means someone somewhere blocks ICMP traffic without distinguishing between ECHO packets (ping) and other stuff.

That makes me very happy that this is not my fault. I am however not willing to dig deeper into this honestly. The root cause has been found, the issue has been fixed and I don't intend to start fighting with Telegram Admins over MTU issues, only to be told off to go an report this to Mikrotik. The Telegram Admins have repeatedly shown that they treat IPv6 as a second class citizen in the GitHub issues for their clients, as such I see no hope in sinking time into this.

I am however not willing to dig deeper into this honestly.

That's alright, it's a legitimate approach of a seasoned support technician who doesn't have enough time to care about the artistic impact of the fix, knowing that it has no substantial side effect. In your particular case, the only thing to consider would be whether forcing even local traffic to use the smaller MTU is a substantial or negligible limitation.

I've posted my comment to let folks who come here googling know what the real issue is, because the broken PMTUD is not just an IPv6 thing nor just a Telegram thing, and instead of reducing the MTU of all interfaces in your network down to the lowest one, you can use action=change-mss rules in mangle to adjust the MSS value in TCP SYN packets as the router forwards them to destinations known to be affected by PMTUD issues. This way, you fix the problem selectively, not affecting the rest of the connections.