Community discussions

MikroTik App
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

IPsec-strange thing to me happening.

Mon Jan 31, 2022 5:53 pm

Please, help.

I have thhose 2 Mikrotiks, and IPsec running.
everything worked fine, unitl i changed ISP. They brought their own router, which they have put it to the bridge, and then i've connected my Mikrotik RB 4011
to PPOE and i have internet. Then i 've configured Ipsec. Only thing that is not working is :
1.can't open web-page from PC2 on the remote side server.(Veeam Enterprise console web-page i've checked that is working from pc on remote side)
2. cant' ping from router R2 to R1

When i do traceroute on R2 to 192.168.97.254 (gateway) traffic goes to internet.--- this part is confusing me.

Everything else is working, like RDP from one side to another, explorer, ping from pc1 to pc2 and vice versa, also ping from client side router also works.


Don't know how to solve this problems.

Please help.
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10618
Joined: Mon Jun 08, 2015 12:09 pm

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 6:44 pm

When you do traceroute in a network configured like this you need to set the source address to the router address on the LAN.
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 7:20 pm

More detail on what @pe1chl wrote:
On R2, the only /ip ipsec policy row says
dst-address=192.168.97.0/24 ... src-address=192.168.79.0/24 tunnel=yes
so only packets with source addresses from 192.168.79.0/24 are matched by that policy.
When the router itself sends a packet, it finds the route to destination, determines the gateway interface, and uses the address associated to that interface as a source one for the packet - unless a pref-src is set on the route, or unless you specify src-address when using commands like /tool traceroute, :ping, or /system ssh, or unless you use a src-nat rule.
=> it is normal that /tool traceroute 192.168.97.22 without specifying src-address=192.168.79.254 leads to getting the response from the WAN gateway.

Regarding the problem to open the web page at PC2, the first suspicion in these cases is always an MTU-related problem and PMTU discovery failing for some reason. The export from the client contains almost nothing, so hard to say whether it is breaking PMTUD or not.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 7:27 pm

You are right!
It works.
But what is with that web page on .97.0 network, that doesn't want to open.( i can't open even the ISP router page).

Is there a way to configure network other (better way) but stil using IPsec. ( you wrote, "when the network configured this way")
Thanks.
You do not have the required permissions to view the files attached to this post.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 7:31 pm

More detail on what @pe1chl wrote:
On R2, the only /ip ipsec policy row says
dst-address=192.168.97.0/24 ... src-address=192.168.79.0/24 tunnel=yes
so only packets with source addresses from 192.168.79.0/24 are matched by that policy.
When the router itself sends a packet, it finds the route to destination, determines the gateway interface, and uses the address associated to that interface as a source one for the packet - unless a pref-src is set on the route, or unless you specify src-address when using commands like /tool traceroute, :ping, or /system ssh, or unless you use a src-nat rule.
=> it is normal that /tool traceroute 192.168.97.22 without specifying src-address=192.168.79.254 leads to getting the response from the WAN gateway.

Is there some resolution for this, or just to leave like this.

Regarding the problem to open the web page at PC2, the first suspicion in these cases is always an MTU-related problem and PMTU discovery failing for some reason. The export from the client contains almost nothing, so hard to say whether it is breaking PMTUD or not.
I'll send another export.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 7:35 pm

Regarding the problem to open the web page at PC2, the first suspicion in these cases is always an MTU-related problem and PMTU discovery failing for some reason. The export from the client contains almost nothing, so hard to say whether it is breaking PMTUD or not.
[/quote]

Sending complete export from client.
You do not have the required permissions to view the files attached to this post.
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 7:35 pm

Is there some resolution for this, or just to leave like this.
There are several possibilities, e.g.:
/interface bridge add name=br-blackhole
/ip route add dst-address=192.168.97.0/24 gateway=br-blackhole pref-src=192.168.79.254
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 7:49 pm

Is there some resolution for this, or just to leave like this.
There are several possibilities, e.g.:
/interface bridge add name=br-blackhole
/ip route add dst-address=192.168.97.0/24 gateway=br-blackhole pref-src=192.168.79.254
Why are you so smart....:)

IT WORKS....unbeleiveable...

and...this web-pages?
 
pe1chl
Forum Guru
Forum Guru
Posts: 10618
Joined: Mon Jun 08, 2015 12:09 pm

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 7:50 pm

Is there a way to configure network other (better way) but stil using IPsec. ( you wrote, "when the network configured this way")
Well, my preferred way of configuring it does not use such a direct IPsec tunnel but rather a GRE/IPsec tunnel between the routers.
When you add a GRE interface at each end and specify the public IP of the other end, and a IPsec secret (some random text) this automatically is created.
Then you set a /30 network on each end of the tunnel using IP->addresses (e.g. 10.0.0.1/30 on one side and 10.0.0.2/30 on the other side).
Now you can set a static route for 192.168.79.0/24 via 10.0.0.2 on one side and similar on the other.
You do not need to have tricky NAT-avoidance rules in this case, and it will likely just work for you.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 8:06 pm

Is there a way to configure network other (better way) but stil using IPsec. ( you wrote, "when the network configured this way")
Well, my preferred way of configuring it does not use such a direct IPsec tunnel but rather a GRE/IPsec tunnel between the routers.
When you add a GRE interface at each end and specify the public IP of the other end, and a IPsec secret (some random text) this automatically is created.
Then you set a /30 network on each end of the tunnel using IP->addresses (e.g. 10.0.0.1/30 on one side and 10.0.0.2/30 on the other side).
Now you can set a static route for 192.168.79.0/24 via 10.0.0.2 on one side and similar on the other.
You do not need to have tricky NAT-avoidance rules in this case, and it will likely just work for you.
Okay, i'll try an setup Network like this, that you told me .

In the meantime, where could i try to modificate MTU for IPSEC?
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 8:11 pm

Anyway, i'm so gratefull for you guys, that you helped me and that you are willing to help others and share your knowledge with others.
This is huge....😁

Two days been going in the circle, couldnt find the exit...and posted here, in two minutes everything was solved.

Would like to buy you a beer.🍻 or lunch...

God bless you guys, and your families.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10618
Joined: Mon Jun 08, 2015 12:09 pm

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 8:20 pm

In the meantime, where could i try to modificate MTU for IPSEC?
The method I described automatically fixes that problem as well.
When it does not, set the MTU on the GRE tunnels to a lower value, e.g. 1400, instead of leaving it blank and using the automatic setting.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 8:37 pm

In the meantime, where could i try to modificate MTU for IPSEC?
The method I described automatically fixes that problem as well.
When it does not, set the MTU on the GRE tunnels to a lower value, e.g. 1400, instead of leaving it blank and using the automatic setting.

Thank you...😁 ✋
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 9:22 pm

Just a remark... There is no advantage of a GRE tunnel as compared to an IPIP one, given that Mikrotik doesn't use the optional ID field allowing to set up multiple tunnels between same endpoints, so what remains different as compared to IPIP is some weird handling by the firewall and some extra overhead bytes.

Other than that, your firewall on both routers is basically non-existent, which probably isn't a big deal at the IPsec initiator as it is behind the ISP modem but the 4011 can be misused for DNS DDoS, also even Mikrotik recommends now not to keep Winbox accessible from the internet. So I'd recommend to have a look at how a firewall should look like. Your 4011 may have actually been infected by malware already, it's incredibly quick to squat in if you give it a chance.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10618
Joined: Mon Jun 08, 2015 12:09 pm

Re: IPsec-strange thing to me happening.

Mon Jan 31, 2022 11:40 pm

An advantage of GRE is that it can transport multicast and IPv6 as well. A bit more future-proof. But when that is not relevant you can use IPIP.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 11:07 am

Just a remark... There is no advantage of a GRE tunnel as compared to an IPIP one, given that Mikrotik doesn't use the optional ID field allowing to set up multiple tunnels between same endpoints, so what remains different as compared to IPIP is some weird handling by the firewall and some extra overhead bytes.

So, if I leave it this way, how i can make those web-pages open normally. (the ISP router one and one from Veeam Enterprise console?)

Other than that, your firewall on both routers is basically non-existent, which probably isn't a big deal at the IPsec initiator as it is behind the ISP modem but the 4011 can be misused for DNS DDoS, also even Mikrotik recommends now not to keep Winbox accessible from the internet. So I'd recommend to have a look at how a firewall should look like. Your 4011 may have actually been infected by malware already, it's incredibly quick to squat in if you give it a chance.
Is there maybe visible sign, that router is been compromised by attacker- in other words how can i check that before i netinstall it?
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 11:23 am

This is what i figured now.
I suppose i should do it with Mangle, right?
You do not have the required permissions to view the files attached to this post.
 
User avatar
nichky
Forum Guru
Forum Guru
Posts: 1406
Joined: Tue Jun 23, 2015 2:35 pm

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 11:33 am

it seems like you have mtu issues. How you actually-mtu looks like on br-blackhole interface?
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 11:54 am

it seems like you have mtu issues. How you actually-mtu looks like on br-blackhole interface?
Here it is.
But if I may ask, isn't this br-blackhole used only to redirect traffic to 79.0 gateway?
i only need to downsize MTU towards 192.168.97.0/24 network, correct me if i'm wrong.
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.  [SOLVED]

Tue Feb 01, 2022 11:57 am

I suppose i should do it with Mangle, right?
Mangle rules are normally a workaround where you cannot make thw PMTUD work properly because most of the network is not under your administration, but yes, it will help here too.

Just bear in mind that MSS is 40 bytes less than MTU, and that the ping size in Windows is the size of the ICMP payload, i.e. 28 bytes less than MTU. So set the new-mss to 12 bytes less than what the maximum ping size.
 
User avatar
nichky
Forum Guru
Forum Guru
Posts: 1406
Joined: Tue Jun 23, 2015 2:35 pm

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 12:00 pm

the best scenario to find when the mtu issue is, first ping your gateway with -l -f max mtu size ,than your other router, and than your last destination
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 12:07 pm

Tadaaaaaaa!!!!


Everything works like a charm.

Thank youuuuuuuuu!!!! :)
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 12:09 pm

How you actually-mtu looks like on br-blackhole interface?
@Nichky, the MTU of br-blackhole s irrelevant, what matters is the actual network path. The TCP packet gets encapsulated into the IPsec transport one, the DF flag is inherited into the transport packet, so if the transport packet has to be fragmented, the ICMP "doesn't fit, MTU xyz here" is sent back. So the way with ping is correct.

If OP is going to accept @pe1chl's advice to replace bare IPsec by an IPsec-encrypted tunnel, there is no point in investigating where the ICMP MTU exceeded gets lost in the current setup.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 12:18 pm

How you actually-mtu looks like on br-blackhole interface?
@Nichky, the MTU of br-blackhole s irrelevant, what matters is the actual network path. The TCP packet gets encapsulated into the IPsec transport one, the DF flag is inherited into the transport packet, so if the transport packet has to be fragmented, the ICMP "doesn't fit, MTU xyz here" is sent back. So the way with ping is correct.

If OP is going to accept @pe1chl's advice to replace bare IPsec by an IPsec-encrypted tunnel, there is no point in investigating where the ICMP MTU exceeded gets lost in the current setup.
Okay, i get it, what @Nichky wanted to say is that packet generated at 97.0/24 network somwhere along the way runs into router that doesn't accept his original size, and sends ICMP message, right?
So what he suggessts is to ping each hop and see who is making problems with size. Correct?
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 12:25 pm

Here it is.
Is it normal that first ping on LAN is not 1500 bytes allowed?
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10618
Joined: Mon Jun 08, 2015 12:09 pm

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 12:30 pm

It means that somewhere you have already set a strange MTU. Either in your PC or in your router.
LAN MTU is normally 1500 with no reason to change that.

Link to the ISP may already have a smaller MTU e.g. in case you have PPPoE without RFC4638 support. MTU will be 1492 or 1480.

That is also often a reason why automatic MTU calculations in the router can fail and you get strange results, especially when there is another router doing the PPPoE.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 1:58 pm

It means that somewhere you have already set a strange MTU. Either in your PC or in your router.
LAN MTU is normally 1500 with no reason to change that.

Link to the ISP may already have a smaller MTU e.g. in case you have PPPoE without RFC4638 support. MTU will be 1492 or 1480.

That is also often a reason why automatic MTU calculations in the router can fail and you get strange results, especially when there is another router doing the PPPoE.
Okay.
So actually, this is from my Mac terminal, I issued a ping to default gateway with 1500b, and it passed without problems.
what it may be is that i'm running windows on parallels desktop, so network adapters are bridged, but maybe in that transition parallels are taking bytes and not allowing real 1500b to be passed on local network.
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10618
Joined: Mon Jun 08, 2015 12:09 pm

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 2:17 pm

Well there is another thing that I forgot about when sending that response: in some ping programs (e.g. on the MikroTik) you specify the size of the entire packet and it is the same as the MTU.
In other programs (e.g. Linux and probably Mac too) you specify the number of data bytes in the ping packet and the total size is 28 more than that.
So 1472 would be normal in that case.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 3:05 pm

I found this on the web.
http://www.dslreports.com/faq/578

I did downloaded and set it like in the picture and it does increased my download speed on paralles desktop vm.
On Mac, speed was always good.

So, today i learned a lot.
:)
Thank you.
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10618
Joined: Mon Jun 08, 2015 12:09 pm

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 3:08 pm

When you get 1492 or 1480 byte MTU on your Internet PPPoE connection you should try to set MTU and MRU to 1500 and see if it sticks after the reconnect is done. I.e. MTU 1500 is still indicated.
If so, that will solve a lot of trouble you have in different situations. If not, ask your ISP to support RFC4638 (and in the meantime switch back to your previous settings).
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 3:14 pm

Well there is another thing that I forgot about when sending that response: in some ping programs (e.g. on the MikroTik) you specify the size of the entire packet and it is the same as the MTU.
In other programs (e.g. Linux and probably Mac too) you specify the number of data bytes in the ping packet and the total size is 28 more than that.
So 1472 would be normal in that case.
But what now? Windows sends 1472 and then adds up 28?
Correct?
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 3:22 pm

When you get 1492 or 1480 byte MTU on your Internet PPPoE connection you should try to set MTU and MRU to 1500 and see if it sticks after the reconnect is done. I.e. MTU 1500 is still indicated.
If so, that will solve a lot of trouble you have in different situations. If not, ask your ISP to support RFC4638 (and in the meantime switch back to your previous settings).
This is my ppoe client info.
Yes MTU here is 1480, but how to set it 1500, is it possible at all?

The previous post, one with dr.TCP picture, that little program helped me to increase download speed on Parallels VM_WIN10. Before was 300 Mbit/s,while on mac was all the time around 850Mbit/s and actually i have 1Gbit/s internet connection.
You do not have the required permissions to view the files attached to this post.
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 3:42 pm

@JosipTopic, your feedback assumes we all know the topology of your network. From the screenshot of the MacOS terminal, it is not clear on which site the Macbook is connected.

There are two distinct situations you can encounter - one of them is that the MTU is indeed lower on some link (and PPPoE is the most likely candidate), and another one is that the interface at one end of a cable has a higher MTU than the interface at the other end of the same cable.

When the MTU is lower than the usual 1500 but same at both ends of the link, any packet size will get through if you don't set the "don't fragment" flag, and with the don't fragment flag set, you find out what the MTU is. If this works, the issue with web pages not opening is caused by the Path MTU Discovery mechanism not working for that particular client or server, i.e. the ICMP packets saying "fragmentation needed" are lost or ignored by the recipient (sender of the oversize packets).

If, however, there is a mismatch of MTU settings between interfaces connected together using a cable, and the MTU configured at the side closer to the sender is higher than the one at the far end of the cable, packets larger than the actual MTU at the far end will be simply dropped, as the interface at the local end will not send back "fragmentation needed" whereas the one at the remote end won't handle them.

So try pinging with 1500-byte IP packets, setting the proper size depending on the operating system you use for pinging, and see whether they pass through all the way (fragmented). If they do, what remains is to find out whether "fragmentation needed" is being sent. Most likely it's the server response which is too large, so by sniffing at the 4011, you should see large packets to arrive from the server and ICMP "fragmentation needed" to be sent to it in response. It may be sent from an unexpected source address, and the server may ignore it. In theory, it should be the 4011 itself sending them, as the WAN interface is a PPPoE one and as the overhead of the IPsec transport occupies part of the MTU.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 4:59 pm

@JosipTopic, your feedback assumes we all know the topology of your network. From the screenshot of the MacOS terminal, it is not clear on which site the Macbook is connected.

My Mac is on 79.0/24 segment, and i use windows 10 as guest operating szstem in parallels.

There are two distinct situations you can encounter - one of them is that the MTU is indeed lower on some link (and PPPoE is the most likely candidate), and another one is that the interface at one end of a cable has a higher MTU than the interface at the other end of the same cable.

When the MTU is lower than the usual 1500 but same at both ends of the link, any packet size will get through if you don't set the "don't fragment" flag, and with the don't fragment flag set, you find out what the MTU is. If this works, the issue with web pages not opening is caused by the Path MTU Discovery mechanism not working for that particular client or server, i.e. the ICMP packets saying "fragmentation needed" are lost or ignored by the recipient (sender of the oversize packets).

How to set DF bit?

If, however, there is a mismatch of MTU settings between interfaces connected together using a cable, and the MTU configured at the side closer to the sender is higher than the one at the far end of the cable, packets larger than the actual MTU at the far end will be simply dropped, as the interface at the local end will not send back "fragmentation needed" whereas the one at the remote end won't handle them.

So try pinging with 1500-byte IP packets, setting the proper size depending on the operating system you use for pinging, and see whether they pass through all the way (fragmented). If they do, what remains is to find out whether "fragmentation needed" is being sent. Most likely it's the server response which is too large, so by sniffing at the 4011, you should see large packets to arrive from the server and ICMP "fragmentation needed" to be sent to it in response. It may be sent from an unexpected source address, and the server may ignore it. In theory, it should be the 4011 itself sending them, as the WAN interface is a PPPoE one and as the overhead of the IPsec transport occupies part of the MTU.
Now i pluged, old LTE router just to see his MTU, and is 1500, so that's why I didn't had problems before with Ipsec and with MTU.
and when i switched to RB4011 and ppoe, MTU changed to 1480, i didn't know that.


Okay, i think that we qare done with this.

Thank you all, once again.
Until next problem. :)
You do not have the required permissions to view the files attached to this post.
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 6:11 pm

How to set DF bit?
That's the -f option to the ping command on Windows, or the do-not-fragment modifier to the ping command in RouterOS. TCP sets it automatically.

Now i pluged, old LTE router just to see his MTU, and is 1500, so that's why I didn't had problems before with Ipsec and with MTU.
and when i switched to RB4011 and ppoe, MTU changed to 1480, i didn't know that.
It's not that easy. Since IPsec is involved in both cases, the overhead of the transport packet always occupies part of the MTU, so the payload packet must be smaller so that the transport one would fit into the MTU of the WAN without fragmentation. Hence the PMTUD is necessary even if the WAN MTU is 1500, you can try that - ping of 1428 bytes of ICMP payload (which means a 1500-byte IP packet) will not get through even via LTE if DF is set. However, for some reason, the server adjusts the packet size according to the ICMP "fragmentation needed" message when LTE is used but doesn't when PPPoE is used. Which is why I suspect that the difference is not the LTE MTU vs. PPPoE MTU but the IP address the router uses as a source one when sending the ICMP "fragmentation needed" message to the server, and how the server treats packets from that address. That is why I've recommended you to sniff what is really happening there as you initiate a connection to the server.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 9:24 pm

How to set DF bit?
That's the -f option to the ping command on Windows, or the do-not-fragment modifier to the ping command in RouterOS. TCP sets it automatically.

Now i pluged, old LTE router just to see his MTU, and is 1500, so that's why I didn't had problems before with Ipsec and with MTU.
and when i switched to RB4011 and ppoe, MTU changed to 1480, i didn't know that.
It's not that easy. Since IPsec is involved in both cases, the overhead of the transport packet always occupies part of the MTU, so the payload packet must be smaller so that the transport one would fit into the MTU of the WAN without fragmentation. Hence the PMTUD is necessary even if the WAN MTU is 1500, you can try that - ping of 1428 bytes of ICMP payload (which means a 1500-byte IP packet) will not get through even via LTE if DF is set. However, for some reason, the server adjusts the packet size according to the ICMP "fragmentation needed" message when LTE is used but doesn't when PPPoE is used. Which is why I suspect that the difference is not the LTE MTU vs. PPPoE MTU but the IP address the router uses as a source one when sending the ICMP "fragmentation needed" message to the server, and how the server treats packets from that address. That is why I've recommended you to sniff what is really happening there as you initiate a connection to the server.
i would gladly do snifing, can you instruct me, what is the best way, and easiest for me to understand.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 10:01 pm

And one more question, now i'm starting to add 3rd location, how should do it?
Same as first client, or should i change something in complete design.
Thanks.
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10618
Joined: Mon Jun 08, 2015 12:09 pm

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 10:14 pm

You should migrate to GRE/IPsec or IPIP/Ipsec.
I am not going to tell it another time! Good luck with your network.
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 10:27 pm

As @pe1chl says - once you start setting up more complex VPN topologies, use of bare IPsec with traffic selectors will quickly go over your head, so the migration to IPIP over IPsec or GRE over IPsec should be your first step, before adding the third site to the mix.

Next, it is important to think about the traffic matrix (i.e. which devices on the various sites need to talk to each other) and therefore the topology. You may be limited by public IPs being available only on a single site, forcing you to send traffic between the other two sites through this one.

Your drawing doesn't contain any of that information.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 10:49 pm

Okay, what actually boders me, it worked with 3 sites for 6 months, but now when i create 3rd site the first one get disconected.
You do not have the required permissions to view the files attached to this post.
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 10:56 pm

And the only change was at the site with the 4011, where the public IP has migrated from the WAN of the old ISP modem (which was a NATing router with port forwarding) directly to the 4011 with PPPoE via new ISP modem in bridge mode? Or the 4011 was not there before and you've copied its configuration from a previous Mikrotik? Configuration of R1 and R3 has remained the same like before?

What means that the first site gets disconnected - does /ip ipsec active-peers at the 4011 not show it as running, or just payload packets stop getting through, which might indicate policies shadowing one another?
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Tue Feb 01, 2022 11:30 pm

And the only change was at the site with the 4011, where the public IP has migrated from the WAN of the old ISP modem (which was a NATing router with port forwarding) directly to the 4011 with PPPoE via new ISP modem in bridge mode? Or the 4011 was not there before and you've copied its configuration from a previous Mikrotik? Configuration of R1 and R3 has remained the same like before?
Instead of RB4011 was Chateau LTE12 with SIM card in it. So on Chateau i had public IP on LTE interface. And Chateu was responder for both locations.

What means that the first site gets disconnected - does /ip ipsec active-peers at the 4011 not show it as running, or just payload packets stop getting through, which might indicate policies shadowing one another?
It means,
site 97.0/24 connects to site 79.0/24
site 97.0/24 connects to site 99.0/24
And when i make that
site 99.0/24 connects to site 79.0/24
then
site 79.0/24 gets disconected from site 97.0/24 wiith error "peer unreachable"
and only site 97.0/24 connect to site 99.0/24
I'm sending old configuration from Chateu, and new configurations from all three sites updated.
You do not have the required permissions to view the files attached to this post.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Wed Feb 02, 2022 12:08 am

I made it to connect to each other.Don't ask me how. :)

But now site 99.0/24 can ping site 79.0/24 and vice versa is not possible at the moment.
i'm sending new configs.
You do not have the required permissions to view the files attached to this post.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Wed Feb 02, 2022 12:31 am

Update:
when i killed connections, now same story like on the begining.

Site 97.0 and 99.0 are conected, and site 79.0 is disconnected from both sites.
If i delete last site, then 97.0 can ping 99.0, and 97.0 can ping 79.0.
good night.
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.

Wed Feb 02, 2022 11:01 am

site 79.0/24 gets disconected from site 97.0/24 wiith error "peer unreachable"
Is this "peer unreachable" a message in the log, or it is a red warning printed next to the peer in the configuration, like this?

Flags: X - disabled; D - dynamic; R - responder
0 R name="peer1" address=192.168.29.0/24 passive=yes profile=default exchange-mode=main send-initial-contact=yes

1 R ;;; This entry is unreachable
name="peer2" address=192.168.29.0/24 passive=yes profile=default exchange-mode=main send-initial-contact=yes


I remember I had an issue like this when experimenting with IPsec and NAT hole punching and using fqdns as peers' addresses, where a peer with address=xxxxxxx.sn.mynetname.net was shadowing the subsequent ones if it was created during boot, before the DNS resolution could be made. So be more specific about the details so that we can investigate in the right direction.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Wed Feb 02, 2022 3:19 pm

site 79.0/24 gets disconected from site 97.0/24 wiith error "peer unreachable"
Is this "peer unreachable" a message in the log, or it is a red warning printed next to the peer in the configuration, like this?
Yes, it is red text, printed.
Flags: X - disabled; D - dynamic; R - responder
0 R name="peer1" address=192.168.29.0/24 passive=yes profile=default exchange-mode=main send-initial-contact=yes

1 R ;;; This entry is unreachable
name="peer2" address=192.168.29.0/24 passive=yes profile=default exchange-mode=main send-initial-contact=yes


I remember I had an issue like this when experimenting with IPsec and NAT hole punching and using fqdns as peers' addresses, where a peer with address=xxxxxxx.sn.mynetname.net was shadowing the subsequent ones if it was created during boot, before the DNS resolution could be made. So be more specific about the details so that we can investigate in the right direction.
I'm sending configurations of 79.0 network (RB4011,my house) and 99.0 network (R1_IVO my brothers house), which bot of them are connected to our office, (R1_Trnava)
You do not have the required permissions to view the files attached to this post.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Wed Feb 02, 2022 3:56 pm

So this is now when everything is connected,
R1_IVO 99.0 network can ping 79.0 network and 97.0 network.
RB4011 can not ping 99.0 network and can ping 97.0 network
on RB4011 there is RED error on Peer for 97.0 network.
That error comes when i create 2nd indentity on RB4011.
If i kill connections, Then R1_IVO can connect to RB4011.

If anything else you want or need to send you, would like to assist.
You do not have the required permissions to view the files attached to this post.
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.

Wed Feb 02, 2022 4:24 pm

Now I finally understand.

You cannot have multiple peers accepting incoming connections from the same range of addresses and with the same exchange-mode. If you do that, exactly this happens, because the peer is chosen when processing the initial packet, and the initial packet doesn't carry enough of the identity information. So the first one is always chosen. If the address of two peers is not identical, the peers are automatically sorted by length of the prefix - individual addresses first, worldwide open (::/0) last.

This is why peer, identity, and active-peers are separate data structures.

What you have to do is to link both the identity rows representing the actual remote peers (presumably, Josip_kuca and Ivo_kuca) to a single peer row on the 4011, with address=::/0.

But since the statically configured policy rows are linked to a particular peer, you have to use dynamic creation of policies from a template. The remote initiator (Josip_kuca or Ivo_kuca) has a statically configured policy, and asks the responder (the 4011) for a corresponding (mirror) one; if the responder has a compatible template, it creates the policy dynamically from that template.

If you want to be more restrictive, you can convert the currently configured policies for Josip_kuca and Ivo_kuca at the 4011 into templates, place each of them into its own dedicated policy template group, and tell each identity row to only search for templates in its dedicated group (you have to create the groups first, then convert policies into templates and choose the appropriate groups for them, and then change policy-template-group from default to the required one on each identity row). This will mean that if one of the initiators eventually asks for some other policy, that request will be denied, while with the default ::/0 <-> ::/0 template, the responder would accept it.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Wed Feb 02, 2022 5:45 pm

Now I finally understand.

You cannot have multiple peers accepting incoming connections from the same range of addresses and with the same exchange-mode. If you do that, exactly this happens, because the peer is chosen when processing the initial packet, and the initial packet doesn't carry enough of the identity information. So the first one is always chosen. If the address of two peers is not identical, the peers are automatically sorted by length of the prefix - individual addresses first, worldwide open (::/0) last.

This is why peer, identity, and active-peers are separate data structures.

What you have to do is to link both the identity rows representing the actual remote peers (presumably, Josip_kuca and Ivo_kuca) to a single peer row on the 4011, with address=::/0.
Do not understand this one...
I mean,i tried this, but winbox isn't allowing me to do this.

But since the statically configured policy rows are linked to a particular peer, you have to use dynamic creation of policies from a template. The remote initiator (Josip_kuca or Ivo_kuca) has a statically configured policy, and asks the responder (the 4011) for a corresponding (mirror) one; if the responder has a compatible template, it creates the policy dynamically from that template.

If you want to be more restrictive, you can convert the currently configured policies for Josip_kuca and Ivo_kuca at the 4011 into templates, place each of them into its own dedicated policy template group, and tell each identity row to only search for templates in its dedicated group (you have to create the groups first, then convert policies into templates and choose the appropriate groups for them, and then change policy-template-group from default to the required one on each identity row). This will mean that if one of the initiators eventually asks for some other policy, that request will be denied, while with the default ::/0 <-> ::/0 template, the responder would accept it.
I did this solution, and 75% is working...:)

Any way here are the pictures.
You do not have the required permissions to view the files attached to this post.
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Wed Feb 02, 2022 5:50 pm

when i do traceroute...goes on internet
You do not have the required permissions to view the files attached to this post.
 
User avatar
sindy
Forum Guru
Forum Guru
Posts: 11541
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec-strange thing to me happening.

Wed Feb 02, 2022 6:43 pm

This:
Do not understand this one...
I mean,i tried this, but winbox isn't allowing me to do this.
and this:
I did this solution, and 75% is working...:)
do not fit together well. Your screenshots show that both policies exist, so the linking of the identities to the same peer must have worked too? What is the current setup then?

Regarding the ping succeeding in one direction but not in the other one, it's due to your lack of systematic approach: at the 4011, you have exempted connections from .79. to .97. from getting src-nated using a selective action=accept rule before the catch-all action=masquerade one in /ip firewall nat, and you have also exempted the .79.<=>.97. traffic from getting handled by connection tracking at all by adding selective action=notrack rules to /ip firewall raw (so the rule in chain=srcnat became useless), but you forgot to do any of those for the .79. <=> .99. traffic.

This is yet another reason why IPsec-encapsulated tunnels are easier to handle than bare IPsec.

Since you do not connect to any 3rd party IPsec VPN, I'd suggest to replace the selective .79.=>.97. action=accept rule in srcnat by one dealing with all IPsec traffic:
chain=srcnat ipsec-policy=out,ipsec action=accept

In general, having no firewall between the sites for anything that goes via IPsec may not be the best idea, with all that cross-platform malware wandering around. And if you want to use a stateful firewall (which is much better than a simple one), bear in mind that exemption from connection tracking prevents the affected traffic from being handled statefully. But if you insist on using the raw variant, replace the two existing rules for .79.<=>.97. by the following ones:
chain=prerouting ipsec-policy=in,ipsec action=notrack
chain=prerouting ipsec-policy=out,ipsec action=notrack
 
JosipTopic
newbie
Topic Author
Posts: 43
Joined: Mon Apr 06, 2020 10:21 pm
Location: Zagreb

Re: IPsec-strange thing to me happening.

Wed Feb 02, 2022 7:26 pm

do not fit together well. Your screenshots show that both policies exist, so the linking of the identities to the same peer must have worked too? What is the current setup then?

Regarding the ping succeeding in one direction but not in the other one, it's due to your lack of systematic approach: at the 4011, you have exempted connections from .79. to .97. from getting src-nated using a selective action=accept rule before the catch-all action=masquerade one in /ip firewall nat, and you have also exempted the .79.<=>.97. traffic from getting handled by connection tracking at all by adding selective action=notrack rules to /ip firewall raw (so the rule in chain=srcnat became useless), but you forgot to do any of those for the .79. <=> .99. traffic.
Again you are the man!!!

This is yet another reason why IPsec-encapsulated tunnels are easier to handle than bare IPsec.

Since you do not connect to any 3rd party IPsec VPN, I'd suggest to replace the selective .79.=>.97. action=accept rule in srcnat by one dealing with all IPsec traffic:
chain=srcnat ipsec-policy=out,ipsec action=accept

In general, having no firewall between the sites for anything that goes via IPsec may not be the best idea, with all that cross-platform malware wandering around. And if you want to use a stateful firewall (which is much better than a simple one),
Please advice...?

bear in mind that exemption from connection tracking prevents the affected traffic from being handled statefully. But if you insist on using the raw variant, replace the two existing rules for .79.<=>.97. by the following ones:
I've deleted RAW, is that OK?
chain=prerouting ipsec-policy=in,ipsec action=notrack
chain=prerouting ipsec-policy=out,ipsec action=notrack


Sending current config.
[/quote]
You do not have the required permissions to view the files attached to this post.