I'll send another export.More detail on what @pe1chl wrote:
On R2, the only /ip ipsec policy row says
dst-address=192.168.97.0/24 ... src-address=192.168.79.0/24 tunnel=yes
so only packets with source addresses from 192.168.79.0/24 are matched by that policy.
When the router itself sends a packet, it finds the route to destination, determines the gateway interface, and uses the address associated to that interface as a source one for the packet - unless a pref-src is set on the route, or unless you specify src-address when using commands like /tool traceroute, :ping, or /system ssh, or unless you use a src-nat rule.
=> it is normal that /tool traceroute 192.168.97.22 without specifying src-address=192.168.79.254 leads to getting the response from the WAN gateway.
Is there some resolution for this, or just to leave like this.
Regarding the problem to open the web page at PC2, the first suspicion in these cases is always an MTU-related problem and PMTU discovery failing for some reason. The export from the client contains almost nothing, so hard to say whether it is breaking PMTUD or not.
There are several possibilities, e.g.:Is there some resolution for this, or just to leave like this.
Why are you so smart....There are several possibilities, e.g.:Is there some resolution for this, or just to leave like this.
/interface bridge add name=br-blackhole
/ip route add dst-address=192.168.97.0/24 gateway=br-blackhole pref-src=192.168.79.254
Well, my preferred way of configuring it does not use such a direct IPsec tunnel but rather a GRE/IPsec tunnel between the routers.Is there a way to configure network other (better way) but stil using IPsec. ( you wrote, "when the network configured this way")
Okay, i'll try an setup Network like this, that you told me .Well, my preferred way of configuring it does not use such a direct IPsec tunnel but rather a GRE/IPsec tunnel between the routers.Is there a way to configure network other (better way) but stil using IPsec. ( you wrote, "when the network configured this way")
When you add a GRE interface at each end and specify the public IP of the other end, and a IPsec secret (some random text) this automatically is created.
Then you set a /30 network on each end of the tunnel using IP->addresses (e.g. 10.0.0.1/30 on one side and 10.0.0.2/30 on the other side).
Now you can set a static route for 192.168.79.0/24 via 10.0.0.2 on one side and similar on the other.
You do not need to have tricky NAT-avoidance rules in this case, and it will likely just work for you.
The method I described automatically fixes that problem as well.In the meantime, where could i try to modificate MTU for IPSEC?
The method I described automatically fixes that problem as well.In the meantime, where could i try to modificate MTU for IPSEC?
When it does not, set the MTU on the GRE tunnels to a lower value, e.g. 1400, instead of leaving it blank and using the automatic setting.
Thank you...![]()
![]()
Is there maybe visible sign, that router is been compromised by attacker- in other words how can i check that before i netinstall it?Just a remark... There is no advantage of a GRE tunnel as compared to an IPIP one, given that Mikrotik doesn't use the optional ID field allowing to set up multiple tunnels between same endpoints, so what remains different as compared to IPIP is some weird handling by the firewall and some extra overhead bytes.
So, if I leave it this way, how i can make those web-pages open normally. (the ISP router one and one from Veeam Enterprise console?)
Other than that, your firewall on both routers is basically non-existent, which probably isn't a big deal at the IPsec initiator as it is behind the ISP modem but the 4011 can be misused for DNS DDoS, also even Mikrotik recommends now not to keep Winbox accessible from the internet. So I'd recommend to have a look at how a firewall should look like. Your 4011 may have actually been infected by malware already, it's incredibly quick to squat in if you give it a chance.
Here it is.it seems like you have mtu issues. How you actually-mtu looks like on br-blackhole interface?
Mangle rules are normally a workaround where you cannot make thw PMTUD work properly because most of the network is not under your administration, but yes, it will help here too.I suppose i should do it with Mangle, right?
@Nichky, the MTU of br-blackhole s irrelevant, what matters is the actual network path. The TCP packet gets encapsulated into the IPsec transport one, the DF flag is inherited into the transport packet, so if the transport packet has to be fragmented, the ICMP "doesn't fit, MTU xyz here" is sent back. So the way with ping is correct.How you actually-mtu looks like on br-blackhole interface?
Okay, i get it, what @Nichky wanted to say is that packet generated at 97.0/24 network somwhere along the way runs into router that doesn't accept his original size, and sends ICMP message, right?@Nichky, the MTU of br-blackhole s irrelevant, what matters is the actual network path. The TCP packet gets encapsulated into the IPsec transport one, the DF flag is inherited into the transport packet, so if the transport packet has to be fragmented, the ICMP "doesn't fit, MTU xyz here" is sent back. So the way with ping is correct.How you actually-mtu looks like on br-blackhole interface?
If OP is going to accept @pe1chl's advice to replace bare IPsec by an IPsec-encrypted tunnel, there is no point in investigating where the ICMP MTU exceeded gets lost in the current setup.
Okay.It means that somewhere you have already set a strange MTU. Either in your PC or in your router.
LAN MTU is normally 1500 with no reason to change that.
Link to the ISP may already have a smaller MTU e.g. in case you have PPPoE without RFC4638 support. MTU will be 1492 or 1480.
That is also often a reason why automatic MTU calculations in the router can fail and you get strange results, especially when there is another router doing the PPPoE.
But what now? Windows sends 1472 and then adds up 28?Well there is another thing that I forgot about when sending that response: in some ping programs (e.g. on the MikroTik) you specify the size of the entire packet and it is the same as the MTU.
In other programs (e.g. Linux and probably Mac too) you specify the number of data bytes in the ping packet and the total size is 28 more than that.
So 1472 would be normal in that case.
This is my ppoe client info.When you get 1492 or 1480 byte MTU on your Internet PPPoE connection you should try to set MTU and MRU to 1500 and see if it sticks after the reconnect is done. I.e. MTU 1500 is still indicated.
If so, that will solve a lot of trouble you have in different situations. If not, ask your ISP to support RFC4638 (and in the meantime switch back to your previous settings).
Now i pluged, old LTE router just to see his MTU, and is 1500, so that's why I didn't had problems before with Ipsec and with MTU.@JosipTopic, your feedback assumes we all know the topology of your network. From the screenshot of the MacOS terminal, it is not clear on which site the Macbook is connected.
My Mac is on 79.0/24 segment, and i use windows 10 as guest operating szstem in parallels.
There are two distinct situations you can encounter - one of them is that the MTU is indeed lower on some link (and PPPoE is the most likely candidate), and another one is that the interface at one end of a cable has a higher MTU than the interface at the other end of the same cable.
When the MTU is lower than the usual 1500 but same at both ends of the link, any packet size will get through if you don't set the "don't fragment" flag, and with the don't fragment flag set, you find out what the MTU is. If this works, the issue with web pages not opening is caused by the Path MTU Discovery mechanism not working for that particular client or server, i.e. the ICMP packets saying "fragmentation needed" are lost or ignored by the recipient (sender of the oversize packets).
How to set DF bit?
If, however, there is a mismatch of MTU settings between interfaces connected together using a cable, and the MTU configured at the side closer to the sender is higher than the one at the far end of the cable, packets larger than the actual MTU at the far end will be simply dropped, as the interface at the local end will not send back "fragmentation needed" whereas the one at the remote end won't handle them.
So try pinging with 1500-byte IP packets, setting the proper size depending on the operating system you use for pinging, and see whether they pass through all the way (fragmented). If they do, what remains is to find out whether "fragmentation needed" is being sent. Most likely it's the server response which is too large, so by sniffing at the 4011, you should see large packets to arrive from the server and ICMP "fragmentation needed" to be sent to it in response. It may be sent from an unexpected source address, and the server may ignore it. In theory, it should be the 4011 itself sending them, as the WAN interface is a PPPoE one and as the overhead of the IPsec transport occupies part of the MTU.
That's the -f option to the ping command on Windows, or the do-not-fragment modifier to the ping command in RouterOS. TCP sets it automatically.How to set DF bit?
It's not that easy. Since IPsec is involved in both cases, the overhead of the transport packet always occupies part of the MTU, so the payload packet must be smaller so that the transport one would fit into the MTU of the WAN without fragmentation. Hence the PMTUD is necessary even if the WAN MTU is 1500, you can try that - ping of 1428 bytes of ICMP payload (which means a 1500-byte IP packet) will not get through even via LTE if DF is set. However, for some reason, the server adjusts the packet size according to the ICMP "fragmentation needed" message when LTE is used but doesn't when PPPoE is used. Which is why I suspect that the difference is not the LTE MTU vs. PPPoE MTU but the IP address the router uses as a source one when sending the ICMP "fragmentation needed" message to the server, and how the server treats packets from that address. That is why I've recommended you to sniff what is really happening there as you initiate a connection to the server.Now i pluged, old LTE router just to see his MTU, and is 1500, so that's why I didn't had problems before with Ipsec and with MTU.
and when i switched to RB4011 and ppoe, MTU changed to 1480, i didn't know that.
i would gladly do snifing, can you instruct me, what is the best way, and easiest for me to understand.That's the -f option to the ping command on Windows, or the do-not-fragment modifier to the ping command in RouterOS. TCP sets it automatically.How to set DF bit?
It's not that easy. Since IPsec is involved in both cases, the overhead of the transport packet always occupies part of the MTU, so the payload packet must be smaller so that the transport one would fit into the MTU of the WAN without fragmentation. Hence the PMTUD is necessary even if the WAN MTU is 1500, you can try that - ping of 1428 bytes of ICMP payload (which means a 1500-byte IP packet) will not get through even via LTE if DF is set. However, for some reason, the server adjusts the packet size according to the ICMP "fragmentation needed" message when LTE is used but doesn't when PPPoE is used. Which is why I suspect that the difference is not the LTE MTU vs. PPPoE MTU but the IP address the router uses as a source one when sending the ICMP "fragmentation needed" message to the server, and how the server treats packets from that address. That is why I've recommended you to sniff what is really happening there as you initiate a connection to the server.Now i pluged, old LTE router just to see his MTU, and is 1500, so that's why I didn't had problems before with Ipsec and with MTU.
and when i switched to RB4011 and ppoe, MTU changed to 1480, i didn't know that.
It means,And the only change was at the site with the 4011, where the public IP has migrated from the WAN of the old ISP modem (which was a NATing router with port forwarding) directly to the 4011 with PPPoE via new ISP modem in bridge mode? Or the 4011 was not there before and you've copied its configuration from a previous Mikrotik? Configuration of R1 and R3 has remained the same like before?
Instead of RB4011 was Chateau LTE12 with SIM card in it. So on Chateau i had public IP on LTE interface. And Chateu was responder for both locations.
What means that the first site gets disconnected - does /ip ipsec active-peers at the 4011 not show it as running, or just payload packets stop getting through, which might indicate policies shadowing one another?
Is this "peer unreachable" a message in the log, or it is a red warning printed next to the peer in the configuration, like this?site 79.0/24 gets disconected from site 97.0/24 wiith error "peer unreachable"
Is this "peer unreachable" a message in the log, or it is a red warning printed next to the peer in the configuration, like this?site 79.0/24 gets disconected from site 97.0/24 wiith error "peer unreachable"
Yes, it is red text, printed.
Flags: X - disabled; D - dynamic; R - responder
0 R name="peer1" address=192.168.29.0/24 passive=yes profile=default exchange-mode=main send-initial-contact=yes
1 R ;;; This entry is unreachable
name="peer2" address=192.168.29.0/24 passive=yes profile=default exchange-mode=main send-initial-contact=yes
I remember I had an issue like this when experimenting with IPsec and NAT hole punching and using fqdns as peers' addresses, where a peer with address=xxxxxxx.sn.mynetname.net was shadowing the subsequent ones if it was created during boot, before the DNS resolution could be made. So be more specific about the details so that we can investigate in the right direction.
I'm sending configurations of 79.0 network (RB4011,my house) and 99.0 network (R1_IVO my brothers house), which bot of them are connected to our office, (R1_Trnava)
Now I finally understand.
You cannot have multiple peers accepting incoming connections from the same range of addresses and with the same exchange-mode. If you do that, exactly this happens, because the peer is chosen when processing the initial packet, and the initial packet doesn't carry enough of the identity information. So the first one is always chosen. If the address of two peers is not identical, the peers are automatically sorted by length of the prefix - individual addresses first, worldwide open (::/0) last.
This is why peer, identity, and active-peers are separate data structures.
What you have to do is to link both the identity rows representing the actual remote peers (presumably, Josip_kuca and Ivo_kuca) to a single peer row on the 4011, with address=::/0.
Do not understand this one...
I mean,i tried this, but winbox isn't allowing me to do this.
But since the statically configured policy rows are linked to a particular peer, you have to use dynamic creation of policies from a template. The remote initiator (Josip_kuca or Ivo_kuca) has a statically configured policy, and asks the responder (the 4011) for a corresponding (mirror) one; if the responder has a compatible template, it creates the policy dynamically from that template.
If you want to be more restrictive, you can convert the currently configured policies for Josip_kuca and Ivo_kuca at the 4011 into templates, place each of them into its own dedicated policy template group, and tell each identity row to only search for templates in its dedicated group (you have to create the groups first, then convert policies into templates and choose the appropriate groups for them, and then change policy-template-group from default to the required one on each identity row). This will mean that if one of the initiators eventually asks for some other policy, that request will be denied, while with the default ::/0 <-> ::/0 template, the responder would accept it.
I did this solution, and 75% is working...
Any way here are the pictures.
and this:Do not understand this one...
I mean,i tried this, but winbox isn't allowing me to do this.
do not fit together well. Your screenshots show that both policies exist, so the linking of the identities to the same peer must have worked too? What is the current setup then?I did this solution, and 75% is working...![]()