Page 1 of 1

ike2, wireguard, mark-routing, two isp and newbie

Posted: Sun Oct 22, 2023 6:49 pm
by ayrsbsdu
Hello!

I want to forward only some destination address to the wireguard table, but I can't figure out the cause of the problems with macOS.
Please tell me what's wrong with the setup
2_result.png

Current CHR:
/interface bridge
add name=bridge1
/interface list
add name=LAN
add name=WAN
/ip ipsec policy group
add name=group1
/ip ipsec profile
add dh-group=ecp256 enc-algorithm=aes-256 hash-algorithm=sha256 name=profile1 prf-algorithm=sha256 proposal-check=strict
/ip ipsec peer
add exchange-mode=ike2 local-address=XXX.XXX.XXX.XXX name=peer1 passive=yes profile=profile1
/ip ipsec proposal
add auth-algorithms="" enc-algorithms=aes-256-gcm lifetime=8h name=proposal1 pfs-group=ecp256
/ip pool
add name=MT ranges=192.168.1.2
add name=macOS ranges=192.168.1.3
/ip ipsec mode-config
add address-pool=MT address-prefix-length=32 name=MT split-include=0.0.0.0/0 static-dns=192.168.1.1 system-dns=no
add address-pool=macOS address-prefix-length=32 name=macOS split-include=0.0.0.0/0 static-dns=192.168.1.1 system-dns=no
/interface list member
add interface=bridge1 list=LAN
add interface=ether1 list=WAN
/ip address
add address=192.168.1.1/24 interface=bridge1 network=192.168.1.0
/ip dhcp-client
add !dhcp-options interface=ether1 use-peer-dns=no use-peer-ntp=no
/ip firewall filter
add action=accept chain=input connection-state=established,related
add action=drop chain=input connection-state=invalid
add action=accept chain=input dst-address=XXX.XXX.XXX.XXX dst-port=500,4500 protocol=udp
add action=accept chain=input dst-address=XXX.XXX.XXX.XXX protocol=ipsec-esp
add action=accept chain=input ipsec-policy=in,ipsec src-address=192.168.1.0/24
add action=drop chain=input in-interface-list=!LAN
add action=drop chain=forward dst-address-list=bad_ipv4
add action=accept chain=forward dst-address=0.0.0.0/0 ipsec-policy=in,ipsec src-address=192.168.1.0/24
add action=accept chain=forward connection-state=established,related
add action=drop chain=forward connection-state=invalid
add action=drop chain=forward connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
/ip firewall mangle
add action=change-mss chain=forward ipsec-policy=in,ipsec new-mss=1360 passthrough=yes protocol=tcp src-address=192.168.1.0/24 tcp-flags=syn tcp-mss=!0-1360
add action=change-mss chain=forward dst-address=192.168.1.0/24 ipsec-policy=out,ipsec new-mss=1360 passthrough=yes protocol=tcp tcp-flags=syn tcp-mss=!0-1360
/ip firewall nat
add action=src-nat chain=srcnat ipsec-policy=out,none out-interface=ether1 src-address=192.168.1.0/24 to-addresses=XXX.XXX.XXX.XXX
/ip ipsec identity
add auth-method=digital-signature certificate=CHR.p12 generate-policy=port-strict match-by=certificate mode-config=MT peer=peer1 policy-template-group=group1 remote-certificate=MT.crt
add auth-method=digital-signature certificate=CHR.p12 generate-policy=port-strict match-by=certificate mode-config=macOS peer=peer1 policy-template-group=group1 remote-certificate=macOS.crt remote-id=ignore
/ip ipsec policy
add dst-address=192.168.1.0/24 group=group1 proposal=proposal1 src-address=0.0.0.0/0 template=yes

My attempts to forward only some destination address to the wireguard (which are not available through the primary ISP):
/interface wireguard
add listen-port=13231 name=wireguard1 private-key=""
/interface wireguard peers
add allowed-address=0.0.0.0/0 endpoint-address=engage.cloudflareclient.com endpoint-port=2408 interface=wireguard1 persistent-keepalive=20s public-key=""
/ip address
add address=172.16.0.2 interface=wireguard1 network=172.16.0.2
/interface list member
add interface=wireguard1 list=WAN
/ip firewall address-list
add address=ping.eu list=list30
/routing table
add fib name=cf
/ip firewall mangle
add action=mark-routing chain=prerouting dst-address-list=list30 new-routing-mark=cf passthrough=yes
/ip route
add check-gateway=none distance=10 dst-address=0.0.0.0/0 gateway=wireguard1 pref-src="" routing-table=cf scope=30 target-scope=10

Re: ike2, wireguard, mark-routing, two isp and newbie

Posted: Mon Oct 23, 2023 2:13 am
by Kentzo
Are you sure the address of ping.eu as resolved by the macOS machine matches the address as resolved by the Windows machine and router?

Re: ike2, wireguard, mark-routing, two isp and newbie

Posted: Mon Oct 23, 2023 9:10 am
by ayrsbsdu
Are you sure the address of ping.eu as resolved by the macOS machine matches the address as resolved by the Windows machine and router?

Thank you for reply, i'm sure
3_result.png

Re: ike2, wireguard, mark-routing, two isp and newbie

Posted: Wed Oct 25, 2023 1:58 am
by Kentzo
i'm sure
The screenshot doesn't show how the Router and Windows machine resolve ping.eu…

What is the IPv4 address that macOS machine obtains from its LAN, can you confirm its network configuration (DHCP?) doesn't overlap with the IPsec network?

Re: ike2, wireguard, mark-routing, two isp and newbie

Posted: Wed Oct 25, 2023 9:17 pm
by ayrsbsdu
The screenshot doesn't show how the Router and Windows machine resolve ping.eu…
What is the IPv4 address that macOS machine obtains from its LAN, can you confirm its network configuration (DHCP?) doesn't overlap with the IPsec network?
_
X:\Windows\System32>ping ping.eu

Pinging ping.eu [88.198.46.60] with 32 bytes of data:
Reply from 88.198.46.60: bytes=32 time=63ms TTL=53
Reply from 88.198.46.60: bytes=32 time=63ms TTL=53
Reply from 88.198.46.60: bytes=32 time=63ms TTL=53
Reply from 88.198.46.60: bytes=32 time=62ms TTL=53

Ping statistics for 88.198.46.60:
	Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
	Minimum = 62ms, Maximum = 63ms, Average = 62ms

[macOS:ping ping.eu
PING ping.eu (88.198.46.60): 56 data
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
Request timeout for icmp_seq 5
Request timeout for icmp_seq 6
Request timeout for icmp_seq 7
Request timeout for icmp_seq 8
Request timeout for icmp_seq 9
^C
--- ping.eu ping statistics ---
11 packets transmitted, 0 packets received, 100.0% packet bytes loss

Judging by the trace in the previous screenshot, the packets from macOS 192.168.1.3 are stuck at CHR 192.168.1.1.
The IPv4 address on mac OS can be any on several ISP, but it does not overlap in any way 192.168.0.0/16.
Maybe CHR cannot route ping.eu into the wireguard1.

Re: ike2, wireguard, mark-routing, two isp and newbie

Posted: Wed Oct 25, 2023 9:59 pm
by Kentzo
From your description it appears to me that the very same route works for the Windows machine. I also don't immediately see in the config that RouterOS would treat Windows traffic any different from macOS traffic.

To rule out the firewall, set identity's `notrack-chain` to `prerouting`. With this setting RouterOS will dynamically add rules in /ip/firewall/raw after the connection is established. Make sure you have something like "chain=forward action=accept connection-state=untracked" high enough for this to work.

Would you mind sharing `netstat -rn -f inet` and `ifconfig -a inet` from the macOS machine after the IPsec connection is established?

Re: ike2, wireguard, mark-routing, two isp and newbie

Posted: Sat Oct 28, 2023 8:10 pm
by ayrsbsdu
From your description it appears to me that the very same route works for the Windows machine. I also don't immediately see in the config that RouterOS would treat Windows traffic any different from macOS traffic.

To rule out the firewall, set identity's `notrack-chain` to `prerouting`. With this setting RouterOS will dynamically add rules in /ip/firewall/raw after the connection is established. Make sure you have something like "chain=forward action=accept connection-state=untracked" high enough for this to work.

Would you mind sharing `netstat -rn -f inet` and `ifconfig -a inet` from the macOS machine after the IPsec connection is established?
I also tried to do it on iOS, the problem is the same - everything works through ike2, but sites from the dst-address-list do not load.
Sorry, i cant see "chain=forward action=accept connection-state=untracked" in raw with enabled identity's "notrack-chain".
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384
	options=1203<RXCSUM,TXCSUM,TXSTATUS,SW_TIMESTAMP>
	inet 127.0.0.1 netmask 0xff000000 
gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280
stf0: flags=0<> mtu 1280
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=424<VLAN_MTU,TSO4,CHANNEL_IO>
	inet 192.168.200.128 netmask 0xffffff00 broadcast 192.168.200.255
utun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1380
utun1: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 2000
utun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1000
ipsec0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280
	options=6463<RXCSUM,TXCSUM,TSO4,TSO6,CHANNEL_IO,PARTIAL_CSUM,ZEROINVERT_CSUM>
	inet 192.168.1.3 --> 192.168.1.3 netmask 0xffffffff 

_

Routing tables

Internet:
Destination        Gateway            Flags           Netif Expire
default            link#8             UCSg           ipsec0       
default            192.168.200.2      UGScIg            en0       
17.167.200.72      link#8             UHWIig         ipsec0       
17.248.214.65      link#8             UHWIig         ipsec0       
17.253.39.201      link#8             UHWIig         ipsec0       
17.253.39.202      link#8             UHWIig         ipsec0       
17.253.39.204      link#8             UHWIig         ipsec0       
17.253.39.207      link#8             UHWIig         ipsec0       
17.253.144.10      link#8             UHWIig         ipsec0       
34.104.35.123      link#8             UHWIig         ipsec0       
54.77.160.229      link#8             UHWIig         ipsec0       
<CHR IPv4>         192.168.200.2      UGHS              en0       
127                127.0.0.1          UCS               lo0       
127.0.0.1          127.0.0.1          UH                lo0       
142.250.74.13      link#8             UHW3Ig         ipsec0   3473
142.250.74.67      link#8             UHW3Ig         ipsec0   3473
142.250.147.94     link#8             UHW3Ig         ipsec0   3473
142.250.178.142    link#8             UHW3Ig         ipsec0   3473
146.59.69.202      link#8             UHWIig         ipsec0       
151.101.85.229     link#8             UHWIig         ipsec0       
159.148.147.239    link#8             UHWIig         ipsec0       
169.254            link#4             UCS               en0      !
169.254.255.255    link#4             UHLSW             en0      !
172.217.18.202     link#8             UHW3Ig         ipsec0   3552
172.217.18.206     link#8             UHW3Ig         ipsec0   3473
192.168.1.1        link#8             UHWIig         ipsec0       
192.168.1.3        192.168.1.3        UH             ipsec0       
192.168.200        link#4             UCS               en0      !
192.168.200.2/32   link#4             UCS               en0      !
192.168.200.2      0:50:56:fd:b9:72   UHLWIir           en0    911
192.168.200.128/32 link#4             UCS               en0      !
192.168.200.255    ff:ff:ff:ff:ff:ff  UHLWbI            en0      !
192.229.221.95     link#8             UHWIig         ipsec0       
216.58.207.202     link#8             UHW3Ig         ipsec0   3534
216.58.207.227     link#8             UHWIig         ipsec0       
216.58.213.65      link#8             UHW3Ig         ipsec0   3473
216.58.214.67      link#8             UHWIig         ipsec0       
216.58.214.164     link#8             UHW3Ig         ipsec0   3590
224.0.0/4          link#8             UmCS           ipsec0       
224.0.0/4          link#4             UmCSI             en0      !
224.0.0.251        link#8             UHmW3I         ipsec0   3557
239.255.255.250    1:0:5e:7f:ff:fa    UHmLWI            en0       
239.255.255.250    link#8             UHmW3I         ipsec0   3560
240.0.0.1          link#8             UHWIig         ipsec0       
255.255.255.255/32 link#8             UCS            ipsec0       
255.255.255.255/32 link#4             UCSI              en0      !
255.255.255.255    link#8             UHW3bI         ipsec0   3486

Re: ike2, wireguard, mark-routing, two isp and newbie

Posted: Sun Oct 29, 2023 8:16 am
by Kentzo
Everything appears as expected. At this point I'd start probing with /tool/sniffer and Wireshark to see where the traffic gets dropped.

Are you positive that the Windows machine indeed routes traffic to ping.eu via IPsec connection? It might be possible that it fails just like macOS / iOS, but then successfully routes (leaks) traffic via its normal gateway.