I followed the wiki at http://wiki.mikrotik.com/wiki/PCC to set up loadbalancing of two internet connections. Because of an ipsec tunnel I added the following:
(1.2.3.0/24 is a subnet including the ipsec remote peer 1.2.3.4. 172.29.5.0/24 is the ipsec remote local subnet)

I suspected it didn't work (ipsec was unstable), so I did some testing. I deleted the two routes to 0.0.0.0/0 without routing-mark and added the following:
Now, when I ping 1.2.3.3 or 1.2.3.4 (or try to establish ipsec from my local side to 1.2.3.4) I see in the log quite frequently that the log display the package will be using ISP2 as out interface, with ISP1 as source address! I think that explains why the ipsec is so unstable. (Further logging show that ipsec packets go out on IPS2 interface, and returning packets come on ISP1 interface.)
And the fact that anything shows up in the log at all proves that the routing-mark is correct.

Then I run the following commands: In my understanding of routing-mark, these two rules should not make any difference at all, as there are other routing rules with lower distance that matches the routing-mark. But they do make a difference. After these two commands are entered the packets to the ipsec peer is sent on the ISP2 interface 100% of the times, except when establishing the tunnel. (For some reason tunnel is established. But all packets thereafter are sent on the wrong interface.) And ping are always sent on the wrong interface.

But if I add the route everything works fine with the ipsec. But still most of the packages to the internet are using ISP2.


So, my question. Is it me doing something completely wrong here, or is routing-mark frequently ignored in the routing table?

The real setup where I experienced the issue was on a RB450G with ROS4.11
The issue is reproduced on a RB751G with ROS 5.11, which I used for all testing/debug. To make sure I didn't fail on translating the addresses in the commands from the wiki, I setup a complete test environment using the exact same addresses as the wiki, using two more rb751g as ISP* routers, and even two more to create a "internet" with an remote ipsec peer.

A litle bit of additional information:
This logging catches the ipsec packets with wrong out-interface: While these two does not: (rules moved to top of list)

SRCNAT is, as far as I know, the last place in the router one can see the packets before they leave. I find it strange that the packets doesn't show up there.

Nobody has experience with routing-mark?

Hi petterg,

I am not sure if my issue is related to the same problem, but I had also problem with marking IPsec packets
and I have found reason/solution/bug ??.

I do not want to paste here all my config, so I will try explain just the problem the best way I can.

I have dual WAN config (no BGP) (with additional scripts for WAN fail-over switching, but it does not matter).

I have mangle rules, based on this site:
https://aacable.wordpress.com/2011/07/2 ... t-by-zaib/

just short of few ones:

/ip firewall mangle
add chain=input in-interface=WAN1 action=mark-connection new-connection-mark=WAN1_conn
add chain=input in-interface=WAN2 action=mark-connection new-connection-mark=WAN2_conn

In my case I have chain=prerouting here, so all incomming connections to the router (both router itself and destined for my local subnets) are connection-marked.

I have VPNs connections with remote locations,
(GRE tunnels working over IPsec).
The GRE tunnels work on local IP's (10.255.255.10/32 - central office, 10.255.255.20/32 - remote office).
The IPsec policies are set up only for those IP's so they are encapsulated and encrypted with IPsec.

I have two GRE tunnels to every remote location (one working on our WAN1 and second working on our WAN2).

What I have discovered, that outgoing IPsec packets that are marked to WAN2 are not really outgoing with correct interface.
(they were outgoing throught WAN1 (default)).
As both our ISP's does not filter our source addresses, we can use IP address assigned by ISP1 and packet can succesfully go out using ISP2 interface and vice-versa. So I was not even conscious, that bad thing is happening..

After 2 day hard investigation, log analisys, I have found the reason !!

All decrypted packets that comes from IPsec tunnel (those one 10.255... on which GRE tunnels are working) in firewall acts with incomming interface = physical interface that IPsec packet was received.
That is actually nothing new for me, I knewed it before and this is probably OK.
This way, according to mangle rules from above, the GRE connections (with local/private IPs) are also connection-marked,
and causes that outgoing GRE packet is correctly marked for correct outgoing WAN interface.

But it completely does not matter, as the outgoing GRE packet is catched by IPsec polisy, encrypted, encapsulated, and whole process starts again. http://wiki.mikrotik.com/images/thumb/2 ... _final.png

and.. here the problem beggins.. !
Now, the outgoing IPsec packet should be also marked and routed via the correct wan interface, but this is not working.
Either route-marking rule is not working or the routing mark is ignored during the routing.

and.. now the solution/reason ! This problem with marking/routing of IPsec packets is casued by previously route-marking of packet that was encapsulated into IPsec !!!!!! (GRE packets in my case)

I have completely no idea.. what is the relation/reason of route-marking packets that comes into IPsec tunnel, with marking IPsec packets itself ???

Solution was very simple, I have updated the above rules with destination IP subnets of WAN links,
so now only real packets coming into WAN interfaces destinet to public IP's are connection marked.
(GRE packets are not marked now).

I have made several tests, not only with GRE but with simple ICMP ping over those private IP's used by GRE.
And I can confirm: every packet that is marked for routing but finally encapsulated into IPsec packet, cause that route-marking for this IPsec packet is not working.

Please somebody guys from MikroTik, confirm if this is a bug or I do not know something ?
(checked on ROS 6.18 and 6.20)

It's 2023 and ROS v7.9, the same thing, still undocumented.

I've resorted to just resetting the routing mark as the first rule in mangle output chain:

Look at the packet flow diagram and specifically ispec example
https://help.mikrotik.com/docs/display/ ... n+RouterOS

If you want to reroute ipsec packets then you have to do routing marks after the ipsec encapsulation. Ipsec doesn't care about any prior marks or prior routing decisions. It encapsulates and sends packet according to its policies.

If you want to reroute incoming decrypted packets then same thing applies but in reverse order, marking must be done after the decryption process.

What is the difference between IPsec tunnel mode and IPsec transport mode?

IPsec tunnel mode is used between two dedicated routers, with each router acting as one end of a virtual "tunnel" through a public network. In IPsec tunnel mode, the original IP header containing the final destination of the packet is encrypted, in addition to the packet payload. To tell intermediary routers where to forward the packets, IPsec adds a new IP header. At each end of the tunnel, the routers decrypt the IP headers to deliver the packets to their destinations.

In transport mode, the payload of each packet is encrypted, but the original IP header is not. Intermediary routers are thus able to view the final destination of each packet — unless a separate tunneling protocol (such as GRE) is used.

Does the posting above means that RouterOS is always using a tunnel? In transport mode only the payload is encrypted so normal routing is used.

Source: https://www.cloudflare.com/learning/net ... -is-ipsec/

It is still ipsec encapsulation/decapsulation and after the encap/decap packet is not the same.