Connection marks not being recognized outbound

marcmerz · Wed Jul 20, 2016 12:06 pm

I am lost.

I configured PCC as per http://wiki.mikrotik.com/wiki/Manual:PCC and this is working fine.

Incoming connections from either WAN interface are marked correctly. The router adress is 10.10.2.253, WAN Uplinks are 10.10.1.1 (ether1-gateway) and 10.10.2.1 (ether5-gateway) and the VPN Server is 10.10.10.1.

When i try to establish a VPN connection via WAN2 (ether5-gateway), the connection is marked with WAN2_conn (but the answer back from the VPN server goes out to ether1-gateway. I have hidden the external IP from the VPN Client:

https://www.dropbox.com/s/yb1yvztjg54cu ... 1.jpg?dl=0

Firewall Connections

The same connection via WAN1 runs perfect.

My config is below:

[admin@MikroTik RB3011] > /ip firewall mangle print
Flags: X - disabled, I - invalid, D - dynamic
 0  D ;;; special dummy rule to show fasttrack counters
      chain=prerouting

 1  D ;;; special dummy rule to show fasttrack counters
      chain=forward

 2  D ;;; special dummy rule to show fasttrack counters
      chain=postrouting

 3 XI  chain=prerouting action=mark-connection new-connection-mark=no-mark passthrough=yes connection-state=new log=no log-prefix=""

 4    chain=prerouting action=accept dst-address=10.10.1.0/24 in-interface=LAG1 log=no log-prefix=""

 5    chain=prerouting action=accept dst-address=10.10.2.0/24 in-interface=LAG1 log=no log-prefix=""

 6    chain=prerouting action=mark-connection new-connection-mark=WAN1_conn passthrough=yes in-interface=ether1-gateway connection-mark=no-mark log=no log-prefix=""

 7    chain=prerouting action=mark-connection new-connection-mark=WAN2_conn passthrough=yes in-interface=ether5-gateway connection-mark=no-mark log=no log-prefix=""

 8    chain=prerouting action=mark-connection new-connection-mark=WAN1_conn passthrough=yes dst-address-type=!local in-interface=LAG1 connection-mark=no-mark per-connection-classifier=both-addresses:2/0 log=no log-prefix=""

 9    chain=prerouting action=mark-connection new-connection-mark=WAN2_conn passthrough=yes dst-address-type=!local in-interface=LAG1 connection-mark=no-mark per-connection-classifier=both-addresses:2/1 log=no log-prefix=""

10    ;;; Mark Connection to dnsomatic for WAN_2
      chain=prerouting action=mark-connection new-connection-mark=WAN2_conn passthrough=yes dst-address-type=!local dst-address-list=host_dnsomatic log=no log-prefix=""

11    chain=prerouting action=mark-routing new-routing-mark=to_WAN1 passthrough=yes in-interface=LAG1 connection-mark=WAN1_conn log=no log-prefix=""

12    chain=prerouting action=mark-routing new-routing-mark=to_WAN2 passthrough=yes in-interface=LAG1 connection-mark=WAN2_conn log=no log-prefix=""

13    chain=output action=mark-routing new-routing-mark=to_WAN1 passthrough=yes connection-mark=WAN1_conn log=no log-prefix=""

14    chain=output action=mark-routing new-routing-mark=to_WAN2 passthrough=yes connection-mark=WAN2_conn log=no log-prefix=""

[admin@MikroTik RB3011] > /ip firewall nat print
Flags: X - disabled, I - invalid, D - dynamic
 0    ;;; defconf: masquerade
      chain=srcnat action=masquerade out-interface=ether1-gateway log=no log-prefix=""

 1    chain=srcnat action=masquerade out-interface=ether5-gateway log=no log-prefix=""

 2    chain=dstnat action=dst-nat to-addresses=10.10.10.1 protocol=udp in-interface=ether1-gateway dst-port=500,4500 log=no log-prefix=""

 3    chain=dstnat action=dst-nat to-addresses=10.10.10.1 protocol=ipsec-esp in-interface=ether1-gateway log=no log-prefix=""

 4    chain=dstnat action=dst-nat to-addresses=10.10.10.1 protocol=udp in-interface=ether5-gateway dst-port=500,4500 log=no log-prefix=""

 5    chain=dstnat action=dst-nat to-addresses=10.10.10.1 protocol=ipsec-esp in-interface=ether5-gateway log=no log-prefix=""

[admin@MikroTik RB3011] > /ip route print detail
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 0 A S  dst-address=0.0.0.0/0 gateway=10.10.1.1 gateway-status=10.10.1.1 reachable via  ether1-gateway check-gateway=ping distance=1 scope=30 target-scope=10 routing-mark=to_WAN1

 1 A S  dst-address=0.0.0.0/0 gateway=10.10.2.1 gateway-status=10.10.2.1 reachable via  ether5-gateway check-gateway=ping distance=1 scope=30 target-scope=10 routing-mark=to_WAN2

 2 A S  dst-address=0.0.0.0/0 gateway=10.10.1.1 gateway-status=10.10.1.1 reachable via  ether1-gateway check-gateway=ping distance=1 scope=30 target-scope=10

 3   S  dst-address=0.0.0.0/0 gateway=10.10.2.1 gateway-status=10.10.2.1 reachable via  ether5-gateway check-gateway=ping distance=2 scope=30 target-scope=10

I would really appreciate if somebody had an idea of what is wrong with the config above.

Many thanks in advance.

magchiel · Wed Jul 20, 2016 12:54 pm

I think you also want to mark on the input chain, not just the prerouting.
Here are my mangle rules which show IPsec connections tagged as wanx->cpu.

/ip firewall mangle
add action=mark-connection chain=input comment=WAN->CPU connection-mark=no-mark in-interface=wan1 new-connection-mark=wan1->cpu
add action=mark-connection chain=input connection-mark=no-mark in-interface=wan2 new-connection-mark=wan2->cpu
add action=mark-routing chain=output connection-mark=wan1->cpu new-routing-mark=wan1_route
add action=mark-routing chain=output connection-mark=wan2->cpu new-routing-mark=wan2_route

marcmerz · Wed Jul 20, 2016 1:52 pm

Thanks for your answer. The incoming connection is marked with WAN2_conn, it is just that the answer back is going to WAN1 regardless.

I have added your rules on the input chain but it does not make a difference.

Furthermore i have observed that i had to activate rule 3 in;

[admin@MikroTik RB3011] > /ip firewall mangle print
Flags: X - disabled, I - invalid, D - dynamic
 0  D ;;; special dummy rule to show fasttrack counters
      chain=prerouting

 1  D ;;; special dummy rule to show fasttrack counters
      chain=forward

 2  D ;;; special dummy rule to show fasttrack counters
      chain=postrouting

 3    chain=prerouting action=mark-connection new-connection-mark=no-mark passthrough=yes connection-state=new log=yes log-prefix="

I read that all connections should be marked with no-mark by default but without rule 3 from above, it seems they are not.

I think that changes on the prerouting chain does cover the input chain as well but my knowledge is far from perfect. Inserting No. 4 and 5 does not make any difference:

[admin@MikroTik RB3011] > /ip firewall mangle print
Flags: X - disabled, I - invalid, D - dynamic
 0  D ;;; special dummy rule to show fasttrack counters
      chain=prerouting

 1  D ;;; special dummy rule to show fasttrack counters
      chain=forward

 2  D ;;; special dummy rule to show fasttrack counters
      chain=postrouting

 3    chain=prerouting action=mark-connection new-connection-mark=no-mark passthrough=yes connection-state=new log=yes log-prefix=""

 4    chain=input action=mark-connection new-connection-mark=WAN1_conn passthrough=yes in-interface=ether1-gateway connection-mark=no-mark log=no log-prefix=""

 5    chain=input action=mark-connection new-connection-mark=WAN2_conn passthrough=yes in-interface=ether5-gateway connection-mark=no-mark log=no log-prefix=""

 6    chain=prerouting action=accept dst-address=10.10.1.0/24 in-interface=LAG1 log=no log-prefix=""

 7    chain=prerouting action=accept dst-address=10.10.2.0/24 in-interface=LAG1 log=no log-prefix=""

 8    chain=prerouting action=mark-connection new-connection-mark=WAN1_conn passthrough=yes in-interface=ether1-gateway connection-mark=no-mark log=no log-prefix=""

 9    chain=prerouting action=mark-connection new-connection-mark=WAN2_conn passthrough=yes in-interface=ether5-gateway connection-mark=no-mark log=yes log-prefix=""

10    chain=prerouting action=mark-connection new-connection-mark=WAN1_conn passthrough=yes dst-address-type=!local in-interface=LAG1 connection-mark=no-mark per-connection-classifier=both-addresses:2/0 log=no log-prefix=""

11    chain=prerouting action=mark-connection new-connection-mark=WAN2_conn passthrough=yes dst-address-type=!local in-interface=LAG1 connection-mark=no-mark per-connection-classifier=both-addresses:2/1 log=no log-prefix=""

12    ;;; Mark Connection to dnsomatic for WAN_2
      chain=prerouting action=mark-connection new-connection-mark=WAN2_conn passthrough=yes dst-address-type=!local dst-address-list=host_dnsomatic log=no log-prefix=""

13    chain=prerouting action=mark-routing new-routing-mark=to_WAN1 passthrough=yes in-interface=LAG1 connection-mark=WAN1_conn log=no log-prefix=""

14    chain=prerouting action=mark-routing new-routing-mark=to_WAN2 passthrough=yes in-interface=LAG1 connection-mark=WAN2_conn log=no log-prefix=""

15    chain=output action=mark-routing new-routing-mark=to_WAN1 passthrough=yes connection-mark=WAN1_conn log=no log-prefix=""

16    chain=output action=mark-routing new-routing-mark=to_WAN2 passthrough=yes connection-mark=WAN2_conn log=no log-prefix=""

Thanks anyway.

marcmerz · Thu Jul 21, 2016 4:09 pm

Btw.: As soon as i enter a route for the VPN client to the gateway 10.10.2.1 (ether5-gateway), it does work instantly. Obviously i can not connect with the same client (IP) via WAN1 because of that route and the client has a dynamic IP. The point is, it is working with that route and it should work as the connection is marked.

Is there a way i can keep track of routing marks as well beside connection marks? packet sniffer does not show this.

TIA

Thu Jul 21, 2016 8:11 pm

(I just glossed over all of this, and I assume that the Mikrotik itself is terminating the VPN connections, not some internal device, right?)

If the above is true, then you need to put route marking rules in your OUTPUT chain of the mangle table in addition to the ones in the prerouting chain.
(you don't need to duplicate the connection mark rules, just the mark-routing rules)

This is because when the Mikrotik itself generates a packet to be sent somewhere, this packet goes through the output chain instead of the prerouting chain.

Basically, mangle can take these paths:
prerouting -> input
prerouting -> forward -> postrouting
output -> postrouting

marcmerz · Thu Jul 21, 2016 8:57 pm

Hmm, i already have routing marks in the output chain as well as in the prerouting chain (11-14 from above):

11    chain=prerouting action=mark-routing new-routing-mark=to_WAN1 passthrough=yes in-interface=LAG1 connection-mark=WAN1_conn log=no log-prefix=""

12    chain=prerouting action=mark-routing new-routing-mark=to_WAN2 passthrough=yes in-interface=LAG1 connection-mark=WAN2_conn log=no log-prefix=""

13    chain=output action=mark-routing new-routing-mark=to_WAN1 passthrough=yes connection-mark=WAN1_conn log=no log-prefix=""

14    chain=output action=mark-routing new-routing-mark=to_WAN2 passthrough=yes connection-mark=WAN2_conn log=no log-prefix=""

I do a port forward to the Mikrotik from my DSL Modem and the Mikrotik then does a port forward via DstNAT to the VPN Server. I also tried a Port Forward directly from the DSL modem to the VPN Server but then even the incoming connection did not get marked at all. This does work for VPN connections to WAN1 though as WAN1 is the default route from the MT.
The infrastructure is 2 DSL Modem and one router:
DSL Modem WAN1 10.10.1.1 => MT RB3011 10.10.1.253 => VPN Server 10.10.10.1 (via ether1-gateway on the WAN side and LAG1 (bonding ether2 and ether3 on the LAN side) which gets marked with WAN1_conn and works perfect.
DSL Modem WAN2 10.10.2.1 => MT RB3011 10.10.2.253 => VPN Server 10.10.10.1 (via ether5-gateway on the WAN side and LAG1 (bonding ether2 and ether3 on the LAN side) which gets marked with WAN2_conn and the answer back from the VPN server => MT => 10.10.2.1 still goes via WAN1 i.e. ether1-gateway.

I have no idea why this rather simple configuration fails in my case.

marcmerz · Sun Jul 24, 2016 12:13 pm

It seems the behavior I see is rather unique...

marcmerz · Sat Aug 06, 2016 12:55 pm

Little update with no real solution (yet):

By coincidence i noticed, that when i leave the In-Interface in

11    chain=prerouting action=mark-routing new-routing-mark=to_WAN1 passthrough=yes in-interface=LAG1 connection-mark=WAN1_conn log=no log-prefix=""

12    chain=prerouting action=mark-routing new-routing-mark=to_WAN2 passthrough=yes in-interface=LAG1 connection-mark=WAN2_conn log=no log-prefix=""

i.e.

11    chain=prerouting action=mark-routing new-routing-mark=to_WAN1 passthrough=yes connection-mark=WAN1_conn log=no log-prefix=""

12    chain=prerouting action=mark-routing new-routing-mark=to_WAN2 passthrough=yes connection-mark=WAN2_conn log=no log-prefix=""

it does work but then i can not connect to my dsl modems (10.10.1.1 and 10.10.2.1) anymore.

The problem seems to be, that i use a couple of vlan's and then the In-Interface is physically LAG1 but logically it is vlan10 (for example).

Oh well...

Connection marks not being recognized outbound

Connection marks not being recognized outbound

Re: Connection marks not being recognized outbound

Re: Connection marks not being recognized outbound

Re: Connection marks not being recognized outbound

Re: Connection marks not being recognized outbound

Re: Connection marks not being recognized outbound

Re: Connection marks not being recognized outbound

Re: Connection marks not being recognized outbound