Hi,

I have a RB with two DSL connections. IP addresses as follows

WAN1: 10.1.1.1/30 gw 10.1.1.2 - ether4
WAN2: 10.1.1.5/30 gw 10.1.1.6 - ether5

Each DSL connection will route packets only for the IP assigned so I need to make sure packets leave on the interface they come in on.

I have applied connection tracking and added several mangle rules:

Mark connection to 10.1.1.1 as dsl1 in prerouting
Mark connection to 10.1.1.5 as dsl2 in prerouting

I then apply packet marks based on connection mark

If connection mark dsl1 mark packet as dsl1 in prerouting
If connection mark dsl2 mark packet as dsl2 in prerouting
If connection mark dsl1 mark packet as dsl1 in output
If connection mark dsl2 mark packet as dsl2 in output

I then apply route marks based on packet marks

If packet mark dsl1 mark route as dsl1 in preouting
If packet mark dsl2 mark route as dsl2 in prerouting

If I apply logs in the filter I can see the marks being applied correctly for all packets

In the routing table I have

0.0.0.0/0 gw 10.1.1.2
0.0.0.0/0 route mark dsl1 10.1.1.2
0.0.0.0/0 route mark dsl2 10.1.1.6

I then add two rules

route mark dsl1 lookup dsl1
route mark dsl2 lookup dsl2

Regardless of what I do all packets leave by the 10.1.1.2 interface even though I see the correct marks in place.

My test is to ping 10.1.1.1 and 10.1.1.5 and always the traffic leaves from the correct source ip but always on ether4. I also try ssh in case the issue was the connectionless nature of icmp but that does not resolve the issue.

Can anyone see a solution?

Also, does the order of routes in the table matter? If so I cannot see any way to change the order of routes. My expectation is that it does matter but even if I have the dsl4 mark route before the default gateway with no mark the same result happens.

Any ideas? All suggestions welcome.

I have just repeated this configuration on a fresh RB150 with system default and latest v3.13 RouterOS firmware. I simply cannot get the packets coming in on ether4 to leave on ether4. At present I have default gateway on ether5 and regardless of what I do the packets leave by ether5.

I have firewall rules in place to log all traffic and I can see inbound traffic with connection marks, packet marks and route marks. The mangle rule packet count also shows the mark for each of connection, packet and route applied.

If I add a firewall rule to log all traffic on output chain then the packet marks, connection marks and route marks are not visible. It appears the mangle rules in the prerouting chain do not get applied to the packets leaving the router itself. Thus ping or ssh packets on input chain are marked correctly and ping or ssh packets on output chain that are replies/related to the incoming connection are not marked and thus (I assume) not routed using the relevant policy route.

I have asked this question before and I have applied the responses that were kindly presented. In fact I have a setup very close to this post

http://forum.mikrotik.com/viewtopic.php ... ltiple+isp

However I simply cannot get it to work and I cannot see why.

Looking at the packet flow diagram it appears to me that the route marks need to be applied at the output chain after egress from output process. However at this point they do not appear to have the connection mark in place and thus I cannot mark based on connection.

I then tried to mark based on the source address using the address of the RB on ether4. I marked packets in the output chain and this did indeed mark the egress packets from local process. I then marked the routing based on packet marks again in the output chain. This appeared to route mark the packets. Looking at firewall rules added to the output chain the packet and route marks were indeed on the reply packets now.

I then applied the policy route using the route mark and clearly selected a gateway on ether4. The packets still leave by the default gateway and if I disable the default gateway I loose all connection from ether4 despite having a separate policy route table.

Clearly I am doing something stupid but I cannot see what. I also am quite frustrated that RouterOS makes it so hard to ensure something that should in my opinion be the default behaviour that packets leave on the interface they came in on unless routed otherwise - exactly as Linux does by default.

Can anyone save my sanity?

the only input i have is this: http://forum.mikrotik.com/viewtopic.php ... 50#p106550

Thanks for the reply. This is driving me mad! It seems crazy to take so much effort to achieve something so simple. Having done all the hard work for quite complex bonded tunnels I cannot solve the simple things!!

I have read your post numerous times and I have made some changes but still cannot get it to work. I have placed a firewall rule in output chain just to log and the packets do indeed have the route mark in place on egress. I have found that it is not enough to route mark in pre-routing, the route marks apply only to the packets if the mangle is also done in the output chain. Your post suggests placing a mangle in the input chain but the router I am using (RB150 using v3.13) allows mangle only in pre-routing and output as also suggested by the packet flow diagram.

The problem is that despite the route marks and the policy routing the packets still leave by the default gateway (dsl1). If I disable the default gateway the packets do not leave on the correct interface despite a policy route in place.

Any chance you could have a look at my config below and suggest if I have done anything stupid or missed the obvious. Another pair of eyes would be very welcome .... and help my sanity!

From route table I can see that the order of the routes may be the issue - default gateway before the route mark rule. I assumed the routing rules would be applied first then "fall through" to the main table. This may be the mistake but I cannot see how to change this. I am not sure how to re-order routes. However if this was the case I assumed that if I disable the default gateway then the route mark route should be used but that does not work.

I think the issue is with the use of the route rule or the table lookup. It seems that the routing table for out-dsl4 is not being used. I cannot find any way to debug this to see if the routing table is being hit. At this point it would be great to have a packet flow trace on the RB

/ip firewall mangle
add action=mark-connection chain=prerouting comment="" disabled=no \
in-interface=ether4 new-connection-mark=in-dsl4 passthrough=yes
add action=mark-packet chain=prerouting comment="" connection-mark=in-dsl4 \
disabled=no new-packet-mark=in-dsl4 passthrough=yes
add action=mark-routing chain=prerouting comment="" disabled=no \
new-routing-mark=out-dsl4 packet-mark=in-dsl4 passthrough=yes
add action=mark-packet chain=input comment="" connection-mark=in-dsl4 \
disabled=no new-packet-mark=in-dsl4 passthrough=yes
add action=mark-packet chain=output comment="" disabled=no new-packet-mark=\
"pm dsl4-out" packet-mark=in-dsl4 passthrough=yes
add action=mark-routing chain=output comment="" connection-mark=in-dsl4 \
disabled=no new-routing-mark=out-dsl4 passthrough=yes

/ip route
add comment="" disabled=no distance=2 dst-address=0.0.0.0/0 gateway=\
81.187.102.18 scope=255 target-scope=10
add comment="" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
81.187.102.30 routing-mark=out-dsl4 scope=255 target-scope=10
add comment="" disabled=no distance=1 dst-address=81.187.102.67/32 gateway=\
81.187.102.22 scope=30 target-scope=10
add comment="" disabled=no distance=1 dst-address=81.187.102.68/32 gateway=\
81.187.102.26 scope=30 target-scope=10
/ip route rule
add action=lookup comment="" disabled=no routing-mark=out-dsl4 table=out-dsl4

How are you checking if your policy routing works?

If its only by pinging from router itself, then it might be because your output chain rules are incorrect:
You only mark with routing mark packets that are marked with connection mark 'in-dsl4', but there is no rule that would set this in output chain.

I am checking by pinging from a remote Linux box. I have added several firewall rules to log traffic based upon the connection mark and the route mark. I have added the following rules

/ip firewall filter
add action=log chain=output comment="" connection-mark=in-dsl4 disabled=no \
log-prefix="cm in-dsl4"
add action=log chain=output comment="" disabled=no log-prefix="rm out-dsl4" \
routing-mark=out-dsl4
add action=log chain=output comment="" disabled=no log-prefix="icmp out" \
protocol=icmp

The log then clearly shows the packet in the output chain with both a connection mark and a route mark. However the third rule logs the icmp traffic and shows the output interface as ether5 and not ether4 as would have been selected if the policy route had been applied.

Since the tests I have now added a rule to the output chain for router generated traffic

/ip firewall mangle
add action=mark-routing chain=output comment="" disabled=no new-routing-mark=\
out-dsl4 passthrough=no src-address=81.187.102.29

I had expected this to tag traffic from the dsl4 IP on the router with the route mark out-dsl4. The log and rules listed above do indeed show this is the case. Ping from the router with src address set as .29 does indeed get the route mark. However it also leaves on ether5 despite the policy rule.

From this my only conclusion is that the policy route is not being applied. From what I can see packets generated on the router either established from the router or related to an inbound connection do not get routed according to the policy routes.

I am very frustrated by this as I am used to linux doing this by default - packets leave on the interface according to their source IP address by default. This seems to be causing a considerable amount of work for what should be a very simple issue. I have done an incredible amount of work on tunnels, bonding, route systems scripts etc. and this last stage of my project is causing me to re-consider the use of RB OS and hardware. Given the hardware and OS is otherwise perfect this is very frustrating - not to mention potentially a lot of time down the drain!

I have searched the forums extensively and I can see this issue has come up many times and there have been many variations on the resolution suggested. Clearly this is an issue in need of a definitive solution and a wiki entry. If I can get get it solved I will document this in the wiki. The problem is I have never had a solution that has been 100% reliable and repeatable. I am fully prepared to pay for support on this issue if I knew how to go about this!

Any comments would be welcome.

I have just carried out some more testing. I deleted the rules for routing and rebuilt them (using CLI) and ended up with the route table as follows:

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=81.187.102.30 routing-mark=out-dsl4 scope=30 \
target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=81.187.102.18 scope=30 target-scope=10
add comment="" disabled=no distance=1 dst-address=81.187.102.66/32 gateway=81.187.102.18 scope=30 \
target-scope=10
add comment="" disabled=no distance=1 dst-address=81.187.102.67/32 gateway=81.187.102.22 scope=30 \
target-scope=10
add comment="" disabled=no distance=1 dst-address=81.187.102.68/32 gateway=81.187.102.26 scope=30 \
target-scope=10
/ip route rule
add action=lookup comment="" disabled=no routing-mark=out-dsl4 table=out-dsl4

I have made sure the policy route rule appears before the default gateway thinking the order mattered.

After a test from external source pinging 81.187.102.29 this is what the log showed

21:34:58 firewall,info rm out-dsl4 output: in:(none) out:ether5, proto ICMP (type 0, code 0), 81.187.102.29-
>90.209.87.201, len 64
21:34:58 firewall,info icmp out output: in:(none) out:ether5, proto ICMP (type 0, code 0), 81.187.102.29->90
.209.87.201, len 64

Still the reply packets leave by ether5 and not ether4. As I have control the DSL gateways I have not enabled IP address rejection so the routers will route packets not from their subnet. If I was to disable this then the system breaks completely.

I am simply at a loss as to why this is happening. I have looked at the documentation yet again for ip route and have followed the example. It appears the router does not do what the documentation says should work.

I do not know what else to try. Any comments would be appreciated.

I think I have found the problem .... and it is interesting!

The log file does not tell the truth. The log file shows egress on ether5. I placed a packet sniffer on ether5 and no ping traffic from my linux host. I placed a sniffer on ether4 and the ping traffic was egress on ether4.

Tomorrow when I have access to physical routers I will install a packet monitor on the switch between the router and the DSL router and dump the exact packet data.

At this stage I have studied the packet flow and I think the following happens.

The RB generates the packet and then makes initial routing decision based on default gateway.
The mangle then changes the packet and this triggers a second pass of the route table.
The log then logs the packet now mangled but still on original egress interface
The second pass of route table applies policy route and changes egress interface
The packet passes to output interface and is despatched.

Based on this sequence of steps the packet would log with ether5 but would actually egress on ether4. This is indeed what seems to be happening.

This leads to several interesting conclusions.

The policy route is applied on a second pass of the route table. This means that if a route does not exist in the first pass the router cannot route the packet. This does indeed happen. Disable the default gateway and despite the policy route the router cannot select a route to host.

The first pass of the route table selects the route almost like a proxy. This can be replaced by policy routing during a second pass after the mangle stage. After initial route selection the egress interface is selected. After initial route selection the packet enters the filters. Thus the filters log and filter on the basis of the initial route decision which is not necessarily the final packet direction.

A mangle and subsequent pass of the route table can change the egress interface. This is not reflected in the logs. The log will reflect the state at the initial route and not the final route.

This means that the filter stage can be confusing. If logging or filtering is applied based on egress interface then it may be changed in mangle and thus not reflect what is expected.

I have now completed some more tests and found some interesting results in the process of making sure packets leave by te interface they come in on.

I have reduced this to one rule and one route statement. In mangle add a rule to the output chain to route mark the packet based on the src ip address of the interface on the routerboard that you want to select. Add a route statement to the route table to route based on the routing mark.

This works perfectly though I have not exhaustively tested this. Packets leave on the interface they come in on and it needs only one mangle and one route. There is no need to mark connections, mark packets or mark anything on the input or pre-routing chains. I will test this comprehensively and provide feedback.

In general you are right - this behaviour for locally generated packets is discussed in other policy routing related topics, e.g. here:

http://forum.mikrotik.com/viewtopic.php?f=2&t=27151

Im not sure if you've done this or not, but in your alternate routing tables are you re-creating the DAC routes that show up in the main table? If the router doesnt even know it's own IP/subnet in the alternate table of course it has to fall back to the main table.

You need to duplicate your LAN and WAN subnets into your alternate routing tables. Make the gateway the routers IP in that subnet.

For instance, if my LAN is 192.168.1.1/24, and that route shows up as DAC in the main table dynamically - I also need to enter a static route for 192.168.1.0/24 into that alternate table and point it to 192.168.1.1.

I had rebuilt the dynamic table in the routing table to ensure access to local subnets.

I seem to have solved most of the issues now. The log shows the packet egress interface before mangle and route policy changes the interface. Torch confirms the packet actually leaves on the interface specified by the policy route. As long as I remember not to be too dependent on the log interface details when policy route is applied.