Strange RP filter behavior

611 · Sun May 12, 2019 4:12 pm

I've been testing "mangling loopback" (known workaround for dstnat not available in output chain + no cDNS + no non-standard winbox port in Dude in ROS v6) configuration on a metarouter (as I wanted a config as generic as possible). Metarouter is running on RB2011, ROS 6.44.3.

The config is a typical implementation of conditional DNS for v6:

a bridge to bind extra loopback address to (bridge-loopback, 127.0.0.2);
two IPIP tunnels looking at each other (ip2t-loopback-out and ip2t-loopback-in);
routes to these tunnels - one with routing mark (rm-loopback), another with fake address (10.53.0.0/16);
L7 pattern (l7-mr1 with 'mr1.test' content) and mangling rule to mark specific DNS requests with said routing mark (log-prefix=mangle1);
three NAT rules:
1. replacing our address with fake one (log-prefix=mangle2);
2. replacing destination address with required one (log-prefix=mangle3);
3. masquerading fake address (log-prefix=mangle4);

plus some logging rules in filter:

/interface bridge
add name=bridge-loopback protocol-mode=none
/interface ethernet
set [ find ] name=ether-intra
set [ find ] name=ether-mgmt
/interface ipip
add local-address=127.0.0.2 name=ip2t-loopback-in remote-address=127.0.0.1
add local-address=127.0.0.1 name=ip2t-loopback-out remote-address=127.0.0.2
/ip firewall layer7-protocol
add name=l7-mr1 regexp=mr1.test
/ip settings
set rp-filter=loose
/ip address
add address=192.168.10.30/24 interface=ether-intra network=192.168.10.0
add address=127.0.0.2 interface=bridge-loopback network=127.0.0.2
/ip dhcp-client
add default-route-distance=100 dhcp-options=hostname,clientid disabled=no \
    interface=ether-mgmt
/ip firewall filter
add action=log chain=forward log=yes log-prefix=packet
add action=log chain=input in-interface=!ether-mgmt log=yes log-prefix=packet \
    protocol=!ipencap
add action=log chain=output log=yes log-prefix=packet out-interface=\
    !ether-mgmt protocol=!ipencap
/ip firewall mangle
add action=mark-routing chain=output dst-port=53 layer7-protocol=l7-mr1 log=\
    yes log-prefix=mangle1 new-routing-mark=rm-loopback protocol=udp
/ip firewall nat
add action=src-nat chain=srcnat dst-port=53 log=yes log-prefix=mangle2 \
    out-interface=ip2t-loopback-out protocol=udp to-addresses=10.53.0.1
add action=dst-nat chain=dstnat dst-port=53 in-interface=ip2t-loopback-in \
    log=yes log-prefix=mangle3 protocol=udp to-addresses=192.168.10.20
add action=masquerade chain=srcnat dst-port=53 log=yes log-prefix=mangle4 \
    out-interface=ether-intra protocol=udp src-address=10.53.0.0/16
/ip route
add distance=1 gateway=ip2t-loopback-out routing-mark=rm-loopback
add distance=1 dst-address=10.53.0.0/16 gateway=ip2t-loopback-in

The environment is a:

intranet between two metarouters (ether-intra), with another metarouter at 192.168.10.20 with external DNS queries enabled and a static entry mr1.test;
management network (ether-mgmt) to keep intranet clean. It has a DHCP server with DNS option enabled, so DNS requests without cDNS would go there (to 192.168.20.1).

When put [resolve mr1.test] is invoked in console (AFAIK it's the only way to nslookup on Mikrotik), the request goes to 192.168.10.20 which replies with it's own address.

But if RP Filer is set to loose, the reply seems to get dropped before reaching input chain:

[admin@mr2] > put [resolve mr1.test]
failure: dns server failure

Log:

05:38:41 firewall,info mangle1 output: in:(unknown 0) out:ether-mgmt, proto UDP, 192.168.20.253:50883->192.168.20.1:53, len 54 
05:38:41 firewall,info mangle2 srcnat: in:(unknown 0) out:ip2t-loopback-out, proto UDP, 192.168.20.253:50883->192.168.20.1:53, len 54 
05:38:41 firewall,info mangle3 dstnat: in:ip2t-loopback-in out:(unknown 0), proto UDP, 10.53.0.1:50883->192.168.20.1:53, len 54 
05:38:41 firewall,info packet forward: in:ip2t-loopback-in out:ether-intra, proto UDP, 10.53.0.1:50883->192.168.10.20:53, NAT 10.53.0.1:50883->(192.168.20.1:53->192.168.10.20:53), len 54 
05:38:41 firewall,info packet forward: in:ip2t-loopback-in out:ether-intra, proto UDP, 10.53.0.1:50883->192.168.10.20:53, NAT 10.53.0.1:50883->(192.168.20.1:53->192.168.10.20:53), len 54 
05:38:41 firewall,info mangle4 srcnat: in:(unknown 0) out:ether-intra, proto UDP, 10.53.0.1:50883->192.168.10.20:53, NAT 10.53.0.1:50883->(192.168.20.1:53->192.168.10.20:53), len 54 
05:38:41 firewall,info packet forward: in:ether-intra out:ip2t-loopback-in, src-mac 02:bf:11:38:5d:c0, proto UDP, 192.168.10.20:53->10.53.0.1:50883, NAT (192.168.10.20:53->192.168.20.1:53)->(192.168.10.30:50883->10.53.0.1:50883), len 70 
05:38:41 firewall,info packet forward: in:ether-intra out:ip2t-loopback-in, src-mac 02:bf:11:38:5d:c0, proto UDP, 192.168.10.20:53->10.53.0.1:50883, NAT (192.168.10.20:53->192.168.20.1:53)->(192.168.10.30:50883->10.53.0.1:50883), len 70

(there are several retries of course)

When RP Filer is set to no, the setup works correctly:

[admin@mr2] > put [resolve mr1.test]
192.168.10.20

Log:

05:38:56 system,info config changed by admin 
05:39:00 firewall,info mangle1 output: in:(unknown 0) out:ether-mgmt, proto UDP, 192.168.20.253:60136->192.168.20.1:53, len 54 
05:39:00 firewall,info mangle2 srcnat: in:(unknown 0) out:ip2t-loopback-out, proto UDP, 192.168.20.253:60136->192.168.20.1:53, len 54 
05:39:00 firewall,info mangle3 dstnat: in:ip2t-loopback-in out:(unknown 0), proto UDP, 10.53.0.1:60136->192.168.20.1:53, len 54 
05:39:00 firewall,info packet forward: in:ip2t-loopback-in out:ether-intra, proto UDP, 10.53.0.1:60136->192.168.10.20:53, NAT 10.53.0.1:60136->(192.168.20.1:53->192.168.10.20:53), len 54 
05:39:00 firewall,info packet forward: in:ip2t-loopback-in out:ether-intra, proto UDP, 10.53.0.1:60136->192.168.10.20:53, NAT 10.53.0.1:60136->(192.168.20.1:53->192.168.10.20:53), len 54 
05:39:00 firewall,info mangle4 srcnat: in:(unknown 0) out:ether-intra, proto UDP, 10.53.0.1:60136->192.168.10.20:53, NAT 10.53.0.1:60136->(192.168.20.1:53->192.168.10.20:53), len 54 
05:39:00 firewall,info packet forward: in:ether-intra out:ip2t-loopback-in, src-mac 02:bf:11:38:5d:c0, proto UDP, 192.168.10.20:53->10.53.0.1:60136, NAT (192.168.10.20:53->192.168.20.1:53)->(192.168.10.30:60136->10.53.0.1:60136), len 70 
05:39:00 firewall,info packet forward: in:ether-intra out:ip2t-loopback-in, src-mac 02:bf:11:38:5d:c0, proto UDP, 192.168.10.20:53->10.53.0.1:60136, NAT (192.168.10.20:53->192.168.20.1:53)->(192.168.10.30:60136->10.53.0.1:60136), len 70 
05:39:00 firewall,info packet input: in:ip2t-loopback-out out:(unknown 0), proto UDP, 192.168.20.1:53->192.168.20.253:60136, NAT 192.168.20.1:53->(10.53.0.1:60136->192.168.20.253:60136), len 70 
05:39:00 firewall,info packet input: in:ip2t-loopback-out out:(unknown 0), proto UDP, 192.168.20.1:53->192.168.20.253:60136, NAT 192.168.20.1:53->(10.53.0.1:60136->192.168.20.253:60136), len 70

Note two last lines - the reply packet reaches input chain. These lines are absent when RP filer is set to loose.

The routing table includes routes to all the addresses used:

[admin@mr2] > ip route print detail 
Flags: X - disabled, A - active, D - dynamic, 
C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, 
B - blackhole, U - unreachable, P - prohibit 
 0 A S  dst-address=0.0.0.0/0 gateway=ip2t-loopback-out 
        gateway-status=ip2t-loopback-out reachable distance=1 scope=30 
        target-scope=10 routing-mark=rm-loopback 

 1 ADS  dst-address=0.0.0.0/0 gateway=192.168.20.1 
        gateway-status=192.168.20.1 reachable via  ether-mgmt distance=100 
        scope=30 target-scope=10 vrf-interface=ether-mgmt 

 2 A S  dst-address=10.53.0.0/16 gateway=ip2t-loopback-in 
        gateway-status=ip2t-loopback-in reachable distance=1 scope=30 
        target-scope=10 

 3 ADC  dst-address=127.0.0.2/32 pref-src=127.0.0.2 gateway=bridge-loopback 
        gateway-status=bridge-loopback reachable distance=0 scope=10 

 4 ADC  dst-address=192.168.10.0/24 pref-src=192.168.10.30 gateway=ether-intra 
        gateway-status=ether-intra reachable distance=0 scope=10 

 5 ADC  dst-address=192.168.20.0/24 pref-src=192.168.20.253 gateway=ether-mgmt 
        gateway-status=ether-mgmt reachable distance=0 scope=10

So loose RP filter should not cause any dropping of the packets.

Any ideas why RP filter behaves this way?

Note that switching RP filter from no to loose are not causing immediate effects. The setup are starting to fail after approx. 10 minutes of idling(!) after setting RP filter to loose.

Actual full configs of metarouters and slightly redacted config of host RB2011 are attached.

(This study is not educational, I'm actually trying to get "mangling loopback" working on my production routers, to no avail for now.)

611 · Sun May 12, 2019 5:41 pm

An update:
Looks like the "mangling loopback" setup was failing to work on my production RB3011 (running the same 6.44.3) for the same RP filter reason.
But unlike the test setup, I had to reboot router after switching RP filter off to get it working. Maybe it's due to existing load, which causes no idling (some idling seems to be required for actual switching).

Tue May 21, 2019 8:41 am

I'm sorry, but i still do not understand - WHY you need this?
I do not know your background, but this is first time i heard about this "know solution of Mangling loopback"..

so please explain functionality that you are trying to achieve

Sob · Tue May 21, 2019 3:27 pm

@macgaiver: Here you go, but be warned, once you see it, you can't unsee it.

CZFan · Tue May 21, 2019 10:33 pm

@macgaiver: Here you go, but be warned, once you see it, you can't unsee it.

Do you and @sindy visit each other in The Matrix for drinks

611 · Tue May 21, 2019 11:02 pm

I'm sorry, but i still do not understand - WHY you need this?
I do not know your background, but this is first time i heard about this "know solution of Mangling loopback"..

so please explain functionality that you are trying to achieve

1. I need conditional DNS (like "*.domain1" is queried from dns1, "*.domain2" - from dns2, others are going default way).
I need it not only for clients, but for router itself (I do like to use names when it's reasonable).
cDNS is not there in ROS DNS resolver, and I'm not sure it would ever be.

2. I'm paranoid. I do change WinBox port to some random number on each my ROS device. I think extra 16 bits of randomness added to the total complexity of getting into my device is nice.
I also need Dude to be able to communicate (sic! not just query availability) to ROS devices on non-standard Winbox port.
The feature was requested circa 8 years ago, it still not there, and I'm not sure it would ever be.

DstNAT in output chain could have solved the issue with less hassle, but it's not supported on ROS, and I'm not sure it would ever be.

The workaround is here, and it works (when RP filter is set to no).
Complexity added is OK, as I'm an old PBX guy and making CO-to-EXT loops (telephone version of ipip tunnels used here) for some hacks was a quite frequent practice.

@Sob: thumbs up!

I've submitted this RP filter issue to support the day I've started this thread. So far no reply (apart from automatic "ticket created" reply).

Sob · Tue May 21, 2019 11:26 pm

@CZFan: This one is on him, I gave him all credit (or let's call it disclaimer). Not that the whole thing would be rocket science, it's relatively simple idea, when you look at it after someone else comes up with it. But somehow it still makes even my brain go in loops.

@611: You can mention to support, that the thing you desperately need is conditional DNS forwarding. And that it's really important, the proof of that being the thing you're trying to do now. Maybe you don't mind, but regular people should not be forced to such desperate measures. It's not just one hack, it's two hacks combined and it still has shortcomings.

sebastia · Wed May 22, 2019 11:58 am

If thinking outside the box is allowed (literally here), dnsmasq is the solution here: it can do selective forwarding for domains & ranges (reverse dns).

Wed May 22, 2019 1:01 pm

I haven't done any actual testing, but most likely issue is with connection tracking way to classify traffic, i had similar setup, where traffic was traversing router twice, connection tracking was unable to classify it for some reason. Trying to assign traffic to same conntrack entry so rp-fiter behavior might be explainable. Why can't you leave it off?

611 · Wed May 22, 2019 6:49 pm

I haven't done any actual testing, but most likely issue is with connection tracking way to classify traffic, i had similar setup, where traffic was traversing router twice, connection tracking was unable to classify it for some reason. Trying to assign traffic to same conntrack entry so rp-fiter behavior might be explainable. Why can't you leave it off?

Most probably it's not the case - I haven't saved the screenshots I've made while debugging the solution, but I'm quite sure I had two distinct entries in connection list for each DNS request - one for original request (SrcNATted and routed to ipip tunnel), another for the same request as it passes the router once again (with SrcNAT and DstNAT).
I thought of conntrack aliasing too, and tried to specify port range in both srcnat operations (so port numbers on first and second pass will differ and conntrack wouldn't mix them) - that hasn't changed anything.

I've actually left RP filter off on my production router after I've found that disabling it will make the solution work.
Honestly, I should have avoided it anyway - my configuration is quite static and firewall in already paranoid enough, so I could have reached the same results with few static rules, without RP filter.

611 · Wed May 22, 2019 7:22 pm

@611: You can mention to support, that the thing you desperately need is conditional DNS forwarding. And that it's really important, the proof of that being the thing you're trying to do now. Maybe you don't mind, but regular people should not be forced to such desperate measures. It's not just one hack, it's two hacks combined and it still has shortcomings.

I've mentioned the need for cDNS as one of the reasons for the case. Honestly, I don't believe the support would get the case this way - most likely they're solving the problems at hand, not thinking of them globally.
I'd agree the solution is a bunch of dirty hacks, and we should have these long-requested the features built in already.

Regarding shortcomings, I've already found them when I've tried to use the same hacks to make Dude connect to non-standard WinBox ports - it turned out that TCP packets other than initial SYN & SYN/ACK do not have proper srcnat/dstnat flags when they're passing firewall, and firewall rules in my production config relied on these flags, thus connection made by Dude were stalling after SYN/ACK.

If thinking outside the box is allowed (literally here), dnsmasq is the solution here: it can do selective forwarding for domains & ranges (reverse dns).

DNS with features I need - outside of the box, OpenVPN and IKEv2 clients able to connect the VPN provider I'm using - outside of the box, monitoring solution which supports my configuration - outside of the box, etc. Well, why do I need the box?

sebastia · Wed May 22, 2019 7:32 pm

Routing & firewalling?

It's core functions, the rest is add-ons.

611 · Fri May 24, 2019 12:34 pm

I've got a reply from support, problem confirmed:

I have managed to reproduce your problem and at the moment it indeed seems to be software related bug which does not comply with loose rp-filter implementation.

However, this parameter functionality in RouterOS works based on Linux Kernel. We will try to look for any possible patches for this problem.

"Solve the problem at hand" approach confirmed too

Routing & firewalling? It's core functions, the rest is add-ons.

Core function of a car is getting you from point A to point B

But when we choose a car we look at everything else.

Strange RP filter behavior

Strange RP filter behavior

Re: Strange RP filter behavior

Re: Strange RP filter behavior

Re: Strange RP filter behavior

Re: Strange RP filter behavior

Re: Strange RP filter behavior

Re: Strange RP filter behavior

Re: Strange RP filter behavior

Re: Strange RP filter behavior

Re: Strange RP filter behavior

Re: Strange RP filter behavior

Re: Strange RP filter behavior

Re: Strange RP filter behavior

Who is online