Valid connection issues when dropping invalid packets in firewall

kmadac · Thu Mar 17, 2016 10:10 pm

Hello,

I have 2 routers CRS125-24G with RouterOS 6.33.3 so 24 one uplink and 23 ports as a switch on each device.

I found out weird thing when routing between networks. I have two networks Device A - 192.168.10.0/24 and Device B - 192.168.100.0/24.

(SSH Client 192.168.10.18) ------ (192.168.10.1 Router A) ------------ (192.168.10.11 Router B 192.168.100.1) ---- (SSH Server 192.168.100.10)

There is a a static route on Router A which routes trafic from 10.0/24 to 100.0/24 over router B:

Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
8 A S 192.168.100.0/24                   192.168.10.11             1

I found out, that when I have firewall rule which drops invalid packets of invalid connections I have problem also with valid ones. Here is that rule

 9    ;;; drop invalid connection
      chain=forward action=drop connection-state=invalid log=yes log-prefix="INVALID"

When it is enabled and ssh from 192.168.10.18 to 192.168.100.10 it takes 7 seconds to connect. When I disable this rule, connection to ssh server is immediate.

I can see following dropped packets in the log if dropping of invalid packets is enabled:

20:31:49 firewall,info INVALID forward: in:bridge out:bridge, src-mac c4:85:08:dd:94:42, proto TCP (ACK), 192.168.10.18:46888->192.168.100.10:22, len 52 
20:31:49 firewall,info INVALID forward: in:bridge out:bridge, src-mac c4:85:08:dd:94:42, proto TCP (ACK,PSH), 192.168.10.18:46888->192.168.100.10:22, len 95 
20:31:50 firewall,info INVALID forward: in:bridge out:bridge, src-mac c4:85:08:dd:94:42, proto TCP (ACK,PSH), 192.168.10.18:46888->192.168.100.10:22, len 95 
20:31:50 firewall,info INVALID forward: in:bridge out:bridge, src-mac c4:85:08:dd:94:42, proto TCP (ACK,PSH), 192.168.10.18:46888->192.168.100.10:22, len 95 
20:31:50 firewall,info INVALID forward: in:bridge out:bridge, src-mac c4:85:08:dd:94:42, proto TCP (ACK,PSH), 192.168.10.18:46888->192.168.100.10:22, len 95 
20:31:50 firewall,info INVALID forward: in:bridge out:bridge, src-mac c4:85:08:dd:94:42, proto TCP (ACK), 192.168.10.18:46888->192.168.100.10:22, len 52 
20:31:51 firewall,info INVALID forward: in:bridge out:bridge, src-mac c4:85:08:dd:94:42, proto TCP (ACK,PSH), 192.168.10.18:46888->192.168.100.10:22, len 95 
20:31:52 firewall,info INVALID forward: in:bridge out:bridge, src-mac c4:85:08:dd:94:42, proto TCP (ACK), 192.168.10.18:46888->192.168.100.10:22, len 52 
20:31:53 firewall,info INVALID forward: in:bridge out:bridge, src-mac c4:85:08:dd:94:42, proto TCP (ACK,PSH), 192.168.10.18:46888->192.168.100.10:22, len 95

I did packets trace and I see many retransmissions and duplicate packets. It is weird that connection is initiated at the end and all further packets in same connection is without latencies. I have connection tracking enabled and I can see connection in connection tracking table.

It seems to me that valid packets are falsely matched by that rule. What could be the reason for such behaviour? Is it a bug, or some misconfiguration on my side?

ShayanFiroozi · Fri Mar 18, 2016 6:12 pm

Hi ,
put this rule the last one on list , first allow your related connections then drop invalids , see what happens

mizeraj · Wed Jun 22, 2016 11:00 am

Hi, same issue here.
But i have to add that it happens only on windows machines, that try to access via ssh or sftp 192.168.100.0 network, added by simple static route rule.
I have no connection issues with generic 192.168.0.0 network, made by mikrotik itself.
Only switching off the "drop invalid" rule helps, moving it does nothing (basically it is already at the bottom of rule list).

nethor · Wed Aug 07, 2019 9:07 am

Any progress on this? Did somebody find the cause of this issue?

I am experiencing the same issue with a setup of two MikroTik Routerboards that have an active OSPF configuration.
I am also observing a delay of around 7 seconds, but only for certain protocols (i.e. REST requests).
This happens only if I connect to devices that are connected to router2 (while router1 is acting as gateway). Direct connections via router2 do not have this delay.
Based on the logs the default firewall forward rule "drop invalid" of router1 is dropping packets, identical to what was posted by kamadac. If I add a custom white-listed rule that allows invalid pakets for that connection, the request go through swiftly, without delay. But this feels like a dirty fix, I would rather like to understand what is causing the packets to be detected as invalid.

I was doing some investigations on the MTU settings of the differecnt components since VPN is involved, but this did not have any impact yet.

sindy · Wed Aug 07, 2019 1:39 pm

Not sure about your (@nethor's) setup, but in the OP from 2016, Router B's interface facing towards the SSH client was in the same subnet like the client. So while the request packets from the client went to Router A (because 192.168.10.1 was configured as the gateway of the default route or at least if a route to 192.68.100.0/24) and were forwarded from there to Router B and further to the destination, the server responses bypassed Router A because Router B has sent them to the client directly. So the firewall of Router A could see the initial SYN from the client but not the SYN,ACK from the server, so any further packets from the client were considered invalid. Also, the accept established or related rule, even if first in the chain, does not match on invalid packets, so that advice wasn't helpful.

Your situation may be completely different, so provide your own network diagram if the above doesn't apply.

pgh321 · Wed Aug 07, 2019 10:04 pm

I've similar issues

viewtopic.php?f=2&t=150601&p=743068#p743068

What can happen if I just remove the rule dropping invalid packets?

sindy · Wed Aug 07, 2019 10:14 pm

What can happen if I just remove the rule dropping invalid packets?

As I wrote just one post above - removing the drop invalid rule doesn't make the invalid packets miraculously be accepted by the accept established,related one. So if your firewall chains don't end with drop the rest rules, removing the drop invalid rules can make your connections work, but if it does, it is just a sign that the firewall is leaky.

pgh321 · Sat Aug 10, 2019 12:09 pm

Sorry I was wrong, my packets are just dropped by the LAST rule "drop and log everything else"

These connections seems "orphaned" like in viewtopic.php?t=3991

DROP: input: in:ether01-gateway out:(unknown 0), src-mac XXX, proto TCP (ACK,PSH), 151.101.X.Y:443->MIK_WAN, len 110

where src-mac is the tplink vdsl router in front of the mikrotik (doing NAT to it) , and MIK_WAN is the external interface of the mikrotik, the one connected to the tplink
The IP 151.101 is from a CDN, I've noticed it is related to my smartphone browsing reddit so seems nothing risky...

I'm not sure if I should do something to packet timeouts, maybe these connections are opened too little or too much and generates this behaviour...
Anyway this is not "always" , because the phone apps and similar things works normally, but I've never noticed this behaviour in the past so this seems related to this phone.
I've never used an android phone before, just tablets wifi, so maybe the phone having different connections does something that generates issues in my network...

EDIT: I've found a better example of my issue

I've seen logs like above for a period of time when my phone was not at home, so mikrotik was "looking for my phone" , but it had not been found in the network, so packet got dropped. Obviously this doesn't happen for PC or tablets that are always inside the home, connected to the same network.
Is this a normal behaviour? Is there a way to avoid this?

I'm a bit confused, and even after much googling I've not found reasonable informations...

pe1chl · Sat Aug 10, 2019 12:50 pm

What usually happens with phones: your phone has connections with all kinds of services all the time. Those are open TCP connections.
Then you walk out of your house and you suddenly lose the WiFi connection without an opportunity to neatly close them.
So, the router and the service still believe these connections are open.
After some time the router deletes the open connection record (one of the timers in the connection tracking has elapsed).
But the service at the other end still sees the idle connection and either tries sending some data on it or sends a "keep alive" packet.
At that time this is an invalid packet and it gets dropped. As the other end still sees no reply, it will keep trying this for some time.
You can help it by adding another rule that rejects TCP packets with connection state "invalid" and use "reject-with: tcp-reset".
That will make the other end understand immediately that there is nothing more to expect from this particular TCP connection.
(of course there can be several connections affected and so this may still repeat a couple of times, but not more than once per connection)

pgh321 · Mon Aug 12, 2019 4:40 pm

There is something I don't understand

The rule blocking invalid packets here is the first one, and without log.
If I see packets in log, they should be blocked by the last rule ( drop all the rest)

So how can I target those packets?
Just above the rule "drop all" but how?
Finding about the 3-4 addresses often logged and dropping with reset them only seems not practical, maybe "out: unknown 0" is the key???

Sorry if I'm a noob

sindy · Mon Aug 12, 2019 11:06 pm

The rule blocking invalid packets here is the first one, and without log.
If I see packets in log, they should be blocked by the last rule ( drop all the rest)

Sounds logical.

So how can I target those packets?
Just above the rule "drop all" but how?
Finding about the 3-4 addresses often logged and dropping with reset them only seems not practical, maybe "out: unknown 0" is the key???

out: unknown only says that the router was unable to route these packets anywhere because they came to its WAN IP but as there was no tracked connection they would match, they could not be "un-src-nated" so the router handled them as input, not forwarded, packets - and input packets have no out-interface.

However, the TCP packets belonging to dead connections are labelled with connection-state=invalid only if loose-tcp-tracking under /ip firewall connection tracking is set to no; otherwise, they are treated as connection-state=new. So add log=yes to the "drop invalid" rule with some distinctive log-prefix value to double-check that it logs nothing, and then do /ip firewall connection tracking set loose-tcp-tracking=yes; after that, you should see them logged by the "drop invalid" rule. And to be clear, we talk about chain=input although the connections which have caused these packets to arrive were handled by chain=forward.

pgh321 · Fri Aug 16, 2019 4:35 pm

Hi, loose tcp tracking was already enabled...

Logging invalid packets I started see only many RST packets, all dropped for the invalid rule, not only for the phone address but also other devices...
Then I tried disabling the "drop invalid" rule, and those RST packets were dropped by the "final drop" rule at the end...

This, for example, was dropped first as invalid and, if not dropped as invalid, was dropped at the end anyway

input: in:ether1-gateway out:(unknown 0), src-mac TPLINK_WAN , proto TCP (RST), 216.58.205.74:443->MIK_WAN:39508, len 40

Valid connection issues when dropping invalid packets in firewall

Valid connection issues when dropping invalid packets in firewall

Re: Valid connection issues when dropping invalid packets in firewall

Re: Valid connection issues when dropping invalid packets in firewall

Re: Valid connection issues when dropping invalid packets in firewall

Re: Valid connection issues when dropping invalid packets in firewall

Re: Valid connection issues when dropping invalid packets in firewall

Re: Valid connection issues when dropping invalid packets in firewall

Re: Valid connection issues when dropping invalid packets in firewall

Re: Valid connection issues when dropping invalid packets in firewall

Re: Valid connection issues when dropping invalid packets in firewall

Re: Valid connection issues when dropping invalid packets in firewall

Re: Valid connection issues when dropping invalid packets in firewall