Community discussions

MikroTik App
 
User avatar
sguox
Trainer
Trainer
Topic Author
Posts: 73
Joined: Fri Mar 09, 2012 6:23 pm
Location: Singapore
Contact:

Firewall Connection, TCP established to non-existent IPs

Tue May 26, 2015 7:29 pm

Hi All,

I am getting a lot of connections in the Firewall which showing same source address and port to random IP in my network (which are not in use). the protocol is TCP and state established.

Any one has similar issues? How could a TCP connection to non-existence IP has state "established" ?
est-hack.png
You do not have the required permissions to view the files attached to this post.
 
User avatar
BartoszP
Forum Guru
Forum Guru
Posts: 2975
Joined: Mon Jun 16, 2014 1:13 pm
Location: Poland

Re: Firewall Connection, TCP established to non-existent IPs

Tue May 26, 2015 8:46 pm

To random LAN IP address or random WAN's ports ?
UDP 7124 is QuickTime Streaming Server but maybe it uses also TCP ?
 
User avatar
pukkita
Trainer
Trainer
Posts: 3051
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Firewall Connection, TCP established to non-existent IPs

Tue May 26, 2015 8:58 pm

Look at the first letter of those connections, U - Unreplied

I also have noticed this and the "stablished" TCP State puzzled me also because I also saw that on non running IPs belonging to our public subnet, from outside hosts scanning whole IP ranges for vulnerabilities...

A search on that returned some unconclusive matches, still looking for the issue.

In the meanwhile you can blackhole your most generic subnet route.
 
jarda
Forum Guru
Forum Guru
Posts: 7756
Joined: Mon Oct 22, 2012 4:46 pm

Tue May 26, 2015 10:57 pm

I tried recently to make a script that would remove the unreplied + established connections from conntrack but I haven't succeeded so far. Looks like the selection doesn't give back numbers to be removed...
 
User avatar
pukkita
Trainer
Trainer
Posts: 3051
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Firewall Connection, TCP established to non-existent IPs

Tue May 26, 2015 11:41 pm

I tried recently to make a script that would remove the unreplied + established connections from conntrack but I haven't succeeded so far. Looks like the selection doesn't give back numbers to be removed...
I wonder what are these connections? How could it be its TCP state is established but unreplied? Why do they "phantom" mangle marking???

I've noticed these connections "build up" also from hosts in my LAN to https servers outside, and have three things in common:

- Unreplied
- TCP State Established
- They don't get properly marked in mangle (neither connection, nor packet) whereas the rest DO get properly marked.

Is this a bug, or something usual with iptables?
 
User avatar
sguox
Trainer
Trainer
Topic Author
Posts: 73
Joined: Fri Mar 09, 2012 6:23 pm
Location: Singapore
Contact:

Re: Firewall Connection, TCP established to non-existent IPs

Wed May 27, 2015 6:13 pm

To random LAN IP address or random WAN's ports ?
UDP 7124 is QuickTime Streaming Server but maybe it uses also TCP ?
it's public ip routing, all destination IP address are not in use, might consider make them honeypots.

Look at the first letter of those connections, U - Unreplied

I also have noticed this and the "stablished" TCP State puzzled me also because I also saw that on non running IPs belonging to our public subnet, from outside hosts scanning whole IP ranges for vulnerabilities...

A search on that returned some unconclusive matches, still looking for the issue.

In the meanwhile you can blackhole your most generic subnet route.
can't blackhole your entire subnet just because only 5% IP are being used.
 
User avatar
pukkita
Trainer
Trainer
Posts: 3051
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Firewall Connection, TCP established to non-existent IPs

Wed May 27, 2015 6:26 pm

can't blackhole your entire subnet just because only 5% IP are being used.
5% used or unused?

What I meant is setting a blackhole route more general than the rest you should have for "active" IPs, that way traffic to unactive IPs will be blackholed.
 
User avatar
sguox
Trainer
Trainer
Topic Author
Posts: 73
Joined: Fri Mar 09, 2012 6:23 pm
Location: Singapore
Contact:

Re: Firewall Connection, TCP established to non-existent IPs

Wed May 27, 2015 6:45 pm

can't blackhole your entire subnet just because only 5% IP are being used.
5% used or unused?

What I meant is setting a blackhole route more general than the rest you should have for "active" IPs, that way traffic to unactive IPs will be blackholed.
5% used, a lot of unused IP that's why Unreplied.

Assume we have 20k IPs where 10% (2k) are unused. Maintaining the blackhole routing will not be an easy job.
 
jarda
Forum Guru
Forum Guru
Posts: 7756
Joined: Mon Oct 22, 2012 4:46 pm

Re: Firewall Connection, TCP established to non-existent IPs

Wed May 27, 2015 8:47 pm

I tried (as mentioned before) such script:
:foreach i in=[/ip firewall connection find tcp-state=established assured=no] do={
/ip firewall connection remove $i};
But it does nothing. How can I automatically remove such records?
 
User avatar
pukkita
Trainer
Trainer
Posts: 3051
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Firewall Connection, TCP established to non-existent IPs

Wed May 27, 2015 8:53 pm

What concerns me more is why do those connections exist (when are initiated by legit hosts from inside to outside) to start, removing it is fixing the symptom not the cause.

I don't know how to remove them apart from manually.
 
jarda
Forum Guru
Forum Guru
Posts: 7756
Joined: Mon Oct 22, 2012 4:46 pm

Firewall Connection, TCP established to non-existent IPs

Thu May 28, 2015 7:22 am

My connections are not suspicious by their addresses. They just seem to be forgotten traffic by their src side but not killed on router level. I don't understand much how it happens.
 
User avatar
pukkita
Trainer
Trainer
Posts: 3051
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Firewall Connection, TCP established to non-existent IPs

Thu May 28, 2015 1:46 pm

My connections are not suspicious by their addresses. They just seem to be forgotten traffic by their src side but not killed on router level. I don't understand much how it happens.
Me neither... and why they seem to dodge through mangle marking rules.

Most are https connections, guess this is related?

An official word from mikrotik would be great to shed some light on this "X Files" issue.
 
jarda
Forum Guru
Forum Guru
Posts: 7756
Joined: Mon Oct 22, 2012 4:46 pm

Firewall Connection, TCP established to non-existent IPs

Thu May 28, 2015 3:51 pm

Many of them in my case are from computer that has more interfaces connected to the lan. When used interface goes down its internal routing table is updated and traffic is redirected through the next interface. It has different ip so router probably keeps already opened connections for some time. Then the first interface goes up, but the programs initiate new connections as the previous were not existing within the computer anymore. Looks like error in conntrack management of ros.
 
User avatar
docmarius
Forum Guru
Forum Guru
Posts: 1224
Joined: Sat Nov 06, 2010 12:04 pm
Location: Timisoara, Romania
Contact:

Re: Firewall Connection, TCP established to non-existent IPs

Thu May 28, 2015 3:57 pm

 
User avatar
pukkita
Trainer
Trainer
Posts: 3051
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Firewall Connection, TCP established to non-existent IPs

Thu May 28, 2015 7:24 pm

It looks so...

My conntrack is set to yes.

And this Unreplied/established TCP-state stale connections build up happens also with regular HTTP.
Looks like error in conntrack management of ros.
Indeed... In my case there isn't any NAT in between, and hosts are connected through just one interface.
 
jarda
Forum Guru
Forum Guru
Posts: 7756
Joined: Mon Oct 22, 2012 4:46 pm

Re: Firewall Connection, TCP established to non-existent IPs

Wed Jun 03, 2015 8:35 am

I can confirm that there is no difference whether the connection tracking is iset to "auto" or to "yes". The effect is the same.

Has anyone created a ticket to mikrotik support about this error?
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1076
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: Firewall Connection, TCP established to non-existent IPs

Wed Jun 03, 2015 12:06 pm

I tried (as mentioned before) such script:
:foreach i in=[/ip firewall connection find tcp-state=established assured=no] do={
/ip firewall connection remove $i};
But it does nothing. How can I automatically remove such records?
Try putting instead of assured=no seen-reply=no

My guess is this are invalid connections generated for whatever reason. If you put in firewall filter a rule with drop action for invalid connections you might see less of these connections.
 
User avatar
pukkita
Trainer
Trainer
Posts: 3051
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Firewall Connection, TCP established to non-existent IPs

Wed Jun 03, 2015 2:04 pm

I tried (as mentioned before) such script:
:foreach i in=[/ip firewall connection find tcp-state=established assured=no] do={
/ip firewall connection remove $i};
But it does nothing. How can I automatically remove such records?
Try putting instead of assured=no seen-reply=no

My guess is this are invalid connections generated for whatever reason. If you put in firewall filter a rule with drop action for invalid connections you might see less of these connections.
I already have drop invalid for forward and input/output chains, making no difference.
 
jarda
Forum Guru
Forum Guru
Posts: 7756
Joined: Mon Oct 22, 2012 4:46 pm

Firewall Connection, TCP established to non-existent IPs

Wed Jun 03, 2015 9:35 pm

I am also dropping invalid by default. And anyway I am dropping by default everything on the end of each chain. Of course I have no rule that would accept invalid packets before.
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1076
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: Firewall Connection, TCP established to non-existent IPs

Thu Jun 04, 2015 1:13 pm

This is a topic about understanding very well ip tables, which I don't :).
Anyway, an established connection is not the same with an established tcp. What could be happening is, if the client closes the http connection without sending an ack packet (or something like that), so the router considers the connection established because it was there before, but unreplied because it is kept alive only from one side of the connection. This is my guess.
I don't think it is a bug, but rather the nature of how router deals with the connections.
This needs a very thorough analyze of the whole connection from the begin to the end of it, in both sites, client and server from someone who has time and good knowledge of how ip tables works :).
 
User avatar
pukkita
Trainer
Trainer
Posts: 3051
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Firewall Connection, TCP established to non-existent IPs

Thu Jun 04, 2015 2:42 pm

This is a topic about understanding very well ip tables, which I don't :).
Anyway, an established connection is not the same with an established tcp. What could be happening is, if the client closes the http connection without sending an ack packet (or something like that), so the router considers the connection established because it was there before, but unreplied because it is kept alive only from one side of the connection. This is my guess.
The only scenario would be an established (I refer to TCP state) connection where the client sends an ACK,FIN or an ACK which is later not replied by the server, or (most probably) a hiccup in conntrack leaving these orphan connections as-is with a 24Hour timeout...
I don't think it is a bug, but rather the nature of how router deals with the connections.
This needs a very thorough analyze of the whole connection from the begin to the end of it, in both sites, client and server from someone who has time and good knowledge of how ip tables works :).
I don't know what is it, but... why doesn't these connections get marked properly?

90% connections get marked properly and behave as expected, w/o these "stalled" connections. Then, amongst the "stalled" ones, only a 5% of these Unreplied, TCP state established "orphan" connections are marked properly, the rest seem to "jump" to the latest marking mangle rule...
 
User avatar
Caci99
Forum Guru
Forum Guru
Posts: 1076
Joined: Wed Feb 21, 2007 2:26 pm
Location: Tirane
Contact:

Re: Firewall Connection, TCP established to non-existent IPs

Thu Jun 04, 2015 9:52 pm


I don't know what is it, but... why doesn't these connections get marked properly?

90% connections get marked properly and behave as expected, w/o these "stalled" connections. Then, amongst the "stalled" ones, only a 5% of these Unreplied, TCP state established "orphan" connections are marked properly, the rest seem to "jump" to the latest marking mangle rule...
That, again, would depend on how the router considers these connection. As we know, the mangle will apply mark according to the chain. In which chain they will belong is kind of mystery to me, or if they do even skip the chain process at all, because the router itself does not know where to put them (maybe?).
 
User avatar
pukkita
Trainer
Trainer
Posts: 3051
Joined: Wed Dec 04, 2013 11:09 am
Location: Spain

Re: Firewall Connection, TCP established to non-existent IPs

Thu Jun 04, 2015 11:29 pm

Let's put it in a clear way:

Public IP -----> Router ------> Internet
a.b.c.d:54321 -------------------> f.g.h.i:443

All connections traversing the router go through the forward chain.
I'm marking on the forward chain based on that criteria (destination port tcp 443).

Say in connections, there are 1000 connections from a.b.c.d to f.g.h.i; what I am seeing is:
- About 9500 of these connections will have consistent TCP states, and would have been marked correctly.
- 50 of these connections will have unconsistent TCP state, BUT still are marked correctly.
- 450 of these connections will have an unconsistent tcp state (established while being unreplied), AND won't be correctly marked.
 
irufan
just joined
Posts: 3
Joined: Thu Nov 24, 2016 4:56 am

Re: Firewall Connection, TCP established to non-existent IPs

Thu Nov 24, 2016 5:16 am

any one found a fix for this issues.
i have the same problem but connection status confirmed and timeout of 24hr.
 
sysPanda
just joined
Posts: 1
Joined: Thu Jul 20, 2017 4:28 pm

Re: Firewall Connection, TCP established to non-existent IPs

Thu Jul 20, 2017 4:42 pm

Somewhat old topic but it might help someone.
Not really a solution but a workaround based on some of the upper posts:
:foreach i in=[/ip firewall connection find protocol=udp seen-reply=no timeout<00:59:30] do={/ip firewall connection remove $i};
:foreach i in=[/ip firewall connection find protocol=tcp seen-reply=no timeout<23:59:30] do={/ip firewall connection remove $i};
This drops all connections that had not received a reply for 30 seconds. Values are based on the default timeout settings.
I have scheduled this to run every minute and it works without problems so far.

Cheers.

Who is online

Users browsing this forum: ismets, kot2905, sindy and 26 guests