Timeout when upgrading

tomZ · Wed Aug 10, 2016 4:27 pm

Hej Community,

our Mikrotik 2011UiAS and 3 cAPs are facing the problem of not being able to connect to the upgrade servers anymore.
Even Tho they receive the changelog correctly, the connection is dropped with a "connection timeout" shortly after pressing download&upgrade.

any suggestions/known issues?

Internet-Connection works of course - even pinging upgrade.mikrotik.com

thanks

pe1chl · Wed Aug 10, 2016 4:50 pm

I experience this problem when the MTU of the connection is below 1500 somewhere along the path,
and the router at that point is not doing TCP clamp MSS to MTU.
It appears that the MikroTik update servers do not handle ICMP "fragmentation needed" replies in a
reasonable way. When such a packet is returned, the next packet from the update server is properly
sized, but immediately thereafter the full-size packets are sent again. Because the router does not
return an ICMP reply every time, the connection is mostly in ever slower re-try mode and as you
describe it dies with a timeout.

GeneralMarmite · Sun Jul 22, 2018 1:17 pm

I have some data on this problem. I'm running 6.42.6 on an RB2011iL. I have a Business DSL connection from BT in the UK. I found, like someone else mentioned, that when I disable the "drop everything else" rule at the bottom of my input ruleset the upgrade succeeded. If I leave that rule in place it fails. I have an answer, but it might only apply to me. Still, this might help others, so here are the details.

My provider assigns me a randomly allocated dynamic /32 IP address, like most residential ISPs do. However, I pay for a /29 range from my provider, and those are the only addresses I use. I have no routes or rules that include the dynamic IP. It's just kinda there. I ignore it. The only rules I have defined are using addresses from my /29. Here's what the list of IPs looks like:

[admin@router] > /ip address print 
Flags: X - disabled, I - invalid, D - dynamic 
 #   ADDRESS            NETWORK         INTERFACE                                                                                                 
 0   172.30.0.1/24      172.30.0.0      bridge-local                                                                                              
 1   172.31.0.1/24      172.31.0.0      bridge-dmz                                                                                                
 2   X.X.X.X/29         X.X.X.X         BTDSLModem                                                                                                
 3   172.31.1.1/24      172.31.1.0      bridge-dmz                                                                                                
 4 D 81.148.161.20/32   81.148.160.1    BTInfinity

When my router initiates the update connection, the source IP of its packets come from this dynamic /32 address. So the SYN/ACKs from Cloudfront are blocked. I have no rule that would allow them to come back. I turned on logging for my drop rule, hit the update button in WebFig, then looked at what got logged by my drop rule. You can see the two rule changes in the log where I enabled logging on this drop rule, then disabled it.

01:12:47 system,info filter rule changed by admin 
01:12:53 firewall,info drop input: in:BTInfinity out:(unknown 0), proto TCP (SYN,ACK), 159.148.172.226:80->81.148.161.20:60822, len 60 
01:12:54 firewall,info drop input: in:BTInfinity out:(unknown 0), proto TCP (SYN,ACK), 159.148.172.226:80->81.148.161.20:60822, len 60 
01:12:55 firewall,info drop input: in:BTInfinity out:(unknown 0), proto TCP (SYN,ACK), 159.148.172.226:80->81.148.161.20:60822, len 60 
01:12:56 firewall,info drop input: in:BTInfinity out:(unknown 0), proto TCP (SYN,ACK), 159.148.172.226:80->81.148.161.20:60822, len 60 
01:12:58 firewall,info drop input: in:BTInfinity out:(unknown 0), proto TCP (SYN,ACK), 159.148.172.226:80->81.148.161.20:60822, len 60 
01:13:08 system,info filter rule changed by admin

It's not clear to me how to control the source IP address that the router uses when it sources packets. For instance, if it is acting as a recursive DNS resolver, it issues DNS queries and it chooses this dynamically assigned IP address as its source IP. When it makes an outbound connection to upgrade.mikrotik.com, it also sources from this dynamic IP address. I want it to use a specific IP address from my network, not the assigned one.

Someone else might have similar problems because of the source IP of connections initiated by the router. I don't know how to write rules that either disable this dynamic IP or make it a valid source for traffic.

pe1chl · Sun Jul 22, 2018 1:39 pm

You can set your preferred source address for outgoing traffic in the IP route table (in the default route, in this case).

GeneralMarmite · Sun Jul 22, 2018 7:50 pm

You can set your preferred source address for outgoing traffic in the IP route table (in the default route, in this case).

I appreciate that suggestion, but in my case I'm struggling to make that happen. Probably because I'm not doing something right. Here's

/ip route print

:

 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          BTInfinity                1
 1 ADC  81.148.160.1/32    81.148.161.20   BTInfinity                0
 2 ADC  172.30.0.0/24      172.30.0.1      bridge-local              0
 3 ADC  172.31.0.0/24      172.31.0.1      bridge-dmz                0
 4 ADC  172.31.1.0/24      172.31.1.1      bridge-dmz                0
 5 ADC  AAA.BB.CC.DD/29    AAA.BB.CC.DD    BTDSLModem                0

Note that my default route has a distance of 1, and all others have a distance of 0. I'm trying a few things and none seem to work.

In WebFig, clicking on any of these routes does not allow me to set the preferred source. The option isn't available to edit.
At the command line it doesn't seem possible to edit the existing routes or set the distances lower.

There are (old) topics on this forum (like this viewtopic.php?t=11433) that suggest it isn't possible to set a source preference for DAC routes. Likewise, I can't seem to create a static route with distance 0, or adjust the dynamic route to have a distance higher than 0. I admit I'm a bit inexperienced in all this. But how is it that I set the source preference in a dynamic route like that?

mkx · Mon Jul 23, 2018 1:54 pm

It seems to me that you only have one default route and that one is using the interface with dynamic address (that is BTInfinity). You will have to add default route, bound to interface with static public IP address (BTDSLModem). To make sure your dynamic IP address is not getting in your way, you can change dhcp-client and add "add-default-route=no" option to it (in webfig, change "Add Default Route" field value to "no").

DrLove73 · Mon Nov 26, 2018 11:57 pm

I have a "ERROR: connection timed out" problem on hEX (RouterBOARD 750G r3) when I try to update via "Check for Updates".
It started recently, I tried to update from 6.42.6 and it failed.
It all started when suddenly connection to my PPTP server on that system from my home Mikrotik "hAP lite" failed to establish. It worked fine for few years, but it stopped this morning after I changed some firewall rules. PPTP server should that TCP connection form my home Mikrotik was established, but nothing more, like it was stuck.
So I tried to update RouterOS and got timeout error.

I messed around and could not solve it, I dissabled fresly made rules, but then I read someone saying that I should disable last DROP rule in INPUT chain.

When I did that, both PPTP connection from my home miraculously connected (disable/enable on PPTP client was needed) AND "Check for Upgrade" worked. I enabled input-drop rule and again both PPTP connection and Check for Update failed.
Then I started to log that input-drop rule and try PPTP connection and clicked "Check for Update".

For PPTP connection, log shows that proto 47 needs to pass, BUT, PPTP server is on that Mikrotik, it should just need open TCP port 1723:

22:45:50 firewall,info DropSveOstalo input: in:ether1-gateway out:(unknown 0), src-mac 2c: ab:eb:27:88:19, proto 47, 212.69.xxx.xx->89.216.yy.yyy, len 59
22:45:50 firewall,info DropSveOstalo input: in:ether1-gateway out:(unknown 0), src-mac 2c: ab:eb:27:88:19, proto 47, 212.69.xxx.xx->89.216.yy.yyy, len 54
22:45:51 firewall,info DropSveOstalo input: in:ether1-gateway out:(unknown 0), src-mac 2c: ab:eb:27:88:19, proto 47, 212.69.xxx.xx->89.216.yy.yyy, len 59

For "Check for Upgrade", it is even weirder, it says that INPUT-DROP rule is catching OUTGOING connection to Mikrotik server (159.148.147.204)!:

22:50:31 firewall,info DropSveOstalo input: in:ether1-gateway out:(unknown 0), src-mac 2c: ab:eb:27:88:19, proto TCP (SYN,ACK), 159.148.147.204:80->89.216.122.99:38422, len 60
22:50:32 firewall,info DropSveOstalo input: in:ether1-gateway out:(unknown 0), src-mac 2c: ab:eb:27:88:19, proto TCP (SYN,ACK), 159.148.147.204:80->89.216.yy.yyy:38422, len 60
22:50:33 firewall,info DropSveOstalo input: in:ether1-gateway out:(unknown 0), src-mac 2c: ab:eb:27:88:19, proto TCP (SYN,ACK), 159.148.147.204:80->89.216.yy.yyy:38422, len 60
22:50:34 firewall,info DropSveOstalo input: in:ether1-gateway out:(unknown 0), src-mac 2c: ab:eb:27:88:19, proto TCP (SYN,ACK), 159.148.147.204:80->89.216.yy.yyy:38422, len 60
22:50:36 firewall,info DropSveOstalo input: in:ether1-gateway out:(unknown 0), src-mac 2c: ab:eb:27:88:19, proto TCP (SYN,ACK), 159.148.147.204:80->89.216.yy.yyy:38422, len 60
22:50:38 system,info filter rule changed by admin

DrLove73 · Tue Nov 27, 2018 12:31 am

Here are my firewall rules, at least as RouterOS reports from terminal:

[admin@Preventiva] /ip firewall> address-list print

Flags: X - disabled, D - dynamic
# LIST ADDRESS CREATION-TIME TIMEOUT
0 PristupSpolja xxx.xxx.xxx.xx/29 nov/26/2018 11:56:16
1 PristupSpolja aaa.aaa.aaa.aaa nov/26/2018 11:57:11
2 PristupSpolja sss.sss.sss.sss nov/26/2018 11:57:25

[admin@Preventiva] /ip firewall> filter print

Flags: X - disabled, I - invalid, D - dynamic
0 X ;;; default configuration
chain=input action=accept protocol=tcp dst-address=89.216.xx.xxx in-interface=ether1-gateway dst-port=80 log=no log-prefix=""
1 X ;;; default configuration
chain=input action=accept protocol=tcp dst-address=89.216.xxx.xx in-interface=ether1-gateway dst-port=443 log=no log-prefix=""
2 ;;; default configuration
chain=input action=accept protocol=icmp log-prefix=""
3 ;;; default configuration
chain=input action=accept connection-state=related in-interface=ether1-gateway log=no log-prefix=""
4 chain=input action=accept protocol=tcp in-interface=ether1-gateway dst-port=1194 log=no log-prefix=""
5 X chain=input action=accept protocol=tcp src-address=xxx.xxx.xxx.xx/29 in-interface=ether1-gateway dst-port=25 log=no log-prefix=""
6 X chain=input action=accept protocol=tcp src-address=xxx.xxx.xxx.xx/29 in-interface=ether1-gateway dst-port=22 log=no log-prefix=""
7 X chain=input action=accept protocol=udp src-address=xxx.xxx.xxx.xx/29 in-interface=ether1-gateway dst-port=25 log=no log-prefix=""
8 ;;; PPTP
chain=input action=accept protocol=tcp in-interface=ether1-gateway dst-port=1723 log=no log-prefix=""
9 ;;; PPTP
chain=input action=accept protocol=gre in-interface=ether1-gateway log=no log-prefix=""
10 ;;; Mikrotik Autoupdate
chain=input action=accept protocol=tcp in-interface=ether1-gateway dst-port=38412 log=no log-prefix=""
11 ;;; Mikrotik Autoupdate
chain=input action=accept protocol=tcp in-interface=ether1-gateway dst-port=38414 log=no log-prefix=""
12 chain=input action=accept protocol=tcp in-interface=ether1-gateway dst-port=8291 log=no log-prefix=""
13 chain=input action=accept protocol=tcp in-interface=ether1-gateway dst-port=8391 log=no log-prefix=""
14 X chain=input action=accept protocol=tcp in-interface=ether1-gateway dst-port=22 log=no log-prefix=""
15 X ;;; Pusti sve sto nije WAN
chain=input action=accept in-interface=ether1-gateway log=no log-prefix=""
16 X ;;; Pusti sve sto nije WAN
chain=forward action=accept in-interface=ether1-gateway log=no log-prefix=""
17 ;;; default configuration
chain=input action=drop in-interface=ether1-gateway log=no log-prefix="DropSveOstalo"
18 X chain=forward action=accept src-address=192.168.9.0/24 dst-address=192.168.2.1 in-interface=eoip-bridge log-prefix=""
19 chain=forward action=accept src-address=192.168.9.10 dst-address=192.168.0.0/16 in-interface=eoip-bridge log-prefix=""
20 chain=forward action=drop src-address=192.168.9.0/24 dst-address=192.168.0.0/16 in-interface=eoip-bridge log-prefix=""
21 chain=forward action=drop src-address=192.168.10.0/24 dst-address=192.168.0.0/16 in-interface=eoip-bridge log-prefix=""

[admin@Preventiva] /ip firewall> mangle print

Flags: X - disabled, I - invalid, D - dynamic

[admin@Preventiva] /ip firewall> nat print

Flags: X - disabled, I - invalid, D - dynamic
0 ;;; default configuration
chain=srcnat action=masquerade to-addresses=0.0.0.0 out-interface=ether1-gateway log=no log-prefix=""
1 chain=dstnat action=redirect to-ports=8291 protocol=tcp src-address=0.0.0.0 in-interface=ether1-gateway dst-port=8391 log=no log-prefix=""
2 chain=dstnat action=redirect to-ports=22 protocol=tcp src-address=0.0.0.0 in-interface=ether1-gateway dst-port=122 log=no log-prefix=""
3 ;;; Indico VM
chain=dstnat action=dst-nat to-addresses=192.168.2.240 to-ports=443 protocol=tcp in-interface=ether1-gateway dst-port=443 log=yes log-prefix="indico"
4 ;;; Indico VM
chain=dstnat action=dst-nat to-addresses=192.168.2.240 to-ports=80 protocol=tcp in-interface=ether1-gateway dst-port=80 log=yes log-prefix="indico"

[admin@Preventiva] /ip firewall> raw print

Flags: X - disabled, I - invalid, D - dynamic

[admin@Preventiva] /ip firewall> service-port print

Flags: X - disabled, I - invalid
# NAME PORTS
0 ftp 21
1 tftp 69
2 irc 6667
3 h323
4 sip 5060
5061
5 pptp
6 udplite
7 dccp
8 sctp

DrLove73 · Tue Nov 27, 2018 12:43 am

So to sum up:
1. Incomming PPTP conection to PPTP server is treated as forwarding? to somewhere inside, needs proto 47 (PPTP pass-through) in INPUT chain for successful connection/tunnel or it hits DROP rule in INPUT chain.
2. Outgoing connection to Mikrotik servers is treated as INCOMING connection and hits DROP rule in INPUT chain.

That DROP rule in INPUT might have had earlier a Connection:"related" checkbox, and that allowed it to work properly, so I returned that when I remembered:

3 ;;; default configuration
chain=input action=accept connection-state=related in-interface=ether1-gateway log=no log-prefix=""

And not PPTP and "Check for Update" work normally.. Why is that connection-state=related so important for thing to work?

pe1chl · Tue Nov 27, 2018 3:02 pm

Because "related" passes traffic that is a reply to the outgoing connections being made. When you do not have "related" as allowed, you will need to have matching input rules for all possible outgoing connections, which gets unpractical very quickly.
Read a document about the packet flow in a MikroTik router or a generic Linux system to understand why you need an input rule for outgoing traffic.

Timeout when upgrading

Timeout when upgrading

Re: Timeout when upgrading

Re: Timeout when upgrading

Re: Timeout when upgrading

Re: Timeout when upgrading

Re: Timeout when upgrading

Re: Timeout when upgrading

Re: Timeout when upgrading

Re: Timeout when upgrading

Re: Timeout when upgrading