Community discussions

MikroTik App
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Topic Author
Posts: 1550
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

How to auto-reboot if remote IP down for 5 minutes

Wed Oct 26, 2011 2:25 am

I started a topic in another section of this forum - but the following actually now belongs here...

This is a step by step procedure to configure a mikrotik to auto-reboot after pings to a remote IP address have gone un-answered for 5 minutes.
Note - The Mikrotk Watchdog "Watch IP address" reboots in 60 seconds - not what I wanted
Note - The Mikrotik Netwatch has a sliding window where a down action could happen in as little as a few seconds - not what I wanted

I wanted a way to auto reboot my mikrotk after 5 minutes. This way I can reboot all kinds of servers and links elsewhere and not trigger my mikrotkks to auto-reboot.


OK - I finally got something working to do what I wanted.

First - what this does:

#1 It will ping test an IP address. This can be a remote IP address
#2 If the remote IP address does not answer any pings for about 5 minutes, then the router reboots.



To set it up - follow these steps using Winbox:

Note: In my example I am testing to remote IP address 192.0.2.254
Note: You will need to change anything I have below from 192.0.2.254 to your IP address you wish to test.
Note: Example if you will be testing 10.10.18.5 then change stuff like what is shown below
Note: "NetWatchBoot-192.0.2.254" would be changed to "NetWatchBoot-10.10.18.5"


A - (Optional) - remove the IP address from System-->Watchdog-->Watch Address

B - Go to System-->Scripts
Click on the red "+"
Give your script a Name (in my example I will use "NetWatchBoot-192.0.2.254"
C - In the Source: area paste in the entire line below:
:if ([/ping 192.0.2.254 interval=5 count=60] =0) do={
log info "my ping watchdog is down" ; /system reboot
}

D - Click the Apply button - then click the OK button

At this time - we have a script - it is not running or active yet.

E - Go to Tools-->Netwatch
Click on the red "+"
On the Host tab - type in 192.0.2.254
(Your IP address may be different - it is the same IP address you used from "B" above)
F - Click on the Down tab
Paste in the line below:
/system script run NetWatchBoot-192.0.2.254
G - Click the Apply button - then click the OK button

---- You are done -----

Your system will now auto reboot only after 5 (up to 6) minutes of the remote IP address not pinging.

If you want to disable the auto-reboot function, all you need to do is:
Tools-->Netwatch and disable the line that shows the IP address you are testing to.

-------------------------------------------------------------------------------
What is nice about this is - if you are testing to a server or something and that server takes longer
than 60 seconds to reboot, your mikrotik will not reboot. The remote IP address has to be down for 5 (to 6) minutes
to trigger an automatic reboot.
---------------------
and yes - i know I could of thrown this is the schedular instead of triggering my script from netwatch - but I like it more this way
FYI - This is my first use of Mikrotik script and Mikrotik netwatch

I hope this help others - if it works for you then please post a reply

Tom Jones
A WISP up in North Idaho
 
User avatar
sadeghrafie
Long time Member
Long time Member
Posts: 514
Joined: Sat Nov 14, 2009 11:28 am
Location: Bushehr, IRAN

Re: How to auto-reboot if remote IP down for 5 minutes

Wed Oct 26, 2011 11:23 am

Good job Tom.
I think it can be a feature of next Mikrotik OS release. :)
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Topic Author
Posts: 1550
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: How to auto-reboot if remote IP down for 5 minutes - Bet

Wed Oct 26, 2011 8:18 pm

I did some clean-up on the scripts so they work a little better now for what I want.

Note - this can be done on client radios, APs and core routers...

Below is my updated script stuff:

First - what this does:

#1 It will ping test an IP address. This can be a remote IP address
#2 If the remote IP address does not answer any pings for about 5 minutes, then the router reboots.


To set it up - follow these steps using Winbox:

Note: In my example I am testing to remote IP address 192.0.2.254
Note: You will need to change anything I have below from 192.0.2.254 to your IP address you wish to test.
Note: Example if you will be testing 10.10.18.5 then change stuff like what is shown below
Note: "NetWatchBoot-192.0.2.254" would be changed to "NetWatchBoot-10.10.18.5"
Note: Again - change all IP address shown below everywhere from "192.0.2.254" to the IP address you are testing

A - Remove the IP address from System-->Watchdog-->Watch Address
(We do not want the System-->Watchdog-->Watch Address to cause a reboot in 60 seconds)
(The Watch Addres box should be empty and should not have a white background)
(There should be a check in the box for Watchdog timer)


B - Go to System-->Scripts
Click on the red "+"
Give your script a Name (in my example I will use "NetWatchBoot-192.0.2.254"

C - In the Source: area paste in the lines below:

:if ([/ping 192.0.2.254 interval=5 count=60] =0) do={
/system reboot
}

(There should be a single blank line under the last } in your script)

D - Click the Apply button - then click the OK button

At this time - we have a script - it is not running or active yet.

E - Go to Tools-->Netwatch
Click on the red "+"
On the Host tab - type in 192.0.2.254
(Your IP address may be different - it is the same IP address you used from "B" above)

F - Click on the Down tab
Paste in the line below:

log info "Netwatch missed a ping to 192.0.2.254 - starting 5 minute timeout script" ; /system script run NetWatchBoot-192.0.2.254

(There should be a single blank line under the line you just put in)

G - Click the Apply button - then click the OK button

---- You are done -----

NOTE: In your Logs, if you see "Netwatch missed a ping to 192.0.2.254 - starting 5 minute timeout script", then this is an indicator the remote IP address you are testing to was not reachable.
If you see many of these messages in your logs, you should examine your network reliability to the device you are testing to.

NOTE: In your Logs, if you see "router rebooted", then your router may of been rebooted by the script "NetWatchBoot-192.0.2.254" however this may not be what caused the reboot - it may of been rebooted for other reasons.

NOTE: In System-->Scripts-->(Jobs tab), if you see "NetWatchBoot-192.0.2.254", then the script "NetWatchBoot-192.0.2.254" is running at that moment because it was called from a missed ping in your Tools-->Netwatch-->192.0.2.254 configuration. The script "NetWatchBoot-192.0.2.254" is currently and actively performing ping tests for 5 minutes. If during the 5 minutes you receive a ping the script will abort. If the script continues to miss test pings for a full 5 minutes, then the router will reboot.


I like this updated version of the script because it now places some information in the logs.

Tom Jones
a WISP in North Idaho

- - - - - - - - - -
Added some information on Oct 31, 2011

Here is an option for the script. If you want your logs for the netwatch script to show up in blue instead of black - then on section "F" above. For me, it is easier to spot the log lines in blue.

change from "log info" to "log warning" as shown on the lines below:

log info "Netwatch missed a ping to 192.0.2.254 - starting 5 minute timeout script" ; /system script run NetWatchBoot-192.0.2.254

change to:

log warning "Netwatch missed a ping to 192.0.2.254 - starting 5 minute timeout script" ; /system script run NetWatchBoot-192.0.2.254


Tom Jones
 
Saasouh
just joined
Posts: 12
Joined: Sat Dec 10, 2011 1:25 pm

Re: How to auto-reboot if remote IP down for 5 minutes

Sat Jan 14, 2012 3:13 pm

Dear friend

I am really in need for that script, I have tried it but it seems that it's not working with me .
I wrote this in Netwatch :


log info "Netwatch missed a ping to 192.168.61.7 - starting 1 minute timeout script" ; /system script run NetWatchBoot-192.168.61.7

and in system script :


:if ([/ping 192.168.61.7 interval=1 count=60] =0) do={
/system reboot
}

please tell me what I did wrong and why the router with IP 192.168.61.9 is not restarting if ping between the two devices is not ok.
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Topic Author
Posts: 1550
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: How to auto-reboot if remote IP down for 5 minutes

Wed Jan 18, 2012 8:57 pm

[quote="Saasouh"]Dear friend
I am really in need for that script, I have tried it but it seems that it's not working with me .

This is for you:

======================================================================
Here is the information:

First - I assume your Mikrotik router performing the test is IP address 192.168.61.9

Second - I assume the remote device you want your Mikrotik router to test to is 192.168.61.7

Here are the instructions:

#1 Winbox into your 192.168.61.9 Mikrotik

#2 Go to Tools - Netwatch

#3 If you have anything in there then remove it

#4 Click on the RED "+"

#5 On the Host tab.
Type in "192.168.61.7"
Interval should be "00:01:00"
Timeout should be "1000"

#6 click on the Down tab.
Paste in the entire line below:
log warning "Netwatch missed a ping to 192.168.61.7 - starting 5 minute timeout script" ; /system script run NetWatchBoot-192.168.61.7

Now push the Enter button (you want your curser on the bottom on a blank line)

What you typed in should look like this:

|-----------------------------------------------
|log warning "Netwatch missed a ping to
|192.168.61.7 - starting 5 minute timeout
|"script" ; /system script run NetWatchBoot-
|192.168.61.7
|
|
|------------------------------------------------
Note - You need to make sure you pushed the Enter button after the last line and hit the "APPLY" button.

**** At this point - you have finished part 1 of 2 things you need to do *****

#7 Go to System - Scripts

#8 If you have anything in there then remove it

#9 Click on the RED "+"

#10 In the name box, paste in the line below:

NetWatchBoot-192.168.61.7

#11 The owner should be "admin" and make sure every check box is checked.

#12 In the Source: box, paste in the following 3 lines:[/color]
:if ([/ping 192.0.2.254 interval=5 count=60] =0) do={
/system reboot
}


Now push the Enter button (you want your curser on the bottom on a blank line)

What you typed in should look like this:
|---------------------------------------------
|:if ([/ping 192.0.2.254 interval=5 count=60]
|=0) do={
|/system reboot
|}
|
|
|--------------------------------------------

Note - You need to make sure you pushed the Enter button after the last line and hit the "APPLY" button.

***** At this point - you have finished part 2 of 2 things you need to do - it is running now *****
INFORMATION - INFORMATION - INFORMATION -

The Netwatch you just configured (part 1 of 2) checks every minute to see if it can ping "192.168.61.7".
If it can ping "192.168.61.7" then the script will exit and auto run another test a minute later.

If the Netwatch script can not ping "192.168.61.7" then it will run the system-script you created (part 2 of 2).
The system-script will try every 60 seconds to ping "192.168.61.7" for five times.
If there is an answer to ping to "192.168.61.7" during this 5-minute period, then the system-script will exit.
If there are no answers to pings to "192.168.61.7" during the 5-minute period, then the Mikrotik router will reboot.

NOTE - NOTE - NOTE
Every time the Netwatch-script fails to ping "192.168.61.7" it will place a blue line in your Mikrotik log.
If you see many blue lines in your log file that look like
Netwatch missed a ping to 192.168.61.7 - starting 5 minute timeout script
Then you should check how good your connections are from your Mikrotik router to the remote 192.168.61.7 device

I hope this helps
Tom Jones - A WISP in North Idaho
 
mdminfo
just joined
Posts: 13
Joined: Sat Jun 09, 2007 2:27 pm

Re: How to auto-reboot if remote IP down for 5 minutes

Tue Apr 03, 2012 1:06 am

great job! it works perfectly!
 
heviejob
Member Candidate
Member Candidate
Posts: 171
Joined: Mon Nov 30, 2009 4:54 pm

Re: How to auto-reboot if remote IP down for 5 minutes

Sun Sep 09, 2012 8:59 pm

I don't mean to hijack but it would be nice to have a script reboot the router if CPU or RAM goes to over 80% - 100%
for 5 mins or so. I have a remote router that is at 90% ram from cacti but now I cannot ssh to it, web access
is failing so is the API.
 
n21roadie
Forum Guru
Forum Guru
Posts: 1949
Joined: Fri Aug 07, 2009 10:36 pm
Location: Limerick,Ireland

Re: How to auto-reboot if remote IP down for 5 minutes

Mon Sep 10, 2012 1:27 pm

I don't mean to hijack but it would be nice to have a script reboot the router if CPU or RAM goes to over 80% - 100%
for 5 mins or so. I have a remote router that is at 90% ram from cacti but now I cannot ssh to it, web access
is failing so is the API.
Not sure if the same would apply to ram but i tried to have a router reboot on high CPU load, http://forum.mikrotik.com/viewtopic.php ... n+high+cpu but the script would not run correctly at 100% CPU load because the script which requires a number of samples cpu % to give a average and if above a certain level would reboot the router even tried reducing the samples.
 
seanfilipin
just joined
Posts: 2
Joined: Wed Oct 21, 2015 7:44 am

Re: How to auto-reboot if remote IP down for 5 minutes

Wed Oct 21, 2015 7:05 pm

Hi,
I was just searching for a solution and I think that with this kind of combination between scripts and netwatch will be perfect my backup solution.
Actually I've got some big problems with my firewall; I have a CCR1036-8G-2S+EM with a dsl modem on port 1 (hopefully in some months i'll be connected with fiber optics but up to know I've got a vdsl with 40 mbps down and 20 mbps) and it happens sometimes (thanks god it happened just once up to know!) that the mikrotik router freezes. Since I've got lot's of services running on my location I cannot have the system down for any time more that 1 - 5 minutes that's why I need a emergency backup line.

My idea was to connect on port two of the modem another mikrotik router (something small that I can buy for something like 70 dollars not the CCR1036-8G-2S+EM (it cost's too much to buy a second one) and upload on this smaller router the same configuration that the main router has.

Actually to connect to the internet and get a static IP on the internet I'm using a ppoe client on the main router so I was thinking of creating the same ppoe client on the smaller router and then, using the netwatch, ping the main router and when this one goes down activate on the secondary router the ppoe client and get the line up again. With this kind of configuration I should get a always on-line solution I hope! (in this way I will still have access to the line and so I will be able to remotely power off and then back on my main router - to reset it)


Up to now all works but I get stack when I need to deactivate the ppoe client on the secondary router if I am able to reset the main one and the main one goes up. What do I have to modify in the script to get this last thing working?

And another question... Do you personally think this would be a good solution?
 
LeGreatMaxiking
just joined
Posts: 2
Joined: Tue Jul 21, 2015 4:00 pm

Re: How to auto-reboot if remote IP down for 5 minutes

Fri Feb 12, 2016 2:50 pm

Can you tell how i make a working script for my MikroTik RouterBOARD RB951G-2HnD my internet connection is a LTE Stick with newest software (hilink) and the internet is not as stable sometimes.. so sometimes it just stop working...

Can you show me a script for: pinging to a website (eg google.com) and if its not reachable it restarts the usb port after 1 min?

THanks for your helps mates!
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Topic Author
Posts: 1550
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: How to auto-reboot if remote IP down for 5 minutes

Fri Feb 12, 2016 6:55 pm

Can you tell how i make a working script for my MikroTik RouterBOARD RB951G-2HnD my internet connection is a LTE Stick with newest software (hilink) and the internet is not as stable sometimes.. so sometimes it just stop working...

Can you show me a script for: pinging to a website (eg google.com) and if its not reachable it restarts the usb port after 1 min?

THanks for your helps mates!
Good question - and I have an answer for you :)

First - There is the Mikrotik watchdog where you can put in an IP address. Although this is just OK, it is not smart and you have little control over how it functions.

Second - My solution works GREAT:
#1 - create a tools-netwatch
give it a name of the IP address you wish to test to.
Here is my example:
Host: 192.0.2.254
Up: blank
Down: (Put in the information below:

log info "Netwatch missed a ping to 192.0.2.254 - starting timeout auto reboot script" ; /system script run NetWatchBoot-192.0.2.254

#2 - create a system-scripts
give it a name of: NetWatchBoot-192.0.2.254
In the Source section, paste in the information below:

:local addresstoping 192.0.2.254;
:local interface "wlan1";
#
:local continue true;
:local counter 0;
:local maxcounter 18;
:local sleepseconds 10;
:local goodpings 0;
:log error "-----> Tom's Netwatch-Script-Warning - Netwatch could not ping $addresstoping - Will begin further testing in $sleepseconds seconds - and will continue for $maxcounter times $sleepseconds seconds";
:while ($continue) do={
:set counter ($counter + 1);
:delay $sleepseconds;
:if ([/ping $addresstoping interval=1 count=1] =0) do={
:log info "----->ping to $addresstoping failed on attempt $counter of $maxcounter -- Will try again in $sleepseconds seconds";
} else {
:log warning "-----> ping success on to $addresstoping attempt $counter of $maxcounter <----- No Further testing needed --- Program will exit -----";
:set continue false;
:set goodpings ($goodpings +1);
/interface wireless monitor $interface once without-paging do={
:local status $"status";
:local band $"band";
:local freq $"frequency";
:local wprotocol $"wireless-protocol";
:local noise $"noise-floor";
:local signal $"signal-strength";
:local snr $"signal-to-noise";
:local thruput $"p-throughput";
:log info "-----> Status: $status --- Band: $band --- Frequency: $frequency --- WProtocol: $wprotocol --- NoiseFloor: $noise";
:log info "-----> Optional Info if Available ---> SignalStrength: $signal --- SNR: $signal --- PThroughput: $throughput";
/interface wireless monitor $interface once
:local txr $"tx-rate";
:local rxr $"rx-rate";
:local sstr $"signal-strength";
:local signoise $"signal-to-noise";
:local curdistance $"current-distance";
:local txccq $"tx-ccq";
:local rxccq $"rx-ccq";
:log info "-----> TxRate: $txr --- RxRate: $rxr --- SignalStreng: $sstr --- SignalToNoise: $signoise --- CurrentDistance: $curdistance --- TxCcq: $txccq --- RxCcq: $rxccq";
};
}
:if ($counter=$maxcounter) do={:set continue false;}
}
:if ($"goodpings" = 0 ) do={
:log info "-----> Rebooting in 15 seconds";
:delay 5;
/file print file=ScriptRebootReason
/file set ScriptRebootReason.txt contents="Rebooted by Toms script on $[/system clock get date] at $[/system clock get time]"
:log error "-----> Rebooting in 10 seconds";
:delay 5;
:log error "-----> Rebooting in 5 seconds";
:delay 5;
:log error "-----> Rebooting now";
:delay 1;
/system reboot
/system reboot
/system reboot
/system reboot
}


---- My notes: ----
My example is testing to IP address: 192.0.2.254
If you want to test to another IP address, then change 192.0.2.254 to the new IP address (including the script name)

In the script - pay attention to the top section:
:local addresstoping 192.0.2.254;
:local interface "wlan1"; --- Log information about this wlan interface
:local maxcounter 18; --- Number of total fails to try prior to rebooting (18 x 10 = 180 seconds) If you want to wait 5 minutes then change to 30
:local sleepseconds 10; --- Time to wait between ping retry


Give it a try and look at your logs and also - look at files.

Your logs will show something like this:
script running
ping try count
success program will exit (and keep a log of the problem)
or a reboot

Your file will show a new file created named: ScriptRebootReason.txt
The contents of this file is not important - what is important is the date & time of this file
You can delete this file any time you want - a new one will be created each time there is a problem.


Please let me know how this works for you. I have been using it for many years on hundreds of Mikrotiks - both APs and customer client devices - If it is a Mikrotik then I install these scripts.

North Idaho Tom Jones
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Topic Author
Posts: 1550
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: How to auto-reboot if remote IP down for 5 minutes

Fri Feb 12, 2016 7:05 pm

I forgot to mention:

With the Netwatch and script:

If you are working on a Mikrotik that can not ping the test IP and you do not want it auto-rebooting while you are working on it - then simply just disable the Netwatch. When you are finished working on it and want it to be active again then just re-enable.

The timeouts I use work great - and even work through an ROS upgrade.

Also - another great thing about this is - if a customer network is saturated and dropping packets - the retry count will keep retrying - where as the default Mikrotik watchdog will simply just give you problems.

North Idaho Tom Jones
 
User avatar
gyropilot
Frequent Visitor
Frequent Visitor
Posts: 64
Joined: Sat Sep 10, 2016 10:49 pm
Location: SE Arizona USA

Re: How to auto-reboot if remote IP down for 5 minutes

Sat Feb 11, 2017 10:02 pm

Hi Tom,

I'm brand new to scripting and have found your examples very helpful. I appreciate you sharing them.

Question: Are the quotes in the following line really supposed to be there?

:if ($"goodpings" = 0 ) do={
I couldn't get the script to work until I removed them.

Thanks in advance,

John
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Topic Author
Posts: 1550
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: How to auto-reboot if remote IP down for 5 minutes

Wed Mar 01, 2017 4:07 pm

Hi Tom,

I'm brand new to scripting and have found your examples very helpful. I appreciate you sharing them.

Question: Are the quotes in the following line really supposed to be there?

:if ($"goodpings" = 0 ) do={
I couldn't get the script to work until I removed them.

Thanks in advance,

John
I am sorry - I don't remember - I wrote that 6 years ago

What that section of code is doing is
- If a good ping then log and exit
- If a bad ping - then log and repeat until a maximum failed ping count.
- At maximum ping count - log - write a file to files then reboot
 
romp
just joined
Posts: 2
Joined: Thu Mar 09, 2017 9:43 am

Re: How to auto-reboot if remote IP down for 5 minutes

Thu Mar 09, 2017 9:50 am

Hi, gents!

Sorry for digging into very old post, but hope someone can reply still.
I have a problem with this script as it's based on presumption that there will be no any reply on ping. In fact, 4 of 5 pings die without reply and 1 out of 5 is replied by router itself with "host unreachable". So far the count of ping replies is >0 and router does not reboot so far. Can someone suggest fix for script?
 
romp
just joined
Posts: 2
Joined: Thu Mar 09, 2017 9:43 am

Re: How to auto-reboot if remote IP down for 5 minutes

Thu Mar 09, 2017 12:51 pm

And finally the solution:
disable ICMP echo reply "destination host unreachable" on router.

/ip firewall filter add protocol=icmp icmp-options=3 chain=output action=drop

now there are no replies from router if host unreachable, so count=0!
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Topic Author
Posts: 1550
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: How to auto-reboot if remote IP down for 5 minutes

Fri Mar 10, 2017 7:13 pm

FYI - I am sometime planning to re-write a 2'nd newer version of my netwatch-script-reboot-on-ping-loss program.

My idea for the 2'nd version is to have the following additional features:
- Option to ping two different IP addresses and then reboot if one-and/or-both IP addresses fail to answer pings
- Option which prior to a reboot is to auto-perform a site-survey and/or frequency-usage and store the results in the files directory - then reboot
- Option to enable sending email and/or logging on ping-drops and/or reboots
- improve the documentation of what does what in the script(s)

Anybody have some ideas for a 2'nd version ?
Would anybody like to write it up themselves (I am sure there are more Mikrotik skilled/experienced script programmers than myself).

North Idaho Tom Jones
 
User avatar
gyropilot
Frequent Visitor
Frequent Visitor
Posts: 64
Joined: Sat Sep 10, 2016 10:49 pm
Location: SE Arizona USA

Re: How to auto-reboot if remote IP down for 5 minutes

Fri Mar 10, 2017 8:40 pm

FYI - I am sometime planning to re-write a 2'nd newer version of my netwatch-script-reboot-on-ping-loss program.
Hi Tom,

I like the idea of adding the option for a second IP address to ping against to make double sure failure of the primary IP address isn't causing a false WAN failure.

I'm using a stripped down (simplified) version of your original script along with Netwatch to restart an attached USB cellular modem when needed. It's been working really well.

Using the script has greatly reduced false triggers which used to occur whenever my cellular provider refreshes the USB modem's DHCP lease... usually daily. That refresh temporarily interrupts WAN access for about 30 to 60 seconds, so I have the script reboot after 2 minutes of unsuccessful pings spaced 10 seconds apart.

Here's the script I'm currently using:
:local continue true;
:local counter 0;
:local maxcounter 12;
:local sleepseconds 10;
:local goodpings 0;
:log error "Script will further test ping to WAN in $sleepseconds seconds and will continue for $maxcounter x $sleepseconds seconds.";
:while ($continue) do={
    :set counter ($counter + 1);
    :delay $sleepseconds;
    :if ([/ping 8.8.8.8 interval=1 count=1] =0) do={
        :log warning "Ping to WAN failed on attempt $counter of $maxcounter - Will try again in $sleepseconds seconds.";
    } else {
        :log warning "Ping to WAN succeeded on attempt $counter of $maxcounter - No Further testing needed and script will exit.";
        :set continue false;
        :set goodpings ($goodpings +1);
        };
    :if ($counter=$maxcounter) do={
        :set continue false;
        }
    }
:if ($goodpings = 0) do={
    :/system routerboard usb power-reset duration=5
    }
Netwatch script executed on Down:
log warning "Netwatch ping to WAN failed.";/system script run WANPing
Netwatch script executed on Up:
/tool e-mail send to=xxxxxxxx@xxxxxxx.net subject="MikroTik Netwatch Ping Failure" body="MikroTik router Netwatch detected a ping to WAN failure and the WANPing script was executed.  Check router log for details.";log warning "Netwatch ping failure notification email sent."
Have a great day,

John
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Topic Author
Posts: 1550
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: How to auto-reboot if remote IP down for 5 minutes

Fri Mar 10, 2017 8:56 pm

gyropilot
+1 to your reputation

thanks for the useful script mods/ideas

North Idaho Tom Jones
 
Sparxx
just joined
Posts: 13
Joined: Wed Mar 18, 2015 7:59 am

Re: How to auto-reboot if remote IP down for 5 minutes

Wed Mar 29, 2017 11:29 am

Hi guys,
related to adding a second address to be checked it's easy, I mean just add an
 \ and [/ping 8.8.4.4 interval=5 count=60] =0 
after the first ping rule and obviously between the parentheses , this way the script reboots the router only if both conditions are true.
 
vitorcsp
just joined
Posts: 4
Joined: Sat May 20, 2017 2:56 am
Location: Rio de Janeiro - RJ
Contact:

Re: How to auto-reboot if remote IP down for 5 minutes

Sun Jun 25, 2017 2:51 am

Hi!
Very nice, Congratulations !
 
maikel
just joined
Posts: 8
Joined: Thu Jun 08, 2017 2:53 pm

Re: How to auto-reboot if remote IP down for 5 minutes

Thu Dec 06, 2018 1:42 pm

Thanks Tom and John,

I can verify this works on a MT RouterBOARD wAP R-2nD with RouterOS v6.43.7
I only commented out the first logging and changed action to "/system reboot". I scheduled for every two minutes

Cheers,
Maikel
FYI - I am sometime planning to re-write a 2'nd newer version of my netwatch-script-reboot-on-ping-loss program.
Hi Tom,

I like the idea of adding the option for a second IP address to ping against to make double sure failure of the primary IP address isn't causing a false WAN failure.

I'm using a stripped down (simplified) version of your original script along with Netwatch to restart an attached USB cellular modem when needed. It's been working really well.

Using the script has greatly reduced false triggers which used to occur whenever my cellular provider refreshes the USB modem's DHCP lease... usually daily. That refresh temporarily interrupts WAN access for about 30 to 60 seconds, so I have the script reboot after 2 minutes of unsuccessful pings spaced 10 seconds apart.

Here's the script I'm currently using:
:local continue true;
:local counter 0;
:local maxcounter 12;
:local sleepseconds 10;
:local goodpings 0;
:log error "Script will further test ping to WAN in $sleepseconds seconds and will continue for $maxcounter x $sleepseconds seconds.";
:while ($continue) do={
    :set counter ($counter + 1);
    :delay $sleepseconds;
    :if ([/ping 8.8.8.8 interval=1 count=1] =0) do={
        :log warning "Ping to WAN failed on attempt $counter of $maxcounter - Will try again in $sleepseconds seconds.";
    } else {
        :log warning "Ping to WAN succeeded on attempt $counter of $maxcounter - No Further testing needed and script will exit.";
        :set continue false;
        :set goodpings ($goodpings +1);
        };
    :if ($counter=$maxcounter) do={
        :set continue false;
        }
    }
:if ($goodpings = 0) do={
    :/system routerboard usb power-reset duration=5
    }
Netwatch script executed on Down:
log warning "Netwatch ping to WAN failed.";/system script run WANPing
Netwatch script executed on Up:
/tool e-mail send to=xxxxxxxx@xxxxxxx.net subject="MikroTik Netwatch Ping Failure" body="MikroTik router Netwatch detected a ping to WAN failure and the WANPing script was executed.  Check router log for details.";log warning "Netwatch ping failure notification email sent."
Have a great day,

John
 
drdedus
just joined
Posts: 14
Joined: Thu Mar 17, 2016 6:24 pm

Re: How to auto-reboot if remote IP down for 5 minutes

Tue Dec 25, 2018 11:05 am

hello and thank you.
i get a message in log that is not enough permissions.
what permissions are need it just to reboot after 5 minutes?
 
User avatar
gyropilot
Frequent Visitor
Frequent Visitor
Posts: 64
Joined: Sat Sep 10, 2016 10:49 pm
Location: SE Arizona USA

Re: How to auto-reboot if remote IP down for 5 minutes

Tue Dec 25, 2018 6:20 pm

hello and thank you.
i get a message in log that is not enough permissions.
what permissions are need it just to reboot after 5 minutes?

I encountered the same problem and just checked the “Don’t Require Permissions” checkbox on the script setup screen. Now I don’t know if that’s considered safe from a router security standpoint, but it eliminated the problem.

John
 
User avatar
TomjNorthIdaho
Forum Guru
Forum Guru
Topic Author
Posts: 1550
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: How to auto-reboot if remote IP down for 5 minutes

Wed Dec 26, 2018 6:51 pm

Updated version(s) of my WatchDog auto-reboot scripts I am now using.

FYI - Back when I first posted my script to auto reboot a Mikrotik when the network lost connectivity , it used to work. Since then (years ago and several ROS versions ago) my scripts ran great - but newer versions of ROS have changed some permissions needed for scripts to perform what my scripts were trying to do. The good news is I have changed and modified my scripts and for me , they now function better.

Below are some instructions and the scripts to configure your Mikrotik to use my WatchDog auto-reboot scripts.

1st - On your core router facing the Internet , create a loopback interface (or a bridge or any interface) that has an ip address of 192.0.2.254 / 32
Note , 192.0.2.254 is a safe IP address to use because it is not on the Internet and the entire 192.0.2.0/24 network is intended to be used for in-house lab testing networks. After you have a 192.0.2.254/32 IP address on your network, every device in your network should be able to ping 192.0.2.254

2nd - Create three scripts on all of your other Mikrotik routers using the following:

Script name: "RebootScript"
Script contents;
/system reboot
/system reboot
#

Check the policy [x] reboot


Script name: "Wlan1-SiteSurvey"
Script contents:
/interface wireless scan wlan1 duration=90 save-file=ManualSiteSurvey-wlan1-90seconds--txt.rtf
Check Polity; ftp , read , policy , password , sensitive , reboot , write , test , sniff , romon

Script name: "Wlan2-SiteSurvey"
Script contents:
/interface wireless scan wlan2 duration=90 save-file=ManualSiteSurvey-wlan1-90seconds--txt.rtf
Check Polity; ftp , read , policy , password , sensitive , reboot , write , test , sniff , romon


3rd , create two netwatch settings;

3A) Netwatch Host (IP address): 192.0.2.254

Interval: 00:05:00

Down action:
#
:local addresstoping 192.0.2.254
:local continue true
:local counter 0
:local maxcounter 60
:local sleepseconds 5
:local goodpings 0
#
:log warning "NetWatchVersion2-SiteSurvey - -----> Tom's NetWatchVersion2-SiteSurvey"
:log warning "NetWatchVersion2-SiteSurvey - -----> Ping testing for $maxcounter times every $sleepseconds seconds"
#
:while ($continue) do={
:set counter ($counter + 1)
:delay $sleepseconds
#
:if ([/ping $addresstoping interval=1 count=1] =0) do={
:log warning "NetWatchVersion2-SiteSurvey - ----->ping to $addresstoping failed on attempt $counter of $maxcounter -- Will try again in $sleepseconds seconds"
} else {
:log warning "NetWatchVersion2-SiteSurvey - -----> ping success on to $addresstoping attempt $counter of $maxcounter <----- No Further testing needed --- Program will exit -----"
exit
quit
end
:set continue false
:set goodpings ($goodpings +1)
}
:if ($counter=$maxcounter) do={:set continue false;}
}
#
:log warning "NetWatchVersion2-SiteSurvey - Performing site survey 1-of-2 using wlan1 for 20 seconds"
/interface wireless scan wlan1 duration=20 save-file=SiteSurvey-wlan1-20seconds--txt.rtf
:log warning "NetWatchVersion2-SiteSurvey - Performing site survey 2-of-2 using wlan1 for 90 seconds"
/interface wireless scan wlan1 duration=90 save-file=SiteSurvey-wlan1-90seconds--txt.rtf
:log warning "NetWatchVersion2-SiteSurvey - Site Surveys completed and saved to files"
:delay 10
exit
quit
end
#

3B) Netwatch Host (IP address): 192.0.2.254

Interval: 00:05:00

Down action:
#
:local addresstoping 192.0.2.254
:local continue true
:local counter 0
:local maxcounter 120
:local sleepseconds 5
:local goodpings 0
#
:log error "-NetWatchVersion2-Reboot-NotConnected - ----> Tom's NetWatchVersion2"
:log error "NetWatchVersion2-Reboot-NotConnected - -----> Ping testing for $maxcounter times every $sleepseconds seconds"
#
:while ($continue) do={
:set counter ($counter + 1)
:delay $sleepseconds
#
:if ([/ping $addresstoping interval=1 count=1] =0) do={
:log error "NetWatchVersion2-Reboot-NotConnected - ----->ping to $addresstoping failed on attempt $counter of $maxcounter -- Will try again in $sleepseconds seconds"
} else {
:log warning "NetWatchVersion2-Reboot-NotConnected - -----> ping success on to $addresstoping attempt $counter of $maxcounter <----- No Further testing needed --- Program will exit -----"
exit
quit
end
:set continue false
:set goodpings ($goodpings +1)
}
:if ($counter=$maxcounter) do={:set continue false;}
}
#
:log error "NetWatchVersion2-Reboot-NotConnected - --- executing reboot script in to seconds"
:delay 10
/system script run RebootScript
:delay 10
:log error "NetWatchVersion2-Reboot-NotConnected - --- I should of rebooted by now - will retry a reboot script againt in to seconds"
:delay 10
/system script run RebootScript
:delay 10
exit
quit
end
#



These scripts and netwatch settings now do the following when a Mikrotik can NOT ping 192.0.2.254

- One netwatch will perform a site-survey and store the results as a file. Using winbox, you can drag the files it creates "SiteSurvey-wlan1-20seconds--txt.rtf" and "SiteSurvey-wlan1-90seconds--txt.rtf" to your windows desktop and double-click on it to open it and see the results of a remote site-survey.

- One netwatch will perform a reboot if the Mikrotik remains unable to ping 192.0.2.254 after a few minutes.

- The reboot function has a long enough period prior to a reboot to allow a remote wireless client Mikrotik to perform a site-survey possibly reconnect.
- The scripts will abort if/when 192.0.2.254 starts to answer pings prior to a reboot

If you wish to test this , try this testing procedure:
- configure your core router to answer a ping to 192.0.2.254
- place the above scripts and netwatch code on a test Mikrotik

FYI - Use these scripts at your own risk and assume I may errors in my code
FYI - If you wish to auto reboot all of your mikrotiks and also have your remote Mikrotiks all perform site-surveys (in my case , hundreds to thousands of customer wireless client Mikrotiks) , simple disable 192.0.2.254 on your core router for about 10 minutes.


Please post any results or your ideas on how to improve these WatchDog auto-site-survey auto-reboot scripts.


North Idaho Tom Jones
 
User avatar
Jotne
Forum Guru
Forum Guru
Posts: 3343
Joined: Sat Dec 24, 2016 11:17 am
Location: Magrathean

Re: How to auto-reboot if remote IP down for 5 minutes

Wed Dec 26, 2018 8:32 pm

script
Can you please code tag your post?
Makes it easier to read.

Like this:
:local addresstoping 192.0.2.254
:local continue true
:local counter 0
:local maxcounter 60
:local sleepseconds 5
:local goodpings 0
#
:log warning "NetWatchVersion2-SiteSurvey - -----> Tom's NetWatchVersion2-SiteSurvey"
:log warning "NetWatchVersion2-SiteSurvey - -----> Ping testing for $maxcounter times every $sleepseconds seconds"
#
:while ($continue) do={
:set counter ($counter + 1)
:delay $sleepseconds
#
:if ([/ping $addresstoping interval=1 count=1] =0) do={
:log warning "NetWatchVersion2-SiteSurvey - ----->ping to $addresstoping failed on attempt $counter of $maxcounter -- Will try again in $sleepseconds seconds"
} else {
:log warning "NetWatchVersion2-SiteSurvey - -----> ping success on to $addresstoping attempt $counter of $maxcounter <----- No Further testing needed --- Program will exit -----"
exit
quit
end
:set continue false
:set goodpings ($goodpings +1)
}
:if ($counter=$maxcounter) do={:set continue false;}
}
#
:log warning "NetWatchVersion2-SiteSurvey - Performing site survey 1-of-2 using wlan1 for 20 seconds"
/interface wireless scan wlan1 duration=20 save-file=SiteSurvey-wlan1-20seconds--txt.rtf
:log warning "NetWatchVersion2-SiteSurvey - Performing site survey 2-of-2 using wlan1 for 90 seconds"
/interface wireless scan wlan1 duration=90 save-file=SiteSurvey-wlan1-90seconds--txt.rtf
:log warning "NetWatchVersion2-SiteSurvey - Site Surveys completed and saved to files"
:delay 10
exit
quit
end
 
Krumpeir
just joined
Posts: 1
Joined: Fri Oct 18, 2019 4:36 pm

Re: How to auto-reboot if remote IP down for 5 minutes

Fri Oct 18, 2019 4:49 pm

Try this. Without "script error: no such item (4)" problem:

1st - add Netwatch with Host: 8.8.8.8 and to down place:
:if ([/system resource get uptime]>30) do={
  :log info "Not ping on 8.8.8.8";
  /system script run AutomaticRestart;
}
2nd - add script with name "AutomaticRestart" and source:
:local PingCount 300;
:local stop false;
:while ([/ping 8.8.8.8 count=1 interval=5]=0 && stop=false) do={
  :set PingCount ($PingCount-5);
  #:log info "Reboot after $PingCount s";
 :if ($PingCount<=0) do={
    :set stop true;
    :log info "NOT PING ON 8.8.8.8 - REBOOT!!!";
    /system reboot;
  };
};

:set PingCount (300-$PingCount);
:if (stop=false) do={:log info "Reboot stop after $PingCount/300 s";};
 
Ksee
just joined
Posts: 2
Joined: Fri Nov 01, 2019 12:28 pm
Location: South Africa

Re: How to auto-reboot if remote IP down for 5 minutes

Fri Nov 01, 2019 12:33 pm

Good day Guys,

I have a similar request but not really the same,

What i would like to do is add a script for when a route becomes unreachable that the Mikrotik reboots.
What i would like to achieve is im having issues with stale voice connections and my PABX requires a reboot before they work again has anyone solved this or have an idea what to do ?
 
indy0077
just joined
Posts: 1
Joined: Thu Nov 18, 2021 10:05 pm

Re: How to auto-reboot if remote IP down for 5 minutes

Fri Dec 17, 2021 6:38 am

Dear friend
I am really in need for that script, I have tried it but it seems that it's not working with me .

This is for you:

======================================================================
Here is the information:

First - I assume your Mikrotik router performing the test is IP address 192.168.61.9

Second - I assume the remote device you want your Mikrotik router to test to is 192.168.61.7

Here are the instructions:

#1 Winbox into your 192.168.61.9 Mikrotik

#2 Go to Tools - Netwatch

#3 If you have anything in there then remove it

#4 Click on the RED "+"

#5 On the Host tab.
Type in "192.168.61.7"
Interval should be "00:01:00"
Timeout should be "1000"

#6 click on the Down tab.
Paste in the entire line below:
log warning "Netwatch missed a ping to 192.168.61.7 - starting 5 minute timeout script" ; /system script run NetWatchBoot-192.168.61.7

Now push the Enter button (you want your curser on the bottom on a blank line)

What you typed in should look like this:

|-----------------------------------------------
|log warning "Netwatch missed a ping to
|192.168.61.7 - starting 5 minute timeout
|"script" ; /system script run NetWatchBoot-
|192.168.61.7
|
|
|------------------------------------------------
Note - You need to make sure you pushed the Enter button after the last line and hit the "APPLY" button.

**** At this point - you have finished part 1 of 2 things you need to do *****

#7 Go to System - Scripts

#8 If you have anything in there then remove it

#9 Click on the RED "+"

#10 In the name box, paste in the line below:

NetWatchBoot-192.168.61.7

#11 The owner should be "admin" and make sure every check box is checked.

#12 In the Source: box, paste in the following 3 lines:[/color]
:if ([/ping 192.0.2.254 interval=5 count=60] =0) do={
/system reboot
}


Now push the Enter button (you want your curser on the bottom on a blank line)

What you typed in should look like this:
|---------------------------------------------
|:if ([/ping 192.0.2.254 interval=5 count=60]
|=0) do={
|/system reboot
|}
|
|
|--------------------------------------------

Note - You need to make sure you pushed the Enter button after the last line and hit the "APPLY" button.

***** At this point - you have finished part 2 of 2 things you need to do - it is running now *****
INFORMATION - INFORMATION - INFORMATION -

The Netwatch you just configured (part 1 of 2) checks every minute to see if it can ping "192.168.61.7".
If it can ping "192.168.61.7" then the script will exit and auto run another test a minute later.

If the Netwatch script can not ping "192.168.61.7" then it will run the system-script you created (part 2 of 2).
The system-script will try every 60 seconds to ping "192.168.61.7" for five times.
If there is an answer to ping to "192.168.61.7" during this 5-minute period, then the system-script will exit.
If there are no answers to pings to "192.168.61.7" during the 5-minute period, then the Mikrotik router will reboot.

NOTE - NOTE - NOTE
Every time the Netwatch-script fails to ping "192.168.61.7" it will place a blue line in your Mikrotik log.
If you see many blue lines in your log file that look like
Netwatch missed a ping to 192.168.61.7 - starting 5 minute timeout script
Then you should check how good your connections are from your Mikrotik router to the remote 192.168.61.7 device

I hope this helps
Tom Jones - A WISP in North Idaho
Hi Tom,
I'm trying your script, but I get an error in the log like this: "could not run script Reboot-ping-failed: not enough permissions"
2.jpg
You do not have the required permissions to view the files attached to this post.

Who is online

Users browsing this forum: No registered users and 8 guests