if the router is connected to a modem that has the gateway ip address, it checks that connection. if the modem has lost internet, that is not tested and the router had no way of knowing that.
I'm using the one located at http://wiki.mikrotik.com/wiki/Failover_Scripting and it seems to be working. for a beginners some extra information would be helpful. 1) does it need to be put in a scheduler? 2) and the policies that need to be checked off.