Using fetch to retrieve IP lists

omega-00 · Tue Jun 03, 2008 3:34 pm

From mikrotik newsletter 8
Fetch Tool
For some time now, RouterOS has a neat little feature - it’s called Fetch. It is a HTTP and FTP client program that allows you to download files to your router.
For example fetch in action:
[admin@I] /tool> fetch address=www.mikrotik.com path=index.php
host=forum.mikrotik.com
status: connecting
status: downloading
downloaded: 7KiB
duration: 1s
You can use Fetch in scripting - for example to download auto-config files and new RouterOS versions.

If I retrieve a plaintext list of IP's from a website, is there any way using the MT scripting that I could add these to an address list?
Can't seem to find any commands to actually parse a file or break it into strings...

also, nothing in the 3.0 documentation about scripting at all.. other than the mikrotik newsletter I would never have even known that a fetch command existed :-/

JJCinAZ · Tue Jun 03, 2008 5:48 pm

String parsing like that would be best done elsewhere. I'd have a php or perl process on a linux box elsewhere do the work and return a prepared list to the mikrotik. The fetch cmd can, effectively, remotely execute the linux process.

omega-00 · Wed Jun 04, 2008 9:13 am

Hmm, the list I'm referring to would already be preformatted, just looking for the commands I'd use to actually read the file using the mikrotik? or should I get the script to prepare it in the form

/ip firewall address-list add address=<address pulled from the list> address-list=abc

what command could I then use to run the file as a script or such?

Wed Jun 04, 2008 9:36 am

can't you do it the other way - instead of making the router connect somewhere, make some nix box connect to the routers and execute some commands on the ROS?

omega-00 · Wed Jun 04, 2008 11:07 am

I could but I figured there may be something i can use to do it on the router(s)
that way I could have multiple routers grabbing different IP lists on a regular basis.

Either way I might as well make it a request... the ability to read a file (or the ability to read address lists from one ROS box on another)

I know that other people do similar things on Linux boxes as normis mentioned (normally for iptables on their actuall box thou) so I figure it would be a good feature to have on the MT's either for system-wide black lists, adding unmetered IP ranges etc etc.

changeip · Wed Jun 04, 2008 9:26 pm

Okay ... here is an example of something I am working on. This works under 3.10, I havent tested earlier versions. I also cannot figure out how to use '/tool fetch' in a script and have it do anything afterwards, so I believe you have to use 2 scripts, one to fetch, and another to parse. If someone wants to explain that let me know.

## Parse DSHIELD feed and build an address-list.
## Written by Sam Norris, ChangeIP.com 2008
## Any comments or suggestions welcome in the forums.
##
## 06/03/08 - Initial list parsing.

#/tool fetch address=feeds.dshield.org host=feeds.dshield.org mode=http path=block.txt

:if ( [/file get [/file find name=block.txt] size] > 0 ) do={ 

  /ip firewall address-list remove [/ip firewall address-list find list=dshield]

  :global content [/file get [/file find name=block.txt] contents] ;
  :global contentLen [ :len $content ] ;

  :global lineEnd 0;
  :global line "";
  :global lastEnd 0;

  :do {
       :set lineEnd [:find $content "\n" $lastEnd ] ;
       :set line [:pick $content $lastEnd $lineEnd] ;
       :set lastEnd ( $lineEnd + 1 ) ;

       :if ( [:pick $line 0 1] != "#" ) do={

	 :if ([:typeof [:toip [:pick $line 0 [:find $line "\t"] ] ] ] != "nil") do={
		:local pos1 [:find $line "\t" 0]
		:local pos2 [:find $line "\t" $pos1]
		:local pos3 [:find $line "\t" $pos2]
		:log info ( "DShield Entry: " . [:pick $line 0 $pos1 ] . "/" . [:pick $line ($pos2+1) $pos3 ] )
		/ip firewall address-list add list=dshield address=( [:pick $line 0 $pos1 ] . "/" . [:pick $line ($pos2+1) $pos3 ] )
         }  

       }

  } while ($lineEnd < $contentLen)

}

Which results in this:

[xxx@rancheros-AP] > /ip firewall address-list print from=[/ip firewall address-list find list=dshield]

Flags: X - disabled, D - dynamic 
 #   LIST                             ADDRESS                        
 0   dshield                          125.211.198.0/24               
 1   dshield                          193.164.131.0/24               
 2   dshield                          81.3.247.0/24                  
 3   dshield                          81.3.248.0/24                  
 4   dshield                          220.128.206.0/24               
 5   dshield                          81.3.244.0/24                  
 6   dshield                          81.3.249.0/24                  
 7   dshield                          81.192.191.0/24                
 8   dshield                          81.3.243.0/24                  
 9   dshield                          81.3.250.0/24                  
10   dshield                          70.21.127.0/24                 
11   dshield                          218.6.8.0/24                   
12   dshield                          202.138.172.0/24               
13   dshield                          81.3.246.0/24                  
14   dshield                          202.99.254.0/24                
15   dshield                          202.6.225.0/24                 
16   dshield                          99.129.23.0/24                 
17   dshield                          76.88.8.0/24                   
18   dshield                          60.222.231.0/24                
19   dshield                          222.187.221.0/24

Once I clean this up a little, and add some other list parsing I will add to the wiki.

Thx,
Sam

changeip · Wed Jun 04, 2008 9:52 pm

Here's the latest with Spamhaus drop.lasso capability. These could be refined if anyone wants to help.

## Parse DSHIELD & Spamhaus feed and build an address-list.
## Written by Sam Norris, ChangeIP.com 2008
## Any comments or suggestions welcome in the forums.
##
## 06/03/08 - Initial list parsing.

#/tool fetch address=feeds.dshield.org host=feeds.dshield.org mode=http path=block.txt
#/tool fetch address=www.spamhaus.org host=www.spamhaus.org mode=http path=drop/drop.lasso

##
## DSHIELD Drop List
##

:if ( [/file get [/file find name=block.txt] size] > 0 ) do={ 

  /ip firewall address-list remove [/ip firewall address-list find list=dshield]

  :global content [/file get [/file find name=block.txt] contents] ;
  :global contentLen [ :len $content ] ;

  :global lineEnd 0;
  :global line "";
  :global lastEnd 0;

  :do {
       :set lineEnd [:find $content "\n" $lastEnd ] ;
       :set line [:pick $content $lastEnd $lineEnd] ;
       :set lastEnd ( $lineEnd + 1 ) ;

       :if ( [:pick $line 0 1] != "#" ) do={

	 :if ([:typeof [:toip [:pick $line 0 [:find $line "\t"] ] ] ] != "nil") do={
		:local pos1 [:find $line "\t" 0]
		:local pos2 [:find $line "\t" $pos1]
		:local pos3 [:find $line "\t" $pos2]
		:log info ( "DShield Entry: " . [:pick $line 0 $pos1 ] . "/" . [:pick $line ($pos2+1) $pos3 ] )
		/ip firewall address-list add list=dshield address=( [:pick $line 0 $pos1 ] . "/" . [:pick $line ($pos2+1) $pos3 ] )
         }  

       }

  } while ($lineEnd < $contentLen)

}

##
## SPAMHAUS.ORG Drop List
##

:if ( [/file get [/file find name=drop.lasso] size] > 0 ) do={ 

  /ip firewall address-list remove [/ip firewall address-list find list=spamhaus.lasso]

  :global content [/file get [/file find name=drop.lasso] contents] ;
  :global contentLen [ :len $content ] ;

  :global lineEnd 0;
  :global line "";
  :global lastEnd 0;

  :do {
       :set lineEnd [:find $content "\n" $lastEnd ] ;
       :set line [:pick $content $lastEnd $lineEnd] ;
       :set lastEnd ( $lineEnd + 1 ) ;

       :if ( [:pick $line 0 1] != ";" ) do={

	 :if ([:len [:pick $line 0 [:find $line ";"] ] ] > 0 ) do={
		:local pos1 [:find $line ";" 0]
		:local entry [:pick $line 0 ($pos1-1) ]
		:if ( [:len $entry ] > 0 ) do={
			:log info "Lasso Entry: $entry"
			/ip firewall address-list add list=spamhaus.lasso address=$entry
		}
         }  

       }

  } while ($lineEnd < $contentLen)

}

omega-00 · Thu Jun 05, 2008 12:07 am

You sir, are a champion

changeip · Thu Jun 05, 2008 12:53 am

if you have any other text lists you want help making a script for let me know. post the URL or a sample.

omega-00 · Thu Jun 05, 2008 1:32 am

## Generic IP address list input
## Based on a script written by Sam Norris, ChangeIP.com 2008
:if ( [/file get [/file find name=node-unmetered-ips.txt] size] > 0 ) do={

 /ip firewall address-list remove [/ip firewall address-list find list=Node-Unmetered]

 :global content [/file get [/file find name=node-unmetered-ips.txt] contents] ;
 :global contentLen [ :len $content ] ;

 :global lineEnd 0;
 :global line "";
 :global lastEnd 0;

 :do {
      :set lineEnd [:find $content "\n" $lastEnd ] ;
      :set line [:pick $content $lastEnd $lineEnd] ;
      :set lastEnd ( $lineEnd + 1 ) ;

      :if ( [:pick $line 0 1] != "#" ) do={

     :local entry [:pick $line 0 ($lineEnd -1) ]
     :if ( [:len $entry ] > 0 ) do={
        /ip firewall address-list add list=Node-Unmetered address=$entry
     }
   }
 } while ($lineEnd < $contentLen)
}

This will do the same thing for a simple list in the format

#This is a comment
1.1.1.1
2.2.2.0/24
3.3.3.3
4.4.4.128/26

Just substitute the address-list and filename you want to pull from (for anyone else who wants to use it with their own generated lists)

maximan · Thu Jun 05, 2008 1:39 am

Thanks changeip, tks for sharing your knowlege .
Max

omega-00 · Thu Jun 05, 2008 2:54 am

Hmm, okay I've got one for you..

Trying to extract addresses from http://tools.pipenetworks.com/lists/qld.txt

## QLD PIPE Address List
## Based on a script written by Sam Norris, ChangeIP.com 2008
:if ( [/file get [/file find name=qld.txt] size] > 0 ) do={

  /ip firewall address-list remove [/ip firewall address-list find list=qld-pipe]

  :global content [/file get [/file find name=qld.txt] contents] ;
  :global contentLen [ :len $content ] ;

  :global lineEnd 0;
  :global line "";
  :global lastEnd 0;

  :do {
       :set lineEnd [:find $content "\n" $lastEnd ] ;
       :set line [:pick $content $lastEnd $lineEnd] ;
       :set lastEnd ( $lineEnd + 1 ) ;

       :if ( [:pick $line 0 1] != "#" ) do={

    :if ([:len [:pick $line 0 [:find $line ","] ] ] > 0 ) do={
      :local pos1 [:find $line "," 0]
      :local entry [:pick $line 0 ($pos1) ]
      :if ( [:len $entry ] > 0 ) do={
         :log info "QLDPIPE: $entry"
         /ip firewall address-list add list=qld-pipe address=$entry
      }
         } 

       }

  } while ($lineEnd < $contentLen)

}

omega-00 · Fri Jul 25, 2008 4:45 am

Have added this to the wiki - http://wiki.mikrotik.com/wiki/Using_Fet ... ress_Lists

che · Fri Jul 25, 2008 12:47 pm

Just minor syntax thing.

Change path=folder/anotherfolder/ipaddress.txt to src-path=folder/anotherfolder/ipaddress.txt

Thanx for great script, Changeip!

omega-00 · Sun Jul 27, 2008 6:16 pm

can't happen to see what I'm doing wrong in my above script eh?

che · Tue Jul 29, 2008 2:54 pm

sergeda · Sat Sep 13, 2008 9:23 am

Hi to all.
Can somebody help me with this script?
I'm trying to modify it to process file from http://www.colocall.net/uaix/prefixes.txt
For fetch script I use "/tool fetch address=www.colocall.net src-path=/uaix/prefixes.txt host=www.colocall.net mode=http" and it works and creates file prefixes.txt
But I can't manage to add info from this file to address list

It seems that content from file can't be retrieved.
This part of code doesn't work and return in log $content=nothing and $contentLen=0 even so I opened file and it is full of lines

:global content [/file get [/file find name=prefixes.txt] contents] ;
:log info $content;
:global contentLen [ :len $content ] ;
:log info $contentLen;

I have Mikrotik 3.13

SurferTim · Sat Sep 13, 2008 5:43 pm

I am not certain you will consider this help, but take a look at
/file print detail
If you do not see a 'contents' listing associated with the filename, it is probably because the file is larger than 4095 bytes.
By experience, if there is no 'contents' listing, it cannot be loaded/accessed/edited by a script. Maybe someone else can be more help.

sergeda · Sat Sep 13, 2008 9:07 pm

Thank you SurferTim for your help.
It was the size of file, when I edited it to 2049 bytes the script correctly processed it and added to address list. But how this limit can be overcomed? I need this big list of networks.

SurferTim · Sat Sep 13, 2008 11:41 pm

Hi sergeda,

Others have the same challenge. I was hoping there was a "readline" command for text files, where you could read a line a a time from large files (like C and other languages), but have not found anything close. Maybe the MikroTik Team will read this and include something in a future release.

ADD: Wouldn't this be great?

:set linecount [:len [:readline test.txt]]
:set thisline [:readline test.txt 20]
:set nextline [:readline test.txt]

It would read the number of lines in test.txt into $linecount and the 20th (ok, maybe 21st) line from test.txt into $thisline, and subsequent calls without a 'line number' parameter retrieves the next line. So the last line is the same as
:set nextline [:readline test.txt 21]

JUST WISHING!!

omega-00 · Mon Sep 15, 2008 4:12 am

Do want, also I've asked for a feature to write to files also.
Would be very useful for exporting address lists to a csv file or similar.

See: http://forum.mikrotik.com/viewtopic.php?f=1&t=25761

SurferTim · Mon Sep 15, 2008 1:17 pm

Hi omega-00

How about writeline? But to simplify matters, it would be easier to use a sequential write rather than random, like "_open +a" in C. Just appends lines to the file, but in the correct format. Editing large text files, especially adding longer lines near the beginning of the file can be memory intensive. Usually the process requires rebuilding the file completely. That ok with you? Like:

:if ([:writeline test.txt $newline] = [:len $newline]) do={:set writeok true} else={:set writeok false}

Creates /file/test.txt if it doesn't exist. Returns the length of the write in bytes. Zero or -1 on fail.

You could use this in conjunction with readline to rebuild large text files on your own. Just watch your memory.

STILL WISHING!!

mrz · Mon Sep 15, 2008 1:47 pm

There is a way to write a file:

#create file
/file print file=test
#write to file
/file set test.txt contents="lalala";
#add another line
/file set test.txt contents=( get test.txt contents ] . "\nlalalala" );

I know it is not a good way to write to file, but still it is something to work with.

SurferTim · Mon Sep 15, 2008 1:50 pm

Now the bad news. 4095 bytes max.

mrz · Mon Sep 15, 2008 1:59 pm

Now the bad news. 4095 bytes max.

Yes, forgot to mention that.

SurferTim · Mon Sep 15, 2008 2:16 pm

Hi mrz

The file we are discussing has about that many lines. It is a list of IP addresses that are to be entered into a list. I presume this is not a list you want to get on. Most of the list is remarks, but there are several hundred IP addresses in the list, and the file size is at about a meg.

It would be unreasonable to expect a single variable 'contents' to hold the entire file, but a line at a time would be manageable, don't you think?

sergeda · Mon Sep 15, 2008 4:52 pm

I think it's clear that we need to be able process big files and Mikrotik please add this as future request. Until that working with file is imposible, what if today file small and processed but tommorow it will be a little bigger and everything will be broken?

omega-00 · Mon Sep 15, 2008 5:41 pm

I agree completely.

rlima · Wed Sep 24, 2008 9:12 pm

Hi to all, as you are able to read form a file, and making with that a command.
Example, in a file " users-list" read a mac form a user and for example deny the connection.

My problem is that i have to read from a file 16Mb of porn urls and put them on the proxy to deny.

1- I don´t know if the web proxy could handled so much information and also if the time of reading this dictionary it´s possible
2- cuold you give me some lines of the script??

Thx

omega-00 · Thu Oct 02, 2008 8:12 am

Its not going to work, as noted above the string will only hold a maximum of 4095 bytes.

If I were you I'd instead try using a service like OpenDNS (http://www.opendns.com) to block these for you.

omega-00 · Fri Nov 21, 2008 4:23 am

mrz, is there any way to use the file-writing you've mentioned to include a variable? I can't seem to get it to accept anything other than direct text in the contents field :-/

mrz · Fri Nov 21, 2008 8:47 am

:global ipAddr 1.1.1.1
/file print file=test
/file set test.txt contents="lalala $ipAddr "
/file :put [ get test.txt contents ]

Output:
lalala 1.1.1.1

So works for me.

omega-00 · Mon Nov 24, 2008 7:25 am

Was my fault, is working fine now. Thanks!

michianawireless · Thu Jun 18, 2009 7:16 am

I tried to use the script for the spamhaus drop list but it does not work. I am running 3.19 on my router. I even tried manually downloading the lasso file and running it and it never populated the firewall address lists. There was no errors in the logs either. Is this script somewhere that is known to work with 3.19 or better?

Thanks

omega-00 · Thu Jun 18, 2009 9:37 am

Try running the script directly from terminal, that'll show you if it stops at some error and if so will give the error line and the reason for stopping.

[admin@Krusher] > system script run not-working-script 
syntax error (line 1 column 9)
[admin@Krusher] >

michianawireless · Wed Jun 24, 2009 11:10 am

The dshield part works but no rules are ever create for the spamhaus... I am running 3.23 and here is what I am running:

/tool fetch address=feeds.dshield.org host=feeds.dshield.org mode=http src-path=block.txt
/tool fetch address www.spamhaus.org host www.spamhaus.org mode http src-path drop/drop.lasso

:if ( [/file get [/file find name=block.txt] size] > 0 ) do={

/ip firewall address-list remove [/ip firewall address-list find list=dshield]

:global content [/file get [/file find name=block.txt] contents] ;
:global contentLen [ :len $content ] ;

:global lineEnd 0;
:global line "";
:global lastEnd 0;

:do {
:set lineEnd [:find $content "\n" $lastEnd ] ;
:set line [:pick $content $lastEnd $lineEnd] ;
:set lastEnd ( $lineEnd + 1 ) ;

:if ( [:pick $line 0 1] != "#" ) do={

:if ([:typeof [:toip [:pick $line 0 [:find $line "\t"] ] ] ] != "nil") do={
:local pos1 [:find $line "\t" 0]
:local pos2 [:find $line "\t" $pos1]
:local pos3 [:find $line "\t" $pos2]
:log info ( "DShield Entry: " . [:pick $line 0 $pos1 ] . "/" . [:pick $line ($pos2+1) $pos3 ] )
/ip firewall address-list add list=dshield address=( [:pick $line 0 $pos1 ] . "/" . [:pick $line ($pos2+1) $pos3 ] )
}

}

} while ($lineEnd < $contentLen)

}

:if ( [/file get [/file find name=drop.lasso] size] > 0 ) do={

/ip firewall address-list remove [/ip firewall address-list find list=spamhaus.lasso]

:global content [/file get [/file find name=drop.lasso] contents] ;
:global contentLen [ :len $content ] ;

:global lineEnd 0;
:global line "";
:global lastEnd 0;

:do {
:set lineEnd [:find $content "\n" $lastEnd ] ;
:set line [:pick $content $lastEnd $lineEnd] ;
:set lastEnd ( $lineEnd + 1 ) ;

:if ( [:pick $line 0 1] != ";" ) do={

:if ([:len [:pick $line 0 [:find $line ";"] ] ] > 0 ) do={
:local pos1 [:find $line ";" 0]
:local entry [:pick $line 0 ($pos1-1) ]
:if ( [:len $entry ] > 0 ) do={
:log info "Lasso Entry: $entry"
/ip firewall address-list add list=spamhaus.lasso address=$entry
}
}

}

} while ($lineEnd < $contentLen)

}

If I run the command via command console I get this:

[admin@GATEWAY] /system script> run droplist
status: finished

status: finished

[admin@GATEWAY] /system script>

But no spamhaus rules...

If I run the command in winbox by clicking run it download and creates the drop files but neither dshield nor spamhause firewall rules and remove or created?

changeip · Wed Jun 24, 2009 7:52 pm

this is another one of those 'RouterOS upgrades breaks my scripts'. I guess we need to start coding for particular versions now... and put that in the script comments. I will try to troubleshoot this sometime next week, no time at the moment. I spend way to much time in this place I am realizing : )

JJCinAZ · Tue Jul 21, 2009 4:17 am

Is the script being "broken" in that the second fetch will not execute? The script engine seems to take the next like as input to the status display of the first fetch, so the second fetch doesn't get executed.

In other words, have you figured out a way to get the script to "pause" until the fetch completes?

changeip · Tue Jul 21, 2009 8:35 am

your second /tool fetch is missing the = signs.

Tue Jul 21, 2009 9:19 am

RouterOS4 will support LUA, which is kind of standard.

omega-00 · Tue Jul 21, 2009 9:50 am

^looking forward to this, removes the 4096 char limitation and gives full file writing support

janisk · Tue Jul 21, 2009 10:58 am

file writing will be limited. and if i am not mistaken then in 4.0beta3 LUA is already included.

mrz · Tue Jul 21, 2009 11:22 am

http://wiki.mikrotik.com/wiki/Scripting ... g_examples

23q · Thu Jan 30, 2014 12:56 pm

## Generic IP address list input
## Based on a script written by Sam Norris, ChangeIP.com 2008
## Edited by Andrew Cox, AccessPlus.com.au 2008
:if ( [/file get [/file find name=ipaddress.txt] size] > 0 ) do={
# Remove exisiting addresses from the current Address list
/ip firewall address-list remove [/ip firewall address-list find list=MY-IP-LIST]

:global content [/file get [/file find name=ipaddress.txt] contents] ;
:global contentLen [ :len $content ] ;

:global lineEnd 0;
:global line "";
:global lastEnd 0;

:do {
:set lineEnd [:find $content "\n" $lastEnd ] ;
:set line [:pick $content $lastEnd $lineEnd] ;
:set lastEnd ( $lineEnd + 1 ) ;
#If the line doesn't start with a hash then process and add to the list
:if ( [:pick $line 0 1] != "#" ) do={

:local entry [:pick $line 0 ($lineEnd -1) ]
:if ( [:len $entry ] > 0 ) do={
/ip firewall address-list add list=MY-IP-LIST address=$entry
}
}
} while ($lineEnd < $contentLen)
}

not work 6.5..