Import list of IP's to ammend Address List

chrisw · Fri Apr 11, 2014 2:54 am

EDIT: Solution Here: http://forum.mikrotik.com/viewtopic.php ... 10#p421517

I have a list of banned IP's that I update all the time for customers who are overdue on their payments. I'm following the instructions on this page http://wiki.mikrotik.com/wiki/Using_Fet ... ress_Lists but having no luck. The fetch script works just fine, my suspendedaccts.txt file shows up in the Mikrotik's files folder and it runs the next script. That next script, however, does not parse my list of IP's and add them in the address-list. What's wrong here?

   :if ( [/file get [/file find name=suspendedaccts.txt] size] > 0 ) do={
   # Remove exisiting addresses from the current Address list
   /ip firewall address-list remove [/ip firewall address-list find list=suspendedaccts]
   
   :global content [/file get [/file find name=suspendedaccts.txt] contents] ;
   :global contentLen [ :len $content ] ;
   
   :global lineEnd 0;
   :global line "";
   :global lastEnd 0;
   
   :do {
         :set lineEnd [:find $content "\n" $lastEnd ] ;
         :set line [:pick $content $lastEnd $lineEnd] ;
         :set lastEnd ( $lineEnd + 1 ) ;
         #If the line doesn't start with a hash then process and add to the list
         :if ( [:pick $line 0 1] != "#" ) do={
   
        :local entry [:pick $line 0 ($lineEnd -1) ]
        :if ( [:len $entry ] > 0 ) do={
           /ip firewall address-list add list=suspendedaccts address=$entry
        }
      }
   } while ($lineEnd < $contentLen)
   }

chrisw · Fri Apr 11, 2014 9:55 pm

It seems the script is hanging at this point at the end:

    :if ( [:pick $line 0 1] != "\n" ) do={
      :if ( [:len $line ] > 0 ) do={ /ip firewall address-list add list=suspendedaccts address=$line }
    }
  } while ($lineEnd < $contentLen)
}

This is where it goes line by line and adds each IP address to an address-list. The text file it parses is a simple .txt file, less than 200KB, and looks like this:

192.168.200.111
192.168.200.112
192.168.200.113

skot · Sat Apr 12, 2014 2:03 am

Unfortunately, there is a 4096 byte read/write limit when working with text files inside RouterOS: http://forum.mikrotik.com/viewtopic.php ... 42#p203311

Reduce your file size, and then pasting this in the terminal. It will show the file contents, unless the file size is too big and then it will return nothing:

:if ( [/file get [/file find name=suspendedaccts.txt] size] > 0 ) do={
   :global content [/file get [/file find name=suspendedaccts.txt] contents] ;
   :put $content
}

The best way to get around this is to generate an importable-ready script server-side using bash / perl / batch / whatever. Then upload using fetch, and it will be executed automatically if you name it something.auto.rsc

So, the text file would look like this:

/ip firewall address-list remove [find list=suspendedaccts]
/ip firewall address-list add list=suspendedaccts address=192.168.200.111
/ip firewall address-list add list=suspendedaccts address=192.168.200.112
/ip firewall address-list add list=suspendedaccts address=192.168.200.113
...

chrisw · Sat Apr 12, 2014 2:03 am

Coming along, but still having issues. I found another website that had slightly different code and it successfully Imports the first IP address on my list - but only the first. Then it throws up the error message "value of address expects range of ip addresses".

:if ( [/file get [/file find name=suspendedaccts.txt] size] > 0 ) do={
# Remove exisiting addresses from the current Address list
/ip firewall address-list remove [/ip firewall address-list find list=suspendedaccts]

:global content [/file get [/file find name=suspendedaccts.txt] contents] ;
:global contentLen [ :len $content ] ;

:global lineEnd 0;
:global line "";
:global lastEnd 0;

:do {
      :set lineEnd [:find $content "\n" $lastEnd ] ;
      :set line [:pick $content $lastEnd $lineEnd] ;
      :set lastEnd ( $lineEnd + 1 ) ;
      :if ( [:pick $line 0 1] != "#" ) do={
     :local entry [:pick $line 0 ($lineEnd -1) ]
     :if ( [:len $entry ] > 0 ) do={
        /ip firewall address-list add list=suspendedaccts address=$entry
     }
   }
} while ($lineEnd < $contentLen)
}

chrisw · Sat Apr 12, 2014 2:08 am

Unfortunately, there is a 4096 byte read/write limit when working with text files inside RouterOS: http://forum.mikrotik.com/viewtopic.php ... 42#p203311

Reduce your file size, and then pasting this in the terminal. It will show the file contents, unless the file size is too big and then it will return nothing:
Code: Select all
:if ( [/file get [/file find name=suspendedaccts.txt] size] > 0 ) do={
   :global content [/file get [/file find name=suspendedaccts.txt] contents] ;
   :put $content
}

The size of my text file is 51 bytes (while its size on disk is 4096 bytes). Pasting what you wrote into the terminal worked, it showed me the IP addresses. However, the actual issue still remains.

skot · Wed Apr 16, 2014 6:22 pm

I was getting the same results as you, and I think I figured out what is happening. The text file line breaks are \r\n, but the script is only checking for \n, so the variables contain line breaks, which are causing the script to fail.

Try this code instead. The starting and ending brackets are necessary if you copy/paste into the terminal, but otherwise they can be removed in your final script.

{
:local content [/file get [/file find name=suspendedaccts.txt] contents]
:local contentLen [:len $content]

:local lineEnd 0
:local line ""
:local lastEnd 0

:while ($lineEnd < $contentLen) do={
	:set lineEnd [:find $content "\r\n" $lastEnd]
	:if ([:len $lineEnd] = 0) do={
	  :set lineEnd $contentLen
	}
	:set line [:pick $content $lastEnd $lineEnd]
	:set lastEnd ($lineEnd + 2)
	/ip firewall address-list add list="suspendedaccts" address=$line
} 
}

n21roadie · Wed Apr 16, 2014 6:43 pm

I have a list of banned IP's that I update all the time for customers who are overdue on their payments. I'm following the instructions on this page http://wiki.mikrotik.com/wiki/Using_Fet ... ress_Lists but having no luck. The fetch script works just fine, my suspendedaccts.txt file shows up in the Mikrotik's files folder and it runs the next script. That next script, however, does not parse my list of IP's and add them in the address-list. What's wrong here?

I use use payment reminder for missed payments, http://wiki.mikrotik.com/wiki/Payment_Reminders, which works OK
but can i ask how does your method work? is the list of banned IP's your customers ip address's?

chrisw · Wed Apr 16, 2014 6:46 pm

The list is for non-paying customers whose IP addresses get added to an address list filtered by a dstNAT rule that redirects all traffic to our pay portal page.

chrisw · Wed Apr 16, 2014 6:50 pm

RESOLVED

Shoutout to /user/netsx on the MikroTik subreddit for the solution.

The problem is because my .txt file is MSDOS line formatted ( "\r\n" ) but it's expecting a Linux/Unix, etc. line format ( "\n" ). Working code is here:

  :if ( [/file get [/file find name=suspendedaccts.txt] size] > 0 ) do={
    /ip firewall address-list remove [/ip firewall address-list find list=suspendedaccts]
    :local content [/file get [/file find name=suspendedaccts.txt] contents] ;
    :local contentLen [ :len $content ] ;
    :put ( "=>" . $content . "<=" )
    :local lineEnd 0
    :local line ""
    :local lastEnd [ :tonum 0 ]


    :do {
      :set line [ :pick $content $lastEnd [ :find $content "\r\n" $lastEnd ] ]
      :set lineEnd [:find $content "\n" $lastEnd ] ;
      :put "lineEnd=$lineEnd< - lastEnd=$lastEnd - line=$line<"
    :if ( [ :len $content ] = 0 ) do={
      :put "UNIX"
      :set line [ :pick $content $lastEnd [ :find $content "\r\n" $lastEnd ] ]
       :set lineEnd [:find $content "\n" $lastEnd ] ;
    }
      :set lastEnd ( $lineEnd + 1 ) ;

      :if ( [:pick $line 0 1] != "\n" ) do={
      :put ">$line<"
        :if ( [:len $line ] > 0 ) do={ /ip firewall address-list add list=suspendedaccts address=$line }
      }
    } while ($lineEnd < $contentLen)
  }

chrisw · Wed Apr 16, 2014 7:57 pm

UPDATE

If you're trying to parse tab-delimited text instead of line-delimited, simply replace every instance of "\r\n" and "\n" in the script with "\t".

n21roadie · Thu Apr 17, 2014 11:13 am

The list is for non-paying customers whose IP addresses get added to an address list filtered by a dstNAT rule that redirects all traffic to our pay portal page.

Ok - Thankfully we have found that once a customer internet service is interrupted by diverting to payment reminder page, they usually don't miss another payment any time soon, however as it only blocks port 80 traffic other programs using different ports can still have internet connectivity, if I could just allow specified on-line banks and PayPal and block everything else.

anisuzzamanb · Thu May 08, 2014 8:46 am

Thanks for the valuable post. I have some other issues on this script. What I need to do if there are a space separated txt file and is it possible to read an space separated CSV file? I need this help since I am receiving space separated txt file or space separated CSV file.

cicserver · Fri Nov 27, 2015 7:23 am

I am trying to use this method (but for importing hotspot walled garden list).
on version 5.x it works fine
BUT on 6.33 version, it goes in LOOP, means the file contains just 4 entries, but the script keep adding them again an again until i stop it manually.

how can I remove this looping bug for 6.x version?

Import list of IP's to ammend Address List

Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Re: Import list of IP's to ammend Address List

Who is online