Feature request (network blacklists)

ngds · Sat Oct 28, 2006 9:21 pm

I'd like to see the ability to import network blacklists such as http://www.spamhaus.org/drop/drop.lasso and http://feeds.dshield.org/block.txt

My main concern is the ability to use firewall rules to block spam email at router level based on the blacklists above, and also to block networks that have a high level of security threats.

There should be an option to import a list of compatible blacklists, and a time setting to grab an updated list every X hours.

pedja · Sun Oct 29, 2006 8:09 am

I second this.

BrianHiggins · Sun Oct 29, 2006 7:40 pm

I 3rd this

changeip · Sun Oct 29, 2006 8:20 pm

i will probably setup a bgp feed of these and allow others access. Once I do this I will post an update. I will also create a script on *nix that runs and ssh's this lists to whatever boxes you have. in 3.0 it would be nice if wget was added and this could be done directly within the router.

Sam

bjohns · Mon Oct 30, 2006 12:33 am

I will also create a script on *nix that runs and ssh's this lists to whatever boxes you have.

Or use the fancy API :)

changeip · Mon Oct 30, 2006 8:00 pm

I will use both : ) But since 2.9 will still be around for a year or so I would like to get something else working for the common people.

cdemers · Fri Dec 01, 2006 7:59 am

As I haven't heard anything about this I scratched up some code together to generate rules, it's based on another script that I found that generated iptables firewall rules. The scripts were tested on FreeBSD but should also work just fine on just about any linux distro. Only requirements are perl and wget.

All that is needed is to ftp upload the output file droproutes.rsc and then import drouproutes.rsc, every time that the script is imported it will remove the previous list and add the new list in case there was any changes.

It could be done manually or done through a scheduled script to do it on it's own.

The last part is to create your firewall rules to drop the traffic or what ever you want it to do with it.

/ ip firewall filter
add chain=forward src-address-list=droproutes action=drop comment="Drop Hijacked \
    networks" disabled=no 
add chain=input src-address-list=droproutes action=drop comment="Drop Hijacked networks" \
    disabled=no

getdrop.sh

#!/bin/sh

wget -O sbl-drop.txt http://www.spamhaus.org/drop/drop.lasso
cat sbl-drop.txt | ./sbl-drop.pl > droproutes.rsc

sbl-drop.pl

#!/usr/bin/perl -w

use strict;
use vars qw{$n $m};

print "/ ip firewall address-list\n";
print ":foreach subnet in [/ip firewall address-list find list=droproutes] do { /ip firewall address-list remove \$subnet }\n";

while(<>) {
    next if m{^;};
    if(($n, $m) = m{(\d+\.\d+\.\d+\.\d+)/(\d+)}) {
   # local sanity check
   die "local network $n" if $n =~ /^127./;

   print "add list=droproutes address=$n/$m disabled=no\n";
    } else {
   print "#??? $_\n";
    }
}

[/code]

changeip · Fri Dec 01, 2006 8:53 am

Here is what I hobbled together on this a while ago and forgot to post...

#!/bin/csh
#
#  This script downloads a few blacklists and imports them into MT.
#
set today = `date "+%m%d%y"`
set workdir = "/home/snorris/mt"
set targets =  ( 10.40.1.1 10.20.1.1 204.123.123.123 )

cd ${workdir}

wget -nv -t 2 -O drop.lasso.txt -U wget-changeip-script http://www.spamhaus.org/drop/drop.lasso
wget -nv -t 2 -O drop.dshield.txt -U wget-changeip-script http://feeds.dshield.org/block.txt

echo :log info \"drop.lasso script import started\" > drop.lasso.rsc

echo :foreach subnet in [/ip firewall address-list find list=drop.lasso] do=\{ /ip firewall address-list remove \$subnet \} >> drop.lasso.rsc

cat drop.lasso.txt | awk '{print $1 " " $3}' | awk -F"/" '{print $1 " " $2 " " $3}' | grep -v ";" | sort +1 | awk '{print "/ip firewall address-list add list=drop.lasso address=" $1 "/" $2 " comment=" $3}' >> drop.lasso.rsc

echo :log info \"drop.lasso script import completed\" >> drop.lasso.rsc

echo :log info \"drop.dshield script begin run\" >> drop.lasso.rsc
echo :foreach subnet in [/ip firewall address-list find list=drop.dshield] do=\{ /ip firewall address-list remove \$subnet \} >> drop.lasso.rsc
egrep "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" drop.dshield.txt | awk '{print "/ip firewall address-list add list=drop.dshield address=" $1 "-" $2}' >> drop.lasso.rsc
echo :log info \"drop.dshield script end run\" >> drop.lasso.rsc

foreach target ($targets)

echo $target
echo "put drop.lasso.rsc" | ftp ftp://admin\:pass\@{$target}:21

ssh -l admin-ssh -i /home/snorris/mt/mt.dsa ${target} "/import drop.lasso.rsc" > /home/snorris/mt/push-lasso.output

end

cdemers · Fri Dec 01, 2006 2:56 pm

Hi changeip,

Very nice, I will have to give that one a try later on. Maybe it can be put in the wiki for everyone else.

Chris

ceL · Thu Apr 19, 2007 4:57 pm

what about the overlapping range problem with the spamhaus drop list, i always get that when i import it and then the script just fails

TomKolb · Sat Apr 21, 2007 3:26 pm

We setup a firewall rule to look at the list of IP in the drop list and within the 1st 12 hours we dropped 125,000 syn connection attempts. They are mostly dns lookups. What was very interesting we had 20 or so customers to call for support stating thay can not surf. We found that the DNS setting of those computers had DNS servers entered that are in the block list. We are working to identify exactly what changed the dns servers on the infected computers.

The code is simple

# apr/21/2007 07:47:14 by RouterOS 2.9.35
# software id = 0DEM-BCT
#

/ ip firewall filter
add chain=forward action=jump jump-target=droplist comment="" disabled=no

add chain=droplist dst-address-list=drop.lasso action=log log-prefix="" \
comment="" disabled=yes
add chain=droplist src-address-list=drop.lasso action=drop comment="Spamhaus \
Drop List 4-19-07" disabled=no
add chain=droplist dst-address-list=drop.lasso action=drop comment="" \
disabled=no
add chain=droplist src-address-list=drop.dshield action=drop comment="Dshield \
Drop List 4-19-07" disabled=no
add chain=droplist dst-address-list=drop.dshield action=drop comment="" \
disabled=no

/ ip firewall address-list
add list=drop.lasso address=123.176.80.0/22 comment="SBL53507" disabled=no
add list=drop.lasso address=128.13.0.0/16 comment="SBL10400" disabled=no
add list=drop.lasso address=132.232.0.0/16 comment="SBL9176" disabled=no
add list=drop.lasso address=134.33.0.0/16 comment="SBL7097" disabled=no
add list=drop.lasso address=138.252.0.0/16 comment="SBL9702" disabled=no
add list=drop.lasso address=143.49.0.0/16 comment="SBL7182" disabled=no
add list=drop.lasso address=148.3.0.0/16 comment="SBL8914" disabled=no
add list=drop.lasso address=152.147.0.0/16 comment="SBL8847" disabled=no
add list=drop.lasso address=167.97.0.0/16 comment="SBL12947" disabled=no
add list=drop.lasso address=170.67.0.0/16 comment="SBL8148" disabled=no
add list=drop.lasso address=192.160.44.0/24 comment="SBL9493" disabled=no
add list=drop.lasso address=192.67.16.0/24 comment="SBL6648" disabled=no
add list=drop.lasso address=193.110.136.0/24 comment="SBL3399" disabled=no
add list=drop.lasso address=193.19.120.0/23 comment="SBL13553" disabled=no
add list=drop.lasso address=193.238.120.0/22 comment="SBL34935" disabled=no
add list=drop.lasso address=193.238.36.0/22 comment="SBL40543" disabled=no
add list=drop.lasso address=194.146.204.0/22 comment="SBL51152" disabled=no
add list=drop.lasso address=195.114.16.0/23 comment="SBL51155" disabled=no
add list=drop.lasso address=195.114.8.0/23 comment="SBL48773" disabled=no
add list=drop.lasso address=195.206.120.0/22 comment="SBL27034" disabled=no
add list=drop.lasso address=195.214.236.0/22 comment="SBL24194" disabled=no
add list=drop.lasso address=195.64.162.0/23 comment="SBL51154" disabled=no
add list=drop.lasso address=195.74.88.0/23 comment="SBL53174" disabled=no
add list=drop.lasso address=195.95.161.0/24 comment="SBL42935" disabled=no
add list=drop.lasso address=196.4.167.0/24 comment="SBL7972" disabled=no
add list=drop.lasso address=198.151.152.0/22 comment="SBL23969" disabled=no
add list=drop.lasso address=198.186.16.0/20 comment="SBL23967" disabled=no
add list=drop.lasso address=198.204.0.0/21 comment="SBL8179" disabled=no
add list=drop.lasso address=199.120.163.0/24 comment="SBL6658" disabled=no
add list=drop.lasso address=199.166.200.0/22 comment="SBL6026" disabled=no
add list=drop.lasso address=199.201.151.0/24 comment="SBL7064" disabled=no
add list=drop.lasso address=199.201.152.0/24 comment="SBL7065" disabled=no
add list=drop.lasso address=199.245.138.0/24 comment="SBL9923" disabled=no
add list=drop.lasso address=199.248.213.0/24 comment="SBL29230" disabled=no
add list=drop.lasso address=199.60.102.0/24 comment="SBL9159" disabled=no
add list=drop.lasso address=200.108.160.0/20 comment="SBL49925" disabled=no
add list=drop.lasso address=200.124.64.0/20 comment="SBL48351" disabled=no
add list=drop.lasso address=201.71.0.0/20 comment="SBL38197" disabled=no
add list=drop.lasso address=202.14.69.0/24 comment="SBL19095" disabled=no
add list=drop.lasso address=202.62.252.0/24 comment="SBL29418" disabled=no
add list=drop.lasso address=203.19.101.0/24 comment="SBL6619" disabled=no
add list=drop.lasso address=203.31.88.0/23 comment="SBL8083" disabled=no
add list=drop.lasso address=203.33.120.0/24 comment="SBL13719" disabled=no
add list=drop.lasso address=203.34.192.0/23 comment="SBL7167" disabled=no
add list=drop.lasso address=203.34.204.0/24 comment="SBL7168" disabled=no
add list=drop.lasso address=203.34.205.0/24 comment="SBL7330" disabled=no
add list=drop.lasso address=203.34.70.0/24 comment="SBL9682" disabled=no
add list=drop.lasso address=203.34.71.0/24 comment="SBL7244" disabled=no
add list=drop.lasso address=203.4.141.0/24 comment="SBL9018" disabled=no
add list=drop.lasso address=203.4.142.0/24 comment="SBL9040" disabled=no
add list=drop.lasso address=203.55.153.0/24 comment="SBL9684" disabled=no
add list=drop.lasso address=203.82.16.0/21 comment="SBL37830" disabled=no
add list=drop.lasso address=204.10.176.0/21 comment="SBL48931" disabled=no
add list=drop.lasso address=204.13.32.0/21 comment="SBL37362" disabled=no
add list=drop.lasso address=204.14.0.0/21 comment="SBL27197" disabled=no
add list=drop.lasso address=204.14.24.0/21 comment="SBL33048" disabled=no
add list=drop.lasso address=204.236.0.0/19 comment="SBL46767" disabled=no
add list=drop.lasso address=204.52.255.0/24 comment="SBL13483" disabled=no
add list=drop.lasso address=204.62.213.0/24 comment="SBL13460" disabled=no
add list=drop.lasso address=204.89.156.0/23 comment="SBL14196" disabled=no
add list=drop.lasso address=204.89.224.0/24 comment="SBL11667" disabled=no
add list=drop.lasso address=205.172.188.0/22 comment="SBL29179" disabled=no
add list=drop.lasso address=205.210.137.0/24 comment="SBL25844" disabled=no
add list=drop.lasso address=205.235.64.0/20 comment="SBL8558" disabled=no
add list=drop.lasso address=205.236.189.0/24 comment="SBL9442" disabled=no
add list=drop.lasso address=206.197.134.0/24 comment="SBL19351" disabled=no
add list=drop.lasso address=206.197.175.0/24 comment="SBL14246" disabled=no
add list=drop.lasso address=206.197.176.0/24 comment="SBL14250" disabled=no
add list=drop.lasso address=206.197.177.0/24 comment="SBL14248" disabled=no
add list=drop.lasso address=206.197.28.0/24 comment="SBL14253" disabled=no
add list=drop.lasso address=206.197.29.0/24 comment="SBL14251" disabled=no
add list=drop.lasso address=206.197.99.0/24 comment="SBL18314" disabled=no
add list=drop.lasso address=206.81.80.0/20 comment="SBL16307" disabled=no
add list=drop.lasso address=207.191.160.0/20 comment="SBL10044" disabled=no
add list=drop.lasso address=208.64.44.0/22 comment="SBL49808" disabled=no
add list=drop.lasso address=209.165.224.0/20 comment="SBL163" disabled=no
add list=drop.lasso address=209.197.192.0/19 comment="SBL6322" disabled=no
add list=drop.lasso address=209.203.160.0/20 comment="SBL8017" disabled=no
add list=drop.lasso address=209.205.192.0/19 comment="SBL41017" disabled=no
add list=drop.lasso address=209.205.224.0/20 comment="SBL41018" disabled=no
add list=drop.lasso address=211.155.128.0/20 comment="SBL42897" disabled=no
add list=drop.lasso address=213.135.80.0/23 comment="SBL28268" disabled=no
add list=drop.lasso address=216.108.224.0/20 comment="SBL11947" disabled=no
add list=drop.lasso address=216.130.192.0/19 comment="SBL4197" disabled=no
add list=drop.lasso address=216.16.128.0/18 comment="SBL48625" disabled=no
add list=drop.lasso address=216.211.144.0/20 comment="SBL29446" disabled=no
add list=drop.lasso address=217.69.112.0/20 comment="SBL13989" disabled=no
add list=drop.lasso address=58.83.4.0/22 comment="SBL51895" disabled=no
add list=drop.lasso address=63.246.32.0/20 comment="SBL5255" disabled=no
add list=drop.lasso address=64.18.240.0/20 comment="SBL41287" disabled=no
add list=drop.lasso address=64.28.176.0/20 comment="SBL36453" disabled=no
add list=drop.lasso address=65.182.128.0/20 comment="SBL12959" disabled=no
add list=drop.lasso address=65.255.32.0/20 comment="SBL14484" disabled=no
add list=drop.lasso address=66.102.32.0/20 comment="SBL6045" disabled=no
add list=drop.lasso address=66.110.72.0/24 comment="SBL20505" disabled=no
add list=drop.lasso address=66.181.160.0/19 comment="SBL11591" disabled=no
add list=drop.lasso address=66.185.112.0/20 comment="SBL38413" disabled=no
add list=drop.lasso address=66.235.128.0/20 comment="SBL11048" disabled=no
add list=drop.lasso address=66.55.160.0/19 comment="SBL11507" disabled=no
add list=drop.lasso address=66.64.96.0/20 comment="SBL11226" disabled=no
add list=drop.lasso address=66.79.96.0/20 comment="SBL49015" disabled=no
add list=drop.lasso address=69.10.0.0/20 comment="SBL8194" disabled=no
add list=drop.lasso address=69.67.64.0/20 comment="SBL12587" disabled=no
add list=drop.lasso address=69.8.176.0/20 comment="SBL15315" disabled=no
add list=drop.lasso address=72.21.128.0/20 comment="SBL23677" disabled=no
add list=drop.lasso address=72.21.64.0/20 comment="SBL22350" disabled=no
add list=drop.lasso address=80.71.64.0/19 comment="SBL9098" disabled=no
add list=drop.lasso address=81.17.16.0/20 comment="SBL48941" disabled=no
add list=drop.lasso address=81.95.144.0/20 comment="SBL43489" disabled=no
add list=drop.lasso address=83.223.224.0/19 comment="SBL18829" disabled=no
add list=drop.lasso address=85.255.112.0/20 comment="SBL36702" disabled=no
add list=drop.lasso address=86.105.230.0/24 comment="SBL50622" disabled=no
add list=drop.lasso address=86.111.128.0/19 comment="SBL39390" disabled=no
add list=drop.lasso address=86.59.128.0/17 comment="SBL28720" disabled=no
add list=drop.lasso address=88.206.0.0/17 comment="SBL41481" disabled=no
add list=drop.lasso address=89.104.112.0/22 comment="SBL46514" disabled=no
add list=drop.lasso address=89.145.128.0/20 comment="SBL49759" disabled=no
add list=drop.lasso address=89.187.192.0/19 comment="SBL47126" disabled=no
add list=drop.lasso address=89.233.64.0/18 comment="SBL42372" disabled=no
add list=drop.lasso address=89.32.203.0/24 comment="SBL48115" disabled=no
add list=drop.lasso address=89.35.0.0/23 comment="SBL47082" disabled=no
add list=drop.lasso address=91.146.112.0/20 comment="SBL48565" disabled=no
add list=drop.dshield address=213.82.59.0/24 comment="" disabled=no
add list=drop.dshield address=61.134.65.0/24 comment="" disabled=no
add list=drop.dshield address=62.94.33.0/24 comment="" disabled=no
add list=drop.dshield address=196.35.64.0/24 comment="" disabled=no
add list=drop.dshield address=61.134.64.0/24 comment="" disabled=no
add list=drop.dshield address=218.106.91.0/24 comment="" disabled=no
add list=drop.dshield address=209.208.170.0/24 comment="" disabled=no
add list=drop.dshield address=220.178.32.0/24 comment="" disabled=no
add list=drop.dshield address=122.32.155.0/24 comment="" disabled=no
add list=drop.dshield address=80.93.223.0/24 comment="" disabled=no
add list=drop.dshield address=203.200.202.0/24 comment="" disabled=no
add list=drop.dshield address=82.12.214.0/24 comment="" disabled=no
add list=drop.dshield address=203.125.242.0/24 comment="" disabled=no
add list=drop.dshield address=203.125.96.0/24 comment="" disabled=no
add list=drop.dshield address=63.245.13.0/24 comment="" disabled=no
add list=drop.dshield address=190.48.21.0/24 comment="" disabled=no
add list=drop.dshield address=59.112.80.0/24 comment="" disabled=no
add list=drop.dshield address=211.138.83.0/24 comment="" disabled=no
add list=drop.dshield address=202.107.229.0/24 comment="" disabled=no
add list=drop.dshield address=61.153.139.0/24 comment="" disabled=no

cdemers · Sat Jan 19, 2008 4:28 pm

Hey changeip,

Would you mind if I make this post a wiki article? As your the original author i should ask before I make one. I have only made one change to the script and that was to fix a format change in the drop.lasso list that caused a blank firewall rule to be included that caused a problem. Of course will add more information about the firewall rules and a basic how to to setup everything to get it all going. It's been working well for myself now for quite a while.

#!/bin/csh
#
#  Original author changeip
#  Minor modifications and commenting by cdemers
#
#  This script downloads a few blacklists and imports them into MT.
#

set today = `date "+%m%d%y"`
set workdir = "/usr/home/admin/work"
set targets =  ( 10.0.0.1 10.0.0.2 10.0.0.3 )

cd ${workdir}

#
# Fetch block lists from sources and store in current folder
#

wget -q -nv -t 2 -O drop.lasso.txt -U wget-changeip-script http://www.spamhaus.org/drop/drop.lasso
wget -q -nv -t 2 -O drop.dshield.txt -U wget-changeip-script http://feeds.dshield.org/block.txt

#
# Begin Processing drop.lasso
#

echo :log info \"drop.lasso script import started\" > drop.lasso.rsc

echo :foreach subnet in [/ip firewall address-list find list=drop.lasso] do=\{ /ip firewall address-list remove \$subnet \} >> drop.lasso.rsc

cat drop.lasso.txt | awk '{print $1 " " $3}' | awk -F"/" '{print $1 " " $2 " " $3}' | grep -v ";" | sort +1 | sed  '/^ *$/d' | awk '{print "/ip firewall addr
ess-list add list=drop.lasso address=" $1 "/" $2 " comment=" $3}' >> drop.lasso.rsc

echo :log info \"drop.lasso script import completed\" >> drop.lasso.rsc

#
# Begin processing drop.dshield
#

echo :log info \"drop.dshield script begin run\" >> drop.lasso.rsc

echo :foreach subnet in [/ip firewall address-list find list=drop.dshield] do=\{ /ip firewall address-list remove \$subnet \} >> drop.lasso.rsc

egrep "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" drop.dshield.txt | awk '{print "/ip firewall address-list add list=drop.dshield address=" $1 "-" $2}'
>> drop.lasso.rsc

echo :log info \"drop.dshield script end run\" >> drop.lasso.rsc

#
# Upload script to router and execute
#

foreach target ($targets)

echo $target
echo "put drop.lasso.rsc" | ftp ftp://ftpuser\:password\@{$target}:21

ssh -l admin-ssh -i ${workdir}/dsa ${target} "/import drop.lasso.rsc" >> ${workdir}/push-lasso.output

end

jp · Sat Jan 19, 2008 6:19 pm

We setup a firewall rule to look at the list of IP in the drop list and within the 1st 12 hours we dropped 125,000 syn connection attempts. They are mostly dns lookups. What was very interesting we had 20 or so customers to call for support stating thay can not surf. We found that the DNS setting of those computers had DNS servers entered that are in the block list. We are working to identify exactly what changed the dns servers on the infected computers.

Customers should be using YOUR dns server(s). Perhaps some malware has changed their DNS settings. Sounds very interesting! Keep us posted!

changeip · Sat Jan 19, 2008 8:31 pm

cdemers - go for it. . . I didnt know if it was 100% stable so I didnt post it yet, but I think its working okay. I really want to get 3.0 to pull these down directly, once I get that going I will post.

cdemers · Sat Jan 19, 2008 9:35 pm

Hi changeip,

It's been completely stable, only change i had to make was recent when drop.lasso list changed something and kept making a bogus rule. I have tested it with up to 2.9.50 but not yet with 3.0, Not sure I'm going to have enough time this weekend but I have it partially already written up for my own documentation. I'm going to upgrade a test router to 3.0 and see if I need to make any modifications. Have to anyways. Will probably take me a couple days and will post it in the wiki for anyone else to use.

jp · Sun Jan 20, 2008 5:03 am

Maybe I'm not polished up on the intricacies of address lists, but I don't see anything in the scripts to remove entries that are no longer in the blacklist. I would think the program should clear the address list, then import the entries.

Things do get erroneously listed, or temporarily listed while there is a problem, and many blacklist users don't use it properly and inadvertently keep blocking things. This causes lingering issues to the person who cleaned up the problem at the source or who inherited an IP address that had been previously blacklisted.

changeip · Sun Jan 20, 2008 5:38 am

echo :foreach subnet in [/ip firewall address-list find list=drop.lasso] do=\{ /ip firewall address-list remove \$subnet \} >> drop.lasso.rsc

I really wish MT would allow us to add a TTL to address-list entries that we add, instead of only allowing that thru the firewall functions.

Sam

Feature request (network blacklists)

Feature request (network blacklists)

Re: Feature request (network blacklists)

Re:

Re: Feature request (network blacklists)

Re: Feature request (network blacklists)

Re: Feature request (network blacklists)

Re: Feature request (network blacklists)

Who is online