Blacklist Filter (Development Topic)

IntrusDave · Wed Jul 11, 2018 3:15 am

Hey guys, I wanted to have this a little more open of a discussion, so I made a new thread.

So I'm starting to plan out the new system and I'm going SQL based. The old system used a boat load of regex, awk, grep, etc. It was pretty dirty, but it worked. The lists generated were stored in a flat file. The new system is going to be way more flexible.

My thoughts are this...
- SQL based realtime list generation
- Subscriber managed private black/white lists (configured per device)
- Subscriber selectable list size (for 32M, 64M, 256M, 512M, 1G+ device)
- Subscriber selectable country blocking (for devices that have enough memory)
Moving to SQL will give this functionality, it will also allow the server to update the blacklists in realtime without blocking downloads. I haven't yet found a way to do non-blocking updates on the client side. (Sorry, no BGP - too complicated to manage, this needs to be fully automated)

So, this is all still only on paper, so if anyone has more ideas, lets hear it.

Here is a form to fill out if you are interested in being notified:
https://goo.gl/forms/UQMYqKJ54E0iV35l2

msatter · Wed Jul 11, 2018 10:20 am

I was charmed by your previous implementation of using DNS to determine which version of the list and partial update (add's) should be provided.

hhgttg42 · Wed Jul 11, 2018 4:17 pm

[...]
- SQL based realtime list generation
[...]

How would this translate into update frequency for the clients?

IntrusDave · Wed Jul 11, 2018 7:41 pm

I do plan on continuing to use DNS for versioning. Ultimate goal will be to have the client send the last update date and time, then request the just the changes from that point.

The effect on the client side would be that the client determines it's own update schedule.

43north · Thu Aug 02, 2018 8:02 am

I am looking forward to it and will definitely be a paying customer!!!!

IntrusDave · Thu Aug 02, 2018 8:27 am

So, I *JUST* started coding this week, so this is really rough.. like pre-alpha. This is going to take some time.. I can code in python, php, perl, and C. but just because I can, doesn't mean I enjoy it. I really hate coding, it's boring and frustrating. Anyway..

I have the database being populated IPs from all the big blacklist sources, I haven't written the module for pulling in the honeypot data. I'm holding off on that because I also want to either rewrite the honeypot code, or move to an open source honeypot with an API.
I'm also pulling in some whitelists. The .rsc will import dynamic address-lists named "IntrusBL" and "IntrusWL". I simply added two RAW rules, 1 to accept the WL one to drop the BL.
I also haven't built any of the accounting, or config system.

If you would like to use the pre-alpha, this script will pull it for you. just change the "priority" to 1, 2, or 3. 1 being the smallest list (about 2k), 2 being middle (18k), and 3 being the whole thing (over 135,000)
I don't recommend the priority 3 list unless you are running servers open to the world with a router that has at least 1GB RAM.
The script doesn't collect anything from your end. yet. As it nears beta, the accounting system will be in place that will require at minimum, the software ID, ether1 MAC address, and maybe CPU type. I will need these to positively identify the router so that the server can generate the router's customized list. I will also be including an opt-in option to provide some "router demographics" so I can generate stats on models/ram/etc.

So, here it is. I make NO PROMISES that it works all the time. my personal router is updating itself every hour, and my development network router is updating every time I make a commit to the source code.

please note that enabling Cloud DDNS is required, no exceptions.

:local destPath "disk1/filterImport.rsc";
:local priority "2";

:local sn [:pick [/ip cloud get dns-name] 0 [:find [/ip cloud get dns-name] "."]];
/tool fetch mode=https url="https://bl.mikrotikfilters.com/secureFetch.php?priority=$priority" http-method=post http-data="$sn" dst-path="$destPath" output=file; /import file-name=$destPath;  /file remove $destPath;

Thu Aug 02, 2018 9:16 am

Dave,

Thank you for your job.

Testing:

[admin@RBTEST] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl.mikrotikfilters.com/fetch.php\?priority=1";
status: failed

failure: closing connection: <500 Internal Server Error> 35.236.78.203:443 (4)
[admin@RBTEST] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl.mikrotikfilters.com/fetch.php?priority=1";
status: failed

failure: closing connection: <500 Internal Server Error> 35.236.78.203:443 (4)

IntrusDave · Thu Aug 02, 2018 9:29 am

fixed. having issues with mysql terminating

Thu Aug 02, 2018 9:58 am

Fixed.

aboiles · Thu Aug 02, 2018 10:35 am

[admin@CHR-O] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl
.mikrotikfilters.com/fetch.phppriority=3";
status: failed

failure: closing connection: <404 Not Found> 35.236.78.203:443 (4)
[admin@CHR-O] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl
.mikrotikfilters.com/fetch.phppriority=3";
status: failed

failure: closing connection: <404 Not Found> 35.236.78.203:443 (4)

IntrusDave · Thu Aug 02, 2018 10:38 am

[admin@CHR-O] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl
.mikrotikfilters.com/fetch.phppriority=3";
status: failed

failure: closing connection: <404 Not Found> 35.236.78.203:443 (4)
[admin@CHR-O] > /tool fetch mode=https dst-path=/disk1/filters.rsc url="https://bl
.mikrotikfilters.com/fetch.phppriority=3";
status: failed

failure: closing connection: <404 Not Found> 35.236.78.203:443 (4)

you are missing the "?" in the url.

kakaxa · Thu Aug 02, 2018 11:21 am

i can't put "?" in terminal
ctrv^v reset "?" in url-address

sid5632 · Thu Aug 02, 2018 12:40 pm

Use \?

The script in post #9 was wrong.

kakaxa · Thu Aug 02, 2018 1:26 pm

Use \?

The script in post #9 was wrong.

Thanks Sid

acortesguasch · Thu Aug 02, 2018 2:17 pm

Just incorporated the script to one router, 12h interval. For now, it seems to works well. We'll see.

IntrusDave · Thu Aug 02, 2018 6:20 pm

Use \?

The script in post #9 was wrong.

The script is not wrong, it's intended to be a script, NOT command line.

jo2jo · Fri Aug 03, 2018 8:26 am

I loved your service and used it, im def. willing to pay once you are live. Do you have anywhere we can sign up for an email alert or some info once the paid service is done/live? (subbing to this thread or the other main/closed thread will produce alot of "noise") tks

expert · Fri Aug 03, 2018 12:19 pm

Hi, since I'm interested about the blacklist service and in order to evaluate whether it's useful to me, I'd like to know, what exactly is blacklisted?
Who/what created such list of IPs? Thanks in advance.

hhgttg42 · Fri Aug 03, 2018 6:23 pm

So, I *JUST* started coding this week, so this is really rough.. like pre-alpha. This is going to take some time.. I can code in python, php, perl, and C. but just because I can, doesn't mean I enjoy it. I really hate coding, it's boring and frustrating. Anyway..

I hear that. Thank you Dave! I will be trying this out tonight to give you some more stress-testing data. Cheers!

jo2jo · Sat Aug 04, 2018 4:03 am

Hi, since I'm interested about the blacklist service and in order to evaluate whether it's useful to me, I'd like to know, what exactly is blacklisted?
Who/what created such list of IPs? Thanks in advance.

the dev of this script/list uses both publicly available lists of "bad" ips (spamhaus , malcode ect), as well as his own "honeypot" devices which look for public IPs that are doing suspicious activities (then adds those IPs to his own "private" list, for distribution to ppl running his script , before he closed the service). Def was a great service + script, and one i plan on paying for once he re-launches.

I think in the main forum thread (ie not this new paid/development thread), the dev lists some of these sources he uses.

aboiles · Sat Aug 04, 2018 6:42 pm

Script is no longer functioning, no updates since last night.

Sat Aug 04, 2018 7:25 pm

Dave,

Have you considered using external to your network honeypots as source of offending IPs?
I use as the first frontier such RAW drop rules and all the time there are some IPs on the list of attackers.

add action=add-src-to-address-list address-list=RAWATTACK2 address-list-timeout=127m chain=prerouting comment=RAW2ADD in-interface-list=WAN_LIST log-prefix="RAW2ADD: " src-address-list=RAWATTACK
add action=drop chain=prerouting comment=RAW2 in-interface-list=WAN_LIST log-prefix="RAW2: " src-address-list=RAWATTACK2
add action=add-src-to-address-list address-list=RAWATTACK address-list-timeout=37m chain=prerouting comment=RAW1ADD dst-port=8291,21,22,23,2000,7547,11211,135,137-139,548,80,8080,81,37215 in-interface-list=WAN_LIST log=yes log-prefix="RAW1: " protocol=tcp
add action=add-src-to-address-list address-list=RAWATTACK address-list-timeout=37m chain=prerouting comment=RAW1ADD dst-port=8291,21,22,23,2000,7547,11211,135,137-139,548,80,8080,81,37215 in-interface-list=WAN_LIST log=yes log-prefix="RAW1: " protocol=udp
add action=drop chain=prerouting disable=yes  comment=RAW1 in-interface-list=WAN_LIST log-prefix="RAW1: " src-address-list=RAWATTACK

IntrusDave · Sat Aug 04, 2018 8:26 pm

Dave,

Have you considered using external to your network honeypots as source of offending IPs?
I use as the first frontier such RAW drop rules and all the time there are some IPs on the list of attackers.

I'm not quite sure I follow what you are saying. I'm always open to more sources. The new system is very modular. So importing another source is as simple as coding an import module for it.

IntrusDave · Sat Aug 04, 2018 8:27 pm

Script is no longer functioning, no updates since last night.

It's running right now.
As stated above, it's still very pre-alpha, so I can't promise that it stays running while I'm making large code changes.

jo2jo · Sat Aug 04, 2018 8:48 pm

dave, maybe make a google forums entry where users (who plan to pay once you go live, which i realize may be a good while away) can submit their email address to you, easily and securely (easy for you to create/keep i mean).
This way when you launch, you can send an email to all those who submitted, and you will have a decent amount of funding coming in at start (vs ppl forgetting about it or loosing track of this thread).
just an idea. tks

Sat Aug 04, 2018 9:36 pm

...Have you considered using external to your network honeypots as source of offending IPs? ...
I'm not quite sure I follow what you are saying. I'm always open to more sources. The new system is very modular. So importing another source is as simple as coding an import module for it.

Could it be possible to send to you lists of attacking IPs from my routers?

kakaxa · Sat Aug 04, 2018 9:46 pm

IntrusDave thx for u service
please prompt how to change a timeout of blacklist lifetime. for example for 7 days

IntrusDave · Sat Aug 04, 2018 10:59 pm

please prompt how to change a timeout of blacklist lifetime. for example for 7 days

The lists are set for a max timeout of 24 hours. This is required so that false positives are not blocked for too long. The system is designed to be update every 1~6 hours.
Once the system goes public, each user will be able to configure the timeout for each router.

Could it be possible to send to you lists of attacking IPs from my routers?

Yes, I am working on that too. My plan is that the routers will add IP's to a dedicated address-list, and then a script will submit that list to the server, just as the honeypots do.

IntrusDave · Sat Aug 04, 2018 11:18 pm

Here is a form to fill out for those that want to be notified

https://goo.gl/forms/UQMYqKJ54E0iV35l2

kakaxa · Sat Aug 04, 2018 11:22 pm

The lists are set for a max timeout of 24 hours. This is required so that false positives are not blocked for too long. The system is designed to be update every 1~6 hours.
Once the system goes public, each user will be able to configure the timeout for each router.

thanks for explanation, Dave

43north · Mon Aug 06, 2018 5:47 am

Just put the script on my home CCR1009 and am sooooo stoked to be using your service again. Just the piece of mind will be huge for me. Will move it into production on my work Tiks after testing a few days at home. EDIT: Also Dave can you educate us on the Priority Levels 1,2,3 that are part of the service, what determines what IP address makes it in to which priority, how are they prioritized?

Dave do you have an email address or a way to touch bases off line? I am not sure why I can no longer send private messages on the forum anymore....

IntrusDave · Mon Aug 06, 2018 5:26 pm

currently, the priorities are pretty basic.
#1 is a short list of about 2000, consisting of just the most common botnet attacks. If I end up offering a free tier, this will be it.
#2 is a longer list of 30,000 to 40,000 IP's and subnets that includes #1, also adds most of the more common crap out there.
#3 is the largest list of 120,000 to 150,000+ IP's and subnets, includes #1 and #2, includes all "known" spammers, as well as unassigned subnets, proxies, etc.

43north · Tue Aug 07, 2018 7:24 am

currently, the priorities are pretty basic.
#1 is a short list of about 2000, consisting of just the most common botnet attacks. If I end up offering a free tier, this will be it.
#2 is a longer list of 30,000 to 40,000 IP's and subnets that includes #1, also adds most of the more common crap out there.
#3 is the largest list of 120,000 to 150,000+ IP's and subnets, includes #1 and #2, includes all "known" spammers, as well as unassigned subnets, proxies, etc.

Thanks for the info. I have been running priority 3 on my 1009 for a couple days now. First time I have used RAW rules as well. Working like a champ!!! Your list is catching everything before anything hits my "blacklist" that I have built over time from things my router has personally seen. Super awesome. Keep up the good work! Once again THANK YOU Dave.

IntrusDave · Tue Aug 07, 2018 7:51 am

If anyone wants to help out more, I need more routers to report some stats to the server. This is part of the health monitoring and alerting system. If you paste the code into a terminal window, it will setup the script and start reporting.

/system scheduler
add interval=1m name=reportStatus on-event="/system script run reportStatus" policy=read,write,policy,test start-time=startup
/system script
add name=reportStatus owner=djoyce policy=read,test source=":local pa\
\_\"\"; :local pb \"\"; :local pc \"\"; :local pd \"\"; :local pe \"\"; :local pf \"\"; :local postdata \"\";\r\
\n:set pa [:tostr [ /system routerboard get ]]; :set pb [:tostr [ /system license get ]];\r\
\n:set pc [:tostr [ /system resource get ]]; :set pd [:tostr [ /system health get ]];\r\
\n:set pe [:tostr [/system identity get ]]; :set postdata [:toarray \"\$pa;\$pb;\$pc;\$pd;\$pe\"];\r\
\n/tool fetch mode=https url=\"https://bl.mikrotikfilters.com/hwstats.php\" http-method=post http-data=\"data=\$postdata\
\" output=file dst-path=hwdata.txt;"

Here is a sample from my personal firewall on what it reports:

board-name=RB1100AHx4 Dude Edition;
current-firmware=6.43rc51;
factory-firmware=3.36.3;
firmware-type=al2;
model=RouterBOARD 1100Dx4;
routerboard=true;
serial-number=735B073F0D77;
upgrade-firmware=6.43rc51;
features=;
nlevel=6;
software-id=NYLS-9KPC;
architecture-name=arm;
board-name=RB1100AHx4 Dude Edition;
build-time=Aug\/01\/2018 09:43:29;
cpu=ARMv7;
cpu-count=4;
cpu-frequency=1400;
cpu-load=0;
factory-software=6.38.4;
free-hdd-space=98365440;
free-memory=1012338688;
platform=MikroTik;
total-hdd-space=134479872;
total-memory=1073741824;
uptime=10:00:11;
version=6.43rc51 (testing);
current=488;
power-consumption=115;
psu1-voltage=243;
psu2-voltage=242;
temperature=53;
voltage=236;
name=Home_Firewall;

Steveocee · Tue Aug 07, 2018 1:29 pm

Just put this onto my CHR home router. Had to fiddle the script a little bit to make it work though which I expected I may need to;
Note, disk1 is not present and I had to add in a "?" after the "fetch.php"

/tool fetch mode=https dst-path=/blacklist/filters.rsc url="https://bl.mikrotikfilters.com/fetch.php\?priority=3";
/import file-name=blacklist/filters.rsc
/file remove blacklist/filters.rsc

In the rsc file it has 4 filter rules at the bottom which didn't apply, I take it you need to add these in manually? Oddly doing a copy & paste didn't add them in so I made these;

/ip firewall raw
add action=drop chain=prerouting comment="DROP intrusBL" src-address-list=intrusBL
add action=drop chain=prerouting comment="DROP intrusBL" dst-address-list=intrusBL

Have stuck the fetch and remove commands into a script (intrus-bl-updater) and added into scheduler running once every 12 hours (a bit longer than suggested I know).

Added in the system reporter as well, it was set to report every minute though so have altered that slightly to 12H intervals

Amazing work Dave!

grusu · Tue Aug 07, 2018 3:30 pm

Hi Dave,

In first list first address is 255.255.255.255 . Is that right?

Thanks,
Geo

tippenring · Tue Aug 07, 2018 5:04 pm

If anyone wants to help out more, I need more routers to report some stats to the server. This is part of the health monitoring and alerting system. If you paste the code into a terminal window, it will setup the script and start reporting.

Running on my home router. Do you really want it reporting every minute?

IntrusDave · Tue Aug 07, 2018 5:28 pm

Running on my home router. Do you really want it reporting every minute?

The reporting and monitoring service is reported every minute. The client side can change that, depending on that type of response time they want.

IntrusDave · Tue Aug 07, 2018 5:32 pm

Hi Dave,

In first list first address is 255.255.255.255 . Is that right?

Thanks,
Geo

Yes. Once the system is complete, you will be able to whitelist if needed. I filter 255.255.255.255 because I'm on a cable network and I see a crap-load of broadcast trash.

expert · Tue Aug 07, 2018 6:17 pm

I see everybody here is amazed how great service it is, but has anybody think about security risks of such service?
Importing third-party script to your router without any validation?

I wonder why this list is not provided as plain list of IPs and let everybody implement custom script parsing and validating the input.

IntrusDave · Tue Aug 07, 2018 6:21 pm

You are 100% welcome to not use it. If you don't have anything to add to the topic, or any input on the development process, please find another topic to post in.

If you followed the previous versions thread, you would see that this has been covered in no less than 5 posts.

Steveocee · Tue Aug 07, 2018 6:50 pm

I see everybody here is amazed how great service it is, but has anybody think about security risks of such service?
Importing third-party script to your router without any validation?

I wonder why this list is not provided as plain list of IPs and let everybody implement custom script parsing and validating the input.

The script is readily available to download and inspect before hand because any self respecting person would do that rather than blindly running it.
Dave has been here for years providing this service to users in the community and is extremely well trusted, just don't pi$$ him off and you'll be fine

IntrusDave · Tue Aug 07, 2018 7:00 pm

I do understand the concern, but after 3 years, I'm tired of explaining myself.
I do what I do because I love the internet and I want it to be a safer place.
My company is based on this principle. I understand that you guys don't know me personally, and you have to trust me.
But do know that once this service goes public, you will have a license agreement that both sides agree to.

That said... I'm curious if the people that question the safety of my service run Windows or macOS...?
Microsoft has displayed a complete disregard for user's privacy and safety. Windows 10 forces updates even if you don't want them. Microsoft has FOUR TIMES invalidated my Volume License Keys because they changed the terms of the contract, effectively telling over 1500 users that the version of windows they are running may be pirated, only to lose a court case and be force to unban the VLK.

....and I'm the bad guy.

Anyway. Use it or don't use it. I'm not interested in providing a service to people that just tech-savvy enough to complain, but not enough to understand and see what's going on.

*MY* service will NEVER be a flat text file for anyone to download and parse. I'm spending thousands in time and money to develop the service, and I would at least like to break even.

IntrusDave · Tue Aug 07, 2018 7:02 pm

on the flip side, if anyone is in Southern California (Rancho Cucamonga / Ontario / Pomona / San Bernardino) you are hit me up and I'd love to grab coffee and chat.

expert · Tue Aug 07, 2018 7:04 pm

The script is readily available to download and inspect before hand because any self respecting person would do that rather than blindly running it.
Dave has been here for years providing this service to users in the community and is extremely well trusted, just don't pi$$ him off and you'll be fine

I don't really downgrade all the effort author put into the service and I still think it can be useful, however definitely not as directly importable (and scheduled!) script.
In my opinion, after every download it must be inspected for malicious content (what if the origin was hacked in meantime?).

You are 100% welcome to not use it.
I will not use it. Or maybe I will, but before I will implement another script that extracts IPs from the file and apply them one by one.

IntrusDave · Tue Aug 07, 2018 7:09 pm

Due to people like this guy ^^^ the beta list will be limited to the "free" list of about 1800 IP's.
I do not want my 150,000 IP's collected by my honeypots being used for other people's services.

IntrusDave · Tue Aug 07, 2018 7:11 pm

I will not use it. Or maybe I will, but before I will implement another script that extracts IPs from the file and apply them one by one.

Now THAT is funny. I can picture you looking through a list of 150,000 IP address every 24 hours.
That's the WHOLE point of this, to have near-realtime protection that doesn't require manually parsing 15 didn't blacklist sources.
But hey, good luck. Now, please don't post here anymore.

kakaxa · Tue Aug 07, 2018 7:21 pm

Due to people like this guy ^^^ the beta list will be limited to the "free" list of about 1800 IP's.
I do not want my 150,000 IP's collected by my honeypots being used for other people's services.

Dave, please don't do it. I very little in all this understand, but your creation is a masterpiece. It isn't necessary to spoil all because of one fool.

I apologize for the Google Translate

IntrusDave · Tue Aug 07, 2018 7:25 pm

Has anyone here worked with tr069 at all? I've never played with it and I'm curious if it offers anything useful to this project.

IntrusDave · Tue Aug 07, 2018 7:42 pm

Oh BTW guys, my Honeypots alone are reporting over 37,000 ACTIVE botnet IP's for the last 12 hours.
Those IP's will NOT be included in the free list.

43north · Tue Aug 07, 2018 7:59 pm

Oh BTW guys, my Honeypots alone are reporting over 37,000 ACTIVE botnet IP's for the last 12 hours.
Those IP's will NOT be included in the free list.

Dave please don't limit the Beta, don't let this guy be the driver for that. It is not worth it and hurts us that are your loyal followers. I am using your Priority 3 list and will start paying for it whenever you want. It is helping me immensely. Please don't limit the beta list to the small list......

IntrusDave · Wed Aug 08, 2018 1:34 am

It's limited for now, hoping to have a very basic auth system in place by tomorrow morning. My server logs show at least 2 people trying VERY hard to figure out how to trick the server to sending the list to a wget/curl client. Sorry, but the blaintant abuse won't be tolerated.
I'll post a simple Google Form for registering ASAP.

Steveocee · Wed Aug 08, 2018 11:03 am

It's limited for now, hoping to have a very basic auth system in place by tomorrow morning. My server logs show at least 2 people trying VERY hard to figure out how to trick the server to sending the list to a wget/curl client. Sorry, but the blaintant abuse won't be tolerated.
I'll post a simple Google Form for registering ASAP.

I was just about to post and say that my blacklist had decreased overnight from 140K+ to 2K!
It's a real shame that people abuse your good nature like this.

43north · Wed Aug 08, 2018 8:44 pm

on the flip side, if anyone is in Southern California (Rancho Cucamonga / Ontario / Pomona / San Bernardino) you are hit me up and I'd love to grab coffee and chat.

Dave, although I am not in your area, I am next door in Idaho. I am very interested in setting up honeypots where I am at to contribute to the database however you see fit. How can we catch up on the phone etc to discuss more information that I don't want to post on the forum?

grusu · Thu Aug 09, 2018 11:02 am

Dave,

Have you considered using external to your network honeypots as source of offending IPs?
I use as the first frontier such RAW drop rules and all the time there are some IPs on the list of attackers.

add action=add-src-to-address-list address-list=RAWATTACK2 address-list-timeout=127m chain=prerouting comment=RAW2ADD in-interface-list=WAN_LIST log-prefix="RAW2ADD: " src-address-list=RAWATTACK
add action=drop chain=prerouting comment=RAW2 in-interface-list=WAN_LIST log-prefix="RAW2: " src-address-list=RAWATTACK2
add action=add-src-to-address-list address-list=RAWATTACK address-list-timeout=37m chain=prerouting comment=RAW1ADD dst-port=8291,21,22,23,2000,7547,11211,135,137-139,548,80,8080,81,37215 in-interface-list=WAN_LIST log=yes log-prefix="RAW1: " protocol=tcp
add action=add-src-to-address-list address-list=RAWATTACK address-list-timeout=37m chain=prerouting comment=RAW1ADD dst-port=8291,21,22,23,2000,7547,11211,135,137-139,548,80,8080,81,37215 in-interface-list=WAN_LIST log=yes log-prefix="RAW1: " protocol=udp
add action=drop chain=prerouting disable=yes  comment=RAW1 in-interface-list=WAN_LIST log-prefix="RAW1: " src-address-list=RAWATTACK

Hi BartoszP,

It's a good starting point. I have used your rules but set it so that I do not block the IPs from where I administer the router.
How can I collect offending IPs?

Thanks,
Geo

Steveocee · Sun Aug 12, 2018 7:25 pm

@IntrusDave
Have you changed the beta availability again? I've just checked my list to make sure it's still updating nicely and noticed I've jumped form some 2K to 16K entries!
Thank you

IntrusDave · Sun Aug 12, 2018 9:56 pm

Yes, I've blocked most of the IP's that are trying to leach the lists.
Still working on an auth system that is reliable. I think it's going to have to be based on the the Cloud DNS.. [/ip cloud set ddns-enable=yes] is going to be required, unless MikroTik gives me a way to authenticate better than that.

Steveocee · Mon Aug 13, 2018 9:44 am

Yes, I've blocked most of the IP's that are trying to leach the lists.
Still working on an auth system that is reliable. I think it's going to have to be based on the the Cloud DNS.. [/ip cloud set ddns-enable=yes] is going to be required, unless MikroTik gives me a way to authenticate better than that.

Good times, thankyou.
“Oh bugger” he says with his CHR! I may end up notifying you of my static IP and hoping you’d do me a solid then?

A question, do I need BOTH a firewall filter rule AND a raw drop rule? I’m currently using just raw drop rules for source and then another for destination of the BL but wondered if there was any gain in running a filter rule of each as well? (Appreciate I could create rules and watch counters but wondering on your recommendation).

Chupaka · Mon Aug 13, 2018 7:03 pm

“Oh bugger” he says with his CHR! I may end up notifying you of my static IP and hoping you’d do me a solid then?

CHR even with trial license has IP Cloud now (starting v6.43)

amity2kare · Thu Aug 16, 2018 3:34 pm

Hi Dave,

I agree with everyone when I say the beta should not be restricted just because one guy decided to go another route. We would be most happy to pay and I would happily be a part of your beta program and even get my clients on board just because it saves them money they would instead spend on a firewall or something.

Regards

Amit

eddieb · Sat Aug 18, 2018 10:00 am

Dave ?

I noticed a problem in my logging, is downloading the ?priority=1 not working anymore ?

Eddie

IntrusDave · Sat Aug 18, 2018 5:08 pm

sorry about that, reboot the server and forgot to start a service.
I don't have anything auto-starting yet.

IntrusDave · Sun Aug 19, 2018 8:02 pm

As you can tell, I've slowed down on development.
Out of the 100+ people who filled out the notification form, more than 60% says they will not pay for this type of service.
Only about 15 say they will pay a commercial product. So, I'm going to take my time with it and try earning some income in other ways.
I'm sure I'll post more updates when I get more time.

43north · Sun Aug 19, 2018 8:27 pm

As you can tell, I've slowed down on development.
Out of the 100+ people who filled out the notification form, more than 60% says they will not pay for this type of service.
Only about 15 say they will pay a commercial product. So, I'm going to take my time with it and try earning some income in other ways.
I'm sure I'll post more updates when I get more time.

Ah man that is crazy, especially for the low cost you were looking to charge. Thank you Dave for the work you are putting in to this. It is a great service. Would still love to learn how to setup a honeypot to help contribute.

IntrusDave · Sun Aug 19, 2018 10:32 pm

I've implemented some of the new security functions. You will need to update the download script.

Cloud DDNS is *REQUIRED*. Which means you need the latest CHR with Cloud support.

/ip cloud set ddns-enabled=yes

Here is the new script.

:local destPath "disk1/filterImport.rsc";
:local priority "2";

:local sn [:pick [/ip cloud get dns-name] 0 [:find [/ip cloud get dns-name] "."]];
/tool fetch mode=https url="https://bl.mikrotikfilters.com/secureFetch.php?priority=$priority" http-method=post http-data="$sn" dst-path="$destPath" output=file; /import file-name=$destPath;  /file remove $destPath;

boldsuck · Mon Aug 20, 2018 3:22 am

As you can tell, I've slowed down on development.
Out of the 100+ people who filled out the notification form, more than 60% says they will not pay for this type of service.
Only about 15 say they will pay a commercial product. So, I'm going to take my time with it and try earning some income in other ways.

Too bad that there is so little interest in this service. There were so many users in the old thread who wanted to pay.
I signed up today as a willing to pay. Actually, this is exaggerated for my 2 Home Router but I would like to contribute something to this project.

Edit:
I mean the huge IP list for my home router, not the money.

dmercer · Tue Aug 21, 2018 5:34 am

My company would like to contribute $1000 to the development of this solution. We are still happy to pay the monthly fee once things are up and running. We have benefitted over the years of many volunteer and or open source projects. We do not have the programming skills to give back but we can offer up some money. I will send you my contact info directly. Please send me an invoice for development work and I will get a cheque sent out in the next week or so

webix · Tue Aug 21, 2018 10:22 am

ok... when i run the script, i get this:

/system script> /tool fetch mode=https url="https://bl.mikrotikfilters.com/secureFetch.php\?priority=$priority" http-met
hod=post http-data="$sn" dst-path="$destPath" output=file; /import file-name=$destPath;  /file remove $destPath;
      status: finished
  downloaded: 0KiBC-z pause]
    duration: 1s


Script file loaded and executed successfully
invalid value for argument numbers

Any clue on what it can be?

hilton · Tue Aug 21, 2018 12:18 pm

I've implemented some of the new security functions. You will need to update the download script.

Cloud DDNS is *REQUIRED*. Which means you need the latest CHR with Cloud support.
/ip cloud set ddns-enabled=yes

Here is the new script.
Code: Select all
:local destPath "disk1/filterImport.rsc";
:local priority "2";

:local sn [:pick [/ip cloud get dns-name] 0 [:find [/ip cloud get dns-name] "."]];
/tool fetch mode=https url="https://bl.mikrotikfilters.com/secureFetch.php?priority=$priority" http-method=post http-data="$sn" dst-path="$destPath" output=file; /import file-name=$destPath;  /file remove $destPath;

Dave does this script replace your old 'blacklistUpdate' script that was scheduled? Must I just schedule this new script as per before?

dmercer · Tue Aug 21, 2018 8:40 pm

for some reason many of my firewalls do not seem to have the version of the code that supports the ddns. So when I go to /ip there is no "cloud". This is true for both x86 versions and CHR running 6.42.7. Has anybody else seen this?

Rico40 · Tue Aug 21, 2018 9:20 pm

For me, CHR works with the rc version

boldsuck · Wed Aug 22, 2018 12:18 am

Dave does this script replace your old 'blacklistUpdate' script that was scheduled? Must I just schedule this new script as per before?

Hi, I'm not Dave but:
Yes and Yes

hilton · Wed Aug 22, 2018 9:14 am

Dave does this script replace your old 'blacklistUpdate' script that was scheduled? Must I just schedule this new script as per before?
Hi, I'm not Dave but:
Yes and Yes :)

Ta!

Steveocee · Wed Aug 22, 2018 10:43 am

for some reason many of my firewalls do not seem to have the version of the code that supports the ddns. So when I go to /ip there is no "cloud". This is true for both x86 versions and CHR running 6.42.7. Has anybody else seen this?

You need 6.43 on your CHR to run IP>Cloud and it has been confirmed there will be no IP>Cloud for x86 platform.

@IntrusDave Can I ask if there is any way to relax this "need" for cloud? With 6.43 being an RC candidate many people won't run this on their "normal" equipment and only on test stuff. I love your script, I really do but I don't want to run a potentially unstable routerOS release on my main router.

Chupaka · Wed Aug 22, 2018 11:56 am

@IntrusDave Can I ask if there is any way to relax this "need" for cloud? With 6.43 being an RC candidate many people won't run this on their "normal" equipment and only on test stuff. I love your script, I really do but I don't want to run a potentially unstable routerOS release on my main router.

Well, the answer is simple: wait for the 6.43 release. Don't you want to run a potentially unstable Blacklist Filter (which is still in development from scratch) on your main router?

Steveocee · Wed Aug 22, 2018 12:29 pm

@IntrusDave Can I ask if there is any way to relax this "need" for cloud? With 6.43 being an RC candidate many people won't run this on their "normal" equipment and only on test stuff. I love your script, I really do but I don't want to run a potentially unstable routerOS release on my main router.
Well, the answer is simple: wait for the 6.43 release. Don't you want to run a potentially unstable Blacklist Filter (which is still in development from scratch) on your main router?

I know Dave's script is fine though. Moving into RC channel and back can cause huge problems and sometimes full re-installation.

wanos · Wed Aug 22, 2018 2:13 pm

Well that was a helluva ride ... I read 700 posts in 3 or 4 days ... still available or not, BGP and DNS shot down, ... fingers itching halfway thru reading to go back to first post and download-install, read code and run ... but I'm patient and waited until the end. And I'm relieved to find this is being revived in a new form.

Looking for blacklists on the net brought me here. I have a rb2011 that will work, and happy to find in an old box ... a micro USB to female USB A adapter cable ... plugged in an old stick and now I have temp storage to wear out instead of NAND on the device.

Being a software dev in a previous lifetime, I have an idea of just how much effort this involves. My hats is off to you Dave. We all owe you a debt of gratitude for the perseverance to keep this project alive for the MikroTik community. Software like this elevates and distinguishes MikroTik from the other major players in a big way. Thank-you. I have no more need for

fyi:
- running the hwlist.txt creation boots the rb2011 cpu to almost 30% for 1 sec. I have changed it to run every 2 days instead. Good enough for home.
- importing level 2, at only 16k addresses currently, keeps the cpu at 100% for about 50 secs. After importing, the cpu rests from 3 to 8%.

Once again, thanks Dave.

IntrusDave · Wed Aug 22, 2018 5:09 pm

Unfortunately, requiring IP Cloud to be enabled is the only way that I have found to try and reducing the amount of piracy and unauthorized redistribution. As at least two of the people in this thread have already shown, they feel my work should be done for free and that they (and the rest of the internet) should have full access to my list.

Using IP Cloud, it allows me to match the device serial number to the registered IP with the sn.mynetname.net service. I would love to use a simple http-auth, but with no way to encrypt the script on the client side, it's too easily forged. There is no great, or even good solution at this time, other than maybe username and password, but those get shared too easily.

I would LOVE it if MikroTik would just build a service into RouterOS to sync address-lists... but I doubt that would happen.

IntrusDave · Wed Aug 22, 2018 5:11 pm

ANd yes, you need the RC for the CHR to get IP Cloud. Given that the development is in "beta" it shouldn't be run on production devices (although I do..), the requirement for IP CLoud isn't going away. As for x86... Well, I have one of those too, but it's being moved to CHR to get pasted the x86 limits.

tippenring · Sun Aug 26, 2018 6:13 am

Please keep up the great work. I've been running the BL on my home router as an experiment for a few weeks now. No trouble so far here.

I would be interested in assisting with dev if I can. I'm not sure what I could do to help though. I'm not a good coder (unless my years-ago basic and quickbasic coding counts). I manage a few hundred routers and networks for clients. I expect many would be interested in a commercial version of your service.

I could easily and securely host a honeypot on my home network or my work network if that would be useful.

HZsolt · Sun Aug 26, 2018 10:38 am

Where can I find the active blacklist script?

Rico40 · Sun Aug 26, 2018 10:50 am

Is on this page

HZsolt · Sun Aug 26, 2018 10:53 am

But I get error in the log: Blacklist Authorization failed

Which is the active and good script?

grusu · Sun Aug 26, 2018 11:27 am

But I get error in the log: Blacklist Authorization failed

Which is the active and good script?

You must enable IP Cloud service first.

HZsolt · Sun Aug 26, 2018 11:37 am

But I get error in the log: Blacklist Authorization failed

Which is the active and good script?
You must enable IP Cloud service first.

Thank you!

Works!!!!

HZsolt · Sun Aug 26, 2018 12:39 pm

Drop or redirect? Which one is better on firewall?

Question for IntrusDave: Do you have any IPv6 blacklist and domain (IPv4/IPv6) blacklist?

IntrusDave · Sun Aug 26, 2018 6:53 pm

Unfortunately, I don't have IPv6 yet. The system is designed for it, but I have no routers in IPv6 networks that I can test with. My home internet supports it, but it's so unstable, I don't bother with it.

HZsolt · Sun Aug 26, 2018 7:46 pm

Unfortunately, I don't have IPv6 yet. The system is designed for it, but I have no routers in IPv6 networks that I can test with. My home internet supports it, but it's so unstable, I don't bother with it.

OK! Thanks!!!

And domain blacklist?

boldsuck · Mon Aug 27, 2018 1:13 am

And domain blacklist?

No only IP based.
But more than 135,000 if you want

and your router can handle. (I get priority "1" on a RB2011UAS.)

This is a further development of the old project / service:
viewtopic.php?f=9&t=98804

IntrusDave · Mon Aug 27, 2018 2:14 am

Unfortunately, I don't have IPv6 yet. The system is designed for it, but I have no routers in IPv6 networks that I can test with. My home internet supports it, but it's so unstable, I don't bother with it.
OK! Thanks!!!

And domain blacklist?

I don't find domain blacklisting very effective. Most botnets and viruses have their own DNS resolver and use hard codes servers, so it doesn't really help at the router level. And more and more are moving to dns over https.

IntrusDave · Mon Aug 27, 2018 2:15 am

Drop or redirect? Which one is better on firewall?

Personally, I use a RAW Drop rule.

IntrusDave · Mon Aug 27, 2018 2:23 am

But I get error in the log: Blacklist Authorization failed

Which is the active and good script?

IP Cloud is used for identification now. Once the service is live, the serial number from IP cloud will be used for authorization. The script gets the serial number from the IP cloud, submits it via the http-post over TLS, this keeps your serial from being sent in the clear. When the server receives the request, the http-post data is pulled, the serial number is then used to do a DNS lookup via {xxxxxxxxx.sn.mynetname.net} and that IP is then matched to the IP that is making the request. If the IP's don't match, then the odds are that the serial number is a fake, or someone is trying to leach the list. It's not a perfect system, but as long as the mynetname service isn't hacked, it should be good enough to stop most from leaching the list.

Other ideas were along the lines of assigning every router a UUID and then sending that, but again, no way to verify that the http-post is authentic. Nothing to keep someone from putting that UUID on other routers, or just faking it and using a script to clone the list.

I would love for MikroTik to put in a service that allows the routers to authenticate themselves, download and apply a list, Hell, I would even code the service for them.. but I'm fairly certain that will never happen.

HZsolt · Mon Aug 27, 2018 8:45 am

Unfortunately, I don't have IPv6 yet. The system is designed for it, but I have no routers in IPv6 networks that I can test with. My home internet supports it, but it's so unstable, I don't bother with it.
OK! Thanks!!!

And domain blacklist?
I don't find domain blacklisting very effective. Most botnets and viruses have their own DNS resolver and use hard codes servers, so it doesn't really help at the router level. And more and more are moving to dns over https.

OK!

Thanks!

For example domain blacklist: https://blog.squidblacklist.org/?p=1658 It would works for effective? The script saves the file to flash.

Download domain blacklist script:

/tool fetch url="https://www.squidblacklist.org/download ... ns-ads.rsc" mode=http;
:log info "tik-dns-ads.rsc from http://www.squidblacklist.org";

Replace downloaded domain blacklist script:

/ip firewall address-list remove [find where comment="sbl ads"]
/import file-name=tik-dns-ads.rsc;
:log info "Removed old DomainBlackList and imported new list";

viewtopic.php?t=113770

43north · Tue Aug 28, 2018 3:35 am

Dave,
Still very interested in learning how to setup a honeypot to collect addresses. Even if you are not to the point to accept other people's honeypot lists, could you do a brief write up to teach us the best way to setup a honeypot? Thanks!

IntrusDave · Tue Aug 28, 2018 5:43 am

i haven't gotten far enough on the honeypot side. I've started from scratch on the RouterOS script. I'll post it once it's stable enough to test.

tippenring · Tue Aug 28, 2018 6:34 am

Unfortunately, I don't have IPv6 yet. The system is designed for it, but I have no routers in IPv6 networks that I can test with. My home internet supports it, but it's so unstable, I don't bother with it.

Have you seen HE's free IPv6 tunnel https://tunnelbroker.net/? I've had one up for nearly a year.

tippenring · Tue Aug 28, 2018 6:37 am

Dave,
Still very interested in learning how to setup a honeypot to collect addresses. Even if you are not to the point to accept other people's honeypot lists, could you do a brief write up to teach us the best way to setup a honeypot? Thanks!

Here are a couple of Honeypot projects from my notes. I'm sure there are many more. It's one of those things I've been wanting to do one of these days.

https://github.com/desaster/kippo
https://trustfoundry.net/honeypi-easy-h ... pberry-pi/

boldsuck · Sat Sep 01, 2018 1:53 pm

Unfortunately, I don't have IPv6 yet. The system is designed for it, but I have no routers in IPv6 networks that I can test with. My home internet supports it, but it's so unstable, I don't bother with it.

If it helps and the IPv4 sevice is done, I can provide an IPv6 router as honeypot.
I get a ::/48 prefix length and could then put a router¹ behind the Mikrotik. @Dave: You can have full admin access on it.
I get a new dynamic prefix from my provider every 36-48 hours. I can get a static IP but I have to pay extra for it. IPv6 has been stable for years, and I've had it since the pilotphase. (Year 2013 / Provider: NetCologne.de)

¹On a UBNT (ER-8) router, a honeypot package can be loaded from the Debian reposity.
Of course, the Mikrotik can serve as honeypot directly, if someone has finished scripts for it.

Steveocee · Mon Sep 10, 2018 2:50 pm

Have just noticed 6.43 has moved into the current branch so have updated accordingly. Can't seem to find IP>Cloud though?? Looking forward to using the IntrusBL again.

**It's not in Winbox but is there in the terminal.

IntrusDave · Mon Sep 10, 2018 8:16 pm

ip Cloud terminal-only when running CHR

IntrusDave · Mon Sep 10, 2018 8:21 pm

I'm in a holding pattern while my lawyer researches the EU "GDPR" laws.
It's looking like I will not be able to use 3rd party honeypots, as the GDPR requires companies to allow users to delete any data collected from there.
That means that anyone with a honeypot running on their router will be able to delete any IP's in the database that their router may have submitted.
While this may not sound like a big deal, it's ENTIRELY possible for a user to fake an update and delete the whole database.

43north · Mon Sep 10, 2018 9:25 pm

I'm in a holding pattern while my lawyer researches the EU "GDPR" laws.
It's looking like I will not be able to use 3rd party honeypots, as the GDPR requires companies to allow users to delete any data collected from there.
That means that anyone with a honeypot running on their router will be able to delete any IP's in the database that their router may have submitted.
While this may not sound like a big deal, it's ENTIRELY possible for a user to fake an update and delete the whole database.

Thanks for the update Dave.

Rico40 · Tue Sep 11, 2018 11:23 am

I also thank you for the update.

idoch · Fri Sep 21, 2018 5:24 pm

I'm in a holding pattern while my lawyer researches the EU "GDPR" laws.
It's looking like I will not be able to use 3rd party honeypots, as the GDPR requires companies to allow users to delete any data collected from there.
That means that anyone with a honeypot running on their router will be able to delete any IP's in the database that their router may have submitted.
While this may not sound like a big deal, it's ENTIRELY possible for a user to fake an update and delete the whole database.

Hey Dave,
I am looking forward to the new service. Obligatory - I am not a lawyer - but... You're over thinking this. An IP address itself is not "personally identifiable" -- until it is associated with other specific data that could be personally identifiable to a natural person (cookies, MAC, RFID, etc.). Classification of an IP as the source of infection, malicious behavior, etc. (or any of the behavior a honeypot would flag) is not at all identifiable to a natural person. Check into Recital 26 in full https://gdpr-info.eu/recitals/no-26/ Here's a piece:

"The principles of data protection should therefore not apply to anonymous information, namely information which does not relate to an identified or identifiable natural person or to personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable. This Regulation does not therefore concern the processing of such anonymous information, including for statistical or research purposes."

If you get Honeypot information secondhand under general classifications of "port scanner" or "wordpress prober" or "SSH Brute Force" -- I can't think of much that could be less personally identifiable to a natural person. You don't know any personally identifiable information about their interactions with the honeypot - all you have is an IP address and a general label. They can't force you to pretend that IP doesn't exist or that you couldn't find their ISP and report them to abuse@ or that you can't make decisions based on anonymized data based on "reputation" of a network -- which (by itself) is not identifiable to a natural person.

Absolute worst case: If your lists were created with an automatic timeout or expiration that was less than the 24 hour requested removal; wouldn't that be compliant? The IP would have been removed from the list after 24 hours (requested or not) -- if they get added back for ongoing bad behavior that's their problem. The list doesn't "track" them at all; each list is an independent serial number.

idoch · Fri Sep 21, 2018 10:30 pm

Showing you further that you (and your subscribers) are in the clear...
https://gdpr-info.eu/recitals/no-49/

szir · Fri Oct 19, 2018 12:14 pm

I'm new to MikroTik and I just found this thread.
I like the work that you do.
Security is important to me, so I would like to use your list. I put together a couple of rules for brute force prevention, (also reported some on abuseipdb) but blocking IPs with malicious activity that others found would be nice.

I read someone suggested DNS for updating the block list (instead of downloading a script). I would also like that. One problem I see is that as far as I know you cannot put an IP range into a DNS A record, which would make blocking whole subnets harder.
msatter suggested DNS as a means to ease the traffic generated by distributing the list.
I would like it for a different reason.

Security is important for me so I don't like the idea of downloading a script form an external source and running it on a schedule. I'm sure you are a nice and trustworthy guy, but I don't know you and don't know what security you put in place that prevents (and will prevent at any point in the future) others from hijacking your update script file. I don't want my router to become part of a botnet because it "accidentally" downloaded the wrong script...

Using DNS to update the list would eliminate the need to download and run unknown scripts, the worst that I can imagine happen in case someone takes over the DNS is that they could block innocent IPs/censor the Internet. (There might be some other way that doesn't require auto-running a downloaded script.)

IntrusDave · Fri Oct 19, 2018 5:02 pm

If using DNS is a requirement for you, I suggest you look elsewhere for the service.
I have no plans to use DNS for this service. It's not a viable distribution method.
DNS is not able to send a response of 200,000+ IP addresses.
BGP is also not going to happen, as it require a large amount of labor on both ends to configure.

I've gotten the server side stable enough to move forward. Though I may be changing the pricing.
My current thought is keeping the small list free for all..
Medium list will be accessible via donations.
Full list and custom configuration will be accessible via monthly subscription.

Steveocee · Fri Oct 19, 2018 6:42 pm

I've watched list "2" slowly grow over time, I think it was "only" around 14,000 entries when you first started this thread off and now it is up to 23,500+ entries. Seriously amazing stuff Dave.

Rico40 · Fri Oct 19, 2018 9:00 pm

Dave since when will be donated for a medium list?

IntrusDave · Fri Oct 19, 2018 9:40 pm

I’m not sure I understand the question

Rico40 · Fri Oct 19, 2018 9:57 pm

I ask since when we pay for the product.
I'm sorry but I'm not good at English.

IntrusDave · Fri Oct 19, 2018 10:52 pm

The pay service will begin on the 1st of the year

Rico40 · Fri Oct 19, 2018 11:22 pm

OK, I'm waiting for information on how to transfer money.

jausovec · Tue Oct 30, 2018 12:24 pm

Hi.

Can someone sum up the latest script/instructions on how to install the new service (and uninstall the old one

)?
Or are we not so far yet, that we could start using it in home environment?
I am also avaiting info about how/where to we can make the payment for the service.

Thank you

boldsuck · Fri Nov 02, 2018 2:02 pm

It's looking like I will not be able to use 3rd party honeypots, as the GDPR requires companies to allow users to delete any data collected from there.

This fucking GDPR

This law harms citizens more than it helps. A shot in the knee.
Some good forums has closed and from Germany you can not order anything from some shops in Switzerland anymore.

@Dave
Do you have IP blacklists from squidblacklist.org in priority 1 or 2?

@all
Because here is increasingly asked for payment:
Just fill out the form by Dave, if you haven't yet. Then you will receive an e-mail in time.
https://goo.gl/forms/UQMYqKJ54E0iV35l2

@jausovec
Disable or delete the old Blacklist script(s) and scheduler and fetch the new.
Post Nr.9 in this topic:
viewtopic.php?f=9&t=136666#p677573
Adjust 'destPath' and 'priority'.
Fix new schedulers or adjust the old ones.

HZsolt · Sat Nov 03, 2018 12:35 pm

For IntusDave:
Do you have any problem or do you update? I run your script but the script didn't download nothing.

I thank you for your help!

Steveocee · Sat Nov 03, 2018 1:27 pm

For IntusDave:
Do you have any problem or do you update? I run your script but the script didn't download nothing.

I thank you for your help!

Are you running IP > Cloud ? Would be the "easiest" thing to check at this point as it is a prerequisite.

HZsolt · Sat Nov 03, 2018 1:43 pm

For IntusDave:
Do you have any problem or do you update? I run your script but the script didn't download nothing.

I thank you for your help!
Are you running IP > Cloud ? Would be the "easiest" thing to check at this point as it is a prerequisite.

Yes, IP -> Cloud is running. DDNS Enabled and updated.

IntrusDave · Sat Nov 03, 2018 5:47 pm

The server is hosted on the google cloud platform. It appears that Google has oversold the zone that my servers are in, and my servers have been taking offline to allow others to run. I'll be moving the server to a different zone ASAP.

IntrusDave · Sat Nov 03, 2018 7:07 pm

@Dave
Do you have IP blacklists from squidblacklist.org in priority 1 or 2?

squidblacklist.org is not included, as it is a pay service. The IP's they have on the free lists are pretty much duplicates of my list 2.

IntrusDave · Sat Nov 03, 2018 8:10 pm

I bought a new network "level 3" honeypots online. These are my first located in "hostile" countries. Bringing the current Level 3 list to over 189,000 entries.

HZsolt · Sat Nov 03, 2018 8:15 pm

The server is hosted on the google cloud platform. It appears that Google has oversold the zone that my servers are in, and my servers have been taking offline to allow others to run. I'll be moving the server to a different zone ASAP.

Thanks!

Your blacklist works well again!

Thanks!

IntrusDave · Sat Nov 03, 2018 8:17 pm

No problem, and I'm sorry about the downtime.
I've changed the policy on my servers so that Google can not longer preempt mine to make room for higher paying customers. Not happy that it's adding another $50/month onto my bill, but I can't have them dropping me because someone bigger wants my cpu or memory.

IntrusDave · Sat Nov 03, 2018 8:56 pm

What are everyone's thoughts on using Patreon for the subscription service?
I've started setting up a page here: https://www.patreon.com/IntrusTechnologies
Once I have everything linked, the existing scripts will stop working and I will post the current scripts on the Patreon page.
The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.
The system will also disable accounts that are using forged serial numbers or IP addresses.

HZsolt · Sat Nov 03, 2018 9:30 pm

What are everyone's thoughts on using Patreon for the subscription service?
I've started setting up a page here: https://www.patreon.com/IntrusTechnologies
Once I have everything linked, the existing scripts will stop working and I will post the current scripts on the Patreon page.
The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.
The system will also disable accounts that are using forged serial numbers or IP addresses.

How much memory use on routers with these Blacklist Filters?

IntrusDave · Sun Nov 04, 2018 9:20 am

It's hard to give accurate numbers, but it looks like the List 1 uses about 768k, List 2 uses 3M, and List 3 uses 54M to load and 50M once the load is done. The other two load too fast on my RB110AHx4 to see the memory load update.

HZsolt · Sun Nov 04, 2018 10:55 am

It's hard to give accurate numbers, but it looks like the List 1 uses about 768k, List 2 uses 3M, and List 3 uses 54M to load and 50M once the load is done. The other two load too fast on my RB110AHx4 to see the memory load update.

What do you think about this service?: viewtopic.php?t=137632

acortesguasch · Sun Nov 04, 2018 12:54 pm

What are everyone's thoughts on using Patreon for the subscription service?
I've started setting up a page here: https://www.patreon.com/IntrusTechnologies
Once I have everything linked, the existing scripts will stop working and I will post the current scripts on the Patreon page.
The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.
The system will also disable accounts that are using forged serial numbers or IP addresses.

Any estimate regarding the end-of-service date as it is? I have to convince my boss of the benefits of being a paying member of the community and I do not want to be caught offguard.

Keep the fantastic work!

IntrusDave · Sun Nov 04, 2018 5:53 pm

It's hard to give accurate numbers, but it looks like the List 1 uses about 768k, List 2 uses 3M, and List 3 uses 54M to load and 50M once the load is done. The other two load too fast on my RB110AHx4 to see the memory load update.
What do you think about this service?: viewtopic.php?t=137632

I think it's a ripoff of my project by someone that hasn't been a part of the community as long as I have. I think I am far more transparent in the development process.

boldsuck · Sun Nov 04, 2018 6:05 pm

squidblacklist.org is not included, as it is a pay service. The IP's they have on the free lists are pretty much duplicates of my list 2.

OK, now I'll be clear here

Thanks.
Will test how much RAM a RB2011 needed. Only with priority 2
or priority 1 + drop.malicious.rsc

PS:
SBL Malicious IP Blacklist from: https://www.squidblacklist.org is free of charge.

IntrusDave · Sun Nov 04, 2018 6:05 pm

What are everyone's thoughts on using Patreon for the subscription service?
I've started setting up a page here: https://www.patreon.com/IntrusTechnologies
Once I have everything linked, the existing scripts will stop working and I will post the current scripts on the Patreon page.
The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.
The system will also disable accounts that are using forged serial numbers or IP addresses.
Any estimate regarding the end-of-service date as it is? I have to convince my boss of the benefits of being a paying member of the community and I do not want to be caught offguard.

Keep the fantastic work!

The goal is NO end-of-service date. This started out as a project out of a personal need, I shared it with the community because I thought others could use it too. Last year, the original service hit just over 17,000 active devices. I realized that several LARGE businesses were using the service and SELLING IT to their customers. At that point I felt that it was "fair" for me to be paying several hundred per month for servers and honeypots out of pocket, while others were making money off it it.

My vision is to have a service that is simple to implement, stable, secure, fast, and self-sustaining. Once the income is able to cover the expenses, and it is able to keep running without me, then I can focus on bringing new features and adding support for new platforms.

I don't want or plan to get rich from this. I want to provide a valuable service at a price that anyone can afford.

As for yearly payments - At this time, I am going to keep it monthly. I don't want to accept a payment for a year of service before I know the service will be able to sustain itself.

IntrusDave · Sun Nov 04, 2018 6:07 pm

squidblacklist.org is not included, as it is a pay service. The IP's they have on the free lists are pretty much duplicates of my list 2.
OK, now I'll be clear here Thanks.
Will test how much RAM a RB2011 needed. Only with priority 2
or priority 1 + drop.malicious.rsc

PS:
SBL Malicious IP Blacklist from: https://www.squidblacklist.org is free of charge.

I compared mine with that one - All of the IP's in that are also in mine. The key difference is the delivery method. My .rsc is much smaller and processed faster.

boldsuck · Sun Nov 04, 2018 6:31 pm

The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.

As for yearly payments - At this time, I am going to keep it monthly. I don't want to accept a payment for a year of service before I know the service will be able to sustain itself.

Would be nice to be able to select priority 1, although you have paid for priority 2.

Uh, monthly. I hope the monthly payment can be automated.

acortesguasch · Sun Nov 04, 2018 6:38 pm

Any estimate regarding the end-of-service date as it is? I have to convince my boss of the benefits of being a paying member of the community and I do not want to be caught offguard.

Keep the fantastic work!
The goal is NO end-of-service date. This started out as a project out of a personal need, I shared it with the community because I thought others could use it too. Last year, the original service hit just over 17,000 active devices. I realized that several LARGE businesses were using the service and SELLING IT to their customers. At that point I felt that it was "fair" for me to be paying several hundred per month for servers and honeypots out of pocket, while others were making money off it it.

When I asked about the end-of-service as it is I was referring when the current scripts will be deactivated and only working via Patreon.

I think I understood since the beginning the kind of project you are running, for you and for the Community, and I cannot see any flaw in you reasoning. You are giving a lot to the Community and it is only fair to try to cover expenses.

IntrusDave · Sun Nov 04, 2018 7:49 pm

My goal is January first.

HZsolt · Sun Nov 04, 2018 8:18 pm

My goal is January first.

After January first the current your script will not work?

IntrusDave · Sun Nov 04, 2018 9:07 pm

The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.

As for yearly payments - At this time, I am going to keep it monthly. I don't want to accept a payment for a year of service before I know the service will be able to sustain itself.
Would be nice to be able to select priority 1, although you have paid for priority 2.

Uh, monthly. I hope the monthly payment can be automated.

Payment is automated via the Patreon page. Each Tier includes the Tier below it. I've updated the tires on the page to better explain what you get.

Rico40 · Sun Nov 04, 2018 10:00 pm

How to make a payment from Poland?

IntrusDave · Sun Nov 04, 2018 10:01 pm

How to make a payment from Poland?

https://www.patreon.com/IntrusTechnologies

You can go to this Patreon page to sign up.

Rico40 · Sun Nov 04, 2018 10:02 pm

OK thanks.

hhgttg42 · Mon Nov 05, 2018 5:05 pm

What are everyone's thoughts on using Patreon for the subscription service?
I've started setting up a page here: https://www.patreon.com/IntrusTechnologies
Once I have everything linked, the existing scripts will stop working and I will post the current scripts on the Patreon page.
The new script will not require any modification, as the server will select the list based on your router's serial number and IP address.
The system will also disable accounts that are using forged serial numbers or IP addresses.

I'm already signed up! Thanks again for the great service Dave.

tippenring · Mon Nov 05, 2018 5:53 pm

OK, now I'll be clear here Thanks.
Will test how much RAM a RB2011 needed. Only with priority 2
or priority 1 + drop.malicious.rsc

I'm using the priority 2 list on an RB2011. Memory is fine. I currently have free 74MB of 128MB with ~30k blacklist entries. The RB2011 is more CPU starved when it's updating the list or if you view the address list.

boldsuck · Mon Nov 05, 2018 7:21 pm

Payment is automated via the Patreon page. Each Tier includes the Tier below it. I've updated the tires on the page to better explain what you get.

Perfect thank you.

Mikrotik forum logged me off last night again and again.

I'm using the priority 2 list on an RB2011. Memory is fine. I currently have free 74MB of 128MB with ~30k blacklist entries. The RB2011 is more CPU starved when it's updating the list or if you view the address list.

I've deleted most of the drop.malicious.rsc address-list entrys (~30k blocklist.de) and switched to priority 2 (also about ~30k). I have 7 MB more Ram free. (Free Memory now 55MB)

anav · Mon Nov 05, 2018 9:45 pm

It's hard to give accurate numbers, but it looks like the List 1 uses about 768k, List 2 uses 3M, and List 3 uses 54M to load and 50M once the load is done. The other two load too fast on my RB110AHx4 to see the memory load update.
What do you think about this service?: viewtopic.php?t=137632
I think it's a ripoff of my project by someone that hasn't been a part of the community as long as I have. I think I am far more transparent in the development process.

Sounds like an emotional response based on little fact. You would be better served by recognizing and supporting a like minded fellow provider who decided to share his work for his customers/clients with the community at large. The development cycle taken matches up with what I would have done, first being exposed to Josh Haven and the various sources, such as the Firehol lists, magically available whether you were on this less and less green earth or not. On his own he matured his script skills to be able to create the database and program with the flexibility and 'scalability' of his clients in mind (hex to larger units) which is critical to many of us with lesser units (not companies with fat wallets). Its stable, it works and its phukking affordable. He obviously has put much time and effort into the program, considering the servers required and the fail over and many other detailed minutia it takes to run a credible service. As to transparency, what are you referring too? He opened up his development to anybody that was interested for testing purpose. He clearly outlines his sources which do not contain 'secret' honeypots.

Don't get me wrong, I support what you are attempting to provide as much as the next person. I just hate to see unwarranted antagonism. If I had the skills I would be tempted to do the same and provide such a service. Good luck on progress in the next months!!

IntrusDave · Mon Nov 05, 2018 9:51 pm

If you want to support him, please support him in his topics.
If you would like to be involved in the me-vs-him debate, please at least do a little background research on the history.
My project has been the subject of "IP Theft" several times, and I do my best to keep my server side tech hidden now because of that.
I shut down my project once before, one of the many factors was other projects taking my lists and pushing them out as their own.

People are free to choose what they want to use, but if you want to talk about his stuff, please do so elsewhere.

IntrusDave · Mon Nov 05, 2018 10:03 pm

Tech Note: 6.44beta20 causes some issues, make sure you update to the current beta, if you are running the betas.

Rico40 · Tue Nov 06, 2018 12:33 am

I'm already signed up payment from January?

RackKing · Tue Nov 06, 2018 6:39 pm

So maybe a dumb question... I did have a look a the Patreon page. What level would you recommend to an integrator like who would offer this to his customers as part of a annual service offering? I would bill them directly and purchase your service. I suppose I could buy a tier and then upgrade as I cross that threshold? Will it be easy to see how many "routers" I have left to use?

I think this is a great project and thank you for all your efforts.

IntrusDave · Tue Nov 06, 2018 10:12 pm

Their are no dumb questions!

I will for sure have a UI for you to manage your routers. My goal is to have the UI finished by mid December. Though I just missed a pretty big deadline, so I may end up having to hire another developer to work on the front end while I continue on the back end. The UI is expected to use the Patreon login, then provide you with the total number of routers you can enroll, as well as the currently enrolled and management of the serial numbers. Authentication of the routers will be handled by the email address you subscribed with, along with the serial number of the router. Each router will have one unique address entry (randomly generated) that will allow me to find users that are misusing or claiming my lists as their own.

Cooperdale · Sun Dec 30, 2018 6:16 pm

Hello, are there any news on this? I can't wait for this service to come alive.

Rico40 · Sun Dec 30, 2018 8:44 pm

The service works well, the first payment has been sent.

boldsuck · Mon Dec 31, 2018 11:03 pm

The service works well, the first payment has been sent.

My Also,

but where to submit my router's Cloud DNS name?

Rico40 · Mon Dec 31, 2018 11:36 pm

let's wait for a message from Dave.

IntrusDave · Tue Jan 01, 2019 10:36 am

Hey guys, sorry for the radio silence, it’s been a pretty tough year but I’m trying to survive it.

I’ve been trying to get the automated registration process done but have ran into a few pretty nasty issues that I didn’t foresee. I’m going to put it on hold and process things manually until I can get some cash to pay another coder to work with me.

Going to spend the rest of the week with my kids, and then start fresh Monday morning.

sjoram · Sun Jan 27, 2019 2:42 pm

I found your Patreon.

I looked at the different 'tiers' - $10 currently works out about £7.50 a month...I'd be more than happy to support your work.

However, I do have a couple of questions (others with knowledge of your project may also have views) - sorry if this is not the best place to ask, but it seemed most likely to get the right visibility:

- I'm a home user, with ROS on two RouterBOARDs. Just about to swap both out with new RB750Gr3. I believe these have 256MB RAM, can these handle the largest tier?
- I found out about your work as I was one of the silly people that when a novice to ROS didn't include a PPPoE WAN interface on the default drop rules AND had some ROS services open to the wild, without port knocking. Also lax with updates, so Winbox vulnerability got exploited on one box. Yes, I know... Now fixed.
- One box particularly has a plethora of open dst-nat rules - mail server and the likes. I'm thinking that despite now having the default drop rules correct such that ROS itself is less vulnerable, your project may still be a benefit in protecting against unknown/unpatched exploits from known rogue addresses, not just for ROS but also the services behind those dst-nat rules... Would you agree?

IntrusDave · Tue Feb 12, 2019 3:21 am

Hey everyone. I'm sorry for being out of touch. Here is an update.

So, life has been pretty rough over the last 12-14 months. I've lost most of my clients, the new government tax code is killing me, and server costs keep going up. On the personal side, I've been dealing some some pretty serious health issues, and I'm not entirely sure how this is going to end for me.

That said, I can't in good conscience bill people for something that I can't guarantee that I can keep working on. I do have my notebook with me all the time, but some of the drugs are simply preventing me from thinking clear enough to do any coding at all.

So, I'm going to leave it open to all, and hope that at least a few of you will donate to keep the servers up and running.
I will do my very best to continue to improve the lists and the script.

I'd like to leave this PayPal Donation button here. Use if you like, don't if you don't like.

Again, as long as the service can support itself (about $200/month USD) I'll leave it running.
I have also taught my 14 year old daughter how to keep the servers running (she's already studding quantum physics) so that if the worst case happens, she can keep it running for you all.

I can certainly promise that NONE of the donations will go to my medical bills. I don't want anyone thinking that I'm using donations for anything other than what they expect.

Thank you, and I hope to keep posting.

https://www.paypal.com/cgi-bin/webscr?c ... source=url

BTW - this is the code that I use, and I prefer anyone to use for the script... make note of the path.

:local destPath "disk1/filterImport.rsc";
:local priority "2";

:local sn [:pick [/ip cloud get dns-name] 0 [:find [/ip cloud get dns-name] "."]];
/tool fetch mode=https url="https://bl.mikrotikfilters.com/fetch.php?priority=$priority" http-method=post http-data="$sn" dst-path="$destPath" output=file; /import file-name=$destPath;  /file remove $destPath;

sjoram · Tue Feb 12, 2019 10:27 am

Hi Dave,

Very sorry to hear of the challenges that life has thrown at you of late. I sincerely wish you and your family all the very best.

Thank you for your work on this, you know yourself how much demand your servers have seen, so I am sure this is benefiting and making life easier for a lot of people!

Rico40 · Tue Feb 12, 2019 9:14 pm

Dave donates funds to patreon and here he will also donate

IntrusDave · Fri Feb 15, 2019 12:59 am

Humans are truly awful.
While at my treatment yesterday, I dozed off. (I'm there for 6 hours every 3 days)
While sleeping, someone stole my backpack with my 6 month old notebook.

I'll still be doing some coding at home, but it's hard to sit at my desktop. I'll keep you all posted.

ihave · Fri Feb 15, 2019 2:45 am

That is truly awful indeed!
Sorry to hear that.

Can someone sum up the latest script/instructions on how to install the new service?
The old install script with blInstaller.rsc doesn't show anything in the log.

Filterimport.rsc is running fine and updating the address list. Am I only missing the scheduler or did the BlInstaller.rsc install additional things I am still missing?

Thank you

boldsuck · Fri Feb 15, 2019 8:10 pm

Can someone sum up the latest script/instructions on how to install the new service?

Script is 4 post higher.

ihave · Fri Feb 15, 2019 9:18 pm

Script is 4 post higher.

That script is running fine. I am just wondering about the frequency of the scheduler and maybe other things that were installed with the old installer.

boldsuck · Sun Feb 17, 2019 3:07 pm

That script is running fine. I am just wondering about the frequency of the scheduler and maybe other things that were installed with the old installer.

OK. Mmmmh, frequency of the scheduler:
That makes everyone different. I update every 12 hours. I think Dave updates the blacklists every hour.

I summarize everything for the people who are new here:

- Enable ddns. I have configured the SNTP client and do not need time here.

/ip cloud set ddns-enabled=yes update-time=no

- The script (destPath and priority may need to be adjusted):

/system script add dont-require-permissions=no name=blacklistScript owner=admin policy=read,write,policy,test source=\
\n"# Intrus Technologies blacklist installer/updater\
\n# \A92017 David Joyce, Intrus Technologies\
\n\
\n:local destPath \"filterImport.rsc\";\
\n:local priority \"2\";\
\n\
\n#If you do not want to delete the script directly after importing, remove the comment and paste one in the last line\
\n#:do { /file remove \$destPath } on-error={};\
\n\
\n:local sn [:pick [/ip cloud get dns-name] 0 [:find [/ip cloud get dns-name] \".\"]];\
\n/tool fetch mode=https url=\"https://bl.mikrotikfilters.com/fetch.php\?priority=\$priority\" http-method=post http-data=\"\$sn\" dst-path=\"\$destPath\" output=file;\
\n/import file-name=\$destPath;\
\n/file remove \$destPath;\
\n"

- The sheduler (After every reboot and then every 12 hours at 6:00 and 18:00) Adjust times as you want:

/system scheduler add interval=12h name=blacklistScriptUpdate on-event="/system script run blacklistScript" policy=read,write,policy,test start-date=jan/01/1970 start-time=06:00:00
/system scheduler add name=blacklistScriptUpdateOnBoot on-event=":delay 30;system script run blacklistScript" policy=read,write,policy,test start-time=startup

- The firewall rule:
Others also filter outgoing traffic (dst-address-list) and/or other interfaces. I block all port scanners, bruteforcer and blacklists in the beginning in the RAW chain. (Connection tracking would be a waste of time

I prefer to use the saved computing time for tarpit rules. I tarpit all WAN connections to unused TCP ports of the router.

/ip firewall raw add action=drop chain=prerouting comment="Drop WAN connections from 'intrusBL' blacklisted hosts <- Src. Address List: intrusBL" in-interface-list=WAN src-address-list=intrusBL

Hope that helps

boldsuck · Sun Feb 17, 2019 4:15 pm

@Dave

Wow, 14 year old woman is interested in coding! Very nice.

Crap. Stolen laptop is one of the worst cases.
I do not know exactly "New US government tax" but I think that's what Mr. Trump introduced. (He needs money for his wall. In Germany, we are glad that the wall is gone!)

Should we still run the reportStatus script? (That from #Post 37)

IntrusDave · Fri Mar 01, 2019 3:16 am

The ReportStatus is 100% optional. It's mostly for stats and being able to ty and tailor the lists to the bulk of the routers. Surprisingly, their are far more CloudCore routers running the blacklist now than the smaller units.

I wanted to thank those of you whom have reached out with support. Every little bit helps! It's been a few weeks and I'm starting to grow hair again... Just when I was getting used to being bald. It won't be long though, treatments start again in 6 weeks.

IntrusDave · Tue Apr 02, 2019 6:05 pm

Good Morning and happy April.

A few updates for you this morning.

Server is running stable with no issues other than the random LetsEncrypt cert error.
Once the list's income manages to cover it's costs, I'll be switching to Digicert for ssl certs.
I've begun work on adding IPv6 to the service, it's pretty easy on the router side, changes things quite a bit on the backend
For those following, the police found my notebook, though it's completely wrecked. they are keeping it as evidence. They didn't find any of my tools.
as for the service pricing, I'm still keeping it as a donation only service. Again, I don't want to bill for a service that I may not be around to keep running.

Rico40 · Tue Apr 02, 2019 9:12 pm

It can be paid by Patreon?

msatter · Tue Apr 02, 2019 11:18 pm

Humans can be truly awfull but using you undergoing your treatment to steal from you then there are no words to describe my feelings about that.

I am sorry to read that you are ill and that the outcome is uncertain. I wish all the strength to overcome this horrible time in your life.

IntrusDave · Wed Apr 03, 2019 1:10 am

It can be paid by Patreon?

Yes, somewhere up above, there is a Patreon link, as well as PayPal links.

IntrusDave · Wed Apr 03, 2019 1:11 am

Humans can be truly awfull but using you undergoing your treatment to steal from you then there are no words to describe my feelings about that.

I am sorry to read that you are ill and that the outcome is uncertain. I wish all the strength to overcome this horrible time in your life.

Thank you. No worries, I’ve survived everything else life has tossed my way. Even an earthquake that brought my apartment down on me.

If I could just keep my fingers and toes warm, I would be happy.

IntrusDave · Fri Apr 26, 2019 12:23 am

Just an update... I'm still here, and I have hair again.

Well, I have hair for a few weeks at least.

I'm still working on the IPv6 version of the list. My home ISP has finally managed to provide a stable v6 connection, so I'll be able to start testing.

boldsuck · Sun May 05, 2019 4:59 pm

I'm still working on the IPv6 version of the list. My home ISP has finally managed to provide a stable v6 connection, so I'll be able to start testing.

IPv6 Yeah! That's very good news.
Have meanwhile a few servers on the run. (Debian, Static IP & IPv6)
If you want to test something on it ...
can I give you access. I only need a pub ssh-key from you.

IntrusDave · Tue May 28, 2019 5:43 am

I regret to inform you all that I have shut down the servers.
With over 35,000 routers hitting the server every few hours, and only 10 supporters (totally $50/month), the expenses are not even remotely being covered. I'm putting out almost $500/month now just in bandwidth costs.

I'm moving the code to my in home server where it will just be supporting my personal units now.

it was a good run. I tried to keep it going, I thought that more people would be willing to help, but sadly not.

wanos · Tue May 28, 2019 2:02 pm

We truly appreciated all your efforts Dave. Well, too few of us apparently did. So many want something for nothing. Sucks.

Thank-you for that and hope all goes well for you and your family.

hhgttg42 · Tue May 28, 2019 4:41 pm

Sorry to hear that Dave. Please let us know if you ever change your mind!

Best of luck.

HZsolt · Tue May 28, 2019 8:08 pm

[DELETED]

IntrusDave · Tue May 28, 2019 8:25 pm

Dude, really? My lively-hood is going down the drain, and you pop up to help push the knife in. Nice.

Rico40 · Wed May 29, 2019 8:11 am

I resigned from patreon in favor of donate paypal.

Chupaka · Sat Jun 08, 2019 11:47 am

Dave, any chance to get the code open-sourced? I mean, maybe someone would like to donate servers/bandwidth instead of money...

pe1chl · Sat Jun 08, 2019 12:43 pm

I regret to inform you all that I have shut down the servers.
With over 35,000 routers hitting the server every few hours, and only 10 supporters (totally $50/month), the expenses are not even remotely being covered. I'm putting out almost $500/month now just in bandwidth costs.

I'm curious how much bandwidth your service consumes... there is another poster complaining about $500/mon bandwidth cost for his service, but frankly I cannot understand how this can happen.
For that $50/mon I would get about 10TB/mon of bandwidth at local cloud hosting companies, and with 35000 routers that would be 285MB per router per month.
When that isn't enough by a factor of 10 I would seriously consider revising the method for distributing the data... e.g. some form of incremental updating.

msatter · Sat Jun 08, 2019 1:41 pm

I helped with an earlier version and it is should be incremental and your get the changes you missed since the last sucessful update you had.

The sheer number of routers connecting still can give a heavy bandwith usage.

Dave is doing a great job despite his personal set backs.

viewtopic.php?t=98804

Rico40 · Sat Jun 08, 2019 8:19 pm

it's more about reducing the cost of the service.

pe1chl · Sat Jun 08, 2019 9:25 pm

Please show a calculation of how much bandwidth per month you need per router using the service, and how much the cost for different amounts of bandwidth per month is.

IntrusDave · Thu Jun 13, 2019 9:00 pm

Sorry, but I feel no need to disclose my stats and financial needs for a service that is free.
I can tell that you 4 servers, 120 honeypots, a CDN, storage and the bandwidth needed for all of it is quite a lot.
I wont be open sourcing the code either. it's 100% written by my with no use of any open source code.
It has a use to me still and I will be keeping it for myself.

suszi · Thu Jun 13, 2019 9:28 pm

the problem is, with RB, that ip firewall raw action=drop doent work with src-address-list=intrusBL
according to documentation:
address-list (string; Default: ) Name of the address list to be used. Applicable if action is add-dst-to-address-list or add-src-to-address-list
https://wiki.mikrotik.com/wiki/Manual:IP/Firewall/Raw

/ip firewall raw
add action=drop chain=prerouting comment="DROP intrusBL" src-address-list=intrusBL
add action=drop chain=prerouting comment="DROP intrusBL" dst-address-list=intrusBL

any way to use RAW? or just regular firewall rule?

suszi · Thu Jun 13, 2019 9:29 pm

I'm updating my second portscanners (TCP SYN) list manually - where can I send it, to be included ?
maybe there is a better way ?

msatter · Thu Jun 13, 2019 11:05 pm

The ADD in the rules is there to add the line to the RAW section in the firewall. After thst it not used anymore.

Dropping unwanted traffic is most efficient in RAW and so it won't reach connection tracking.

Chupaka · Fri Jun 14, 2019 12:38 am

the problem is, with RB, that ip firewall raw action=drop doent work with src-address-list=intrusBL
according to documentation:
address-list (string; Default: ) Name of the address list to be used. Applicable if action is add-dst-to-address-list or add-src-to-address-list
https://wiki.mikrotik.com/wiki/Manual:IP/Firewall/Raw
Code: Select all
/ip firewall raw
add action=drop chain=prerouting comment="DROP intrusBL" src-address-list=intrusBL
add action=drop chain=prerouting comment="DROP intrusBL" dst-address-list=intrusBL
any way to use RAW? or just regular firewall rule?

Address-list, src-address-list and dst-address-list are three different parameters. You're talking about one and look at the description of another.

boldsuck · Thu Jun 20, 2019 10:42 pm

I thought that more people would be willing to help, but sadly not.

me too.

I have now unsubscribed at Patreon. ;-(

Thanks for everything Dave. You and this thread inspired me to refine my own raw bruteforce & portscan rules.

I'm trying to do something similar. With fail2ban (0.10 has ipv6 support) My servers run as fail2ban reporting service anyway. For months there is already a UBNT ER-Pro 8 around here. On EdgeOS, packages from the debian archives can be installed. Fail2ban should then be easy to install on the router.