Problem with StevenBlack list in DNS Adlist

ianiovski · Fri Jul 12, 2024 8:13 pm

Hello, are you having problem adding url list of StevenBlack ? When I add it to Adlist it doesn't recognize any results and matches. I think it's because the sheet starts with this:

# Title: StevenBlack/hosts
#
# This hosts file is a merged collection of hosts from reputable sources,
# with a dash of crowd sourcing via GitHub
#
# Date: 11 July 2024 19:19:26 (UTC)
# Number of unique domains: 155,096
#
# Fetch the latest version of this file: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
# Project home page: https://github.com/StevenBlack/hosts
# Project releases: https://github.com/StevenBlack/hosts/releases
#
# ================================================ ==============

127.0.0.1 localhost
127.0.0.1 localhost.localdomain
127.0.0.1 local
255.255.255.255 broadcasthost
::1 localhost
::1 ip6-localhost
::1 ip6-loopback
fe80::1%lo0 localhost
ff00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
0.0.0.0 0.0.0.0

# Custom host records are listed here.


# End of custom host records.
# Start StevenBlack

#=====================================
# Title: Hosts contributed by Steven Black
# http://stevenblack.com

0.0.0.0 ck.getcookiestxt.com
0.0.0.0 eu1.clevertap-prod.com
0.0.0.0 wizhumpgyros.com
0.0.0.0 coccyxwickimp.com
0.0.0.0 webmail-who-int.000webhostapp.com
...........................

If I make my own file and copy only the useful information from its list things work.

0.0.0.0 XXXXXXXXXXXX

Because in the video tutorial things work out, but I don't know then if the leaf looked like this .

https://help.mikrotik.com/docs/display/ROS/DNS

Here is my situation:

 url="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts" ssl-verify=no 
     match-count=0 name-count=0

anav · Sat Jul 13, 2024 2:09 am

Perhaps should ask Steven??

ianiovski · Sat Jul 13, 2024 9:02 am

How do I contact him?

loloski · Sat Jul 13, 2024 9:44 am

I think the problem is you are using this on lower end device that's why the list won't populated try this on RB5009 it should work

jfim88 · Sat Jul 13, 2024 9:55 am

I tried it on my ax2 and it worked.

Anyway I use NextDNS.

ianiovski · Sat Jul 13, 2024 10:21 am

You're right, I tried it on the RB750Gr3, but I also have a HAP AX³ and it worked, got the list without problems. Well, I will use it for DNS. Thank you very much for the help.

anav · Sat Jul 13, 2024 1:33 pm

This works for smaller devices, as allowances are made for the limitations. Also describes how its done.
https://itexpertoncall.com/additional_info/moabpre.html

quote: "MOAB has two tracks, one for MikroTik Routers like the hEX, hAP ax 2, hAP ax 3, the Audience - tracking between 5K and 16K ipset entries - MikroTik Router models like the CHR, RB3011, RB4011, RB5009, RB1100 and all CCR - tracking between 35K and 60K ipset entries. Once we know which MikroTik Router model you have we will decide if your model qualifies and which track to put you on. Both tracks cover over 600 million IP addresses of known perpetrators.

For MikroTik Routers like the CHR, RB3011, RB4011, RB5009, RB1100 and all CCR models the maximum download file size is 1 MB or less - 3 times daily.

For MikroTik Routers like the hEX, hAP ax 2, hAP ax 3, the Audience - the maximum download file size is 0.5 MB or less - 3 times daily."

netbus · Sat Jul 13, 2024 11:13 pm

I have a similar problem and no matches on a hap ax3

> ip/dns/adlist/print
Flags: X - disabled
0 url="https://raw.githubusercontent.com/Steve ... ster/hosts" ssl-verify=no match-count=0 name-count=100908

Any idea?

rdrafal · Sun Jul 14, 2024 12:25 pm

Hi.

I have the same problem on CCR1009.

ianiovski · Sun Jul 14, 2024 12:42 pm

You have to set your DHCP server or the hosts themselves manually, the DNS server to be the IP address of the Routerboard on which you use Adlist, then you will start to have reporting. The interesting thing is that for me it gives me 154749 on hAP AX3, and on my virtual machine I installed RouterOS with x86 architecture and there it recognizes them as 17266 entries, and for you it gives them 100908. It's different for everyone, I can't explain it.
hAP AX3

Flags: X - disabled 
 0   url="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts" 
     ssl-verify=no match-count=2368 name-count=154749

Virtual RouterOS x86

Flags: X - disabled 
 0   url="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts" 
     ssl-verify=no match-count=0 name-count=17266

patrikg · Sun Jul 14, 2024 2:52 pm

Lets see:

curl -s https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts | grep -v '#' | sed '/^$/d' | wc -l
154749

It's not correct because it removes the lines with hash tags even if the hash tag come after correct value.
Like this lines.

0.0.0.0 iesnare.com # See http://www.codingthewheel.com/archives/online-gambling-privacy-iesnare
0.0.0.0 www.iesnare.com # See http://www.codingthewheel.com/archives/online-gambling-privacy-iesnare

So these lists have to be massaged a little bit before going into the mikrotik device.

mkx · Sun Jul 14, 2024 4:48 pm

Lets see:

Let me fix it for you:

curl -s https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts | sed -e 's/#.*$//' -e '/^$/d' | wc -l
155122

patrikg · Sun Jul 14, 2024 11:39 pm

@mkx Thx,
I am not the best regex script kiddies

But if Mikrotik device count is false, it's also removes all the lines with hash tags.
But i not getting it with this low value 17266. Maybe it's not getting the hole file.

mkx · Mon Jul 15, 2024 8:02 am

I've never used adlist feature, but I'd expect it to emit some kind of diagnostic messages upon importing the list. At least, say, number of items successfully imported in info channel and any crucial problem in error channel (e.g. if import breaks in the mid of file due to lack of memory or some such). Without it it's impossible to tell why some hosts have lower number of entries than others (and I'd assume that file parser acts the same on all MT platforms so the difference should then be tied to individual device state differences).

BTW, I don't expect comments to be the problem, at least config file parser is well able to ignore comments on the code lines.

Mon Jul 15, 2024 8:35 am

FIRST increase the DNS cache value a lot. Only then enable it. If you see no matches, 99% it's because of that.

djferdinad · Mon Jul 22, 2024 4:57 am

I would like to give an update about this:

I have an Hex S, indeed it does not allow big files (5MB) when using an URL.

But, for small files it DOES download and functionally works from a local server using the URL.

Same file, on GitHub. DOES NOT work.

codelogic · Mon Jul 22, 2024 7:24 am

The problem is poor/stupid programmer or design decision to download adlist files first to device storage instead of memory.... Since the HexS only has 16 MB of storage, and *at best* only a few MB free with ROS 7, you see the problem.

I use this script I made, which downloads the target file into *memory* and not the permanent storage:

:delay 10s
# Check cache size and adjust if not large enough
:if ([ip/dns/get cache-size]!=32768) do={/ip/dns/set cache-size=32768}
# Remove all current adlists
/ip/dns/adlist remove [find]
# Fetch new adlist source file(s)
/tool/fetch url=https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts output=file dst-path=sblack-unified-adlist
# Short delay to ensure file is created in RAM
:delay 1s
# Create new adlist from file
/ip/dns/adlist/add file=sblack-unified-adlist
# Remove temp file from RAM
/file remove sblack-unified-adlist

And it works just fine. That list on my HexS shows:

[admin@RB760iGS] > /ip/dns/print
                   cache-size: 32768KiB
                cache-max-ttl: 1w
      address-list-extra-time: 0s
                          vrf: main
                   cache-used: 17819KiB
[admin@RB760iGS] > /ip/dns/adlist/print
Flags: X - disabled
 0   file=sblack-unified-adlist match-count=0 name-count=158886

Have fun.

Edit: This also gives added bonus of not needlessly wearing out your flash storage by downloading adlist over and over again to it…

BillyVan · Mon Jul 22, 2024 11:20 am

# Create new adlist from file
/ip/dns/adlist/add file=sblack-unified-adlist

Maybe change it to

# Create new adlist from file
/ip/dns/adlist/add file=sblack-unified-adlist ssl-verify=no

Mikrotik says no about ssl here
https://help.mikrotik.com/docs/display/ROS/DNS

massinia · Mon Jul 22, 2024 11:44 am

@codelogic
Thank you!
Finally I can also use it with hAP AC2, manually updating the host list every time was a nightmare...
I created a scheduler with your script and it works perfectly.

codelogic · Mon Jul 22, 2024 5:47 pm

@BillyVan
Since we're loading the adlist from a file in memory we've downloaded, ssl-verify has no bearing or effect.

@Massinia
Thanks for confirming. I've had it running on my HexS for a couple of weeks without issue as well.

Perhaps my post should be marked as real solution here?

anav · Mon Jul 22, 2024 6:32 pm

SOLUTION, I almost sprayed my coffee on the screen

Try WORK AROUND.

A solution is for Mikrotik to recognize this issue and to make a decision.

a. Default adlist to Memory (volatile?)
b. Default adlist to storgage device (non-volatile?)

Assuming your work-around is viable
c. ADD USER SELECTABLE option when executing adlist to choose which is the preferred method.
d. Something else, that is way smarter than what I came up with in a nano-second.

@Normis response ???

Valerio5000 · Fri Aug 09, 2024 1:24 pm

The problem is poor/stupid programmer or design decision to download adlist files first to device storage instead of memory.... Since the HexS only has 16 MB of storage, and *at best* only a few MB free with ROS 7, you see the problem.

I use this script I made, which downloads the target file into *memory* and not the permanent storage:
Code: Select all
:delay 10s
# Check cache size and adjust if not large enough
:if ([ip/dns/get cache-size]!=32768) do={/ip/dns/set cache-size=32768}
# Remove all current adlists
/ip/dns/adlist remove [find]
# Fetch new adlist source file(s)
/tool/fetch url=https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts output=file dst-path=sblack-unified-adlist
# Short delay to ensure file is created in RAM
:delay 1s
# Create new adlist from file
/ip/dns/adlist/add file=sblack-unified-adlist
# Remove temp file from RAM
/file remove sblack-unified-adlist
And it works just fine. That list on my HexS shows:
Code: Select all
[admin@RB760iGS] > /ip/dns/print
                   cache-size: 32768KiB
                cache-max-ttl: 1w
      address-list-extra-time: 0s
                          vrf: main
                   cache-used: 17819KiB
[admin@RB760iGS] > /ip/dns/adlist/print
Flags: X - disabled
 0   file=sblack-unified-adlist match-count=0 name-count=158886
Have fun.

Edit: This also gives added bonus of not needlessly wearing out your flash storage by downloading adlist over and over again to it…

My compliments! On my AC2 it works perfectly, just one thing_ to update the file automatically I imagine I have to create a scheduler in ROS to run the script every certain amount of time (2 times a day?)?

beadon · Wed Aug 21, 2024 8:38 am

Just confirmed the problem with the support team. They report that the adblock feature will get some attention on this issue in the near future.

alphaQ · Wed Sep 04, 2024 2:16 pm

The problem is poor/stupid programmer or design decision to download adlist files first to device storage instead of memory.... Since the HexS only has 16 MB of storage, and *at best* only a few MB free with ROS 7, you see the problem.

I use this script I made, which downloads the target file into *memory* and not the permanent storage:
Code: Select all
:delay 10s
# Check cache size and adjust if not large enough
:if ([ip/dns/get cache-size]!=32768) do={/ip/dns/set cache-size=32768}
# Remove all current adlists
/ip/dns/adlist remove [find]
# Fetch new adlist source file(s)
/tool/fetch url=https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts output=file dst-path=sblack-unified-adlist
# Short delay to ensure file is created in RAM
:delay 1s
# Create new adlist from file
/ip/dns/adlist/add file=sblack-unified-adlist ssl-verify=no
# Remove temp file from RAM
/file remove sblack-unified-adlist
And it works just fine. That list on my HexS shows:
Code: Select all
[admin@RB760iGS] > /ip/dns/print
                   cache-size: 32768KiB
                cache-max-ttl: 1w
      address-list-extra-time: 0s
                          vrf: main
                   cache-used: 17819KiB
[admin@RB760iGS] > /ip/dns/adlist/print
Flags: X - disabled
 0   file=sblack-unified-adlist match-count=0 name-count=158886
Have fun.

Edit: This also gives added bonus of not needlessly wearing out your flash storage by downloading adlist over and over again to it…

THANKS,
it worked on low-end devices

kwade · Wed Sep 25, 2024 6:09 pm

I agree that THIS should NOT be marked as the solution:

I think the problem is you are using this on lower end device that's why the list won't populated try this on RB5009 it should work

The OP seems to have disappeared, but the reply that should be marked as the solution is the script by codelogic. Thank you for a very functional workaround.

voip · Tue Oct 01, 2024 4:43 pm

@codelogic Thank you very much ;)

My hEX S | RB760iGS got swamped with ERROR messages in Log file:

[adlist] no space to store a file

cache full, not storing [ignoring repeated messages]

Shame that mt.lv manual pages for the DNS
do not provide solution, but rather ONLY give an advise:
"Adlist is stored on device's internal memory. Ensure that there is enough free space to save the desired adlist."

Also had to manually enter the commands one by one as it always finished with error on ROS 7.16
if I paste the whole things into the bash..........

# Check cache size and adjust if not large enough

:if ([ip/dns/get cache-size]!=32768) do={/ip/dns/set cache-size=32768}

# Remove all current adlists

/ip/dns/adlist remove [find]

# Fetch new adlist source file(s)

/tool/fetch url="https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts" output=file dst-path=sblack-unified-adlist

# Create new adlist from file

/ip/dns/adlist/add file=sblack-unified-adlist

# Remove temp file from RAM

/file remove sblack-unified-adlist

# Check the "DNS" Settings

/ip/dns/print

# Check the "AdList"

/ip/dns/adlist/print

darkmanlv · Sat Oct 05, 2024 4:27 am

tried today this script on hex s 7.16, adlist loading to memory, works perfect! thanks!

Silar · Sun Jan 19, 2025 3:30 pm

I believe this has now been fixed with Router OS 7.17 (2025-Jan-16 10:19):
"*) adlist - optimized import on system with low disk space;"

UkRainUa · Thu Jan 23, 2025 11:54 am

Hi! Saving lists to USB drive will save space on the router and the resource of the router's storage chip, will allow you to load lists immediately after rebooting the router:

:if [/ip/dns/adlist find] do={/ip/dns/adlist reload};

If you have a USB in your router and a flash drive, add this script to the scheduler (once a day), you will always have an updated file on the USB drive::

# MikroTik Adlist USB by UkRainUa (without external functions)
# hardcoded: url, usb1-part1
# hAP ac2 7.16, 7.17 - test ok

:local scriptName "adlist";
/log debug "$scriptName started";
# update adlist function
:local UpdateAdlist do={
	:local scriptName "$1->UpdateAdlist->$description";
	:do {
		/log debug "$scriptName started";
		# fetch new adlist source file
		/tool/fetch mode=https check-certificate=yes url=$url output=file dst-path=$description as-value;
		# short delay to ensure file is created
		:delay 1s
		# check adlist
		:if ([/file/get [find name="$description"] value-name=size] > 1000000) do={
			:if [/ip/dns/adlist find where file=$description] do={
				/ip/dns/adlist reload;
				/log debug "$scriptName adlist reloaded";
			} else={
				# Create new adlist from file
				/ip/dns/adlist add file=$description ssl-verify=no;
				/log debug "$scriptName adlist added";
			};
		} else={
			/ip/dns/adlist remove [find where file=$description];
			/file/remove [find name=$description];
			/log warning "$scriptName runtime error: file not created or corrupted, adlist removed";
		};
		/log debug "$scriptName executed";
	} on-error={
		/log warning "$scriptName runtime error";
	};	
};
# check cache size and adjust if not large enough
:if ([ip/dns/get cache-size]!=32768) do={/ip/dns/set cache-size=32768};
# StevenBlackFGP: Unified hosts + fakenews + gambling + porn
:do {
	# run adlist update function
	$UpdateAdlist $scriptName url=https://raw.githubusercontent.com/StevenBlack/hosts/master/alternates/fakenews-gambling-porn/hosts description=usb1-part1/StevenBlackFGP;
} on-error={
	/log warning "$scriptName StevenBlackFGP update failed";
};
# StevenBlack: Unified hosts (adware + malware)
# :do {
	# # run adlist update function
	# $UpdateAdlist $scriptName url=https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts description=usb1-part1/StevenBlack;
# } on-error={
	# /log warning "$scriptName StevenBlack update failed;
# };
# HaGeZiLight
:do {
	# run adlist update function
	$UpdateAdlist $scriptName url=https://raw.githubusercontent.com/hagezi/dns-blocklists/refs/heads/main/hosts/light.txt description=usb1-part1/HaGeZiLight;
} on-error={
	/log warning "$scriptName HaGeZiLight update failed";
};
/log debug "$scriptName executed";

If fetch ends with an error, the script does nothing but report an error, the file will be downloaded and reloaded next time. A script is possible with checking the message in the log about the success of the fetch execution, it will be a little more complicated, it is wise to use a universal external log analysis function for this.

Problem with StevenBlack list in DNS Adlist [SOLVED]

Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist [SOLVED]

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist

Re: Problem with StevenBlack list in DNS Adlist