Hello everyone,

I am trying to edit my routers firewall address-list via the REST API.
So far I have managed to get everything running via PUT & DELETE but those methods are rather slow for adding and removing 180k entries.

Can anyone shine a light on how to use the POST command for that?
I am struggeling at Postman as either the router is not accepting it due to "invalid JSON" or "missing =list=".

Unfortunately I was only able to find how to add a single address at the forum, but not tons of them.

If there is an intend to add some entries and then after time remove them... You can set an expiration when adding the entry, that save having to delete them.

To use POST, it's still a two step operations. First, you still need to get the list of id from GET (*or POST .../address-list/print). You can optimize this part by adding a ".proplist" in JSON request with only ".id". And, then to remove them, you provide the list of .id in the POST .../remove. NOW... the tricky part is POST wants a comma-separated list of .id in its JSON — not a JSON array of .id. And it's this latter step to get a "CSV" to provide as String type to POST's .id field.

So I suspect you're trying to provide an "array of .id" in your call to POST /rest/ip/firewall/address-list/remove — when it wants a comma-separated string of .id. The reason for this is POST is more a "wrapper" over the lower-level API & that deals in strings. Now it likely should do that for you IMO, but does not.

Here roughly what it should look like: Note: .id is a string with comma separated values.

I actually cannot say if providing all .id in a POST .../remove is that much quicker than a loop over the simple DELETE. But there is also another consideration when using POST to remove — it is NOT transactional. So if you provide an already deleted .id or bad value, it will stop processing the rest of the list, and return a 404 at that point. Anything already delete is deleted, anything after the missing .id would not be deleted.

I think you're likely better off using the built in TTL mechanism vs ANY scripting — if you can. Maybe one POST with a 180K .id is quicker, but even that I'm not necessarily sure & provided a string list of 180K records may have different problems for REST API. So.... if you need remove, a loop over DELETE give you feedback if anything was not removed, even if it takes more time.

And to be clear, .query does NOT work on a /remove. So this DOES NOT WORK:
curl ... https://.../ip/firewall/address-list/remove —json '{".query": "list=mylistname"}'

But that's the command you may be trying to find. And it doesn't not work like that.

Again, POST wraps API, this part is referenced in https://help.mikrotik.com/docs/spaces/R ... -Queryword :

Currently, only the print command handles query words.

Thanks @Amm0 for your suggestions.
In general deleting works pretty okay for me.

My intention was more into uploading a list to the router via the API and wait until he finished with the logic.

So far I was able to to get it running at a proper speed but this comes at some costs:
Running a list with native ROS script language costs around 30 % CPU on my CCR2004.
Compared to that with a more or less equal speed at adding entries via the REST API with PUT it maxes out to ~90 % CPU utilization.

Not sure if this is OK or if I am torturing my device.

Running a list with native ROS script language costs around 30 % CPU on my CCR2004.
Compared to that with a more or less equal speed at adding entries via the REST API with PUT it maxes out to ~90 % CPU utilization.

What specific commands are you running? In general you can model them with REST POST.

The main difference is when using REST is each request is a new connection, so >100K you hitting router with that many TCP connections, and it's having to do a lot of more string parsing than CLI would.

The other approach be to put your add/remove commands into is a /system/script via , then run it. So you could use REST to PUT a script, then POST /system/script/run as alternative.

In general my Python script does three different things:

1) Getting the the current address-list from router via GET https://router/rest/ip/firewall/address ... =blocklist
Parsing the JSON and comparing the current addresses list with a new address list.
Cost ~30 % CPU

Every new IP is being written to an array and every IP which is no longer on the new list (= to be deleted) is being written to another array.

2) Looping through the array to add new IPs:
POST https://router/rest/ip/firewall/address-list/add with a JSON body including {"address": ip, "list": list_name}.
So one request per new IP.
Cost ~90 % CPU

3) Loopin through the other array to delete IPs:
DELETE https://router/rest/ip/firewall/address-list/{.id}
Cost ~90 % CPU

Initial I tested a script found on Github (https://github.com/multiduplikator/mikrotik_blocklist) which also does the job but only on the router itself via the scripting language.
It loads all IPs into a global array and comparse it locally on ROS. Then it adds or deletes IPs.
This script consumes around 30 % CPU but takes sometimes long.

My intention was to do the comparing on a dedicated device (VM, Raspberry PI, WSL,....) to do the heavy job and then simple tells ROS "add this IPs and delete those".

Well at the end of the day, processing a lot of records is going to use the CPU. The only other thing to do if you want to "optimize" is use the native API. Since you're using Python, Mikrotik has a "library" (well a class, with an example main) here: https://help.mikrotik.com/docs/spaces/R ... n3+Example.

I don't necessarily think you're going to see a night-and-day difference... But likely at least some over REST since API will skip the REST JSON to the native API =list=mylist string, since you're Python will do that. Also with API the 100K requests can be on same connection/socket, so you'll save open/closing 100K sockets by using API, over REST. More work however.

Thanks but I wanted to explicit use the REST API as GET, POST, and so on are common operations and are easier to understand compared to a dedicated API.
I am just curious why there is such a big difference between local script execution with Mikrotiks scripting language vs a REST API (~ 60 % CPU difference).

Is there any place where I may share my scripts so other may or may not use them?

Hello everyone,

I am trying to edit my routers firewall address-list via the REST API.
So far I have managed to get everything running via PUT & DELETE but those methods are rather slow for adding and removing 180k entries.

Can anyone shine a light on how to use the POST command for that?
I am struggeling at Postman as either the router is not accepting it due to "invalid JSON" or "missing =list=".

Unfortunately I was only able to find how to add a single address at the forum, but not tons of them.

I have no expertise to help solve your problem, but I am indeed fascinated the project.

Would you mind elaborating on the situation or condition that gives rise to the need to make 180,000 firewall address-list changes?

With these large lists, I can only guess at the why script would be fast or slow. But with REST API each call has to both get authenticated and JSON is even more string parsing. That's overhead that both API and scripting don't have, since auth is done once.

Some speculation... With some RSC script with 100K list of /ip/firewall/address-list/add's that runs from /system/script will be parsed once from script, than executed, with auth being aligning the UNIX user id. If you want to get really dorky, scripting, config and CLI all all get converted some "s-expression like thing" that scripting typeof call "(code)":
So I view the (evl (())) things as the lowest level before any user thing hits RouterOS C code. And when you "run" a script it's first converted into a very long (evl (()())) as the first phase, and that's whole thing is handed off to just run. NOW... when you call REST... track your TCP, check /users each time, convert the JSON into the API "string sentences"....and API likely has same/similar string process to get a similar IL like (evl ()()). Basically with REST, you'll add 100K auth calls, and 100K*2 (200K) extra string transformations when starting at REST.. API get you to 1 auth call, and 100K extra string transformation. CLI would also have a similar amount string transformation (once, so 100K), except it's only parsing it once, not per request (or API sentence) so there is no thread context switches when using /system/script.

Now... the corollary here... you may not want to drive the CPU to 100%... so may want to throttle anyway to avoid the CPU hit that could impact routing.

In terms of posting your code, feel free to do it here. Or, just start a new topic like "Scripting address-list with Python with example", or whatever, if you want a more organized presentation of it.

There is an aggregated list of IPs and IP ranges which may be blocked due to "bad" traffic: https://github.com/stamparm/ipsum
My intention was to import the list and keep it up to date in an easy manner

There is an aggregated list of IPs and IP ranges which may be blocked due to "bad" traffic: https://github.com/stamparm/ipsum
My intention was to import the list and keep it up to date in an easy manner

Thank you.

What prompted you to make banning these bad traffic IPs a priority?

My intention is/was to minimize attack vectors - e.g. known bots which are trying to install a backdoor on your device.

@An5teifo Generally calling API requests as replacement for ROS script which performs many config updates is bad idea, especially using many API requests in loops. Difference in CPU load is because there is much more overhead on API request vs local command, CPU is used for firewall, networking, API service processing - parsing JSON metadata, executing local command, process it and return response.
In you case you can create hybrid solution like:

1. fetch current address list with API request and compare it against new list
2. generate ROS script file with commands for adding and deleting IPs into address list by difference from 1st step
3. upload script file into ROS (for eg. using SFTP) and perform POST /import API request for uploaded script. It is also possible to execute ROS script automatically after upload without calling API, then script file must be named in format "[someting].auto.rsc"

Hello optio,

thanks for your suggestion as this is also my conclusion after a weekend scripting
I posted my scripts and the logic here: viewtopic.php?t=215239

My current workflow is to download lists on a device, aggregate them into a single big file which is being a RouterOS array and upload it to my Git repo.
On my routers I am using an adopted script from here https://github.com/multiduplikator/mikrotik_blocklist (2c - fancy list stuff).

This works so far okay.
My original intention was to do the heavy job of comparing IPs from two different lists rather on a utility device instead of my router.

My original intention was to do the heavy job of comparing IPs from two different lists rather on a utility device instead of my router.

I understand, hybrid solution will work in this case, comparing lists and generating new one can be done outside ROS (CPU load will not be on ROS device), but importing it should be done with ROS script with only add and remove address list commands in it to avoid heavy CPU load on ROS device by performing many API requests.

So simple add and remove commands would be the most resource saving method to work it?

Yes, because there will no be additional logic in ROS script for processing lists to create aggregated list, just commands for adding / removing address list items.

Thanks, I will give that a try!

@optio: This solutions is the most performant one!
My "mikrotik util" VM will download the existing list via REST API from the device, parse the IP adresses & ranges, compares it with my new list and simply create an add or remove command.
The command list is being pushed to my git repository from where my router will download and execute it.

Back to the first post as I am just curious:

1. How many routers do you have to update?
2. How much time takes to update the fastest one?

For me the best solution would be the iBGP used to distribute blackhole list of forbidden addresses prepared on a local device.
One iBGP router rules them all

Currently I have two routers to manage.
The fastest import took around 3 minutes.

iBGP is a nice solution - didn't thought about it as I was too focused on scripting.
I guess it would make sense to import the list on the fastest device and the share it via iBGP with the other non so performant devices, right?

Yes, exactly that way.

Yes, exactly that way.

As I haven't created such blackhole routes so far, can you shine some light on how to create them?
I know how to do iBGP in general but not how to tell the receiving router to block them.

I have now tried to distribute the routes via iBGP but it seems that BGP is somehow broken on 7.18.1?!
Both routers are established but the address-list is not being distributed?!

There are basically 2 ways to use BGP to mitigate DDOS:

• Flowspec, which I dont think Mikrotik implements
  BGP black hole community and RTBH

Setting up Remotely Triggered Black Hole (TRBH) means injecting routes for all the addresses you want to black hole with the “blackhole” community set, then null routing them so that they dont have a route for them. You then have to turn on Reverse Path Forwarding (RPF) so that it will drop all packets it doesnt have routes for.

It can be done, but is a bit tricky. RPF also can bite in ways you dont expect.

Here’s a good article on how to do it with Cisco routers. I’ll leave it up to you all to translate.

https://learningnetwork.cisco.com/s/article/remotely-triggered-black-hole-filtering-dos-mitigation

I already have my own list which I regular update.
The thing which I found is that the IPs that are available at /ip/firewall/address-list needs also to be available at /ip/route.

So I guess I need to rewrite my script to add/remove the IPs to the firewall address-list as well as the IP routes - right?


How would I "block" such IP addresses after that?

Once there is no route for an address (or a null route) you can drop all packets from that address by turning on RPF (which Mikrotik calls /ip/settings/rp-filter).

Try “rp-filter loose” first, strict really tends to break things.

All RPF is tricky though. It tends to drop packets you wouldn’t expect.

rp-filter (loose | no | strict; Default: no) Disables or enables source validation.
no - No source validation.
strict - Strict mode as defined in RFC3704 Strict Reverse Path. Each incoming packet is tested against the FIB and if the interface is not the best reverse path the packet check will fail. By default failed packets are discarded.
loose - Loose mode as defined in RFC3704 Loose Reverse Path. Each incoming packet's source address is also tested against the FIB and if the source address is not reachable via any interface the packet check will fail.
The current recommended practice in RFC3704 is to enable strict mode to prevent IP spoofing from DDoS attacks. If using asymmetric routing or other complicated routing or VRRP, then the loose mode is recommended.
Warning: strict mode does not work with routing tables

Okay thanks for the info but how can I keep the address-list and the routes in sync?

Okay, I finally found a solution for me which works like a charme:

On my "Mikrotik Util" VM I download various bad IP & network lists and aggregate them.
I then announce those routes via Exabgp as "blackhole" community and peer with the nearest Mikrotik router.
This router then redistribute the BGP routes via iBGP and iBGP RR to my other Mikrotik routers.

Sounds like a great solution.

Did you have to somehow point these to a null route, and use RPF to discard? I would be interested to see how that configuration looks in. Mikrotik.

I am using a BGP input filter with following settings: This imports any routes with this community as blackholed which I guessed would be the same logic as a null route.
RPF is set to loose altought I did not see any difference.

Nice to hear that you successed implementing iBGP solution.