find addresses with same octets

Sertik · Fri Jul 28, 2023 12:44 pm

You need to get a list of IP addresses from the address list in which the first three octets of the IP addresses are the same, but only the last one is different and there are more than four such addresses.
I understand that you need to use posix regex, but I'm not on friendly terms with this at all. I ask for the help of our dear Rex.

msatter · Fri Jul 28, 2023 2:43 pm

Ok. ;-)

Amm0 · Fri Jul 28, 2023 6:02 pm

Is there any reason the "in" operator wouldn't work? The "in" checks if a type=ip/ip6/ip-prefex is included in a type=ip-prefix. Depending on what you're doing, might be easier then mucking with regex to parse IP addresses.

e.g.

:put (1.1.1.1 in 1.0.0.0/8)
#true
:put (1.1.1.1 in 8.0.0.0/8)
#false

rextended · Sat Jul 29, 2023 8:14 pm

The OP question is not really clear to me...

Ok.

@msatter is right, ask everyone, this is not my personal forum...

@Amm0 give you the soluction:

{
/ip firewall address-list
:local pool [find where (address in 1.2.3.0/24) or (address in 4.5.6.0/24) or (address in 7.8.9.0/24)]
}

Sertik · Mon Jul 31, 2023 9:17 am

You need to get a list of IP addresses from the address list myAddr, where the first three octets of the IP address are the same, but only the last one differs.

for example: 192.168.88.3, 192.168.88.67; 192.168.88.12; 192.168.0.11; 192.168.67.5; 192.168.88.34

From these, addresses with the first three single octets should be selected, that is, in this case, from the 88/0 network

Amm0 · Mon Jul 31, 2023 9:59 am

for example: 192.168.88.3, 192.168.88.67; 192.168.88.12; 192.168.0.11; 192.168.67.5; 192.168.88.34

{
:local listname octettest3
/ip/firewall/address-list
:foreach i in={192.168.88.3;192.168.88.67;192.168.88.12;192.168.0.11;192.168.67.5;192.168.88.34} do={add list=$listname address=$i}  
:foreach a in=[find (address in 192.168.88.0/24) and (list=$listname)] do={:put [get $a address]}
remove [find list=$listname]
}

Which outputs the items in bold:
192.168.88.3
192.168.88.67
192.168.88.12
192.168.88.34

Sertik · Mon Jul 31, 2023 10:12 am

Dear Amm0. If it were that easy, I wouldn't be asking. Addresses and subnets are not known in advance. Dear Amm0. If it were that easy, I wouldn't be asking. Addresses and subnets are not known in advance. This address list was given as an example

Amm0 · Mon Jul 31, 2023 10:21 am

Dear Amm0. If it were that easy, I wouldn't be asking. Addresses and subnets are not known in advance. Dear Amm0. If it were that easy, I wouldn't be asking. Addresses and subnets are not known in advance. This address list was given as an example

I guess I'm not understanding. If there is no address nor subnet... What are you trying to match? I just populated a new list to verify/show.

If the address-list was from DNS, it still search the dynamic entires with IP. But yes you'd have to know the subnet you were looking for.

Sertik · Mon Jul 31, 2023 10:27 am

I'll try to explain again...
Let's say we have a certain address list, to which arbitrary addresses are added by some kind of firewall rule. Of course, we do not know these addresses in advance. Among them, there may be the same and similar. We need to catch addresses in this list with the same first three octets, for example 45.140.168.XXX. And detect the moment when there are more identical ones, for example 5 (that is, someone knocked from the same subnet more than 5 times).

rextended · Mon Jul 31, 2023 1:04 pm

The most important thing is to explain yourself well, not only in words, but with concrete examples of how it is before and how you want it afterwards.

What I understand is that you want an IP aggregator so that many IP's of the same subnet are represented only once, to reduce the number of items in the firewall.

This requires enormous computing power each time the list is processed,
which at which point you better do things with common sense: close open world services, as it should be,
and drop-all-at-the-end that's more than enough.

Sertik · Mon Jul 31, 2023 1:21 pm

I don't know what this is for, one user from the Russian-speaking forum needs it. I don't care what he wants to do with it. God bless them with computing power. We need a script with probably a posix regexp template that allows you to select addresses with the same first three octets.

rextended · Mon Jul 31, 2023 1:55 pm

We are always there, this magical RegEx, what is it supposed to do?
Automatically select the IPs and divide them by subnet, and then merge them back into /24?
As I said: concrete example, not novels.

It should probably be taken one IP at a time in the list, converted to x.x.x.0/24, and looked for how many other x.x.x.y addresses exist.
Above a certain threshold, add x.x.x.0/24 to a temporary address list, if it doesn't already exist on that list, and so on...

Sertik · Mon Jul 31, 2023 3:24 pm

Yes, probably so. I was thinking how can I use posix patterns to compare addresses...
And yes, probably, as you said, it will heavily load the processor

rextended · Mon Jul 31, 2023 5:16 pm

Posix? Not needed at all the RegEx...

This example display only text, do not alter nothing, and display the network only if more than one occurrance exist of x.x.x.0/24

example code

/ip firewall address-list {
    :local ip2net do={:return [[:parse ":return $($1 & 255.255.255.0)/$2"]]}
    :local addr    0.0.0.0
    :local net     0.0.0.0
    :foreach item in=[find where ([:typeof [:toip $address]]="ip")] do={
        :set addr [get $item address]
        :set net  [$ip2net $addr 24]
        :if ([:len [find where $address in $net]] > 1) do={
            :put "\r\nSearch results for $net"; print where $address in $net
        }
    }
}

rextended · Mon Jul 31, 2023 6:07 pm

I need help writing an efficient algorithm to convert (for example) this:

start block 1 code

192.168.88.0
192.168.88.1
192.168.88.2
192.168.88.3
192.168.88.4
192.168.88.5
192.168.88.6
192.168.88.7
192.168.88.8
192.168.88.9
192.168.88.10
192.168.88.11
192.168.88.12
192.168.88.13
192.168.88.14
192.168.88.15
192.168.88.16
192.168.88.17
192.168.88.18
192.168.88.19
192.168.88.20
192.168.88.21
192.168.88.22
192.168.88.23
192.168.88.24
192.168.88.25
192.168.88.26
192.168.88.27
192.168.88.28
192.168.88.29
192.168.88.30
192.168.88.31

To this:

wanted end result 1 code

192.168.88.0/27

Using only RouterOS functions.

Or for example, from this:

start block 2 code

192.168.88.0
192.168.88.1
192.168.88.2
192.168.88.3
192.168.88.5
192.168.88.6
192.168.88.7
192.168.88.8
192.168.88.9
192.168.88.10
192.168.88.11
192.168.88.12
192.168.88.13
192.168.88.14
192.168.88.15
192.168.88.16
192.168.88.17
192.168.88.19
192.168.88.20
192.168.88.21
192.168.88.26
192.168.88.27
192.168.88.28
192.168.88.29
192.168.88.30
192.168.88.31

To this:

wanted end result2 code

192.168.88.0/30 (4 IPs)
192.168.88.5/32 (single IP, not aggregable)
192.168.88.6/31 (2 IPs)
192.168.88.8/29 (8 IPs)
192.168.88.16/31 (2 IPs)
192.168.88.19/32 (single IP, not aggregable)
192.168.88.20/31 (2 IPs)
192.168.88.26/31 (2 IPs)
192.168.88.28/30 (4 IPs)

This can be used, for example, for aggregate routes or IPs inside address list.

For example, as first step, try to convert all /32 to /31, like

What is the next IP for the /31 of 192.168.88.0/32? Is 192.168.88.1/32.
192.168.88.1 is on the list?
Yes, so delete 192.168.88.0/32 and 192.168.88.1/32 and add 192.168.88.0/31
...
Do all this for every /32 left,
And at the end are present only /32 and /31.

Next step, ignoring the /32, is to join the /31 if are present the next /31 and so on, until all the possibilities from 31 to /8 (or bettere less, more is excessive)
are tryed for compact the IPs....

So, from this

start example 26 lines code

192.168.88.0
192.168.88.1
192.168.88.2
192.168.88.3
192.168.88.5
192.168.88.6
192.168.88.7
192.168.88.8
192.168.88.9
192.168.88.10
192.168.88.11
192.168.88.12
192.168.88.13
192.168.88.14
192.168.88.15
192.168.88.16
192.168.88.17
192.168.88.19
192.168.88.20
192.168.88.21
192.168.88.26
192.168.88.27
192.168.88.28
192.168.88.29
192.168.88.30
192.168.88.31

1st step 31 now 14 lines code

192.168.88.0/31
192.168.88.2/31
192.168.88.5/32
192.168.88.6/31
192.168.88.8/31
192.168.88.10/31
192.168.88.12/31
192.168.88.14/31
192.168.88.16/31
192.168.88.19/32
192.168.88.20/31
192.168.88.26/31
192.168.88.28/31
192.168.88.30/31

2nd step 30 now 10 lines code

192.168.88.0/30
192.168.88.5/32
192.168.88.6/31
192.168.88.8/30
192.168.88.12/30
192.168.88.16/31
192.168.88.19/32
192.168.88.20/31
192.168.88.26/31
192.168.88.28/30

3rd step 29 now 9 lines code

192.168.88.0/30
192.168.88.5/32
192.168.88.6/31
192.168.88.8/29
192.168.88.16/31
192.168.88.19/32
192.168.88.20/31
192.168.88.26/31
192.168.88.28/30

And on this case, a /29 is the last step, because /28 and /27 do not produce any results.

If at this point 192.168.88.4 must be added to the list, first check if 192.168.88.4 is already inside on any subnet,
then add the 192.168.88.4/32
then calc again the aggregation:
from

another example code

192.168.88.0/30
192.168.88.4/32 (just added)
192.168.88.5/32
192.168.88.6/31
192.168.88.8/29
192.168.88.16/31
192.168.88.19/32
192.168.88.20/31
192.168.88.26/31
192.168.88.28/30

to

another result code

192.168.88.0/28
192.168.88.16/31
192.168.88.19/32
192.168.88.20/31
192.168.88.26/31
192.168.88.28/30

because
on 1st step 192.168.88.4/32 + 192.168.88.5/32 = 192.168.88.4/31
on 2nd 192.168.88.4/31 + 192.168.88.6/31 = 192.168.88.4/30
on 3rd 192.168.88.0/30 + 192.168.88.4/30 = 192.168.88.0/29
on 4th 192.168.88.0/29 + 192.168.88.8/29 = 192.168.88.0/28
and can not be aggregated more.

Amm0 · Mon Jul 31, 2023 6:52 pm

Boy that's above my pay grade...

But the Ubuntu "aggregate" command seem to take the approach to convert IP to int, in big endian, so normal compare operator work on these ints (e.g. called network byte order for a reason): https://git.launchpad.net/ubuntu/+sourc ... ggregate.c

rextended · Mon Jul 31, 2023 7:23 pm

/* now hunt for adjacent entries that can be combined */

From the comment it seems thet the author reason the same way I do...
A lot of functions used on RouterOS don't exist, on purpose I wrote "an algorithm that uses only functions present on RouterOS"....

Since I write something to aggregate /32 to /31 and /31 to /30, is a start to do a full aggregation, from /32 to /0.
As I already wrote, /9 can already be excessive, for not deplete uselessly the CPU must be ckecked a less number of bits.

Amm0 · Mon Jul 31, 2023 8:14 pm

LOL, well you start with...

I need help writing an efficient algorithm to convert (for example) this:

I'm not even sure that's the best C example, I just know the answer involved converting to big endian. My thought is most algorithms in C (but NOT the code) can be model in RouterOS script so that's were I start when looking for an algorithm (not code)... e.g. some rextended script function that does some as c-runtime's ntohl(), and any C struct can be modeled as array maps

I get your requirement for RouterOS script... But RouterOS script is NOT particular "fast" however, even if well-written. So I suspect starting a container to run some aggregation command would likely be much fast (e.g. efficient, in some definitions) than executing the amount of script code required. But then again that only be only true on ARM & V7...

rextended · Mon Jul 31, 2023 9:01 pm

I suspect starting a container to run some aggregation command would likely be much fast

Uhm........

Amm0 · Mon Jul 31, 2023 9:30 pm

I suspect starting a container to run some aggregation command would likely be much fast
Uhm........

* For the generic problem of aggregating prefixes...and your list large (thousands)... I'd bet on container**
** with Alpine or distroless, not starting a entire Ubuntu instance to do ip prefix aggregation

Each script line takes many milliseconds to execute and script goes through a lot of parsing. It's the lack of some ":toip-prefix" that adds even more since you'd like have to use the return//global/parse that reduce performance even more. It would not be your script code...

Since I'm guessing there is implict requirement of V6 and all platforms...

I'm sure script can do it & not sure of a better way than what you describe without seeing code.

rextended · Mon Jul 31, 2023 10:45 pm

I have a starting point now, and I'm stopped at /29, but now is clear where I must do the cycle, for now take that:

starting data code

/ip firewall address-list
add address=192.168.88.0 list=test_Start
add address=192.168.88.1 list=test_Start
add address=192.168.88.2 list=test_Start
add address=192.168.88.3 list=test_Start
add address=192.168.88.5 list=test_Start
add address=192.168.88.6 list=test_Start
add address=192.168.88.7 list=test_Start
add address=192.168.88.8 list=test_Start
add address=192.168.88.9 list=test_Start
add address=192.168.88.10 list=test_Start
add address=192.168.88.11 list=test_Start
add address=192.168.88.12/31 list=test_Start
add address=192.168.88.14 list=test_Start
add address=192.168.88.15 list=test_Start
add address=192.168.88.16 list=test_Start
add address=192.168.88.18 list=test_Start
add address=192.168.88.19 list=test_Start
add address=192.168.88.20 list=test_Start
add address=192.168.88.21 list=test_Start
add address=192.168.88.26 list=test_Start
add address=192.168.88.27 list=test_Start
add address=192.168.88.28 list=test_Start
add address=192.168.88.29 list=test_Start
add address=192.168.88.30 list=test_Start
add address=192.168.88.31 list=test_Start

working code

/ip firewall address-list {
    :local toipprefix do={:return [[:parse ":return $1"]]}
    :local IPmustMask "^((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\/(3[0-2]|[0-2]\?[0-9])\$"
    :local IPoptiMask "^((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])(\\/(3[0-2]|[0-2]\?[0-9])){0,1}\$"
    :local IPwoutMask "^((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\$"

    :local field 0.0.0.0/0
    :local sub   0.0.0.0/0
    :local sub1  0.0.0.0/0
    :local sub2  0.0.0.0/0
    :local temp  0.0.0.0

# from 32 to 31
    :local addrarray [:toarray ""]
    :local newarray  [:toarray ""]
    :foreach item in=[print as-value where address~$IPoptiMask and list="test_Start"] do={
        :set addrarray ($addrarray , ($item->"address"))
    }
    :foreach item in=$addrarray do={
        :set field $item
        :if ($field~$IPwoutMask) do={:set field [:toip $field]} ; :if ($field~$IPmustMask) do={:set field [$toipprefix $field]}
        :if ([:typeof $field] = "ip-prefix") do={
            :if ($field~"/31\$") do={
                :if ([:find $newarray $field] = [:nothing]) do={
                    :set newarray ($newarray , $field)
                }
            }
        }
        :if ([:typeof $field] = "ip") do={
            :set temp $field
            :set sub  [$toipprefix ("$($temp & 255.255.255.254)/31")]
            :set sub1 ($temp & 255.255.255.254)
            :set sub2 (($temp & 255.255.255.254) + 1)
            :if (([:find $newarray $sub] = [:nothing]) and ([:find $newarray $field] = [:nothing])) do={
                :if (([:find $addrarray $sub1] = [:nothing]) or ([:find $addrarray $sub2] = [:nothing])) do={
                    :set newarray ($newarray , $field)
                } else={
                    :set newarray ($newarray , $sub)
                }
            }
        }
    }

# useless, just for debug
    :foreach item in=$newarray do={
        add list="test_Inter31" address=$item
    }

# from 31 to 30
    :local addrarray $newarray
    :local newarray  [:toarray ""]
    :foreach item in=$addrarray do={
        :set field $item
        :if ($field~$IPwoutMask) do={:set field [:toip $field]} ; :if ($field~$IPmustMask) do={:set field [$toipprefix $field]}
        :if ([:typeof $field] = "ip") do={
            :if ([:find $newarray $field] = [:nothing]) do={
                :set newarray ($newarray , $field)
            }
        }
        :if ([:typeof $field] = "ip-prefix") do={
            :if ($field~"/30\$") do={
                :if ([:find $newarray $field] = [:nothing]) do={
                    :set newarray ($newarray , $field)
                }
            }
            :if ($field~"/31\$") do={
                :set temp [:toip [:pick $field 0 [:find $field "/" -1]]]
                :set sub  [$toipprefix ("$($temp & 255.255.255.252)/30")]
                :set sub1 [$toipprefix ("$($temp & 255.255.255.252)/31")]
                :set sub2 [$toipprefix ("$(($temp & 255.255.255.252) + 2)/31")]
                :if (([:find $newarray $sub] = [:nothing]) and ([:find $newarray $field] = [:nothing])) do={
                    :if (([:find $addrarray $sub1] = [:nothing]) or ([:find $addrarray $sub2] = [:nothing])) do={
                        :set newarray ($newarray , $field)
                    } else={
                        :set newarray ($newarray , $sub)
                    }
                }
            }
        }
    }

# useless, just for debug
    :foreach item in=$newarray do={
        add list="test_Inter30" address=$item
    }

# from 30 to 29
    :local addrarray $newarray
    :local newarray  [:toarray ""]
    :foreach item in=$addrarray do={
        :set field $item
        :if ($field~$IPwoutMask) do={:set field [:toip $field]} ; :if ($field~$IPmustMask) do={:set field [$toipprefix $field]}
        :if ([:typeof $field] = "ip") do={
            :if ([:find $newarray $field] = [:nothing]) do={
                :set newarray ($newarray , $field)
            }
        }
        :if ([:typeof $field] = "ip-prefix") do={
            :if ($field~"/(29|31)\$") do={
                :if ([:find $newarray $field] = [:nothing]) do={
                    :set newarray ($newarray , $field)
                }
            }
            :if ($field~"/30\$") do={
                :set temp [:toip [:pick $field 0 [:find $field "/" -1]]]
                :set sub  [$toipprefix ("$($temp & 255.255.255.248)/29")]
                :set sub1 [$toipprefix ("$($temp & 255.255.255.248)/30")]
                :set sub2 [$toipprefix ("$(($temp & 255.255.255.248) + 4)/30")]
                :if (([:find $newarray $sub] = [:nothing]) and ([:find $newarray $field] = [:nothing])) do={
                    :if (([:find $addrarray $sub1] = [:nothing]) or ([:find $addrarray $sub2] = [:nothing])) do={
                        :set newarray ($newarray , $field)
                    } else={
                        :set newarray ($newarray , $sub)
                    }
                }
            }
        }
    }

    :foreach item in=$newarray do={
        add list="test_End" address=$item
    }
}

end result code

add address=192.168.88.0/31 list=test_Interm31
add address=192.168.88.2/31 list=test_Interm31
add address=192.168.88.5 list=test_Interm31
add address=192.168.88.6/31 list=test_Interm31
add address=192.168.88.8/31 list=test_Interm31
add address=192.168.88.10/31 list=test_Interm31
add address=192.168.88.12/31 list=test_Interm31
add address=192.168.88.14/31 list=test_Interm31
add address=192.168.88.16 list=test_Interm31
add address=192.168.88.18/31 list=test_Interm31
add address=192.168.88.20/31 list=test_Interm31
add address=192.168.88.26/31 list=test_Interm31
add address=192.168.88.28/31 list=test_Interm31
add address=192.168.88.30/31 list=test_Interm31

add address=192.168.88.0/30 list=test_Interm30
add address=192.168.88.5 list=test_Interm30
add address=192.168.88.6/31 list=test_Interm30
add address=192.168.88.8/30 list=test_Interm30
add address=192.168.88.12/30 list=test_Interm30
add address=192.168.88.16 list=test_Interm30
add address=192.168.88.18/31 list=test_Interm30
add address=192.168.88.20/31 list=test_Interm30
add address=192.168.88.26/31 list=test_Interm30
add address=192.168.88.28/30 list=test_Interm30

add address=192.168.88.0/30 list=test_End
add address=192.168.88.5 list=test_End
add address=192.168.88.6/31 list=test_End
add address=192.168.88.8/29 list=test_End
add address=192.168.88.16 list=test_End
add address=192.168.88.18/31 list=test_End
add address=192.168.88.20/31 list=test_End
add address=192.168.88.26/31 list=test_End
add address=192.168.88.28/30 list=test_End

Creating different lists for debugging is wanted, of course...
It's easier than I thought...
After 3 cycles the pattern is clear.
I'll finish it as soon as I have time.

Sertik · Tue Aug 01, 2023 9:20 am

Oh, friends, you've written something complicated here... It seems to me that my task is easier - to find addresses with the same first three octets in one particular NameList address sheet and mark the moment when there are more of them, say 5. The subnet mask is not taken into account, it can be discarded immediately, Apparently Rex wants to do something universal, as always!

rextended · Tue Aug 01, 2023 2:10 pm

Oh, friends, you've written something complicated here... say 5...

Nono, what follow is the next step, I have already wrote something easy:
viewtopic.php?t=198167#p1016214
Just replace "> 1" with "> 4" on example and elaborate further....

Apparently Rex wants to do something universal, as always!

Rex Tended

msatter · Tue Aug 01, 2023 2:48 pm

I have to admit to see all those postings was funny, while I already posted here the script for that, years ago.

But then, I would out of place to post a direct link.

Sertik · Tue Aug 01, 2023 2:58 pm

Yes, no why, give a link, please, it might be useful.

rextended · Tue Aug 01, 2023 4:06 pm

I already posted here the script for that, years ago.

For what? The OP request
viewtopic.php?t=152953#p755080
or for address-list aggregator of existent IPs?

Amm0 · Tue Aug 01, 2023 10:15 pm

For what?

Well, since your over-engineering @Sertik's problem (e.g. solving the more generic "prefix aggregation in address-list" problem)...

I wrote an over-engineered test function $fantasylist to generate random IPs in an address-list, e.g. ones that might require aggregation – since I'm curious on the performance...but you kinda need to create a large list to do that....

The function $fantasylist below follows the RSC-GOST standard to include a help:

$fantasylist help

Usage:
$fantasylist [spread=4] [density=50] [list=$fantasylist] [replace=yes] [ip=169.254.0.0] [fidelity=10]
        spread=         num of /24's to distribute random entires over (e.g. how many / 254)
        density=        percentage (as int) of used address over the total range (i.e. 50 = 50% of possible IP)
        list=           default is $fantasylist but can be any /ip/firewall/address-list
        ip=             the first possible IP address to use (e.g. 169.254.0.0 )
        fidelity=       during IP randomization, dups can happen...
                but on-error is slow, so use lower fidelity=1 to speed creation
                (at expense of accuracy to number of IPs requested by density=)
        replace=        any previous list created by $fantasylist is removed,
                use 'replace=no' to keep an old entires in list

To see it in action... the basic uses "spread=" to control the range of IPs added to list (by /24 subnet's...,so spread=2 mean a /23) & density=10 means the range is populated with 10% of possible IP addresses in the resulting address-list.

# create 25 IPs (density=10 is 10%) in an address-list over 169.254.0.0/24 (spread=1 is 254 IPs)
$fantasylist spread=1 density=10
     remove  previous list: $fantasylist
     adding  169.254.0.100   in $fantasylist (25 / 25)
     done! requested 25 and added 25 random IPs (off by 0) to $fantasylist (len=25) after 00:00:02.517758220

# create 645 IPs (density=1 is 1%) in an address-list over 169.254.0.0/16 (so spread=254 is /16)
$fantasylist spread=254 density=1
    remove  previous list: $fantasylist
    adding  169.254.134.194 in $fantasylist (645 / 645)
    done! requested 645 and added 645 random IPs (off by 0) to $fantasylist (len=645) after 00:00:03.615506500

The needed function code to do this is here:

:global fantasylist do={
    :local tstart [:timestamp]

    :local listname [:pick $0 1 255]
    :if ([:typeof $list] = "str") do={
        :set listname $list
    }
    :local lspread 4
    :if ([:typeof [:tonum $spread]] = "num") do={
        :set lspread [:tonum $spread]
    }
    :local ldensity 50 
    :if ([:typeof [:tonum $density]] = "num") do={
        :set ldensity [:tonum $density] 
    }
    :local start 169.254.0.0
    :if ([:typeof [:toip $ip]] = "ip") do={
        :set start [:toip $ip] 
    }
    :local retries 10
    :if ([:typeof [:tonum $fidelity]] = "num") do={
        :set retries [:tonum $fidelity]
    }
    :local clean true
    :if ($replace = "no") do={
        :set clean false
    } 
    :if ($1 = "help") do={
        :put "Usage:\r\n$0 [spread=$lspread] [density=$ldensity] [list=$0] [replace=yes] [ip=$start] [fidelity=$retries]"
        :put "\tspread=\t\tnum of /24's to distribute random entires over (e.g. how many / 254)"
        :put "\tdensity=\tpercentage (as int) of used address over the total range (i.e. 50 = 50% of possible IP)"
        :put "\tlist=\t\tdefault is $0 but can be any /ip/firewall/address-list"
        :put "\tip=\t\tthe first possible IP address to use (e.g. $[:tostr $start] )"
        :put "\tfidelity=\tduring IP randomization, dups can happen...\r\n\t\tso fidelity=$retries means try a new random IP $retries times\t\r\t\tbut on-error is slow, so use lower fidelity=1 to speed creation\r\n\t\t(at expense of accuracy to number of IPs requested by density=)"
        :put "\treplace=\tany previous list created by $0 is removed,\r\n\t\tuse 'replace=no' to keep an old entires in list"
        :return
    }

    :local possible (254*$lspread)
    :local howmany ($possible*$ldensity/100)

    /ip/firewall/address-list {
        :if ($clean) do={
            remove [find list=$listname]
            :put "remove\tprevious list: $listname"
        }
        :put ""
        :local numadded 0
        :for listitem from=0 to=($howmany-1) do={
            :retry max=$retries {
                :local rndip ($start + [:rndnum from=0 to=$possible])
                add address=$rndip list=$listname
                /terminal/cuu 
                :put "adding\t$rndip\tin $listname\t($($listitem+1) / $howmany)"
                :set numadded ($numadded+1)
            } on-error={
                :put "skipping number $listitem - no unique random ip after $retries tries (perhaps use fidelity=$($retries*2))\r\n"
            }
        }
        :put "done! requested $howmany and added $numadded random IPs (off by $($howmany-$numadded)) to $listname (len=$[:len [find list=$listname]]) after $([:timestamp]-$tstart)"
    }
}

# show help
$fantasylist help

# create 25 IPs (density=10 is 10%) in an address-list over 169.254.0.0/24 (spread=1 is 254 IPs)
$fantasylist spread=1 density=10

# create 645 IPs (density=1 is 1%) in an address-list over 169.254.0.0/16 (so spread=254 is /16)
$fantasylist spread=254 density=1

N.B. what curious is the performance is :retry... an error can happen since it's possible same IP was generated twice and so the 2nd "add" will fail. So instead of checking, a potential large list to know if an IP exists before doing the "add", I just used :retry to create a new random IP and try to add that instead. The fidelity= option controls how many times it will retry the "add" using a new IP. Basically as spread= goes down, and density= goes up...the slower the $fantasylist function gets, because the chance of a duplicate goes up as the list fills up (e.g. the birthday paradox)

# to see how slow on-error= is, use higher density= 
# which means a greater chance that random IP is already present...
# code will re-try another random IP, but as more IP...
# but harder to find unique one randomly

$fantasylist spread=1 density=75 fidelity=2

     remove  previous list: $fantasylist
     adding  169.254.0.176   in $fantasylist (29 / 190)
     skipping number 29 - no unique random ip after 2 tries (perhaps use fidelity=4)
     adding  169.254.0.201   in $fantasylist (44 / 190)
     skipping number 44 - no unique random ip after 2 tries (perhaps use fidelity=4)
     adding  169.254.0.112   in $fantasylist (55 / 190)
          [...]
     done! requested 190 and added 161 random IPs (off by 29) to $fantasylist (len=161) after 00:01:06.894532020

Above takes >1 minute on an RB1100 – with some variability in the time based on how big the terminal windows is... If we change the fidelity= even higher, with a higher density= number, there more retries to achieve the desired density...so takes more time...

$fantasylist spread=1 density=75 fidelity=10
     remove  previous list: $fantasylist
     adding  169.254.0.146   in $fantasylist (190 / 190)
     done! requested 190 and added 190 random IPs (off by 0) to $fantasylist (len=190) after 00:02:43.722691640

So.... to test this use this to test @rextended prefix aggregator (viewtopic.php?p=1016529#p1016300)... since he only deals with up to /29s right now, so this gets "good" chance of some needed for shortening ip prefixes from address-list with ip... using "list=" so the generated test list is "compatible" with his code (wrapped in $ippa fn below):

{ $fantasylist list=test_Start spread=16 density=75 fidelity=10; $ippa}  
#and 42 minutes later...

done! requested 3048 and added 3034 random IPs (off by 14) to test_Start (len=3034) after 00:42:59.374805740

# @rextended prefix aggregator then runs...in 1 minute 16 seconds...
before @rextended aggregation there are 3034 IPs
@rextended completed in 00:01:16.116526360
after @rextended aggregation there are 1532 IPs

On a Mac using homebrew's aggregate commands and SSH to Mikrotik to get same address-list... it finds one less...

time ssh admin@router '/ip/firewall/address-list/print proplist=address where list=test_Start' | awk '{ print $2"/32" }' | aggregate | wc -l

aggregate: maximum prefix length permitted will be 32
1531
ssh admin@router 0.02s user 0.01s system 3% cpu 0.848 total
awk '{ print $2"/32" }' 0.01s user 0.00s system 2% cpu 0.846 total
aggregate 0.04s user 0.00s system 4% cpu 0.871 total
wc -l 0.00s user 0.00s system 0% cpu 0.870 total

BUT it's off by one ONLY because the Mac CLI command finds a /28 somewhere in the same set — @rextended stops at /29.... But the Mac completes the aggregation of an address-list in about 1-2 seconds as see by `time`...

Sertik · Wed Aug 02, 2023 10:23 am

Thanks a lot to Amm0 for supporting my idea of $1="help"!

I'm trying to use a piece of Rex'a to implement the initial task of the topic. There is a mistake somewhere ...

:local list "inlist"
:local listblock "dahgers-list"
:local count 4

/ip firewall address-list {
    :local ip2net do={:return [[:parse ":return $($1 & 255.255.255.0)/$2"]]}
    :local addr    0.0.0.0
    :local net     0.0.0.0
    :foreach item in=[find where ([:typeof [:toip $address]]="ip") and (list=$list)] do={
        :set addr [get $item address]
        :set net  [$ip2net $addr 24]
        add address=$addr list=$listblock timeout=24h
        :if ([:len [find where $address in $net]] > $count) do={
            :log info "\r\nSearch results for $net"; print where $address in $net

        }
    }
}

# /ip firewall filter add action=drop chain=input comment="Dropping dangerous adresses" src-address-list=$list

For some reason, the search restriction and list=$list does not work and the search is carried out in all address lists...

rextended · Wed Aug 02, 2023 10:24 am

@Amm0
I hope you have disabled the two debug lines in the calculation....

I'm happy with the speed, it has room for improvement,
but especially if it runs later in the list, if you add more addresses, it's faster because most of them are already aggregated.

Especially if you avoid adding IPs to the list that aren't already in the list with larger prefixes.
(it is enough that in the rule of the firewall that adds the IPs, it is put that it is not already present on the same list)
This is a RouterOS bug that allows you to add, for example, 192.168.88.2 and 192.168.88.0/24 in the same list, as different objects...
(and here it is sufficient that the scripts that import the blacklists check the IPs before inserting them in the list)

Sertik · Wed Aug 02, 2023 10:38 am

Renamed one variable, it seems to work:

:local Inlist "inlist"
:local listblock "dahgers-list"
:local count 4

/ip firewall address-list {
    :local ip2net do={:return [[:parse ":return $($1 & 255.255.255.0)/$2"]]}
    :local addr    0.0.0.0
    :local net     0.0.0.0
    :foreach item in=[find where ([:typeof [:toip $address]]="ip") and (list=$Inlist)] do={
        :set addr [get $item address]
        :set net  [$ip2net $addr 24]
        add address=$addr list=$listblock timeout=24h
        :if ([:len [find where $address in $net]] > $count) do={
            :log info "\r\nSearch results for $net"; print where $address in $net

        }
    }
}


# /ip firewall filter add action=drop chain=input comment="Dropping dangerous adresses" src-address-list=$Iistblock

rextended · Wed Aug 02, 2023 10:39 am

@Sertik

[…]
:local count 4
[…]
        :if ([:len [find where $address in $net]] > $count) do={
[…]

The condition is extremely slow...

I also try this, but still slow.

part code

            […]
            :local pat   [:pick $sub 0 ([:len $sub] - 4)]
            :local count 0
            :foreach obj in=$addrarray do={
                :if ($obj~$pat) do={:set count ($count + 1)}
            }
            :if ($count > 4) do={
                […]
            }
            […]

rextended · Wed Aug 02, 2023 12:08 pm

I solved the speed problem...

Fake24 aggregator, now is really fast...

fake24 code

# If the Start and End lists are the same,
# at the end of the script the Start list will be deleted and recreated with the content of the processing.
:local startList "inlist"
:local endList   "dahgers-list" ; # warning: at the end the whole End list is deleted and replaced
:local count     4

/ip firewall address-list {
    :local toipprefix do={:return [[:parse ":return $1"]]}
    :local IPmustMask "^((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\/(3[0-2]|[0-2]\?[0-9])\$"
    :local IPoptiMask "^((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])(\\/(3[0-2]|[0-2]\?[0-9])){0,1}\$"
    :local IPwoutMask "^((25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\\.){3}(25[0-5]|(2[0-4]|[01]\?[0-9]\?)[0-9])\$"

    :local field 0.0.0.0/0
    :local sub   0.0.0.0/0
    :local sub1  0.0.0.0/0
    :local sub2  0.0.0.0/0
    :local temp  0.0.0.0

# from 32 and /24 to Fake24
    :local addrarray [:toarray ""]
    :local newarray  [:toarray ""]
    :foreach item in=[print as-value where address~$IPoptiMask and list=$startList] do={
        :set addrarray ($addrarray , ($item->"address"))
    }
    :foreach item in=$addrarray do={
        :set field $item
        :if ($field~$IPwoutMask) do={:set field [:toip $field]} ; :if ($field~$IPmustMask) do={:set field [$toipprefix $field]}
        :if ([:typeof $field] = "ip-prefix") do={
            :if ($field~"/24\$") do={
                :set field [:tostr $field]
                :if ([:typeof ($newarray->$field)] = "nil") do={:set ($newarray->$field) 0}
                :set ($newarray->$field) (($newarray->$field) + 1)
            }
        }
        :if ([:typeof $field] = "ip") do={
            :set temp $field
            :set sub  [:tostr [$toipprefix ("$($temp & 255.255.255.0)/24")]]
            :if ([:typeof ($newarray->$sub)] = "nil") do={:set ($newarray->$sub) 0}
            :set ($newarray->$sub) (($newarray->$sub) + 1)
        }
    }
    remove [find where list=$endList]
    :foreach x,y in=$newarray do={
        :if ($y > $count) do={
            add list=$endList address=$x comment=$y timeout=1d
        }
    }
}

diamuxin · Wed Aug 02, 2023 3:34 pm

I solved the speed problem...

Fake24 aggregator, now is really fast...

This script, what does it really do?
can you give an example to understand it? thanks.

rextended · Wed Aug 02, 2023 4:14 pm

I solved the speed problem...

Fake24 aggregator, now is really fast...
This script, what does it really do?
can you give an example to understand it? thanks.

It cover only the OP request: If 5 or more IPs from same /24 try to attack, put all /24 on blacklist.
Is why the /24 is Fake, because if are present just 5 IPs the /24 is created, no matter if are present or not all IPs on the /24.

The unfinished TRUE aggregator, now only to /29 max, instead it actually works as an aggregator, because aggregate only if the IPs are really present.
A separate topic, and link on my snippets, for that is created when is done.

Amm0 · Wed Aug 02, 2023 4:47 pm

The unfinished TRUE aggregator, now only to /29 max, instead it actually works as an aggregator, because aggregate only if the IPs are really present.
A separate topic, and link on my snippets, for that is created when is done.

The QA department awaits the TRUE aggregator.

BTW, in my fantasy list creator $fantasylist found the same issue with [find] being slow... I'd thought just "randomly picking, and if error try another" would avoid slow [find] ... but turns out that's not true. Basically all the operations slow down non-linearly, as list size increases...including "add". Anyway, I'll post another version of $fantasylist – since that's a bit off topic here. But since RSC is quick a loops, going through all possible address, then apply the "odds" appears as the quickest method to add random ones, but will post separately.

rextended · Wed Aug 02, 2023 4:51 pm

BTW, in my fantasy list creator $fantasylist found the same issue with [find] being slow...

The technique used for speed the program is avoid like the pest any "print" or "find" over the first needed for read the list....

This:

snippet code

            :if ([:typeof ($newarray->$sub)] = "nil") do={:set ($newarray->$sub) 0}
            :set ($newarray->$sub) (($newarray->$sub) + 1)

is the more elegant way I find for avoid "find if exist more than 4", instead, I add to the address-list, at the end, only the entry that have counter > 4

Optimizing the code is still possible, like

example code

            :if ([:typeof ($newarray->$sub)] = "nil") do={:set ($newarray->$sub) 1} else={:set ($newarray->$sub) (($newarray->$sub) + 1)}

and also on other points, but require time...

Amm0 · Wed Aug 02, 2023 6:59 pm

BTW, in my fantasy list creator $fantasylist found the same issue with [find] being slow...
[...] avoid like the pest any "print" or "find" over the first needed for read the list....

@rextended, I wrote new version my random address-list generator and put in separate topic here: viewtopic.php?t=198298
But new version it's WAY faster than my approach above. It took just 1 second to generate 2000 address. And 54 seconds to aggregate the prefixes using your "up to /29" code (without the test_Inter3x debug code)

>      $FAKEAL list=test_Start spread=16 density=75; $Pippa

using   address-list test_Start
remove  test_Start done   
adding  100%    169.254.15.223 (added 2050 at 4063 of 4064)             
runtime 00:00:01
done!   wanted 3048 got 2050 (off by 998) in test_Start (length 2050)
before @rextended aggregation there are 2050 IPs
completed in 00:00:53.501365120
after @rextended aggregation there are 1502 IP

rextended · Wed Aug 02, 2023 7:18 pm

I already have some ideas and room for improvement in the overall speed of the aggregating function.

Now I'm interested in making a version that works up to "/0" (then maybe later limited to /9 or /10 or maybe much less, for obvious reasons of uselessness).
Once I've done a nice loop from /31 to /0 that works fine,
I optimize the cycle in order to increase its speed.

msatter · Wed Aug 02, 2023 7:56 pm

Doing this for years now, the list is contains now almost 2200 /24 ranges collected since September 2021.

Underneath the log when a range is placed on the block list when reaching the set limit which is set to three.

PermBlock.JPG

The last address ranges added to the block list:

PermBlock2.JPG

rextended · Wed Aug 02, 2023 10:48 pm

2200??

Who knows if using aggregation they become less, some /24 could become /23, and why not some /23 become a /22 and so on...

I actually use https://tehnoblog.org/ip-tools/ip-address-aggregator/ for do that.
But if I automatize that on RouterOS is better than export, convert and re-import... (or write a script for interact with the page and elaborate the result)