I apologize for not paying attention to the thread before I got sick, I had some thoughts.
I'm glad that we made progress in solving the problem and that I was able to help a little.
I think it's a good idea to use the two lists together, it bridges the time between copying and stabilizes the route.
There may be a few more pitfalls:
- interruption of a long work process due to a route change
- due to established connections, conntrack insists on ip addresses
- several servers can operate on one IP address, with several names, therefore traffic cannot be separated
- unless you want to ban the name, an nxdomain might be a better solution (be careful, there is no dnsview)
- more dns requests, although this might matter less
I like simple, even one-liner type solutions.
If there are fewer instructions, the operation can be faster.
I also didn't think that it is possible to modify the data to be copied after the copy-from...
/ip firewall address-list
:foreach i in=[find where list=vpn1_list] do={
:do {
add copy-from=$i list=vpn1_list2 timeout="1h"
} on-error={remove $i}
}
We also delete the vpn1_list from the list (the copies that go wrong), the number of cycles and errors could be less during the next run. For me, the first list became empty in 2-3 rounds, the script had nothing to do. I only use the "vpn_list2" list as a collection list and block the traffic in RAW prerouting chain.
Regarding duplication:
I used :do {} on-error commands because if the address is already in the list, "add" causes an error and the script stops running.
And if we search for the address in the target list with find, it takes a long time, especially for a collection list with many items.
In list management, mikrotik has an internal solution, because it manages what the dns request adds and is already on the list.
Some people use huge address lists, e.g. RBL, territorial list per country, even internet full route.
As a result of the dns search, we get /32 addresses, the size of large lists can be reduced by address aggregation, which, however, takes a long time to prepare.
I got some ideas:
What takes a lot of work is that we periodically take the elements of vpn1_list2 in order, try to resolve their names, ping them or add them to netwatch as a monitored ip, and if there is no response, the target is not alive, we delete them from the list.
This may not be worth it... Monitoring hundreds for a few IPs...
Another idea, you need these IP addresses for routing, perhaps by deleting the second list, you could transfer them to the routing table with the appropriate gateway. I didn't think about when and what should happen to the addresses - recursive route? black hole? remove?
The result can be seen in the firewall and cpu load, fewer packets need to be marked for routing, in exchange for a larger routing table, address aggregation would be useful here as well.
Maybe you don't want to make the "change back" conditional, is it good if the route is reset when the addresses expire?
But that's another script...
Third idea:
In the address lists, we can use not only ip addresses, ip domains, but also dns names.
When I manually add the dns name to a list in winbox, it enters it and gets a dynamic indicator, it resolves and the ips are displayed.
The luck is that the comment contains a dns name that we got with the "Address List" (thanks mikrotik!).
The comment looks like "created for example.com.".
If you need the exact name...
/ip firewall address-list
:foreach i in=[find where list=vpn1_list] do={
:do {
:local cfqdn [get $i comment]
add address=[:pick $cfqdn 12 ([:len $cfqdn] - 1)] list=vpn1_list2 timeout="1h"
} on-error={remove $i}
}
If the vpn1_list were big and we would do it faster:
- we are not precise, we keep the dot that closes the domain name, it does not cause a problem in name resolution
- we do not count the length of the name
- we make bad use of the possibility that the pick does not indicate an error if the end of the string is greater than the length (be careful, not expected operation)
- we save the variable
- fqdn longer than 255 characters is not very common...
It's not "pretty", but within the right limits it works for me...
/ip firewall address-list
:foreach i in=[find where list=vpn1_list] do={
:do {
add address=[:pick [get $i comment] 12 255] list=vpn1_list2 timeout="1h"
} on-error={remove $i}
}
The problem you call "Dead Zone effect" (cool name, could even be the title of a game or demo)
Maybe /ip dns cache flush would solve it, because then the resolver cannot resort to anything other than external servers and its own static entries. When this should be done is a good question, but it is certain that it will cause an increase in dns traffic and some delay in name resolution.
If you shorten the "Cache Max TTL", which is the default 7d, you don't have to empty the entire dns cache, but you will get fewer hits.
Quite a few dns requests later....
Testing:
It is an "external dns name", there is an entry in the static <fqdn> fwd <external dns server> address-list <list name>.
:put [:resolve dnsname] does not always use the "local dns server", it takes a few repetitions to get into the cache,
therefore :put [:resolve fqdn server=127.0.0.1] works better.
- you make a dns request, it enters the cache, it adds the ip to the list, you remove it manually, but the next dns request doesn't include it
The TTL is still ticking in the dns cache, so it is not added, because it would be double....
dns flush cache, deletes cache and removes the ip from the list
Another request, it will be added
, it works...
- you make a dns request, it is added to the cache, the ip is added to the list, you rename the list, the ip is not removed from the old list
but it will be uploaded again
But, if you rename it, the TTL or flush cache is ticked in the cache, it will be removed
- I disable the record, flush, I make a dns request, it is put in the cache, TTL starts ticking,
I enable the fwd record with the list, then it is not listed yet,
dns request, it is added to the list, but even though the TTL expires, it is not removed and the cache flush does not remove it either.
I manually delete the ip from the list, it is removed, it is not in the cache.
Dns request, cache, is added to the list, flush, cached, deleted from the list.
With a static "A record" (you manually overwrote an ip), the problem is that you cannot automatically remove the ip from the list.
- it's already in the cache, dns request, it picks up an ip, I disable the record (of course, the management of the list ends with this), it drops out of the cache
but it stays on the list
- I manually delete the ip from the list, enable the record, it is put in the cache, no matter how many requests I make, it is no longer on the list
.
Well, the flush doesn't help with this either, because there will be no static, TTL expiration, no trigger.
If I give another list or ip, "it works the same way"
Is the ip on some internal list? Do you have any hidden flags in the cache?
Well, only restarting the router helps with this...
The ip included in the address list has a dynamic flag, which should be understood as "this managed by a service and preferably not tampered".
But it allows you to remove it, but then the router does not think that it has expired in the cache, there is no trigger. According to him, the ip is still on the list, so the dns request does not add it again.
Maybe that's why there is no timeout value in the list either, because the detour is not done by the list. If the timeout is shorter than the TTL, the dns request will not be added to the list, and if it is longer, it will collide.
I think avoiding multiple additions was solved this way, without searching the entire list.
This is too long and complicated for me, so I google translated it.
