v7.15.3 [stable] is released!

Maggiore81 · Fri Aug 09, 2024 12:04 pm

No logs after reboot. The internal LTE card seem disappeared. I have taken another WAP and it sees the LTE PCIe card.
Tried downgrading, upgrading etc, no issue. I have opened the unit the card is nicely warm, everything is connected there... maybe the card just died ?

infabo · Fri Aug 09, 2024 12:15 pm

/system/resources/usb/print

Does the device show up there?

Maggiore81 · Fri Aug 09, 2024 12:21 pm

/system/resources/usb/print

Does the device show up there?

No, 0.
I think the PCIe card died...
I will try next week to disassemble another one and replace the card to see if it is the card. So I can order one separately.

toxicfusion · Fri Aug 09, 2024 5:53 pm

I will also chime in and mention we're also experiencing Wireless stability issues since upgrading CAPS to 7.15.3, up from 6.49.X [legacy??] routerOS....

Clients appears to be constantly connecting/disconnecting. We're going to try downgrading to 7.14.X build to see if alleviates. if not, we're rolling back to 6.49.X release, and use the regular wireless package. We had also installed the wifi-qcom-ac driver package and had fun time manually configuring those CAPS to support SSID/VLANs -- but not also without issues.

What a mess. We're at a point of just dropping MT for another wireless vendor for client access.

infabo · Fri Aug 09, 2024 8:57 pm

try 7.13.5 before downgrading to 6.x again. It is not clear to me if you used the wireless package on 7.x. The wireless package on 7.15 hould be as stable as on 6.x.

optio · Sat Aug 10, 2024 2:50 pm

Please check issue related to container mounts, issue is described here: viewtopic.php?p=1090584#p1090584

kryztoval · Tue Aug 13, 2024 6:43 am

I will also chime in and mention we're also experiencing Wireless stability issues since upgrading CAPS to 7.15.3, up from 6.49.X [legacy??] routerOS....

Clients appears to be constantly connecting/disconnecting. We're going to try downgrading to 7.14.X build to see if alleviates. if not, we're rolling back to 6.49.X release, and use the regular wireless package. We had also installed the wifi-qcom-ac driver package and had fun time manually configuring those CAPS to support SSID/VLANs -- but not also without issues.

What a mess. We're at a point of just dropping MT for another wireless vendor for client access.

Yeah, I love the Mikrotik routers and in general their devices. But when it comes to wireless they only play nice with themselves. And with the latest update they barely do that. I have spun up a few third party APs that do nothing else but wireless because of this. My main Router is a RB5009UG+S+ and for APs I was using RBD52G-5HacD2HnD (AC^2), RDB53iG-5HacD2HnD (AC^3) and 2xC52iG-5HaxD2HaxD (AX^2) bridged wirelessly but those devices kept disconnecting. I have a few AX^2 that I bought when it was released that I have not been able to set up because of this wireless incompatibility, instability and lack of 4-mac modes, besides the misisng usb port that the AC^2 did have.

ahmedelbarbary · Tue Aug 13, 2024 6:16 pm

Hello Mikrotik Support
Since 7.15.3 PPPOE not stable, Clients disconnect and connect again, 7.15 was fine not have any issues, Could you check this and fix it ?

dfdf · Sun Aug 18, 2024 12:34 pm

Observing strange behavior on CRS328-24P-4S+ running ROS (not SWOS) 7.15.3. Device configured as a switch (by nature of configuration, despite running ROS - intention is to use scripts later). All ports are in one bridge, no routing, no vlans, just a few fw rules to prevent port scanning. After about a day running well all/part devices behind the switch stop getting ip traffic - I can ping devices connected to switch only from switch. I can connect to switch itself (via winbox for ex.) from a device on the same ip subnet (device is not connected directly to switch), but no connection to any device (on the same ip subnet) behind the switch works. Meanwhile, I STILL can ping/connect (for ex - ssh) to device behind the switch from the switch itself, I also can connect if device connecting from is attached to the same switch. Rebooting switch solves problem, but after some time (about a day running) it's coming back. It looks like a some BRIDGE firewall rule is being activated about after one day running (but i have no any), Reverting to 7.14.3 seems to resolve the issue. My config (very simple indeed) attached. UPD: Reverting to 7.14.3 didn't help :( Will either try another switch or change to SWOS for detecting problem is soft or hard. If SWOS will work ok, than supout.sif and bug report. UPD2: seems found the root of the issue: there was an HP MF device connected directly to switch, and on the HP MF device ethernet port settings were 100M Half Duplex. Got a warning about it in MK log (see attachment). After warning, some time later, got a problem with ip traffic. Changed port setting on HP MF to "Auto". Switched corresponding port via Winbox on MK to disabled, than back to enabled. No more problems till now, Uptime of MK is 10 days.

infabo · Sun Aug 18, 2024 12:58 pm

Looks like your device does more as you described. Watch memory usage. Maybe your are running short of RAM because of your firewall rules.

dfdf · Sun Aug 18, 2024 8:05 pm

Looks like your device does more as you described.

Nope, just a few rules, see attached conf. And than why downgrading to 7.14.3 solves the issue?

infabo · Sun Aug 18, 2024 8:22 pm

there is at least an ovpn server configured. infamous detect internet as well. please have a look at your own file.

Basically you answered your own question: when downgrading to 7.14 resolved the issue - then there is no config issue we need to review in public here. File a bug report.

dfdf · Sun Aug 18, 2024 11:11 pm

there is at least an ovpn server configured.

/interface ovpn-server server set auth=sha1,md5

you're wrong, it's not about any vpn config at all. and this is the only string concerning vpn slightly with no consequences.
I did review my config before posting it. Idea to file a bug report is known, nothing new. I thought maybe someone else already have had this issue too.

infabo · Sun Aug 18, 2024 11:46 pm

🤷..

dang21000 · Tue Aug 20, 2024 9:59 pm

Hi,

Strange thing on 7.15.3 with user active and logout.

Here, user "promtheus" via unknow must be "rest-api" ; display well on 7.16rc2.
But no way to request logout with always this error.

[admmikrotik@router50] > /user/active/print
Columns: WHEN, NAME, ADDRESS, VIA
# WHEN                 NAME         ADDRESS          VIA      
0 2024-08-20 20:51:13  prometheus   192.168.175.113  (unknown)
1 2024-08-20 20:51:13  prometheus                    api      
2 2024-08-20 20:51:24  admmikrotik  192.168.70.105   winbox   
3 2024-08-20 20:53:09  admmikrotik  192.168.70.105   ssh      


[admmikrotik@router50] > /user/active/request-logout numbers=0
action failed (6)

DjM · Tue Aug 20, 2024 10:57 pm

@dang21000:
viewtopic.php?t=207040

dang21000 · Wed Aug 21, 2024 6:29 pm

@dang21000:
viewtopic.php?t=207040

Thank you

pe1chl · Fri Aug 23, 2024 3:45 pm

*) bgp - correctly synchronize input.accept-nlri address list;

What does this change mean?
In the past I have configured an input.accept-nlri on one peer, referring to an address list that has a single /16 entry, and I think at that time it meant that all subnets WITHIN that /16 (so also e.g. a /24 or /28 within that /16) would be accepted.
Now I should admit that I have never fully analyzed that to see if it really worked.

But today I ran into an issue where this peer became important, and I found that this input.accept-nlri does not work correctly!
When the address list with the single /16 item is specified, it accepts a route for the full /16 network, and also for /32 addresses within that network, but NOT for /24 or /28 routes within the /16. These are simply dropped and only appeared after I removed the input.accept-nlri .

Is that related to this "fix" and has that now maybe been broken? Because I think it worked OK before, although I am not sure.

brunoforcajovem · Mon Aug 26, 2024 7:46 pm

Several pppoe drops with 7.15.3

Thu Aug 29, 2024 2:01 am

*) bgp - correctly synchronize input.accept-nlri address list;
What does this change mean?
In the past I have configured an input.accept-nlri on one peer, referring to an address list that has a single /16 entry, and I think at that time it meant that all subnets WITHIN that /16 (so also e.g. a /24 or /28 within that /16) would be accepted.
Now I should admit that I have never fully analyzed that to see if it really worked.

But today I ran into an issue where this peer became important, and I found that this input.accept-nlri does not work correctly!
When the address list with the single /16 item is specified, it accepts a route for the full /16 network, and also for /32 addresses within that network, but NOT for /24 or /28 routes within the /16. These are simply dropped and only appeared after I removed the input.accept-nlri .

Is that related to this "fix" and has that now maybe been broken? Because I think it worked OK before, although I am not sure.

1. i think NLRI does not apply inmediatly, resend/refresh on BGP session is neeeded to enable/disable NLRI or any changes to it
2. i have seen that NLRI does not drop /32 in ipv4 and/or /128 on ipv6 i dont know why, in mi case that is not a problem, i filter that routes

pe1chl · Thu Aug 29, 2024 2:19 pm

1. i think NLRI does not apply inmediatly, resend/refresh on BGP session is neeeded to enable/disable NLRI or any changes to it
2. i have seen that NLRI does not drop /32 in ipv4 and/or /128 on ipv6 i dont know why, in mi case that is not a problem, i filter that routes

Correct, I submitted the same as a support case and the answer was that they were very surprised that I would assume that a /16 entry in the address list would admit all /24 entries "under" that /16 (while I would think that would be natural for many use cases), and when I replied that it DOES admit the /32 routes under the /16 network, it was said that this is a known bug.
So I had to remove my input.accept-nlri configuration because it is useless for me. Fortunately I was able to put an output filter at the other end, so I do not have 800 "filtered" routes in the table on this router (something I tried to avoid using input.accept-nlri )

kiler129 · Thu Aug 29, 2024 10:51 pm

*) poe-out - moved "PoE LLDP" property from "/interface/ethernet/poe" to "/ip/neighbor/discovery-settings" and enable it by default;

While this is definitely a good change, I think such modifications should be handled more gracefully. I am not sure if/what is the deprecation-removal policy on ROS specifically but such modifications result in instant syntax errors. While our automatic test suite detected this it would be great to keep these moved/renamed options and throw a warning "XYZ was deprecated, use ABC instead" and keep it for some time.

pe1chl · Fri Aug 30, 2024 12:16 am

+1
This kind of changes causes problems when importing older exports into devices with newer version.
(of course when upgrading a device there is automatic conversion)
It happened before e.g. with "/tool email set"

kiler129 · Fri Aug 30, 2024 1:41 am

This kind of changes causes problems when importing older exports into devices with newer version.

This is actually even bigger of a problem when using any sort of automation or config management. With lack of transition period, the iffology for version if a bit painful at times. While I try to schedule regular maintenance windows at all sites I manage, such changes make these operations much harder.

Ideally, where possible, old behavior should probably be auto-converted on the fly for some time. Given that ROS doesn't seem to follow semantic versioning, some even very loose policy of "hey, we will keep it for a year" would be better than nothing. In case of LLDP and move from per-port to per-device where conversion isn't possible really, leaving such option even as no-op would make life 100x easier: we can throw a ticket to deprecate it internally in the configs and update docs rather than leaving note "hey, if it's v7.15+ it will work differently".

infabo · Fri Aug 30, 2024 8:12 am

Mikrotik is not willing to change anything in either their changelog structure, versioning, handling of breaking changes nor will they introduce deprecation warnings. That's not in their software engineering manual dated from 1999.

kcarhc · Sat Aug 31, 2024 9:55 am

I suspect that there might be a memory leak in DNS on version 7.15.3.
The DNS cache has 4,000 entries, but the cache used is 29,093 KB.
After flushing the cache, there are 500 entries, but the cache used is still 28,338 KiB, only releasing 1,000 KiB.

kcarhc · Wed Sep 04, 2024 5:52 am

please check SUP-164084
RouterOS 7.15.3 dns memory leak

kcarhc · Wed Sep 04, 2024 5:57 am

The runtime is 20 days, and currently, the DNS cache has grown to 42,375 KiB.
The DNS memory leak in RouterOS 7.15.3 is continuously occurring.

nmt1900 · Wed Sep 04, 2024 11:53 am

Can this be related to "adlist" functionality? I use my own manually managed blacklist which has entries as NXDOMAIN and do not see this problem.

Doing "adlist" vith A entries '0.0.0.0' is probably not the best and most efficient solution after all...

Jotne · Wed Sep 04, 2024 11:55 am

The runtime is 20 days, and currently, the DNS cache has grown to 42,375 KiB.
The DNS memory leak in RouterOS 7.15.3 is continuously occurring.

Why have you set Cache size to 64MB? It seems that default on my routers is 2MB.
Will not not the router use up to 64MB if its set to 64MB? (If you have a lot of requests)

mkx · Wed Sep 04, 2024 3:19 pm

The runtime is 20 days, and currently, the DNS cache has grown to 42,375 KiB.
The DNS memory leak in RouterOS 7.15.3 is continuously occurring.
Why have you set Cache size to 64MB?

This.

It's not a memory leak if service uses up to amount of RAM assigned. In this particular case, even if some DNS cache entries are expired it's not required to actually purge them if feature/service uses less RAM than assigned. Maybe those entries remain in the cache but when they get hit again, DNS service actually goes out to refresh the status ... and updating cache entry might be faster (less resource hungry) than creating a new one (not to mention periodic purging which costs CPU resources as well). Search has to be performed anyway (even if only to discover that wanted entry doesn't exist).

Jotne · Wed Sep 04, 2024 8:05 pm

That was my thought as well.

kcarhc · Wed Sep 04, 2024 11:22 pm

Can this be related to "adlist" functionality? I use my own manually managed blacklist which has entries as NXDOMAIN and do not see this problem.

Doing "adlist" vith A entries '0.0.0.0' is probably not the best and most efficient solution after all...

Currently, no adlist functionality is being used, so the issue is not caused by the adlist.
The design of the RouterOS DNS adlist is still incomplete.
It can only block domains by resolving them to 0.0.0.0, which may cause some apps to continuously send requests and try to connect.
I anticipate that, in the future, it will likely support feedback methods similar to other ad-blocking software, such as NXDOMAIN or simply not responding, which would improve the situation.

kcarhc · Wed Sep 04, 2024 11:28 pm

Why have you set Cache size to 64MB?
This.

It's not a memory leak if service uses up to amount of RAM assigned. In this particular case, even if some DNS cache entries are expired it's not required to actually purge them if feature/service uses less RAM than assigned. Maybe those entries remain in the cache but when they get hit again, DNS service actually goes out to refresh the status ... and updating cache entry might be faster (less resource hungry) than creating a new one (not to mention periodic purging which costs CPU resources as well). Search has to be performed anyway (even if only to discover that wanted entry doesn't exist).

if I don't keep increasing the size of the DNS cache, I get an error message saying "DNS cache full."
At that point, even if you execute a DNS -> flush cache command, it still doesn't significantly reduce the "Cached Used" value—it might decrease slightly, but not by much.
At the very least, I think the continuous, never-ending growth is abnormal. I haven't encountered a similar issue in versions prior to 7.15.

kcarhc · Wed Sep 04, 2024 11:33 pm

The runtime is 20 days, and currently, the DNS cache has grown to 42,375 KiB.
The DNS memory leak in RouterOS 7.15.3 is continuously occurring.
Why have you set Cache size to 64MB? It seems that default on my routers is 2MB.
Will not not the router use up to 64MB if its set to 64MB? (If you have a lot of requests)

it will show cache full, an error message saying "DNS cache full" appears, which I hadn't encountered before. Moreover, when the cache is full, you'll notice that the DNS dynamic servers are much more likely to crash randomly. I suspect that when the cache is full, it causes certain anomalies that trigger these crashes. As a result, I have to keep increasing the DNS cache size to avoid the series of issues caused by the "DNS cache full" error. These issues did not exist before version 7.15, so with the same usage, I believe that version 7.15.3 has a memory leak in the DNS cache.

sirbryan · Thu Sep 05, 2024 6:52 am

I've submitted a ticket, but wanted to post here just in case someone else has seen a similar problem.

I have five CCR2116's in a full iBGP mesh. Three are peers with other providers, two sit in our core. We take full routes, but filter out AS-PATH's longer than 2 ASN's.

For a couple of years this has worked fine. But in the last number of weeks, one or more of them has rebooted randomly. Today, four of the five all suffered a kernel failure at the same exact time, created no support files, and rebooted at the same time due to watchdog timeout. The three borders did the same thing a few weeks ago; unfortunately, two had watchdog disabled and required truck rolls to reboot them. It appears I upgraded them all 25 days ago to 7.15.3, so I'm restarting them into their 7.14.x partitions. (Not sure if the first reboot happened under 7.14... prior to that they were running 7.12.)

Because of the simultaneous reboot, and since they're all speaking BGP in the same ASN, I suspect an errant BGP message is the cause.

I have a number of CRS300's also running 7.15.3, and they're randomly rebooting every few days due to kernel failures as well, but those ones are likely the memory leak introduced in 7.15 (reportedly addressed in 7.16).

Other CCR2116's carrying similar amounts of traffic as our BGP mesh (the CGNAT routers) have been up and solid for those same three weeks since updating.

Jotne · Thu Sep 05, 2024 7:57 am

it will show cache full, an error message saying "DNS cache full" appears, which I hadn't encountered before...

It may have something to do with you config. You can post it here.

oeyre · Thu Sep 05, 2024 5:06 pm

I've submitted a ticket, but wanted to post here just in case someone else has seen a similar problem.

I have five CCR2116's in a full iBGP mesh. Three are peers with other providers, two sit in our core. We take full routes, but filter out AS-PATH's longer than 2 ASN's.

For a couple of years this has worked fine. But in the last number of weeks, one or more of them has rebooted randomly. Today, four of the five all suffered a kernel failure at the same exact time, created no support files, and rebooted at the same time due to watchdog timeout. The three borders did the same thing a few weeks ago; unfortunately, two had watchdog disabled and required truck rolls to reboot them. It appears I upgraded them all 25 days ago to 7.15.3, so I'm restarting them into their 7.14.x partitions. (Not sure if the first reboot happened under 7.14... prior to that they were running 7.12.)

Because of the simultaneous reboot, and since they're all speaking BGP in the same ASN, I suspect an errant BGP message is the cause.

I have a number of CRS300's also running 7.15.3, and they're randomly rebooting every few days due to kernel failures as well, but those ones are likely the memory leak introduced in 7.15 (reportedly addressed in 7.16).

Other CCR2116's carrying similar amounts of traffic as our BGP mesh (the CGNAT routers) have been up and solid for those same three weeks since updating.

Since upgrading to 7.14.3 (from either 7.12 or 6.49) we've noticed some of our CCR1036/1072 rebooting due to kernel failure. From initial analysis this seems to coincide with IGP/LDP instability on the network casued by mmwave radio backhauls flapping.

Support has advised from our supout file that the router received a packet of 65534 bytes length. This is impressive since our network is L2 transport only so frame size should be limited by the ingress interface MTU.

sirbryan · Thu Sep 05, 2024 5:50 pm

After further inspection, CRS300's (CRS310, NetPower16/CRS318) that are participating in OSPF/BGP had really low RAM available numbers, related to the number of days of uptime (4 days = 64MB of RAM left, 8 days uptime = only 22MB of RAM left), whereas those acting as switches are fine (160-170MB of RAM free). Routers with 1GB of RAM (AX3, 4011, 5009) had less than half of their RAM available after 25 days of uptime. The 2116's that haven't rebooted didn't really show any noticeable depletion because they have 16GB of RAM.

All of the above were running 7.15.3 and have been moved to 7.16rc4 for now. Prior to upgrading them to 7.15.3 three weeks ago, they had been running 7.11.2, and one had an uptime of 258 days before needing a reboot due to resource exhaustion of some kind (got laggy and unresponsive; had to power-cycle it).

makrob · Mon Sep 09, 2024 12:12 pm

I have problem-issue on RB5009UG+S+, when using usb flash disk Kingston Data Traveler 3.0 (USB3.20 5000Mbps).
When I reboot the router the disk gets disconnected on Diks Lists shows as not recognized.
I must manually pull it out and push it back into usb port.
I have tested it on two usb flash disk same model one 32GB second 64GB, all were formatted ext4 file system.
Does anyone have a similar problem?
I have found a second problem, when using smb shares with Require Encryption enabled. After accessing the share from windows device the router reboots !!!
Thanks.

Mon Sep 09, 2024 12:14 pm

It might be a known issue.
Are you sure the disk has not been mounted as USB2 (and different label, e.g. disk2) ?
If so, USB reset should solve this issue.

Support is aware of the issue, it happens with some brands of USB disks.
No ETA yet on the solution.

patrikg · Tue Sep 10, 2024 12:08 pm

Maybe this is because, how the ext4 drive partitioned gpt mbr ?

My suggestion to try to wipefs (linux-command) or even use windows diskpart and do "clean" first and the try to format the drive in Mikrotik device.
So the partition will be created on the Microtik device, not any another equipment like Linux computer.
I don't know if the Mikrotik device use any partitions.

kryztoval · Fri Sep 13, 2024 8:31 pm

I am wondering why it keeps disconnecting every 3 hours give or take.

I think I have a reason for this. I may be wrong, I am waiting on support to confirm this.

It seems that leaving the wireless channel on auto will make the device try to scan for the channels every 3 hours or so. And either that process kicks all the clients or it re-sets the same channel and it kicks all the clients.

This doesn't seem to happen if you set a specific channel to the wireless device.

I did not test it but it may also be that setting the interval for the channel check may prevent it from doing the test so often or outside of that interval.
So far I have only seen this behavior on the AX^2. Could extend to all wifiwave2 devices but I have not tested the AX^3 nor the hap ax.lite

globalmedia · Mon Sep 16, 2024 7:21 am

Hi,

I'm not able to get work Netwatch/ fetch to send any notification. I had fetch thorough Teams I did not touch the config but it is not working anymore after package upgrade to version 7.15.3. Today I tried set it up with Telegram and there is info Fetch failed with status 400 or Mode not specify.

Is here anyone with working Netwatch/Fetch? Anyone know how to solve it?

sindy · Mon Sep 16, 2024 6:06 pm

Is here anyone with working Netwatch/Fetch? Anyone know how to solve it?

I do send Telegram notifications using /tool/fetch in 7.15.3 and I do not suffer from this kind of problems:
/tool fetch url="https://api.telegram.org/$botId:$botPwd ... MarkdownV2" output=none check-certificate=yes-without-crl

To stay on topic here, please open a new topic for further discussion regarding this.

Rox169 · Mon Sep 16, 2024 9:57 pm

Is here anyone with working Netwatch/Fetch? Anyone know how to solve it?
I do send Telegram notifications using /tool/fetch in 7.15.3 and I do not suffer from this kind of problems:
/tool fetch url="https://api.telegram.org/$botId:$botPwd ... MarkdownV2" output=none check-certificate=yes-without-crl

To stay on topic here, please open a new topic for further discussion regarding this.

Hi,
your fetch command is not working for me...I have only changed the chat:id. What else should I change?

Thank you

sindy · Mon Sep 16, 2024 10:41 pm

What else should I change?

The topic, please. This one is not the right place for discussing this, create a new one.

Amm0 · Mon Sep 16, 2024 10:52 pm

What else should I change?
The topic, please. This one is not the right place for discussing this, create a new one.

Very true. Just to close this out... since the permission have changed somewhat recently (but NOT in this release). Poster with fetch is using netwatch, which has restricted permissions. It also may not be the fetch per se but another restricted operation too.

Anyway. The answer lies in the docs netwatch docs (https://help.mikrotik.com/docs/display/ROS/Netwatch) and searching forum for "netwatch fetch" - or as noted SEVERAL times as a NEW post.

kryztoval · Tue Sep 17, 2024 3:43 am

It seems that leaving the wireless channel on auto will make the device try to scan for the channels every 3 hours or so. And either that process kicks all the clients or it re-sets the same channel and it kicks all the clients.

There is also another process that seem to be running every 40 hours (give or take) and I am not sure what that could possibly be now. But this also resets the wireless interface.

Tue Sep 24, 2024 9:39 am

RouterOS v7.16 has been released
viewtopic.php?t=211157