ip route cache BUG

ujin · Tue Oct 22, 2013 2:13 pm

The problem is following: The oldest entries are not flushed from cache.

During one day we got:
/ip route cache print
cache-size: 52442
max-cache-size: 65536

This bug is observed only on OS version 6.x.

OS version 5.x did not have such issue.

Please provide workaround HOW TO FLUSH CACHE.

ujin · Tue Oct 22, 2013 3:25 pm

> /ip route cache print
cache-size: 53528
max-cache-size: 65536

ujin · Wed Oct 23, 2013 10:04 am

Please help, routers rebooted regularly!

pacas · Fri Nov 08, 2013 5:33 pm

i use this script scheduled at 04:00 nightly:

:global datum [/system clock get date];
:global time [/system clock get time];

:local percentused ((100 * [/ip route cache get cache-size]) / [/ip route cache get max-cache-size])

:log info "RouteCacheUsed: $percentused %";

:if ($percentused > 70) do={
/tool e-mail send server=10.10.10.10 to="monitoring@email" subject=[/system identity get name] from="router@email" body=(" on $datum at $time the route_cache on router got to ",[$percentused],"%" )
/system reboot ;
};

nigelvh · Tue Nov 12, 2013 8:17 am

I'm seeing this as well on a couple of RB2011's. Networking stops working, and if I get in through the console, I see:

[user@Router] > /system resource print        
                   uptime: 2w10h57m37s
                  version: 6.4
               build-time: Sep/12/2013 13:52:41
              free-memory: 95.3MiB
             total-memory: 128.0MiB
                      cpu: MIPS 74Kc V4.12
                cpu-count: 1
            cpu-frequency: 600MHz
                 cpu-load: 1%
           free-hdd-space: 110.6MiB
          total-hdd-space: 128.0MiB
  write-sect-since-reboot: 24631
         write-sect-total: 106765
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: RB2011UAS
                 platform: MikroTik
[user@Router] > ip route cache print   
      cache-size: 16383
  max-cache-size: 16384
[user@Router] > ping 8.8.8.8
HOST                                     SIZE TTL TIME  STATUS                                                           
8.8.8.8                                                 timeout                                                          
8.8.8.8                                                 timeout                                                          
8.8.8.8                                                 timeout                                                          
                                                        132 (No buffer space available)                                  
                                                        132 (No buffer space available)                                  
                                                        132 (No buffer space available)                                  
    sent=6 received=0 packet-loss=100% 

[user@Router] >

pacas · Mon Nov 18, 2013 3:41 pm

guys, are you using bgp and ipv6 ?

nigelvh · Mon Nov 18, 2013 4:47 pm

In my case, we are not presently using BGP or IPv6, we are however using OSPF over a number of wireless links, which may cause routes to be changing relatively frequently. We plan to start using BGP soon, and IPv6 may come farther down the road, but for right now, it's v4 only and OSPF.

nigelvh · Mon Nov 25, 2013 7:47 am

We've now confirmed this in RouterOS 6.6, as we upgraded one of the routers from 6.4. I've also manged to get a supout file and will be emailing it to support.

[user@Router] > /ip route cache print
      cache-size: 16383
  max-cache-size: 16384
  
[user@Router] > /system resource print
                   uptime: 1w4d14h11m49s
                  version: 6.6
               build-time: Nov/07/2013 13:04:08
              free-memory: 96.3MiB
             total-memory: 128.0MiB
                      cpu: MIPS 74Kc V4.12
                cpu-count: 1
            cpu-frequency: 600MHz
                 cpu-load: 0%
           free-hdd-space: 108.6MiB
          total-hdd-space: 128.0MiB
  write-sect-since-reboot: 54179
         write-sect-total: 168056
               bad-blocks: 0%
        architecture-name: mipsbe
               board-name: RB2011UAS
                 platform: MikroTik

iprob · Mon Feb 03, 2014 3:00 pm

I am experiencing this in 6.7 on one particular router running as an x86 VM. I was able to see the route cache completely full and networking shut down. Since it was a VM, I was able to reboot it and resolve the issue. It does happen regularly with this router. Other than a scheduled reboot, are there other workarounds?

See this thread: http://forum.mikrotik.com/viewtopic.php ... &start=100

It suggests the issue is fixed in 6.5, but I don't think it is. It is still evident in 6.7.

iprob · Mon Feb 03, 2014 4:07 pm

I read in this thread (http://forum.mikrotik.com/viewtopic.php?f=2&t=51127) that some were running dynamic protocols, and some were not. I noticed on the router that I am experiencing the issue is not running any dynamic firewalls but there are frequent PPTP connections. The PPTP server does create quite a few dynamic interfaces and routes on the router and I'm wondering if maybe that is the issue.

Is anyone else using dynamic protocols or inbound VPN connections when they experience the problem?

nigelvh · Mon Feb 03, 2014 5:21 pm

Yes, the issue is still very evident. We've got most of our network running 6.7 and it does still occur. We believe we've narrowed down the issue in our case to IPIP tunnels. We have a large number of devices that don't run IPIP tunnels, but do run dynamic protocols like OSPF and have not seen this issue. However, ALL of the devices running IPIP tunnels have run-away route caches and will eventually stop working properly unless rebooted.

For now we have implemented a reboot script that prevents the device from hanging altogether, but is still VERY annoying in the bit about our routers rebooting every so often. I will see about upgrading one of our IPIP enabled routers with 6.9 to see if it remains an issue.

We've also added monitoring of the route cache to our Cacti server, which does help to keep an eye on these issues. I wasn't able to find an OID for it, so I implemented it via a script that logs in with a read only user and parses the output for Cacti.

timk · Mon Mar 10, 2014 1:53 am

I am hitting this bug too, running RouterOS 6.10.

I have three RB2011s with similar configs, however the problematic one is running an L2TP/IPSec VPN and the other two are not.

I obtained this info via the serial console, all other networking is unavailable:

uptime: 2d20h16m
version: 6.10

cache-size: 16384
max-cache-size: 16384

Other two routers which appear stable:

uptime: 2w4d17h15m15s
version: 6.10

cache-size: 716
max-cache-size: 16384

uptime: 2w3d15h19m13s
version: 6.10

cache-size: 53
max-cache-size: 16384

A reboot will fix the problem for about a day.

Cheers

iprob · Fri Apr 04, 2014 1:36 am

This problem appears to be much worse in 6.11. See the thread here: http://forum.mikrotik.com/viewtopic.php ... 43#p419643.

All indications are that it is a bug in the PPP code from MikroTik. My problem routers are the ones with L2TP set up.

nigelvh · Mon May 05, 2014 8:56 pm

It looks like this issue is still evident in 6.12. I had hoped with the changelog noting:

*) l2tp - fixed "no buffer space available" problem;

That this issue might be resolved for tunnels in general, but it appears the issue is still around on devices using IPIP / IPsec tunnels.

Mikrotik: PLEASE PLEASE PLEASE PLEASE PLEASE FIX THIS. Or at least give us some form of communication that you're working on the issue!! I've emailed support twice about this issue and beyond the automatic response have gotten NO communication back regarding this.

Hotz1 · Mon Jun 12, 2017 1:55 pm

"The more things change, the more they stay the same." Welcome to 2017.

After 4 years in production, we have one RB2011 that suddenly developed this same problem shortly after upgrading from 6.37.5 to 6.39.1. (Upgrading to 6.39.2 didn't help.)

MT looked at the supout and pointed out that the route cache was full--which I hadn't even heard of before this--but their explanations (hacking or torrents?) don't fit the situation. This router manages traffic for a hotel, and is at least three private-IP hops away from an edge router with public IP. It isn't using BGP, IPv6, or PPP; just OSPF, bridges, VLANs, and some queues to prioritize SIP--just like we have in every building on our network. And in this case, there is only one route out of the building, so routes shouldn't be changing; everything is either on a local interface, or goes through a single interface on a rooftop backhaul router.

Taking their advice, I set / ip settings route-cache=no. cache-size would sit in the single digits for hours, then suddenly start climbing into the thousands, and over the course of ~15 minutes fill completely (in this case, 16K). With / ip settings route-cache = no!

I downgraded everything in the building to 6.37.5, and replaced the 2011 with an 1100XH2 (cache size 512K). 16 hours later, I'm seeing cache sizes in the 2K-3K range (with route-cache=yes). It is set up to start sending me emails if it exceeds 10K. If this proves stable, I'll upgrade to 6.39.2 and post an update.

Fingers crossed that it keeps working (so we have a stable fallback position), but I'm never thrilled when we don't have an explanation for a problem.

dgnevans · Sun Aug 13, 2017 8:50 pm

Recently I experience this issue on the ccr 1009 7G and the ccr 1009 8G. I had a number of tunnels flapping due to service providers instabilities. I am running OSPF so that routes come back in and out automatically. The cache grows quickly when this happens and the router stops responding on all ports. Disabling IP route cache resolves the issue. Tried different router os versions. no change. Currently running on the latest Bugfix version. With tunnels enabled and cache enabled the router cache sits at around 90. if the tunnels drop in and out it quickly grows. I disabled the cache again once I saw it go above 1000 in less than 30 seconds. It would be great if a solution could be found for this.

Hotz1 · Sun Aug 13, 2017 9:09 pm

Recently I experience this issue on the ccr 1009 7G and the ccr 1009 8G. I had a number of tunnels flapping due to service providers instabilities. I am running OSPF so that routes come back in and out automatically. The cache grows quickly when this happens and the router stops responding on all ports. Disabling IP route cache resolves the issue. Tried different router os versions. no change. Currently running on the latest Bugfix version. With tunnels enabled and cache enabled the router cache sits at around 90. if the tunnels drop in and out it quickly grows. I disabled the cache again once I saw it go above 1000 in less than 30 seconds. It would be great if a solution could be found for this.

I now have this problem on a RouterBoard that provides backhaul to a motel via 5GHz. It's running OSPF just like the rest of our equipment, including devices on other buildings doing the same thing with the same configuration.

With route-cache enabled, the cache grows consistently by about 1,000 per day. So it can run for ~2 weeks before it has to be rebooted, to clear the route cache.

With route-cache disabled, the cache grows consistently by about 1 per second. 4.75 hours later, like clockwork, it's full and has to be rebooted. Ran it like this for days at a time, and saw no more than 15 minutes variation in uptime. :-/

Something is wrong there. But without being able to view the cache contents for clues, and without any way to purge it without rebooting, what to do? I leave route-cache enabled, and schedule a script to run daily at 3am: Check the cache size, and if it's over 14000/16384, reboot. It's a band-aid, but at least it's an automatic band-aid.

dgnevans · Sun Aug 13, 2017 9:20 pm

My routers with OSPF max at around 154 - 200 routes cached with ip route cache disabled. How many devices do you have on either side sharing ospf routes.

Hotz1 · Sun Aug 13, 2017 9:36 pm

My routers with OSPF max at around 154 - 200 routes cached with ip route cache disabled. How many devices do you have on either side sharing ospf routes.

We have a couple hundred MT devices with OSPF enabled--a couple hundred examples working perfectly well. With the exception of our edge routers, all our MTs have <1K routes cached (some <20) with route-cache enabled. And this one isn't even an especially busy one. The motel is at the end of its particular branch, so the link isn't carrying traffic for any other location: just for the motel itself. Couple Mbps each way, nothing much. (It's directly on the beach, so most guests aren't there for the Wifi.)

A dozen messages have passed between me and MT about this issue. They're convinced there must be something about the traffic at that location that is causing the route cache to continually grow. I suspect there's a bug behind some combination of settings that must be unique to this board, because the route cache fills 75x faster when route-cache is disabled. (Not 75% faster; 75 times faster.)

bertj · Thu Aug 17, 2017 10:05 am

Hi guys!
I have the same issue on my CCR1009-7G-1C-1S+ ROS 6.40.1
Support case open: #2017071322000426 and answer from support: "Waiting for problem re-appears"
My router operating properly all time till:
- 2 months and crash,
- 4 days and crash,
- 3 weeks and crash,
- now 12 days without failures.

[admin@mikrotik] > /ip route cache pr
cache-size: 143684
max-cache-size: 262144

With / ip settings route-cache =yes it's increase ~12 000 per day!

While problem occurs, router has total disconnecting on all interfaces. Interfaces status has connecting, but when I've try sent ping from CLI, the status was: 105 (No buffer...
I don't use OSPF, IPV6, BGP, only static routing and OVPN.

Waiting for solution...

pe1chl · Thu Aug 17, 2017 11:28 am

The route cache feature was removed from Linux in version 3.6, apparently because it was not very effective and it caused
problems and vulnerabilities all the time. However, RouterOS is still using an older kernel. Maybe MikroTik could
consider backporting the removal to their older kernel, or speed up the migration to a more recent kernel.

This is of course all "planned for RouterOS v7", which will have a new kernel and thus no more route cache.
But we know that is postponed all the time.

As I have written before, I think they should release a new version with updated versions of all the components that are
used without major modifications, and then introduce the new features incrementally during minor releases and optional
packages, instead of the promised "you'll get v7 and it will solve all problems because major subsystems have been
rewritten from scratch". That model just will never work, and result in indefinite postponement of v7.

Of course by now they can not even release a v7.0 that is just a v6.xx with updated kernel and tools, because of the
highly risen expectations....

giallone · Wed Oct 04, 2017 10:20 am

Any news with this?
I am experiencing the same issue on 6.40.2

andredossantos · Sat Oct 27, 2018 2:56 am

With /ip settings set route-cache=no, why is the cache-size not always zero?

Hotz1 · Sat Oct 27, 2018 3:55 am

With /ip settings set route-cache=no, why is the cache-size not always zero?

In fact, in my experience setting it to "no" tends to make the route cache fill faster.

Out of nearly 200 ROS devices on our network, only three have this problem of continuous route cache filling. They are all different hardware (RB2011, OmniTik ac, RB433GL) performing different network roles, none carrying an enormous amount of network traffic. They can be running any ROS version from 6.39 to the present, and the most it seems to affect is the rate of filling. I have seen one take two weeks, and I have seen one take five hours.

The frustrating part is that our only option of dealing with this condition, even if we see it coming, is to reboot the device.

I've been adding a new supout to my correspondence with Mikrotik almost daily, along with regular appeals to implement:

/ ip route cache clear

I have yet to receive a response that addresses this idea, or even acknowledges that this is a problem that they need to look into. I can't see how a "clear" command would be difficult to implement--and even if doing so were to take the router offline for 30 seconds to repopulate its local routes and OSPF to settle down again, that's still a big improvement over rebooting.

andredossantos · Sat Oct 27, 2018 6:31 am

If you change the hardware, keeping the same configuration, do you have the same issue?
Can you reproduce this in a VM?

Hotz1 · Sat Oct 27, 2018 7:42 am

If you change the hardware, keeping the same configuration, do you have the same issue?
Can you reproduce this in a VM?

Replacing the hardware with similar or identical devices has not eliminated the problem. It's also unclear whether this problem started as soon as the devices were installed, or whether it developed over time. We also have devices that used to have this problem, that no longer do, without any hardware or configuration changes. For these reasons, we can't rule out interactions with other devices (e.g., OSPF) nearby, as a cause.

The three devices having this problem today are all in production, so we aren't in a position to perform extensive experiments. Devices on the test bench with identical configurations don't exhibit this problem, but we also aren't prepared to duplicate the hardware environment and traffic encountered.

giallone · Sat Oct 27, 2018 9:07 am

If you change the hardware, keeping the same configuration, do you have the same issue?
Can you reproduce this in a VM?

My idea to address the issue was to move from a VM to a physical hardware, but I have the issue in both cases.
My device is a OVPN server with 10Mb/s avg traffic. It fills in less than a week.
From my experiments, when the cache reaches 25% (both VM and Physical HW, that's strange) the bug arise.
The only workaround is to schedule the following running every night:

:local uptime [/system resource get uptime];
:local date [/system clock get date];
:local time [/system clock get time];
:local identity [/system identity get name];
:local freememory [/system resource get free-memory];
:local cpuload [/system resource get cpu-load];
:local freehdspace [/system resource get free-hdd-space];
:local cache [/ip route cache get cache-size];
:local maxcache [/ip route cache get max-cache-size];
:global datum [/system clock get date];
:local percentused ((100 * [/ip route cache get cache-size]) / [/ip route cache get max-cache-size]);
:log info "RouteCacheUsed: $percentused %";
:if ($percentused > 17) do={
/tool e-mail send server="smtp.gmail.com" port=587 start-tls=yes user="XXXXXXXXX" password="XXXXXXX" from="XXXXXX@gmail.com" to="XXXXXXX@gmail.com" subject="Cache ip4you server full" body="Date: $date, Time: $time\n\nResources\n\nuptime: $uptime\nfree memory:$freememory\ncpu load: $cpuload\nfree hdd space: $freehdspace\ncache: $cache\nmax cache: $maxcache\nPercent used: $percentused %"
:log info "reboot";
/system reboot;
};

pe1chl · Sat Oct 27, 2018 12:17 pm

I have several routers, including CCR1009, operating in various different scenario's which do lots of routing and also VPN (GRE and L2TP) and I have never seen this problem.
Checking the CCRs in operation the use of the route cache is very small compared to the size.
I think there must be something particular to the configuration or usage of the affected routers.

Hotz1 · Sat Oct 27, 2018 5:22 pm

I have several routers, including CCR1009, operating in various different scenario's which do lots of routing and also VPN (GRE and L2TP) and I have never seen this problem.
Checking the CCRs in operation the use of the route cache is very small compared to the size.
I think there must be something particular to the configuration or usage of the affected routers.

Our other 150+ ROS devices don't have this problem either. Most of them operate for weeks and months at a time, and their route caches never come close to 1,000.

But three of them do. And as far as we can tell, none of them are configured in any unusual way; use any ROS feature that the other 150+ don't use; perform any particular function or carry any particular traffic that sets them apart from the rest. If there were anything distinctive about them, that's where I'd be looking. But even near-identical ROS devices nearby--the OmniTik on the roof of the next building, performing exactly the same functions; the ROS device at the other end of the 5GHz link, carrying exactly the same traffic--don't exhibit this problem.

So, until someone can figure out what particular to the configuration or usage is affecting them, our only option is to reboot when necessary; if possible, at an hour when it will not cut off someone's phone call or leave them offline for a couple frustrating minutes.

mrz · Mon Oct 29, 2018 10:56 am

Currently it is known that OVPN interface reconnects are responsible for route cache leaks.

giallone · Mon Oct 29, 2018 11:56 am

Currently it is known that OVPN interface reconnects are responsible for route cache leaks.

this is a very good news, I was not aware of this, thanks.
Do you think there is a better workaround (I mean script so schedule frequently) to avoid rebooting when cache reaches some size?

Hotz1 · Mon Oct 29, 2018 2:20 pm

Currently it is known that OVPN interface reconnects are responsible for route cache leaks.

Unfortunately in our case, we don't use OVPN anywhere.

Hotz1 · Thu Dec 06, 2018 8:43 pm

As a follow-up, we have identified two causes for our route-cache problem: one of which fills the cache quickly, and one which we believe fills the cache slowly:

We found two RBs with the same OSPF router-id in the same area. Resolving that conflict vastly reduced the rate at which certain route caches were filling: from a matter of hours to a matter of weeks.
We suspect, but have yet to confirm, that our remaining route-cache problem is caused by using dynamic WDS on links that drop and reconnect every hour or two.

cgood · Fri Mar 15, 2019 12:05 pm

Currently it is known that OVPN interface reconnects are responsible for route cache leaks.

Hello, how to fix this issue?
Mikrotik CHR -> ip route cache full 512k .. OVPN/L2TP server

MichalP · Mon Aug 12, 2019 5:06 pm

Currently it is known that OVPN interface reconnects are responsible for route cache leaks.

Is there any plans to resolve it in near future?
Routers can't even work 1 week with many ovpn tunnels... ip route cache get full and router needs restart...
@Mikrotik please react.

xootraoox · Mon Sep 23, 2019 6:01 pm

Hello, I'm running ROS 6.45.6 on CCR1009 and problem still persist... the cache increase when the OVPN clients try to connect, for example if add various clients (in my case more than 100 PPP > Secrets) and disable, the clients try to connect, this start to increase the route cache, the unique solution is a script to reboot the CCR when the cache is running low.

Please fix this issue. my clients don't stop crying.

pe1chl · Mon Sep 23, 2019 6:46 pm

Offload your OpenVPN connects to some external server until this is fixed. OpenVPN on RouterOS v6 is a joke anyway.
It should improve on v7.

avaz · Fri Nov 06, 2020 3:14 am

Hello everyone,

I have some equipment (CCR, 4011, 2011) that all have active PPP services (openvpn / pptp ..etc).

We noticed that even using version 6.47.7 which until the day of this post is the latest stable version available, we still find problems with the uncontrolled increase of the routing cache.

Does anyone know of a possible correction date or a possible trick to solve the problem (without a reboot).

Maggiore81 · Wed Jan 13, 2021 2:16 pm

I have disabled everywhere the route cache
no issues
I dont need fasttrack or fastpath so it is disabled with no issues.

Hotz1 · Wed Jan 13, 2021 3:08 pm

Even on 6.47.8, we're still getting cache overflows on the same devices. They tend to be close (topologically) to radio links that are a bit unstable, which generate numerous OSPF updates.

It was quite a while ago that we last disabled route caching, and found that the cache filled up FASTER than when it was on.

We now run a script hourly on those RBs to notify us when the route cache has exceeded a threshold, and automatically reboot at 3am. So we know the problem is still happening, but (fortunately!) on devices that aren't mission critical.

Who is online