Hi,

We started deploying 2004s into our network and have issues with one we are trying to add into our bgp core.

It rebooted every 10-14 days so we took it out and replaced in with another one. We can ran memory test and got memory errors on the first one.

This was 12 days ago....

Today the new rebooted same way making lots of issues in OSPF 0.0.0.0 to the extent we had to reboot 2 other routers running 6.45.9 to make traffic resume again. Another CCR1016 got so corrupted that /export compact, snmp etc dident work.

The second CCR2004 (6.47.1) is connected with console cable now and doing remote memory test we get errors on this one too. Broken batch or something else? Feels like too much coincidence perhaps?

Error in address=0x00000000C0004768, W=0xC0004768 R=0x00000000 X=0xC0004768
Error in address=0x00000000C000476C, W=0xC000476C R=0x00000000 X=0xC000476C
Error in address=0x00000000C0004770, W=0xC0004770 R=0x00000000 X=0xC0004770
Error in address=0x00000000C0004774, W=0xC0004774 R=0x00000000 X=0xC0004774

etc etc etc.

/M

Something like this is better sent to support@mikrotik.com to start a real case - this is a discussion forum not a proper support channel.

Hi,

I asked the question here because it's a forum. I'm fully aware that it's not a support channel.

Mikrotik has no answer for the reboots. Seems they were able to reproduce the memory tester issue so it's confirmed that the memory tester is broken.

They don't think it's related to the reboot.

The system reboots randomly but seldom. I'm trying to attach a console to get output but this is at production site.

If anyone experiences 2004 reboots and tests memory like I did it's not faulty memory being the cause.

The one we have issues with runs 3 full bgp feeds but not much traffic.

The 2004s with no bgp has not rebooted.

/Mikael

I'm having lockups on one of our two 2004s as well. It's an edge router with 4 bgp sessions (no full tables). About every 36 hours, it locks up where we can't access it via winbox, ssh and snmp stops. Sometimes it passes traffic through it, other times no traffic will pass.

We got a console server on it now. Last lockup reported nothing at all in the console but I still had console access. When I tried to get it to generate a supout.rif file via console, that failed but immediately after, it came back to life and I could generate one via winbox.

Our crash 2 days ago it was passing traffic but we lost all access to it including console. Had to pull power to reboot it.

Support ticket is open but so far, no info.

I have been told by Mikrotik the bootloader memory test has been fixed and fix will be included at next release.

As for the logging we have connected to another mikrotik at the same site and logging to locai file - hoping for a crash to happen.


/M

What are you all logging to "echo" in hopes of getting useful info in case of a crash? We have "critical, warning, health, system and event" echoing and nothing was on the console at all for our last crash.

Not from the log but supposedly there should come some crash information on the console disregarding logging settings.

Bootloader fix was included in 6.47.2

We are seeing same issues with one of 8 CCR2004s: uninvited reboots 1-2 weeks apart. Each one is running 6.47.1, each one is running BGP, so it would seem that it may be a hardware problem. SFP28 is in use on all, so it's also not the reason.

I am shipping one 2004 back under RMA and our other one reboots every 1 -2 weeks. Was on 6.47 and I just put it on 6.47.2 two days ago. Both are running BGP and OSPF.

Mikrotik is running some special debug packages on one of our routers. Its either a software bug or something deep in the hardware since they mentioned involving the CPU vendor.

Without the debug packages nothing came on console at crash time, so hoping for a new crash soon so this can be resolved. It has currently been up 7 days.

We have 24 2004s in boxes so would be nice to solve this issue.....

we are running the same debug firmware on ours. Every one of our deployed 2004s is having some sort of problem (either random reboots or crashes or both).

2004 that used to reboot every now and then seems stable so far:

I gave it something to do, so it didn't die out of boredom:
...but it seems there's either no correlation between load and reboots, or there is and load prevents reboots ;)

Anyway, my $0.03 to the case. No instabilities seen on other 2004s either.

We are seeing the same problems on two CCR 2004 our of 10 deployed. No Connection tracking enabled. Support says that unless we have the debug package installed with the console, there's no way to catch the problem. Today it happend again on the same units with version 6.47.2: it looks like an hardware issue to me, we will deploy the debug and console and see if we can help finding the problem.

We just had our 2004 crash with the debug firmware installed. The last line of the console output is:
Nothing before that for hours. After the crash, we got 2 physical link up/down messages in console (about 2 minutes after the crash). Nothing else. Router will not respond to console input and we can't log into it. Nor is it passing traffic. I have sent the debug log to support with our open ticket with them. Hopefully this console message is useful to them and they can get this fixed.

Any new info about this?


I wonder if this has something to do with it ->

What's new in 6.48beta40 (2020-Sep-14 13:34):
*) arm64 - improved reboot reason reporting in log;

That is related. It was added to help support troubleshoot this. We are running that firmware at the request of Mikrotik to gather more information when it crashes. Has not done anything yet to stop the crashing.

We also have issues with CCR2004, there is no BGP/OSPF in our case. Reboot fully random, sometimes few times per day, sometimes once in two weeks.
CPU load do not exceed 6%, average load 20-25Mbps with rare spikes to 50Mbps. ROS: 6.47.4.
We love MT devices, ...but those reboots are horrible.

Hi,

We are running 6.48beta48 on some of the 2004s that was rebooting, It seems to have solved the reboots, but it seems that we now face an issue where the routing protocols stops working instead.

Are any of you experiencing the same?

/Mikael

Just had the first unexpected reboot on our CCR2004. Running 6.47.4, no autosupout.rif on flash. Very light load, 20-30Mbit/s of traffic.

After almost a month of troubleshooting with Mikrotik, our 2004's have been RMAd. I would strongly encourage all of you with 2004s that are rebooting, freezing, etc to open support tickets with Mikrotik. They need data to solve this problem and the more data, hopefully the faster it can be solved.

I opened a ticket with Mikrotik and sent a support file and have not heard back. Did they require that you upgrade to the beta before RMA?

We did not go to ROS 7 beta but we did run a couple different version 6 betas as part of the test prior to RMA.

Experienced our second crash in one week today. No autosupout.rif was generated, sent manual supout.rif to support@mikrotik.com.

Hi,

Right now we have one 2004 not reachable by winbox, but routing works. It runt 6.48beta48.

The same box dropped ospf last time it had issues.

Previously is rebooted when it had issues and I think that was actually better.

We swapped this with another 2004 so its either a general hardware issue or a software isssue.

Did any of you get improvements after swapping for RMAd boxes as this might have been an issue with early production?

/Mikael

Hi!

Did anyone here get any more updates from MT on this? We are seeing processor hangs in the beta and reboots in stable still. We have a lot of 2004s waiting to go to production, so its frustrating :)

/M

Nothing with us. RMA'd two of ours and bought CCR 1036s to go in their place. My gut tells me it's going to be a while.

We have been having similar issues, our previous 2004 just rebooted randomly, we also had problems with PPPoE accounts connecting showing Dynamic but not Running, we then replaced it and it went well for about two weeks. We just had the same issue with the PPPoE accounts again and had to do a reboot to get it working.

Reading this topic, i am on hold, before buying... I need 6 of them

We were told the new beta solved the issue, but we did some fast testing and the whole router behaves strange and becomes unreachable even with mac telnet and romon when OSPF is running.

So test very carefully....

6.47.8 (stable) also includes the fix for the freeze/reboot issue and I can confirm that it installs fine and no issues with OSPF/BGP.
We are now testing it on 3 units starting today.

6.47.8 (stable) also includes the fix for the freeze/reboot issue and I can confirm that it installs fine and no issues with OSPF/BGP.
We are now testing it on 3 units starting today.

Same here, we started with two units (doing OSPF and BGP).

regards,
Glueck

We have rolled out on a couple too.

I did see this in the release that someone seems to have experienced a reboot even with the new software.

I got the feeling also that BGP convergece with full tables was much slower, but dident time it. Ill try to get some time in the lab to reproduce if possible.

/M

I need to upgrade the CCR2004 in my lab and see if the stability improves. We've used them for several clients but have also had some stability issues.

I think this router will be amazing after a few more months of bug fixes from MikroTik. This is pretty typical of a new router release...it takes a little while to get stable.

4 months since the issue started - so im hoping the "amazing" starts now :)

So far its been stable for us since upgrade.

/M

I have two 2004s (and a few spares on the shelf), which are neighbors to each other in a small (mid 20s) mpls network, not a lot of throughput, not a lot of firewall rules(typical usage ~1%). Started with 6.47, tried 6.47.7 (which was better) but every so often they would just reboot. And typically within 12 hours of each other. I cant have them doing that, so Ive also ordered a few 1036s as well, and some should be here tomorrow. I also hadnt previously seen when I chose them, the limitations with using rj01 sfps (close proximity and power issues within sfp chassis).

Before the one rebooted this morning, I had logged into it ( I had made a few changes just prior) and saw that overall cpu utilization had jumped to 25% (one of the cores was maxed). It did finally crash and reboot, which seemed to take about 5 minutes for it to come back. Quite honestly I thought I was going to have to netinstall. Before it crashed, I tried to get a support file, but it crashed before it could complete. I had sent in autosup files into them, and they suggested newer versions for debug reasons. Chalk it up to experience I guess, and hang on to them until they become a better work horse.

I have two 2004s (and a few spares on the shelf), which are neighbors to each other in a small (mid 20s) mpls network, not a lot of throughput, not a lot of firewall rules(typical usage ~1%). Started with 6.47, tried 6.47.7 (which was better) but every so often they would just reboot. And typically within 12 hours of each other. I cant have them doing that, so Ive also ordered a few 1036s as well, and some should be here tomorrow. I also hadnt previously seen when I chose them, the limitations with using rj01 sfps (close proximity and power issues within sfp chassis).

Before the one rebooted this morning, I had logged into it ( I had made a few changes just prior) and saw that overall cpu utilization had jumped to 25% (one of the cores was maxed). It did finally crash and reboot, which seemed to take about 5 minutes for it to come back. Quite honestly I thought I was going to have to netinstall. Before it crashed, I tried to get a support file, but it crashed before it could complete. I had sent in autosup files into them, and they suggested newer versions for debug reasons. Chalk it up to experience I guess, and hang on to them until they become a better work horse.

Hi this seems to have been fixed in latest stable - 6.47.8

Hi,
Our CCR2004 has rebooted without proper shutdown during night with 6.47.8. Not fixed yet. It's not a power problem because this device is in a rack with dual power circuit.
Reboot happens after 3.8 days of running time the 04 December

Hi this seems to have been fixed in latest stable - 6.47.8

Thanks for the update, but with the old version it would last up to ~20 some days before it would quit. Sometimes as quick as 8 or so.I do hope they figure it out soon, I have about 10 of these now that will hold paper down, until they are proven again.

Hi,
Our CCR2004 has rebooted without proper shutdown during night with 6.47.8. Not fixed yet. It's not a power problem because this device is in a rack with dual power circuit.
Reboot happens after 3.8 days of running time the 04 December

Thats what I was afraid of. Were you able to log into it shortly before it rebooted? Or do you keep cpu statistics? If so were they higher than normal before reboot?
Hi this seems to have been fixed in latest stable - 6.47.8

Thats what I was afraid of. Were you able to log into it shortly before it rebooted? Or do you keep cpu statistics? If so were they higher than normal before reboot?

Yes, of course, nothing really special. It's not related to trafic, because it happens during night when it's much lower.

We just had a 2004 have all interfaces go down and up again. Also seems OSPF dident fully recover, so in the end we rebooted it.

/Mikael

Went and looked at the uptimes of our six longest-deployed 2004s:

11w3d18h44m51s, S/N D4F00...
10w6d18h10m42s, S/N D4F10...
8w1d16h32s, S/N D4F10...
5w1d16h1m47s, S/N D4F10...
1w6d17h29m41s, S/N D4F00...
1d22h28m59s, S/N C8A60...

The last two have clearly rebooted without human intervention, not 100% sure about the rest.

Thats what I was afraid of. Were you able to log into it shortly before it rebooted? Or do you keep cpu statistics? If so were they higher than normal before reboot?

Yes, of course, nothing really special. It's not related to trafic, because it happens during night when it's much lower.

CCR2004_Uptime.jpg

CCR2004_trafic.jpg

CCR2004_CPU.jpg

CCR2004_Memory.jpg

CCR2004_winbox.jpg

I was more curious if yours had a spike in "unclassified" on one of the cores like I caught mine doing this last time.

Have you guys seen this about the S+RJ10? Not sure if it affects you, you may be using fiber, I know I didnt catch it until recently.

https://wiki.mikrotik.com/wiki/S%2BRJ10 ... l_guidance

We've experienced all that everyone has listed including;

• Random Reboots

• Interface Resets

• System Deadlocks

• System Slow Downs

We've tried special firmware's, Debugging firmware's, Logging console outputs for months and unfortunately this issues caused out CCR1072's to be unstable and experience similar issues.

We've pulled all CCR2004's from our network and we have stability back again however due to the lack of support and willingness to fix these issues as a priority and getting support responses back like "Are you sure it rebooted" has caused us to make the rash decision to drop Mikrotik from our Core network and spend Hundreds of Thousands of dollars and switch to Juniper.

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.

What version were you on previously?

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.

What version were you on previously?

6.47.4

Upgraded one of my CCR2004s to 6.47.8 and just saw an unexpected reboot.

What version were you on previously?

6.47.4

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.

The longest uptime I saw was 14 or so days. I did NOT upgrade the routerboard firmware yet, just the OS. I have gone ahead and ordered a different CCR model to replace the 2004.

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.

The longest uptime I saw was 14 or so days. I did NOT upgrade the routerboard firmware yet. I have gone ahead and ordered a different CCR model to replace the 2004.

I think my longest was ~mid 20 days. Dont think I ever reached anything above 30. Was there a reason you upgraded to the new version?

How long was 6.47.4 stable for you? I went right from 6.47 -> 6.47.7, then 6.47.8 and experienced the reboots in all.

The longest uptime I saw was 14 or so days. I did NOT upgrade the routerboard firmware yet. I have gone ahead and ordered a different CCR model to replace the 2004.

I think my longest was ~mid 20 days. Dont think I ever reached anything above 30. Was there a reason you upgraded to the new version?

The new version was said to have improvements in stability for the arm based devices.

Hi,

We havent seen any reboots in 6.47.8 but one 2004 dropped and reconnected all intefaces same second. Was asked by Mikrotik if it was a cable issue - which I dont think since nobody was in the rack or the other sites it connects to. Also disconnecting and reconnecting 8 cables same second would be the fastest person alive.

Besides this we do have some oddity in the ospf in network since a couple versions. Having more than 500 mikrotiks in my network this is honestly not what we are used to. Its been quite stable for years.

Im also surprised by the speed Mikrotik is attacking these issues - Its been months. And when you have an issue like that you need to talk with your customers, not just tell them it will be fixed when found even if it was done in a very polite way.

/M

Hi,

We havent seen any reboots in 6.47.8 but one 2004 dropped and reconnected all intefaces same second. Was asked by Mikrotik if it was a cable issue - which I dont think since nobody was in the rack or the other sites it connects to. Also disconnecting and reconnecting 8 cables same second would be the fastest person alive.

Besides this we do have some oddity in the ospf in network since a couple versions. Having more than 500 mikrotiks in my network this is honestly not what we are used to. Its been quite stable for years.

Im also surprised by the speed Mikrotik is attacking these issues - Its been months. And when you have an issue like that you need to talk with your customers, not just tell them it will be fixed when found even if it was done in a very polite way.

/M

Did you just upgrade the RouterOS or did you upgrade the firmware as well?

Did you just upgrade the RouterOS or did you upgrade the firmware as well?

Both

/Mikael

Hi,

We havent seen any reboots in 6.47.8 but one 2004 dropped and reconnected all intefaces same second. Was asked by Mikrotik if it was a cable issue - which I dont think since nobody was in the rack or the other sites it connects to. Also disconnecting and reconnecting 8 cables same second would be the fastest person alive.

Besides this we do have some oddity in the ospf in network since a couple versions. Having more than 500 mikrotiks in my network this is honestly not what we are used to. Its been quite stable for years.

Im also surprised by the speed Mikrotik is attacking these issues - Its been months. And when you have an issue like that you need to talk with your customers, not just tell them it will be fixed when found even if it was done in a very polite way.

/M

I may have seen something similar.

When an even happens on one ospf port, it sometimes seems to happen simultaneously on one or several other ports. And the site I've had this happen at has 24/7 camera monitoring, so I know nobody touched ours either.

I may have seen something similar.

When an even happens on one ospf port, it sometimes seems to happen simultaneously on one or several other ports. And the site I've had this happen at has 24/7 camera monitoring, so I know nobody touched ours either.

If you feel like it then please mail Mikrotik support and reference SUP-35544 which is the ticket we have with this behavior.

/M

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.

Have y'all had these issues with the v7 betas?

I know it's a hard question since v7's own bugs can mask this. But as the 2004 was designed for ROS7, I wonder if some of these problems are specific to ROS6.

Unfortunately I don't have any spares to try v7 on...

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.

How much? ;-)

Have y'all had these issues with the v7 betas?

I know it's a hard question since v7's own bugs can mask this. But as the 2004 was designed for ROS7, I wonder if some of these problems are specific to ROS6.

Unfortunately I don't have any spares to try v7 on...

Dont try beta3 - It gives massive pingloss for 2004s both on IP and ARP. Same with latest 6.48beta.

/M

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.

What was the reason for rejecting?

/Mikael

I've replaced the CCR2004 with a CCR1072 and it's running like a charm !

I hope that Mikrotik will soon consider making a premium hardware product line in parallel of the cheap hardware race !
RouterOS has so much potential, it's a shame that this software has no more premium rock solid hardware...

I've replaced the CCR2004 with a CCR1072 and it's running like a charm !

I hope that Mikrotik will soon consider making a premium hardware product line in parallel of the cheap hardware race !
RouterOS has so much potential, it's a shame that this software has no more premium rock solid hardware...

+1

You are atacking the wrong problem: this looks like software related, not hardware. I agree that it should be more stable, but (barring some defective units) this kind of complain is usually solved down the line, with a new RoS version.

I fully agree this is software, but actually MT was a little unsure of this in the beginning so even hardware could be solved with software at some cost I suspect.

I have two wishes this year - First is hoping for stable 2004s on 6.47.x soon and of course stable ROS 7 for new year, but most importantly stable 2004s.

/M

If you feel like it then please mail Mikrotik support and reference SUP-35544 which is the ticket we have with this behavior.

/M

Ill have to put them back in service to do that lol. Ive since replaced them. I will have to try them and either with the older version you started with, or wait for a newer release.

You are atacking the wrong problem: this looks like software related, not hardware. I agree that it should be more stable, but (barring some defective units) this kind of complain is usually solved down the line, with a new RoS version.

No, it shouldn't happen - You are quite right about it. But is the software testing that needs improving...

Wrong to you or not, when you've try a few versions of software, and NEED your equipment to continue running, you have to look for alternative hardware until they have something stable for that hardware. When the problem goes away, you know it was a combination of that hardware/software. When you don't have a unit that's stable, you have to try something else.

Have y'all had these issues with the v7 betas?

I know it's a hard question since v7's own bugs can mask this. But as the 2004 was designed for ROS7, I wonder if some of these problems are specific to ROS6.

Unfortunately I don't have any spares to try v7 on...

Can't run v7 here, running mpls/vpls, and from what I have seen thats not supported yet. Hopefully soon!

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.

What was the reason for rejecting?

/Mikael

Nothing wrong with the hardware....

We tried to RMA our units as not fit for use and it was rejected so we now have 4 of them just sitting collecting dust.

What was the reason for rejecting?

Nothing wrong with the hardware....

Thats probably true and they will claim software is without warranty I guess which is the real issue. While correct according to license I think they dont understand that we are returning customers. Hopefully.,,

We have 14 of them if it makes you feel better - One we use to test stability.

/M

Hi again,

Today we again experienced a drop of interfaces on a 2004. The drop this time was not all the interfaces, but still all ospf was lost. The other 2004s we are running in test has less traffic and no problems, but one thing I realized is that we are using an SFP28 (DAC Cable) port and that the sfpplus10 is a S-RJ01 which draws much more power than the rest of the interfaces that are normal optical SFPs.

As you can see this happened very fast but because of BGP convergence with CCRs plus no graceful restart it becomes a much longer outage.

Has anyone else seen something like this? We have on the positive side seen no reboots with 6.47.8

13:49:03 snmp,warning timeout while waiting for program 20
13:52:18 interface,info sfp-sfpplus10 link down
13:52:18 interface,info sfp28-2.transit.AS65600 link down
13:52:18 route,ospf,info OSPFv2 neighbor 185.230.33.240: state change from Full to
Down
13:52:18 route,ospf,info OSPFv2 neighbor 192.168.33.235: state change from Full to
Down
13:52:18 route,ospf,info OSPFv2 neighbor 192.168.33.8: state change from Full to Do
wn
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.1
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.8
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.3
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.6
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.234
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=185.230.33.40
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=185.230.33.240
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.231
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.235
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=5.157.4.1
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=193.110.13.254
13:52:18 route,bgp,error HoldTimer expired
13:52:18 route,bgp,error RemoteAddress=192.168.33.2
13:52:18 route,bgp,error Unexpected UPDATE message
13:52:18 route,bgp,error RemoteAddress=192.168.33.3
13:52:18 route,bgp,error Unexpected UPDATE message
13:52:18 route,bgp,error RemoteAddress=192.168.33.235
13:52:18 route,bgp,info Connection terminated
13:52:18 route,bgp,info RemoteAddress=5.157.4.1
13:52:18 route,bgp,info Connection opened by remote host
13:52:18 route,bgp,info RemoteAddress=193.110.13.254
13:52:18 route,bgp,info Connection opened by remote host
13:52:18 route,bgp,info RemoteAddress=5.157.4.1
13:52:18 route,ospf,info Discarding Database Description packet: wrong neighbor st
ate
13:52:18 route,ospf,info state=Down
13:52:18 route,ospf,info Discarding Database Description packet: wrong neighbor st
ate
13:52:18 route,ospf,info state=Down
13:52:20 interface,info sfp28-2.transit.AS65600 link up (speed 10G, full duplex)
13:52:21 interface,info sfp-sfpplus10 link up (speed 1G, full duplex)
13:52:40 route,bgp,info Connection opened by remote host
13:52:40 route,bgp,info RemoteAddress=192.168.33.234
13:52:41 route,bgp,info Connection opened by remote host
13:52:41 route,bgp,info RemoteAddress=185.230.33.240
13:52:41 route,bgp,info Connection opened by remote host
13:52:41 route,bgp,info RemoteAddress=185.230.33.40
13:52:42 route,bgp,info Connection opened by remote host
13:52:42 route,bgp,info RemoteAddress=192.168.33.3
13:52:42 route,bgp,info Connection opened by remote host
13:52:42 route,bgp,info RemoteAddress=192.168.33.6
13:52:44 route,bgp,info Connection opened by remote host
13:52:44 route,bgp,info RemoteAddress=192.168.33.235
13:52:45 route,bgp,info Connection opened by remote host
13:52:45 route,bgp,info RemoteAddress=192.168.33.1
13:52:47 route,bgp,info Connection opened by remote host
13:52:47 route,bgp,info RemoteAddress=192.168.33.231
13:53:20 route,bgp,info TCP connection established
13:53:20 route,bgp,info RemoteAddress=192.168.33.8
13:53:20 route,bgp,info TCP connection established
13:53:20 route,bgp,info RemoteAddress=192.168.33.2
13:53:20 route,bgp,info Connection opened by remote host
13:53:20 route,bgp,info RemoteAddress=192.168.33.8

Hi again,

Today we again experienced a drop of interfaces on a 2004. The drop this time was not all the interfaces, but still all ospf was lost. The other 2004s we are running in test has less traffic and no problems, but one thing I realized is that we are using an SFP28 (DAC Cable) port and that the sfpplus10 is a S-RJ01 which draws much more power than the rest of the interfaces that are normal optical SFPs.

As you can see this happened very fast but because of BGP convergence with CCRs plus no graceful restart it becomes a much longer outage.

Has anyone else seen something like this? We have on the positive side seen no reboots with 6.47.8

13:49:03 snmp,warning timeout while waiting for program 20

It was my understanding the newer version was supposed to have numerous snmp fixes. I saw that issue lock myself out of a few 1100ahx4's of mine, requiring reboot to regain access.

When mine dropped bfd/ospf connectons, it was using fiber sfp's. Even the two routers connected together with a short fiber jumper. (The other was a longer run to a separate building, though underground and a dark fiber link) I have previously found out that the RJ-01's dont seem to like to hold stable, when used to connect between two routers (RJ-01s on each end), hence why I switched that to fiber. But they seem fine going to a standard ethernet port on one end of the cable. I just didnt have enough fiber sfps on original install, but had RJ-01s.

I have also been experimenting with passive DACs, and they seem better, but I also get them to mess up (ospf/bfds) with the 2004s. (Running 6.47.8) Though I have been pushing them harder for testing purposes.

When mine dropped bfd/ospf connectons, it was using fiber sfp's. Even the two routers connected together with a short fiber jumper. (The other was a longer run to a separate building, though underground and a dark fiber link) I have previously found out that the RJ-01's dont seem to like to hold stable, when used to connect between two routers (RJ-01s on each end), hence why I switched that to fiber. But they seem fine going to a standard ethernet port on one end of the cable. I just didnt have enough fiber sfps on original install, but had RJ-01s.

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.

I note you write dropped ospf/bfd but did your interfaces go down too?

/M

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.

I note you write dropped ospf/bfd but did your interfaces go down too?

/M

I have over 100 wireless microwave paths in my network that spans across the state, I cant live without bfds! Im going from memory, but I believe I have seen the interface go down, but IIRC it wasnt as common as the ospf/bfd drops. IIRC when the interface went down there was usually a RJ-01 involved. Id really like to buy another ~20 or so of these 2004s, they really seem like a nice router, especially at their price point, but I need the reliability for constant voice traffic.

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.

BFD typically works fine on CHR, and most ARM based Mikrotik's. I use it with no issue on CHR & RB4011. I haven't picked up a 2004 yet as I typically let them stabilize a year before putting them in production.

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.

BFD typically works fine on CHR, and most ARM based Mikrotik's. I use it with no issue on CHR & RB4011. I haven't picked up a 2004 yet as I typically let them stabilize a year before putting them in production.

It might be tilera specific -

> -----Original Message-----
> From: Maris B. [MikroTik Support] <support@mikrotik.com>
> Sent: 04 January 2019 11:05
> Subject: Re: [Ticket#2019010322004734] Router acting wierdly
>
> Hello,
>
> Currently there are known problems with BFD on CCR series routers. I would suggest
> to turn off BFD on those devices until problem is resolved.
>
> Best regards,
> Maris B.
>
> 01/03/2019 23:18 - Mikael Hugo wrote:
>
> > Hi,
> >
> > See this autosupout. The router started loosing OSPF neighbours ang
> > giving BFD errors on multiple links with some hours inbetween.
> >
> > We upgraded to .10 and rebooted.

and lastly a follow up on the progress of fixing it from 24th july 2020.

There were no changes regarding BFD in ROS v6.
It may work and it may not work in specific setups, so general recommendation is not to use it in ROSv6.
Māris B.

/Mikael

We have been advised against using bfd in Mikrotiks by Mikrotik helpdesk. Its been broken for a long time and will not be fixed in 6.x was the answer when I asked again some months ago.

BFD typically works fine on CHR, and most ARM based Mikrotik's. I use it with no issue on CHR & RB4011. I haven't picked up a 2004 yet as I typically let them stabilize a year before putting them in production.

It might be tilera specific -

> -----Original Message-----
> From: Maris B. [MikroTik Support] <support@mikrotik.com>
> Sent: 04 January 2019 11:05
> Subject: Re: [Ticket#2019010322004734] Router acting wierdly
>
> Hello,
>
> Currently there are known problems with BFD on CCR series routers. I would suggest
> to turn off BFD on those devices until problem is resolved.
>
> Best regards,
> Maris B.
>
> 01/03/2019 23:18 - Mikael Hugo wrote:
>
> > Hi,
> >
> > See this autosupout. The router started loosing OSPF neighbours ang
> > giving BFD errors on multiple links with some hours inbetween.
> >
> > We upgraded to .10 and rebooted.

and lastly a follow up on the progress of fixing it from 24th july 2020.

There were no changes regarding BFD in ROS v6.
It may work and it may not work in specific setups, so general recommendation is not to use it in ROSv6.
Māris B.

/Mikael

There have been some ospf changes from 6.47 to 6.47.7 and 6.47.8. I only recently noticed when setting up some test environment equipment, that it no longer complains about l2mtu mis-match between neighbors anymore. As you likely know, it will cause you issues if they dont match, that isnt apparent at first. I went to update that in the thread about 6.47.8 but it appears to be locked. I am not sure which version killed it, I went straight from 6.47 to 6.47.7 and then to .8 when they had arm stability upgrades in the change log. I am not sure what all other ospf changes were made, but I had realized that one.

I have ospf\bfds running on few CCR (tile) processor devices, and havent noticed issues, until the 2004 (which is arm64 actually)

Hi,

Are any of you experiencing reboots still? We actually havent had any on 6.47.8 yet and the interfaces going down could have been from an RJ45 sfp plug which does makes me wonder about power in the sfp cages on the 2004, but it wasent even connected to anything so was easy to remove.

/Mikael

Hi,

Are any of you experiencing reboots still? We actually havent had any on 6.47.8 yet and the interfaces going down could have been from an RJ45 sfp plug which does makes me wonder about power in the sfp cages on the 2004, but it wasent even connected to anything so was easy to remove.

/Mikael

I have one left in the field, and it has 6.47.8, I just looked and it has an uptime of 20 days. This has been about the upper limit from before. This device not really being used hard at the moment, though it is part of a smaller ospf/mpls network. We shall see if it exceeds ~35 days or so. Its going to become more important soon, so I will start to monitor it a little more closely. The ports have gone up and down some, but thats because the backhaul has been changing, and equipment being moved around there, so those details are not worth watching at the moment, unfortunately.

We've had one reboot since being on 6.47.8

Device was installed on the 12th this month and rebooted on the 14th :(

We've had one reboot since being on 6.47.8

Device was installed on the 12th this month and rebooted on the 14th :(

Do you have any logging running? CPU performance, etc? Traffic?

What sfp's are you using?

We've had one reboot since being on 6.47.8

Device was installed on the 12th this month and rebooted on the 14th :(

Do you have any logging running? CPU performance, etc? Traffic?

What sfp's are you using?

2 x 10gbps 10Gtek SFP+ MM Modules in a LAG

Logs show nothing per normal on Disk or Syslog just "Router was rebooted without a proper shutdown"

Syslog: Device rebooted after 2 days 14 hours 33 minutes -> 202s

2 x 10gbps 10Gtek SFP+ MM Modules in a LAG

Sorry to hijack the thread, but what's the performance like on that LAG, and the CPU load?

I've been reluctant to set up bonding on my 2004s as its done in software vs my current setup of hardware bonding on a CRS317.

Still seeing reboots. Max uptime so far on 6.47.8 is 8 days

Still seeing reboots. Max uptime so far on 6.47.8 is 8 days

We have three of these in production no bgp just OSPF routing and vlan, we still seeing random reboots every X days....

Hi,

Ours have not yet rebooted - all have full BGP - 25 days since last.

We run "simple" ospf/bgp routing. Nothing else. Not too many interfaces (all optical) in either and only 500 mbit of traffic.

The ones with reboots - what differs from ours?

/M

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.

What types of sfp's is yours using ?

All fs.com optics in ours.

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.

What types of sfp's is yours using ?

I am using the mikrotik gigabit RJ45 SFPs (S-RJ01)

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.

What types of sfp's is yours using ?

I am using the mikrotik gigabit RJ45 SFPs (S-RJ01)

Do you notice any of your interfaces dropping (going up and down) randomly when not expecting it?

On a positive note, the one I still have in service is up to 26 days of uptime.

Mine is running an apartment complex providing internet to tenants. Nothing fancy. Just a few VLANs, NAT and firewall rules. Peak traffic is about 500mbps.

What types of sfp's is yours using ?

I am using the mikrotik gigabit RJ45 SFPs (S-RJ01)

Do you notice any of your interfaces dropping (going up and down) randomly when not expecting it?

On a positive note, the one I still have in service is up to 26 days of uptime.

We did with S-RJ01 on the only 2004 that had one and disabled it because it wasent even connected. We have had no interface drops since then.

Im reading in the changelog of 6.48 and im not sure why it states arm and arm64 stability since these were already in 6.47.8. Does anyone know if they fixed even more stuff?

/Mikael

Im reading in the changelog of 6.48 and im not sure why it states arm and arm64 stability since these were already in 6.47.8. Does anyone know if they fixed even more stuff?

+1

Hi,

We had 2 reboots on 6.47.8 in the last 24 hours. Seems they last longer but still reboots.

/Mikael

Hi,

We had 2 reboots on 6.47.8 in the last 24 hours. Seems they last longer but still reboots.

/Mikael

Mine was going strong until you said that, it died 2.5 hours ago lol, time to pull it I suppose. Has anyone had better success 6.46 long term versions?

Mine was going strong until you said that, it died 2.5 hours ago lol, time to pull it I suppose. Has anyone had better success 6.46 long term versions?

Please send ticket to Mikrotik too so they dont think its just me :)

6.46.x is worse im afraid.

/M

Sent another one in.

SUP-37419 (new)
SUP-35544 (yours)
SUP-30924 (my previous)

Sent another one in.

SUP-37419 (new)
SUP-35544 (yours)
SUP-30924 (my previous)

Yeah and sent in some more too - one asking if 6.48 does in fact solve anything more but got a wierd response so I've asked for clarification.

No responses on the main issue in the reboot.

/M

Sent another one in.

SUP-37419 (new)
SUP-35544 (yours)
SUP-30924 (my previous)

Yeah and sent in some more too - one asking if 6.48 does in fact solve anything more but got a wierd response so I've asked for clarification.

No responses on the main issue in the reboot.

/M

I havent tried 6.48, but the thread from 6.48 doesnt look stable at all.

I pulled mine last 2004 from service for now until they are stable. I think I have about 6 of the dang things. They suggested custom firmware for them, but I cant let it keep running like that. I will have to setup a separate test setup to do so.

I havent tried 6.48, but the thread from 6.48 doesnt look stable at all.

I pulled mine last 2004 from service for now until they are stable. I think I have about 6 of the dang things. They suggested custom firmware for them, but I cant let it keep running like that. I will have to setup a separate test setup to do so.

Ive been watchinh the 6.48 thread too and it doesent look too good.

Did they suggest custom firmware recently? The only feedback ive been really recieving is that it should be fixed and then no more response on the older tickets.

If it makes you feel any better we have 20 of them :)

I havent tried 6.48, but the thread from 6.48 doesnt look stable at all.

I pulled mine last 2004 from service for now until they are stable. I think I have about 6 of the dang things. They suggested custom firmware for them, but I cant let it keep running like that. I will have to setup a separate test setup to do so.

Ive been watchinh the 6.48 thread too and it doesent look too good.

Did they suggest custom firmware recently? The only feedback ive been really recieving is that it should be fixed and then no more response on the older tickets.

If it makes you feel any better we have 20 of them :)

Yes, they suggested custom firmware and wanted serial out information. Though the response was unfortunately after I pulled the unit from service. This unit was located about two hours from my office. A little too far to gamble. I had two in a redundant setup that were ten minutes from my office, which would have been more convenient. But when they started to reboot within hours of each other, I just couldn't risk it.

Yes, they suggested custom firmware and wanted serial out information. Though the response was unfortunately after I pulled the unit from service. This unit was located about two hours from my office. A little too far to gamble. I had two in a redundant setup that were ten minutes from my office, which would have been more convenient. But when they started to reboot within hours of each other, I just couldn't risk it.

Please give them my ticket id. We have a couple that can run the custom firmware.

Seems the MT support suggests very different things depending on who picks up the ticket.

Yes, they suggested custom firmware and wanted serial out information. Though the response was unfortunately after I pulled the unit from service. This unit was located about two hours from my office. A little too far to gamble. I had two in a redundant setup that were ten minutes from my office, which would have been more convenient. But when they started to reboot within hours of each other, I just couldn't risk it.

Please give them my ticket id. We have a couple that can run the custom firmware.

Seems the MT support suggests very different things depending on who picks up the ticket.

I had given yours and my prior. I wasnt thinking at the time or I would have update my prior. So I am at fault as well.

Hi,

Got confirmation that the fixes are in 6.47.8 so no additional fixes in 6.48.

Mikrotik is under the impression this is fixed so please keep reporting if it crashes for you.

We had 2 reboots with 6.47.8, but it's much better and some routers has been up for 38 days now.

/Mikael

Two crashes so far on 6.48 - doesn't seem fixed to me. Hugely disappointing.

Same here, 6..48 and a lot of reboots.

Same here, 6..48 and a lot of reboots.

We had 2 reboots on 6.47.8 14 days ago in same day for 2 routers. After that nothing. We don't run 6.48.

Are you making tickets? We need to keep the pressure on mikrotik so they realize it's an ongoing issue.

/M

Just opened and supout sent, i believe it is something to do with ospf and or mpls. I have other 2004 routers but they don't reboot that often, some have uptime as much as 20 days. these instead restart every 7-8 hours (OSPF + MPLS, no firewall, mix of 10G 1G interfaces)

Same here, 6..48 and a lot of reboots.

We had 2 reboots on 6.47.8 14 days ago in same day for 2 routers. After that nothing. We don't run 6.48.

Are you making tickets? We need to keep the pressure on mikrotik so they realize it's an ongoing issue.

/M

Just opened and supout sent, i believe it is something to do with ospf and or mpls. I have other 2004 routers but they don't reboot that often, some have uptime as much as 20 days. these instead restart every 7-8 hours (OSPF + MPLS, no firewall, mix of 10G 1G interfaces)

Same here, 6..48 and a lot of reboots.

We had 2 reboots on 6.47.8 14 days ago in same day for 2 routers. After that nothing. We don't run 6.48.

Are you making tickets? We need to keep the pressure on mikrotik so they realize it's an ongoing issue.

/M

The ones I had in service were running ospf, mpls, vpls, and not even high volumes of traffic. They had a max uptime of ~30 days before rebooting. Two were side by side in an effective redundant setup, and within ~8 hours of one, the second would reboot. Again after an max of ~30 days. Unfortunately, mine all have been pulled from service at this time. I had better results with 6.47.8, and Im not sure 6.48rc1 was any different in uptime with no other config changes.

Just opened and supout sent, i believe it is something to do with ospf and or mpls. I have other 2004 routers but they don't reboot that often, some have uptime as much as 20 days. these instead restart every 7-8 hours (OSPF + MPLS, no firewall, mix of 10G 1G interfaces)

Hi - we actually removed MPLS from our network recently because it was misbehaving and we used it so little. This could be a reson the reboots ceased to occur as often.

The routers rebooting for me on 6.47.8 runs ospf and full BGP. The load is only a couple of hundred megabits. The odd thing was it was only 2 of them. The rest are now above 40 days some reaching 50 days uptime.

2004s with no routing just standing idle never reboots.

/Mikael

No OSPF or MPLS on mine. Averages between 7 and 30 days between reboots. Low traffic load (averages probably 30 to 50 mbit).

Created SUP-38895.

We have 7 of these routers in production.
My experience with them is:
1 we use as a route reflector. 115 days uptime. (never rebooted). roughly 2 mill routes in RT
4 we use for passing traffic for clients. 3-13 days uptime. (only local routes and default)
2 we have sitting waiting to be put into production. No reboots since we turned them on 37 days ago.

It definitely seems to be related to when these routers are passing traffic. Not really how much traffic, but just passing traffic.

Of the ones passing traffic, we pass between a few 100 mbit to 2 gigabits/s. and they alle seem to reboot randomly equally between them regardless of load.

We have 7 of these routers in production.
My experience with them is:
1 we use as a route reflector. 115 days uptime. (never rebooted). roughly 2 mill routes in RT
4 we use for passing traffic for clients. 3-13 days uptime. (only local routes and default)
2 we have sitting waiting to be put into production. No reboots since we turned them on 37 days ago.

It definitely seems to be related to when these routers are passing traffic. Not really how much traffic, but just passing traffic.

Of the ones passing traffic, we pass between a few 100 mbit to 2 gigabits/s. and they alle seem to reboot randomly equally between them regardless of load.

Hi,

We have a similar experience. We have some with full bgp not passing traffic that never reboots, but the ones that do will reboot but some.of these too have never rebooted since 6.47.8. It feels very random like a lockup in fib when traffic has some specific pattern.

/Mikael.

Hi,

We had one crash today - first in some time. Uptime 34 days. The 2004 next to it has been up 45 days and pushes even more traffic so no clear difference.

Made new ticket as we are not getting response on the old ones - SUP-39418

/Mikael

It would be great to get a comment from Mikrotik on this matter - It would also be nice to get a refund on the hardware we cannot use due to it being "Faulty"

It would be great to get a comment from Mikrotik on this matter - It would also be nice to get a refund on the hardware we cannot use due to it being "Faulty"

Mikrotik cannot reproduce the issues so they need our help.

"RouterOS 6.48 has improvements for arm64 stability, which for some of our clients has solved an unexpected reboot issue. So if you say that other issues could you please provide all possible information about them and how we can reproduce them at our lab.

About interface drops, we still do not know what may cause them."

So please make tickets and feed them info. We havent had reboots (6.47.8 and 6.48) for more than 30 days so we have had nothing to feed back.

/Mikael

We've reported it to Mikrotik on multiple occasions an they've been able to replicate it.

Unfortunately now we have all the CCR2004's out of production so we cannot provide any further production testing - The risk was too high leaving it in the network.

We've reported it to Mikrotik on multiple occasions an they've been able to replicate it.

Unfortunately now we have all the CCR2004's out of production so we cannot provide any further production testing - The risk was too high leaving it in the network.

Hi,

Did you try with 6.48 that has fixes for these issues? I have several running 6.47.8 that had some fixes too and most 2004s hasent rebooted since. 3 of them have all and some are not at 65 days uptime. 2 with 35 and one without almost any traffic that rebooted 8 days ago.

/M

Yep Tried 6.48 and crashed 5 days later.

We've reported it to Mikrotik on multiple occasions an they've been able to replicate it.

Unfortunately now we have all the CCR2004's out of production so we cannot provide any further production testing - The risk was too high leaving it in the network.

This is where I am at on the subject as well. My longest was low 30s for days of uptime. Even with 6.48rc1

I have two ccr2004 in our network with traffic of about 1.5gb many problems of packet loss without solution until the moment!

I have two ccr2004 in our network with traffic of about 1.5gb many problems of packet loss without solution until the moment!

Hi,

Are you using 1gbit sfps and especially any rj45s plugs? There is a separate thread for that topic, but when we had them in lab we had issues with rj45 plugs.

/Mikael

Could be interesting

6.49 released.

switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device;

viewtopic.php?f=21&t=172259&sid=b361e20 ... ddcaa12707

It's a beta, so test it out first.

Could be interesting

6.49 released.

switch - improved packet transmit between CPU and 98PX1012 for CCR2004-1G-12S+2XS device;

viewtopic.php?f=21&t=172259&sid=b361e20 ... ddcaa12707

It's a beta, so test it out first.

We tried it and rolled back. It lost one of the 1 gig ports that started going up and down a lot. Rest seemed ok.

Think its too many changes in one. Would be nice if they could have a 2004 branch based on long-term.

Do NOT buy CCR2004's.

I have x4 CCR2004's (version 6.47.1) in production running OSFP and x2 Full BGP tables and I am experiencing these issues.

Last night I experienced a full OSPF/BGP lockup where the Routers had to be manually rebooted to recover (1 hour drive to the Data Centre at god forsaken hours). Last night was the first time the Routers didn't recover themselves after this weird behaviour. Last Friday night was the first time I noticed the OSPF/BGP lockup, which it auto recovered without intervention from me. I thought this was a DDOS attack or a SNMP issue because we are running Full BGP tables and had SNMP turned on and https://wiki.mikrotik.com/wiki/Manual:BGP_HowTo_%26_FAQ (scroll to the very bottom) says to not have SNMP turned on with Full BGP tables so I turned off SNMP on all our Routers. I experienced another OSPF/BGP lockup on Wednesday that auto recovered. Saying this a few months ago I noticed one of our CCR2004 Router reboot, which I didn't look that far into due to me still setting them up and configuring them. Something worthy of note is that I do notice x1 CPU core maxing out all of the time. It is almost as though every time our eBGP feeds have prefix updates x1 CPU core is maxing out, strange stuff.

Due to this complete OSFP/BGP lockup I decided to go full cowboy and upgrade the version to 6.49beta11. You should NOT run 6.49beta11 in PROD since our BGP peers started flapping like mad on this version. I have now upgraded the version of all our x4 CCR2004's to 6.48.1, which according to this thread still experienced this intermittent OSFP/BGP lockup and reboots. I did want to be a completely insane cowboy and run v7, but I saw someone posted there is a traffic flow bug that we require to work.

It looks like I purchased these CCR2004 Routers on the 6th of July 2020 before this thread was made (this helps clear my conscience on the decision a little). I am incredibly disappointed in these Routers and the only thing that makes sense to me that is causing these issues is the ARM 64bit CPU. I assume other Mikrotik CCR Routers don't have such a critical flaw right? My number one priority is replacing these CCR2004's with something else. I am thinking either the CCR1016-12S-1S+ or CCR1072-1G-8S+ might be a good option? The CCR1072-1G-8S+ is very pricey so I might go with the CCR1016-12S-1S+ once I figure out if all of my 10Gbps cross-connects will be compatible with 1000BASE-LX instead. We require Routers that can handle QinQ, OSPF, x2 Full BGP feeds, 200 ish mangle rules, x8 simple queue's for each customer and about 100Mbps - 1Gbps of traffic.

I run five 2004s in production with full BGP tables (and a couple more without BGP) and they don’t suffer from the reboots. I’ve tried to pattern match the things people are reporting in this thread and OSPF stands out—I don’t run OSPF but many people reporting reboots here seem to.

What version of RouterOS are you running? I might swap to that.

I run five 2004s in production with full BGP tables (and a couple more without BGP) and they don’t suffer from the reboots. I’ve tried to pattern match the things people are reporting in this thread and OSPF stands out—I don’t run OSPF but many people reporting reboots here seem to.

If you are running a RouterOS version described in this thread that is having the issues I might look at re-designing my network to use RIP/BGP instead of OSFP/BGP lol. Or even just BGP... I am pretty sure Mikrotik don't support EIGRP/IS-IS right?

What version of RouterOS are you running? I might swap to that.

This bug claims to be fixed in 6.47.8 and then some more in 6.48. Ive had reboots on 6.48 but not on 6.47.8 so running 6.47.9 now.

These is also a version 6.49rc11 that fixes some more, but it broke sfp support.

/Mikael

Ive had reboots on 6.48 but not on 6.47.8 so running 6.49.9 now.
[/quote]

Saying that do you think 6.47.8 is the gold standard since you haven't had any OSPF/BGP lockups or reboots on it?

I am very curious to know what version @markonen is running since he hasn't suffered OSPF/BGP lockups or reboots.

I have two that are running 6.47.8 with 75 days and 66 days of uptime, as well as a bunch that I just upgraded from 6.47.4 to 6.48.1 that were between 80 and 120 days of uptime before the upgrade.

Just BGP, static routes and raw firewall rules.

I have two that are running 6.47.8 with 75 days and 66 days of uptime, as well as a bunch that I just upgraded from 6.47.4 to 6.48.1 that were between 80 and 120 days of uptime before the upgrade.

Just BGP, static routes and raw firewall rules.

Out of curiosity - how much bgp do you run on them? Full feeds?

/Mikael

Full feeds from one or two transits (over both IPv4 and IPv6, so 2-4 sessions) + some peering. So one CPU core is more or less pegged doing BGP.

Full feeds from one or two transits (over both IPv4 and IPv6, so 2-4 sessions) + some peering. So one CPU core is more or less pegged doing BGP.

Thank you so much for this information @markonen

My plan of attack is to downgrade to 6.47.8 with OSPF and x2 Full BGP Tables.

If the OSPF/BGP lockup or reboot happens again I will re-design my network to be a RIP/BGP or Static Routes/BGP design.

@markonen what are your BGP peer uptimes? Maybe you have experienced a BGP lockup without noticing it?

I have been impacted by OSPF/BGP lockups a lot more than reboots.

Longest peer uptime is 32 days but IIRC they had scheduled maintenance then. I haven't really paid attention to these enough to really say whether it's the remote or some change we did that reset the other sessions (or if it was indeed a lockup, I guess).

Very interesting, It potentially could of been a lockup. I really am hoping it wasn't because that would mean I have to buy new Routers :(

Very interesting, It potentially could of been a lockup. I really am hoping it wasn't because that would mean I have to buy new Routers :(

This could be related to port extender issues. See 6.49 beta. We are running 3 of these in production but beware there is a bug if any of the interfaces at 1G and not 10G. They all come up in 10G and becomes unstable.

/M

What version of RouterOS are you running? I might swap to that.

This bug claims to be fixed in 6.47.8 and then some more in 6.48. Ive had reboots on 6.48 but not on 6.47.8 so running 6.47.9 now.

These is also a version 6.49rc11 that fixes some more, but it broke sfp support.

/Mikael

@mhugo In regards to the reboots you experienced on 6.48, how certain are you these weren't power related? Is your Router in a Data Centre power connected to A and B rails?

Full feeds from one or two transits (over both IPv4 and IPv6, so 2-4 sessions) + some peering. So one CPU core is more or less pegged doing BGP.

Out of interest, what is your route count? and do you see any packet loss?

Full feeds from one or two transits (over both IPv4 and IPv6, so 2-4 sessions) + some peering. So one CPU core is more or less pegged doing BGP.

Out of interest, what is your route count? and do you see any packet loss?

We see slight packet loss on routers even with 150k routes. I don't think it's more or less on the ones with 2-3 full.

Got a reboot on one now on 6.49beta11.

/Mikael

I had another OSPF/BGP lockup running 6.48.1, it auto recovered.

I have just re-designed my network to be Static Routes/BGP instead of OSPF/BGP.

I hope removing OSPF from our design has resolved these issues. If not I think my only option is to replace these CCR2004's.

Does anyone have any thoughts on Full BGP tables being too much for the CCR2004's to handle (CPU, etc)?

I have just re-designed my network to be Static Routes/BGP instead of OSPF/BGP.

I hope removing OSPF from our design has resolved these issues. If not I think my only option is to replace these CCR2004's.

Does anyone have any thoughts on Full BGP tables being too much for the CCR2004's to handle (CPU, etc)?

I dont have lockups in ospf/bgp. We have seen interface down/ups and resets.

/Mikael

6.48 ospf and bgp
no snmp, firewall, nat

reboots once per day

SOLUTION: turn off routeros monitoring in dude .

I dont have lockups in ospf/bgp. We have seen interface down/ups and resets.

Hmm, maybe that is what I have been experiencing too. I haven't noticed any syslogs for the interface going up/down though. The syslogs that I have noticed are "OSFPv2 neighbor x.x.x.x; state change from Full to Down.

I'll have to wait and see if removed OSPF from my network design has resolved the issue or not. @markonen is running Static Routes/BGP and hasn't experienced any of these issues we are, fingers crossed he was onto something.

I dont have lockups in ospf/bgp. We have seen interface down/ups and resets.

Hmm, maybe that is what I have been experiencing too. I haven't noticed any syslogs for the interface going up/down though. The syslogs that I have noticed are "OSFPv2 neighbor x.x.x.x; state change from Full to Down.

I'll have to wait and see if removed OSPF from my network design has resolved the issue or not. @markonen is running Static Routes/BGP and hasn't experienced any of these issues we are, fingers crossed he was onto something.

BGP on loopback is not tied to the interface like OSPF is.

This interface flapping is supposedly fixed in 6.49beta11.

/M

I tried 6.49beta11 and BGP was flapping like mad on it, certainly don't run that version in production.

I tried 6.49beta11 and BGP was flapping like mad on it, certainly don't run that version in production.

Are you using 1Gs - optics or rj45?

Are you using 1Gs - optics or rj45?

Yes, some of my cross-connects are 1000BASE-LX 1Gbps. I am pretty sure the Router <-> Router 10Gbps iBGP was going up and down on the 6.49beta11.

Also faced this problem. It worked for more than a year, first at CCR1009 - uptime for more than 365 days. About a month ago, they installed CCR2004, on the same kind of config, worked for 3 weeks, rebooted. Today again, less than a week has passed, rebooted.

The config is as minimal as possible, only BGP and iBGP. 4 active BGP sessions, 2 full-view. Latest long term firmware. Traffic is about 1.5 gigabits at its peak. After rebooting, one of the peering session does not get up, you need to switch it by hand. The link is connected with copper, through the sfp module 10G GL-CC-SFP-030.

SNMP enabled. Firewall, OSPF disabled

Are you using 1Gs - optics or rj45?

Yes, some of my cross-connects are 1000BASE-LX 1Gbps. I am pretty sure the Router <-> Router 10Gbps iBGP was going up and down on the 6.49beta11.

1G in 6.49beta11 did not seem to fully work as they negotiated in 10G even when setting to low speed.

Just swapped some 2004s for 1072. Feels sad when router is running 200-300 mbit to have to buy 1072 with less ports and higher cost due to Mikrotiks inability to fix the problem and even worse not really communicate about it.

Starting to feel like having stable 2004s require routeros 7. I would suggest mikrotik to skip mpls for the first release and make a 7.1 with that to focus efforts and perhaps solve some peoples issues who can do without mpls from the beginning.

/M

Just swapped some 2004s for 1072.

That is sad you had to resort to replacing the CCR2004's. Do you have any plan for returning them and getting your money back or anything along those lines?

Starting to feel like having stable 2004s require routeros 7

Yes I am starting to get that feeling as well. If I get one more lockup or reboot after changing my network design from OSPF/BGP to Static Routes/BGP then I am swapping in either CCR1036-8G-2S+'s or CCR1072-1G-8S+'s.

If ROS 7 is required for stable CCR2004's then I hope they get OSPF, BGP, VLANs, Traffic Flow, QinQ, DHCP, Simple Queues, Firewall, Routing Filters, and RADIUS operational.

I now have one CCR2004 in prod. Uptime only 10 days, but no issues so far.
Running 6.48.1.
Very simple configuration-
Only two static routes, some mangle rules and a few firewall rules.

I now have one CCR2004 in prod. Uptime only 10 days, but no issues so far.
Running 6.48.1.
Very simple configuration-
Only two static routes, some mangle rules and a few firewall rules.

I dont think (or at least hope) you will have any issues. I have had no reboots on the 2004s with simple configs. The issues are the ones with ospf (we have 50 in some areas) and/or BGP with full or at least 150K routes.

/Mikael

I had another lockup after changing from a OSPF/BGP to Static Routes/BGP Network Design.

CCR1036-8G-2S+'s have been ordered and I will be swapping out all of my CCR2004's ASAP.

In my opinion Mikrotik should stop selling CCR2004's until ROSv7 is up to scratch and has confirmed resolved these issues. Having CCR2004's in my network has damaged my ISPs reputation, I am not happy.

Very simple configuration-
Only two static routes, some mangle rules and a few firewall rules.

For your sake I hope the issues I have experienced are caused due to running x2 Full BGP tables. If I was you I would be planning for it to experience lockups/reboots.

Very simple configuration-
Only two static routes, some mangle rules and a few firewall rules.

For your sake I hope the issues I have experienced are caused due to running x2 Full BGP tables. If I was you I would be planning for it to experience lockups/reboots.

It's not I think. We have had issues with less than full bgp at 100k and no issues with routers with multiple full bgp.

/M

CCR1036-8G-2S+'s have been ordered and I will be swapping out all of my CCR2004's ASAP.

Thats the route I went (replaced with 1036s). No reboots since, 100+ days. Not running bgp here, but an ospf/mpls network. Max uptime was 30ish days with 2004s for me. No matter what version.

We pulled both our 2004's from service and replaced them with 1036's as well.

One of our 2004s was sent to Mikrotik (at their request through a SUP). We opened the ticket back in July. The RMA was approved in August and the 2004 was boxed and shipped to Baltic Networks for return to Mikrotik in early September. Last week (March), it was shipped back to me. The box was never opened and the router was never touched. They shipped it back to Baltic Networks stating it was repaired so Baltic returned it to me.

So unimpressed with this entire process. Sorry for all of you that bought more of these and are trying to run businesses with them. I'm, lucky I only got stuck with two of them.

One of our 2004s was sent to Mikrotik (at their request through a SUP). We opened the ticket back in July. The RMA was approved in August and the 2004 was boxed and shipped to Baltic Networks for return to Mikrotik in early September. Last week (March), it was shipped back to me. The box was never opened and the router was never touched. They shipped it back to Baltic Networks stating it was repaired so Baltic returned it to me.

It sounds like I have no chance at getting my money back then :( I guess I have just sunk $4,000 down the drain until ROS7 is up to scratch.

Thats the route I went (replaced with 1036s). No reboots since, 100+ days. Not running bgp here, but an ospf/mpls network.

It sounds like I have made the right choice getting 1036s then.

I am a new Mikrotik customer so getting CCR2004's hasn't been a good first time experience.

Tested the latest RC Friday just past and the 2004 lasted all of 8 hours before rebooting...

Tested the latest RC Friday just past and the 2004 lasted all of 8 hours before rebooting...

For me they have been stable - we have 7 running RC and it solved most of packetloss.

/Mikael

Hello, 3 CCR2004 Here, we are experiencing packet loss issues :



We do have another issue related to OSPF and neighbors getting down for no reason :



i tried woth both 6.49 Beta and 6.48.2 stable, but i didn't see any difference. Is anyone at MikroTik working on this issue ?

Hello, 3 CCR2004 Here, we are experiencing packet loss issues :

What speeds are going through it...what protocols are you running?

Hello, 3 CCR2004 Here, we are experiencing packet loss issues :

What speeds are going through it...what protocols are you running?

Less than 1gbps per router, with OSPF and 200k routes v4+v6 BGP.

Anybody tried 6.48.2 (2021-Apr-09 10:17)?

*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
*) switch - improved system stability with 98PX1012 switch chip for CCR2004-1G-12S+2XS device;

Running 14 on 6.48.2 they modified the timing so we have had two issues with them getting bad routes after large network changes but not reboots.

Waiting for a stable 7 for some working bgp....

Waiting for a stable 7 for some working bgp....

"Soon" :D

I have now three different units running on 6.48.2 still experiencing random reboots: they last from a few hours (6 hours) to several days before rebooting: they are rebooted by watchdog timer (router was rebooted without proper shutdown by watchdog timer) - I have console cables on all of them logging on a server the output, but there's nothing printed there before the reboot.
I also offered to Support several times to install the debug firmware, but they are not responding to tickets anymore.

I have now three different units running on 6.48.2 still experiencing random reboots: they last from a few hours (6 hours) to several days before rebooting: they are rebooted by watchdog timer (router was rebooted without proper shutdown by watchdog timer) - I have console cables on all of them logging on a server the output, but there's nothing printed there before the reboot.
I also offered to Support several times to install the debug firmware, but they are not responding to tickets anymore.

Exact same scenario here, single unit, very low throughput use case. Lasts anywhere from 4 to 20 days on average before a reboot. Nothing dumped to the console. I'm definitely getting tired of this router and I've lost confidence with Mikrotik as a result of using it.

Exact same scenario here, single unit, very low throughput use case. Lasts anywhere from 4 to 20 days on average before a reboot. Nothing dumped to the console. I'm definitely getting tired of this router and I've lost confidence with Mikrotik as a result of using it.

We actually dont have that issue and have 20 in production. We run full bgp and ospf but nothing else on them. How are you using the router?

/Mikael

Exact same scenario here, single unit, very low throughput use case. Lasts anywhere from 4 to 20 days on average before a reboot. Nothing dumped to the console. I'm definitely getting tired of this router and I've lost confidence with Mikrotik as a result of using it.

We actually dont have that issue and have 20 in production. We run full bgp and ospf but nothing else on them. How are you using the router?

/Mikael

I use it as a relative simple NAT firewall and edge router running a 1000mbit down/50 up commercial cable connection with a /29 public and a couple dozen internal private addresses and about 50 rules in the firewall. No BGP, no OSPF.

I am small ISP from India.
I bought 2004 for better specs.
but now, this CCR reboots in week.
Previously, I was on 6.47.7 and now on 6.48.3.
but no improvements.
Actual 6.48.3 reboots more often.
.
1G SFP modules on 3 sfp slots.
traffic between 100Mbps to 200Mbps. 170+ PPP clients.

I am small ISP from India.
I bought 2004 for better specs.
but now, this CCR reboots in week.
Previously, I was on 6.47.7 and now on 6.48.3.
but no improvements.
Actual 6.48.3 reboots more often.
.
1G SFP modules on 3 sfp slots.
traffic between 100Mbps to 200Mbps. 170+ PPP clients.

After a couple of weeks of running 6.48.3 I can report the same. Reboots are more frequent. I'm just about ready to throw this thing out the window.

@hoeser:

Could you please use "Post reply" button instead of quotting previous posts all the time.

Anybody tried 6.48.2 (2021-Apr-09 10:17)?

*) switch - improved resource allocation on 98PX1012 switch chip for CCR2004-1G-12S+2XS device;
*) switch - improved system stability with 98PX1012 switch chip for CCR2004-1G-12S+2XS device;

6.48.2 still packet loss (0.4 %) on SFP28 10 giga and still reboot around every 2 weeks: 2.5 Gb/s peak traffic and OSPF, no bgp

For the diehards still trying to use these, any updates on stability? Been quiet in here.

Not much. So reboots continues.
This is with the latest stable release (6.38.3)

Regards,
stuckoff

Hi All,

Is there anyone that is using the LTS release (6.47.10) on CCR2004 with BGP?
Is it unstable as the "stable" release?

Regards,
stuckoff

We tried several versions and now we are on the last, but still reboots every 5/10 days and packet loss <%0,2

Up and Running.. 2 Full tables and both 10G and 1G Links. To me, most stable is 6.47.8.

Vendor/Hardware Mikrotik CCR2004-1G-12S+2XS
Operating system Mikrotik RouterOS 6.47.8 (Level 6)
Uptime 47D 6h 6m 51s

Hi All,

Is there anyone that is using the LTS release (6.47.10) on CCR2004 with BGP?
Is it unstable as the "stable" release?

Regards,
stuckoff

I am becoming crazy with reboots anyone has better results with 6.48.3?

anyone has better results with 6.48.3?

https://tryitands.ee

Anyway, considering all the fixes in 6.48.3, I would expect there to be some improvement with the CCR2004.

Wish I knew about this thread before buying a 2004.

Now sitting with an unresponsive 2004 in a datacentre, having to route our traffic through a colleagues network.

Running 6.48.3 / BGP and OSPF.

Had one reboot with just OSPF running, no packet loss or interface drops like others. Just watchdog timer rebooting. Now that BGP is on, it lasts a few days between reboot (and stall as it is).

No copper, just 3x UF-SM-10G

Guys, any update about reboot issue ?
I raised ticket and they are now asking me shitty CLI things, as if this is unknown to them.
If such poor support from mikrotik, why network guys suggest it ?
I am totally disappointed by mikrotik and never buy any device again because poor support.
Also, referring same to others, don't buy mikrotik

... and they are now asking me shitty CLI things...
... why network guys suggest it ?

Mikrotik devices (running ROS in particular) are not really something to recommend to people with attitude towards CLI as you have. Most network guys, who know their stuff (be it Cisco, Juniper, ... or Mikrotik), like CLI very much, hence recommendations for Mikrotik as home router.

Experiencing continued CCR2004 watchdog reboots on 6.48.3. Using BGP and OSPF. No copper SFPs, all 10G fiber transceivers. Average reboot every 4-12 days.

CCR2004 trash hardware not usable in a professional network.

Guys, any update about reboot issue ?
I raised ticket and they are now asking me shitty CLI things, as if this is unknown to them.
If such poor support from mikrotik, why network guys suggest it ?
I am totally disappointed by mikrotik and never buy any device again because poor support.
Also, referring same to others, don't buy mikrotik

CCR2004 trash hardware not usable in a professional network.

What's the alternative? The equivalent Cisco would cost 100 times as much.

What's the alternative? The equivalent Cisco would cost 100 times as much.

I agree but we have 4 of these in production and it's a disaster long watchdog reboots every weeks, we move back to old CCR1016 for non 10 gbits link for 10 gibt routed link we are looking for a solutions

It seems be people are using this router for BGP sessions to upstream AND NAT/Connection State tracking.

NAT/Connection Tracking should be disabled on edge routers: https://mum.mikrotik.com/presentations/ ... 231376.pdf