The big CCR2004 reboot thread (was 2004 hardware issues?)

mikeeg02 · Fri Oct 22, 2021 3:52 am

Dont worry, on v6 they will, sometimes within 12h of eachother. They are special like that. Hope they arent important.

server8 · Mon Oct 25, 2021 1:21 pm

Here the same 6.48.4, reboot after 11 days of uptime

RohanAJoshi · Mon Oct 25, 2021 4:03 pm

I am still runing 7.1rc3 on this router and set tcp established timeout to 1hr
That's all i changed.. currently 33d and 15hr uptime
FYI

We have still no reboot after 39d 13h still running 7.1rc3 only thing we changed is the tcp established timeout.
We also updates a second CCR2004 it is running 7.1rc4 for 6d 9h now.

We dit not set the queue/pfifo setting.

any reboot till now on both routers ?

server8 · Tue Oct 26, 2021 8:36 am

6.48.4 this morrning another reboot less than a day of uptime. Opened a new ticket #[SUP-64102]:

I am borried

sander123 · Tue Oct 26, 2021 10:52 am

FYI

We have still no reboot after 39d 13h still running 7.1rc3 only thing we changed is the tcp established timeout.
We also updates a second CCR2004 it is running 7.1rc4 for 6d 9h now.

We dit not set the queue/pfifo setting.
any reboot till now on both routers ?

7.1rc3 rebooted after 42 days.
It was after a lot of OSPF changes because i try to update one of our BGP routers and it give me some problems in the network.

mikeeg02 · Tue Oct 26, 2021 12:05 pm

May as well reboot it twice more and see if 7.1rc4 does anything for ya.

RohanAJoshi · Tue Oct 26, 2021 12:35 pm

No hope for 7.1rc5 also, rebooted twice,
support asking me console output with watchdog disabled, which means I need manual restart when device get hang.
it will take 2 hrs downtime for my services, So, I am not doing this.
Shifted to old ccr1009

mhugo · Tue Oct 26, 2021 10:39 pm

We are not seeing any reboots on 2004s running 7.1rc at all.

We do however see some oddness when upgrading - we need to reboot them several time to get what I think is the layer2 working.

Units are almost not reachable even with mac telnet.

But after a couple of reboots they suddenly work with no changes and are stable.

Has anyone else seen this - It happens bascially every time we upgrade a 2004 with a new rc. We have 3 in test.

/Mikael

vasa85 · Sat Oct 30, 2021 10:34 pm

We recently bought two of these for use as BGP routers and I just discovered this topic. We haven't implemented them yet and now I'm a bit anxious about it. We might implement one of them just to see how it behaves.

mikeeg02 · Sun Oct 31, 2021 12:58 am

We recently bought two of these for use as BGP routers and I just discovered this topic. We haven't implemented them yet and now I'm a bit anxious about it. We might implement one of them just to see how it behaves.

If youre ok with random reboots and can tolerate that. Go ahead. Otherwise best advice is to wait until r7 is a little further along. Imo.

stealth · Tue Nov 02, 2021 3:53 pm

We are also in random reboots club. Device rebooted after 16 days 21 hours 34 minutes 24 seconds.
We have no software queue, only HW queues. Simple NAT router with one 10g trunk interface. Nothing fancy. Angry as hell.

Tue Nov 02, 2021 4:00 pm

We are also in random reboots club. Device rebooted after 16 days 21 hours 34 minutes 24 seconds.
We have no software queue, only HW queues. Simple NAT router with one 10g trunk interface. Nothing fancy. Angry as hell.

What RouterOS version do you use? Have you tried to set proposed enhancement in previous posts?

server8 · Tue Nov 02, 2021 8:01 pm

What RouterOS version do you use? Have you tried to set proposed enhancement in previous posts?

Sergejs have you read this thread? None has solved reboots problem whatever ros version used with or without tricks

Wed Nov 03, 2021 8:29 am

What RouterOS version do you use? Have you tried to set proposed enhancement in previous posts?
Sergejs have you read this thread? None has solved reboots problem whatever ros version used with or without tricks

Yes, I did it, as well as I worked with tickets that are related to the problem, currently there are very less 2004 that are rebooting (actually it helped for biggest portion of devices) and we are desperately looking for the best possible ways how to fix it.
If you want to help us with it, it would be great you can apply settings from the previous posts, make sure you have 6.49 for RouterOS version and firmware, send us support output file in case of any issue.

RohanAJoshi · Wed Nov 03, 2021 9:02 am

Sergejs have you read this thread? None has solved reboots problem whatever ros version used with or without tricks
Yes, I did it, as well as I worked with tickets that are related to the problem, currently there are very less 2004 that are rebooting (actually it helped for biggest portion of devices) and we are desperately looking for the best possible ways how to fix it.
If you want to help us with it, it would be great you can apply settings from the previous posts, make sure you have 6.49 for RouterOS version and firmware, send us support output file in case of any issue.

Ticket : SUP-51953
That's exactly what I did gave you, lot of supouts from start, no resolution, then you asked me to try 7.1rc, all rcs including latest rc5 rebooted for me, 3 times day.
You asked me console output and said to disable watchdog, this means I need to manual restart of device if issue occurs, my location is far from me, remote area, it will take 2 hrs to reach there.
So, I didn't do this, and shifted to ccr1009, currently looking for alternative from other brand.
.
So, don't be in disguise, look here, no one solved there problem, you are not getting replies for tickets because they are used to for this issue or they are moved from ccr2004.
Open your eyes and see.
User @sander123 also got reboot after 42 days uptime on 7.1rc3
So, your conception about very less 2004 are rebooting is wrong, almost all ccr2004 are rebooting, no one recommending this model for new customers.
So, please give something solid, no more this, please.
This topic is open from more that 1 year, yet no resolution.

Wed Nov 03, 2021 9:13 am

Have you tried 6.49 and
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default
on all queues as well dynamic PPPoE ones.
As I do not see such settings in your support output file.

RohanAJoshi · Wed Nov 03, 2021 9:30 am

Have you tried 6.49 and
/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default
on all queues as well dynamic PPPoE ones.
As I do not see such settings in your support output file.

Yes, I did above 2 queue changes, how to do same for ppoe ones ?

Wed Nov 03, 2021 9:52 am

/ppp/profile> set <profile-name> queue-type=<queue-type-name>

server8 · Wed Nov 03, 2021 10:02 am

Yes, I did it, as well as I worked with tickets that are related to the problem, currently there are very less 2004 that are rebooting (actually it helped for biggest portion of devices) and we are desperately looking for the best possible ways how to fix it.
If you want to help us with it, it would be great you can apply settings from the previous posts, make sure you have 6.49 for RouterOS version and firmware, send us support output file in case of any issue.

We have removed all CCR2004 less one from our network, actually I am on 6.48.4 and I opened several tickets without success. Reboot random some day ore some weeks, Tomorrow morning I 'll upgrade to 6.49 but I am quite sure it's useless

Wed Nov 03, 2021 10:06 am

Yes, I did it, as well as I worked with tickets that are related to the problem, currently there are very less 2004 that are rebooting (actually it helped for biggest portion of devices) and we are desperately looking for the best possible ways how to fix it.
If you want to help us with it, it would be great you can apply settings from the previous posts, make sure you have 6.49 for RouterOS version and firmware, send us support output file in case of any issue.
We have removed all CCR2004 less one from our network, actually I am on 6.48.4 and I opened several tickets without success. Reboot random some day ore some weeks, Tomorrow morning I 'll upgrade to 6.49 but I am quite sure it's useless

Provide us with support output, you can mention ticket number here. We will look into it.

server8 · Wed Nov 03, 2021 10:15 pm

We have removed all CCR2004 less one from our network, actually I am on 6.48.4 and I opened several tickets without success. Reboot random some day ore some weeks, Tomorrow morning I 'll upgrade to 6.49 but I am quite sure it's useless
Provide us with support output, you can mention ticket number here. We will look into it.

I have already did it a week ago [SUP-64102]

jespernissen · Thu Nov 04, 2021 1:41 pm

I've got the same issue with our CCR2004's.
We've got four units all with issues regarding random reboots by watchdog.
We've got two CCR1036 running almost exactly the same config as the CCR2004's, they've been running stable for the last two years without a single unexpected reboot.
Clearly something is bugged on the CCR2004 platform.

I've been following this thread for quite a while hoping for a solution, but now I've decided coming out from hiding

I've submitted a support message SUP-65050. Now with two support files.

Sasright · Fri Nov 05, 2021 11:49 am

On v6.49 and uptime is 9d

Running with:
1. Queue solution
2. tcp-established-timeout 1h
3. Disabled watchdog

I know for some of you disabling the watchdog is not an option but I’m just sharing

Will update soon again

10-NOV-21 update: 14d uptime.
15-NOV-21 update: 19d uptime.

RohanAJoshi · Wed Nov 10, 2021 2:13 am

SUP-51953
Guys, suddenly support closed my ticket without proper resolution.
Anyone getting same ?
What's meaning of this ?
@sergejs please tell me why happening this ?

server8 · Wed Nov 10, 2021 8:28 am

2 years without solutions means that it was an hardware issue:-(

jplitza · Fri Nov 12, 2021 6:16 pm

We had two units, A and B, both with completely different use cases and configurations. Both of them rebooted randomly.

We replaced one of them (A) with a CCR1016-12S-1S+ and opened support case SUP-60993 for the other (B). Support told us to RMA, so we did, but replaced it with A to continue debugging. Support told us to disable watchdog and attach serial console, so we did. A few days ago, it hung for the first time in weeks (!). Since watchdog was disabled, it simply hung there for 30 minutes (which in our case isn't worse than rebooting, since BGP has already moved on at that point), then toggled all its ports and continued operating, re-establishing all BGP sessions and such. Without any helpful logs, not even on serial. Weird.

mikeeg02 · Tue Nov 16, 2021 12:11 am

I do have good up time on ros7.1rc4. Its missing features that I need for production (that and using RC software in production is foolish) In current use its not doing much (just using at home now) but Ive never achieved this long of uptime. Not using cake, but am using queues for upload and ospf over a wireless link.

They were originally removed from production because of that.

2004.PNG

fcolang · Wed Nov 17, 2021 12:03 am

It is not clear to me whether the problems described in this thread occur on both CCR2004-16G-2S+ and CCR2004-1G-12S+2XS. Or it doesn't matter, they both have the same issue.

Maggiore81 · Sun Nov 21, 2021 12:17 pm

I never had reboot on 2004, but packet loss.
I use the latest lonterm+latest routerboot and it works flawlessy.
we have a policy that reboot each router every 14 days at night to avoid issues.
0 issues till now.

on the 7.1rc4 it was very unstable on some test units, rebooted on the bench, upgraded to rc6 and it doesnt reboot.

mikeeg02 · Sun Nov 21, 2021 6:56 pm

I never had reboot on 2004, but packet loss.
I use the latest lonterm+latest routerboot and it works flawlessy.
we have a policy that reboot each router every 14 days at night to avoid issues.
0 issues till now.

on the 7.1rc4 it was very unstable on some test units, rebooted on the bench, upgraded to rc6 and it doesnt reboot.

You've never had a reboot, but you just said you've never had more than 14 days of uptime. As YOU reboot it every 14 days.

What issues did you have with 7.1rc4?

What all is your router doing? (ospf, mpls, bgp, etc)

Splash · Sat Nov 27, 2021 12:04 pm

I have now found the Mikrotik don't even respond to tickets logged regarding random reboots.

Running any track of RouterOS, various sub-versions all produce the same issue with our devices.

Do you think this device was build for V7 and that V6 is running in compatibility mode (32bit) on a 64bit CPU?

murrayis · Mon Nov 29, 2021 2:32 am

I gave up on my tickets with Support regarding the issues. We had 6 CCR2004 all rebooting randomly - Usually when OSPF or BGP has a big update but some time doing nothing just would either lockup or reboot.

I had one this morning which had been online for "Device rebooted after 90 days 8 hours 56 minutes 12 seconds -> 259s"

Traffic Past 24 hours (Reboot was just prior to 0900 Mon)

As you can see this route does nothing... Yet it still reboots with 3 BGP session; 80k Routes in the Table and 3 OSPF areas.

Splash · Mon Nov 29, 2021 10:06 am

I wonder if this helps narrowing down what causes the router to reboot?

[*]Is this a batch of routers with this issue?
[*]Is this a software issue in general?
[*]Is this related to a BGP issue. (We have them rebooting with no OSPF)

@mikrotik?

mducharme · Mon Nov 29, 2021 12:59 pm

Do you think this device was build for V7 and that V6 is running in compatibility mode (32bit) on a 64bit CPU?

It definitely is designed for v7 and runs in compatibility mode in v6, which is why it cannot use all of the RAM on v6. And, mikeeg02 is finding that the CCR2004 seems to not crash on v7 at all, suggesting that the issue is something related to the kernel version or this compatibility mode.

I suspect they probably won't come up with a fix for this on v6 and will ask people to upgrade to v7 to get rid of the issue. If the fix is some change that was introduced to the kernel in the past decade, it may be easier to just upgrade the kernel to fix it instead of trying to track down what change made in the past 10 years fixed it and apply it to v6.

Splash · Mon Nov 29, 2021 1:13 pm

Makes sense. I am hoping the stable release of V7 will be soon as its quite disruptive to have a core device just randomly reboot

OlofL · Mon Nov 29, 2021 4:00 pm

I have no issues. I run 6.48, and only a small OSPF network.
65 days uptime. I have not seen random reboots at all.

GjergjiP · Wed Dec 01, 2021 3:24 am

I have the same problem with my rb 2004 . All the ports that are connected are up but it won't send or receive any data, also i cannot connect to the router. After many times with this problem, i think it is an hardware problem because it happen when the AC Power source goes down and the inverter goes to bypass mode. After this situation it will go up again after a manual restart (remove power and plug it again). I am going to change it because i have no hope that it will work well after 1 year of stress work.

yaorushdie · Wed Dec 01, 2021 5:20 am

Tried 6.49 - no luck, reboot after 8 days.

Can't test 7.1rc4 - VPLS does not pass MTU > 1500, looks like it's a bug.

Changing the MTU and L2 MTU to 9216 that all interfaces you run as LDP interface.
You can try this working or not.

korsakik · Wed Dec 01, 2021 9:26 am

Mikrotik, how about money back?? No resolution - no customer.
I dont need this trash anymore.

server8 · Wed Dec 01, 2021 4:09 pm

Mikrotik, how about money back?? No resolution - no customer.
I dont need this trash anymore.

forgot it

Wed Dec 01, 2021 4:12 pm

In case any of you are experiencing issue with random reboots on 2004 and 6.49.1/7.1rc7
Please, contact us (support@mikrotik.com) with the information and attached support output files.

mhugo · Sat Dec 04, 2021 5:53 pm

In case any of you are experiencing issue with random reboots on 2004 and 6.49.1/7.1rc7
Please, contact us (support@mikrotik.com) with the information and attached support output files.

Does this mean that some fix for this has been applied in 6.49.1 and 7.1rc7 and that fix is not in 6.48.5?

As a information to everyone else - we are seeing 2004s become sluggish after upgrades to new versions in 7 branch including 7.1. Router is hard to reach with massive packetloss and requires multiple reboots to suddenly become normal. Feels like something doesent get initialized right or silicon is storing something not getting flushed on reboot. I suspect all the way down to L2 as mactelnet, romon etc is affected too.

It does not happen every time, but in 20% of upgrades and not on specific router but all random.

Mikrotik has open tickets from us but cannot reproduce so please observe and make supouts if this happens to you.

StubArea51 · Sat Dec 04, 2021 6:09 pm

It definitely is designed for v7 and runs in compatibility mode in v6, which is why it cannot use all of the RAM on v6. And, mikeeg02 is finding that the CCR2004 seems to not crash on v7 at all, suggesting that the issue is something related to the kernel version or this compatibility mode.

I suspect they probably won't come up with a fix for this on v6 and will ask people to upgrade to v7 to get rid of the issue. If the fix is some change that was introduced to the kernel in the past decade, it may be easier to just upgrade the kernel to fix it instead of trying to track down what change made in the past 10 years fixed it and apply it to v6.

Agree 100%

mhugo · Sat Dec 04, 2021 6:27 pm

I suspect they probably won't come up with a fix for this on v6 and will ask people to upgrade to v7 to get rid of the issue. If the fix is some change that was introduced to the kernel in the past decade, it may be easier to just upgrade the kernel to fix it instead of trying to track down what change made in the past 10 years fixed it and apply it to v6.
Agree 100%

IMHO I agree the best path is for Mikrotik to focus all resources on getting 7.1.x on par with 6.49.1 and not do anything more on 6.x or add any new features besides any blocking an upgrade from 6.x

Im running 12 2004s on 7.1rc7, 7.2beta17 and 7.1 in production. In a network using only ospf and bgp its working great now with the last ospf fixes. No reboots but we have been upgrading quite often so uptime is max 10 days.

The comment above about crashes to be mailed to mikrotik made me a little worried as I have been told to apply same fixes to 7.1 by support, as we hoped 7.x would fix both the reboots and the packetloss.

Splash · Sat Dec 04, 2021 6:36 pm

We applied the suggested Ethernet hardware queues changes to 6.49 and the router still rebooted.... We are now running 6.49.1 with the suggested fix and will monitor it over the coming days.

RohanAJoshi · Mon Dec 06, 2021 11:31 am

We applied the suggested Ethernet hardware queues changes to 6.49 and the router still rebooted.... We are now running 6.49.1 with the suggested fix and will monitor it over the coming days.

don't get fooled, they are suggesting same changes from 6.48, still issue remains same.

Splash · Mon Dec 06, 2021 11:37 am

/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default

jespernissen · Mon Dec 06, 2021 8:10 pm

In case any of you are experiencing issue with random reboots on 2004 and 6.49.1/7.1rc7
Please, contact us (support@mikrotik.com) with the information and attached support output files.

I've sent multiple support output files now, in SUP-65050, but every time i just get the answer "The supout rif file does not contain any hardware or software errors, the device was rebooted by watchdog timer", which is exactly the issue I'm trying to explain. I have four of these devices, and every single one of them reboots randomly with no pattern what so ever.
Once i was also told to disable the watchdog timer. My best guess is that this will lead to the device crashing without any possibility to regain access than physically unplug the device.
This is not an option as they are running in production on remote sites.

Three of them is running each with a single full BGP table, both IPv4 and IPv6. One of them is running with around 250K routes in the BGP table.
My CCR1036's running exactly the same configuration, it's a 1:1 configuration, doesn't have any issues and has never rebooted once, unless i told them to.

Something is obviously rotten on the CCR2004 platform, and i don't understand why this issue hasn't been fixed for more than a year.

As of now i have no idea whether i need to buy other CCR's or to completely switch to another brand, this is unacceptable, and of course noticeable for our customers.

guipoletto · Tue Dec 07, 2021 2:23 pm

"The supout rif file does not contain any hardware or software errors, the device was rebooted by watchdog timer",

That's what i see here too. No autogenetrated supout.rif on reboot.

I have not configured any watchdog, it is in the default state:

[admin@CCR2004] > system watchdog export verbose
# dec/07/2021 09:22:08 by RouterOS 6.49.1
# software id = MFVV-BXU3
#
# model = CCR2004-1G-12S+2XS
# serial number = D4F00C0EB777
/system watchdog
set auto-send-supout=no automatic-supout=yes ping-start-after-boot=5m ping-timeout=1m watch-address=none watchdog-timer=yes
[admin@CCR2004] >

mikruser · Tue Dec 07, 2021 4:27 pm

Something is obviously rotten on the CCR2004 platform, and i don't understand why this issue hasn't been fixed for more than a year.
As of now i have no idea whether i need to buy other CCR's or to completely switch to another brand, this is unacceptable, and of course noticeable for our customers.

Are you new here? Mikrotik had a lot of fuckups before. Everyone has known for a long time - MT routers cannot be used in enterprise. Its only for SOHO with low-speed wan. Its too bugged and have very poor support.

ak4020 · Tue Dec 07, 2021 6:39 pm

So my best decision was to switch to Mikrotik, in all core data centers in Europe a for example CCR1009-7G-1C-1S+ already in operation for years

since then zero problems and 100 percent satisfaction...

if the hardware goes haywire buy a current ccr and you will be happy again for many years to come

StubArea51 · Tue Dec 07, 2021 7:15 pm

Everyone has known for a long time - MT routers cannot be used in enterprise. Its only for SOHO with low-speed wan. Its too bugged and have very poor support.

Have to disagree, I've put MIkroTik CCRs into plenty of large enterprises for critical roles.

In one specific example, we put 4 x CCR1036 routers into the flagship data center of a 19 billion dollar publicly traded company because the cisco ASRs were 90 days behind schedule - and it ran perfectly. we used BGP/OSPF and DSCP based QoS for voice/video. More than 30,000 employees of this company relied on that data center for day-to-day operations.

I've consulted for a number of US Fortune 500 companies and whether they use Cisco, Juniper, Aruba or MikroTik, the process is the same:

You must understand at an expert level:

1) The hardware platform and its limitations
2) The software lifecycle and which versions to use
3) Proper config and protocol selection

As it relates to the new CCR2xxx series

The CCR2004 will get there as ROS 7.x stabilizes - just like the CCR1036 did when we moved from ROS 5.x to 6.x ten years ago.

Splash · Wed Dec 08, 2021 9:42 am

I think the main note to take is that the CCR2004 platform is designed as a 64 bit device (ARM64), and running the operating system in 32bit compatibility mode is probably the primary issue its experiencing. For example it only exposes 50% of the devices RAM to the operating system in V6.

I think the 2004 will become a power house with the new V7 operating system, the question will be how long will it take to stabilize.

mikruser · Sat Dec 11, 2021 7:24 pm

IPANetEngineer
Have to disagree, I've put MIkroTik CCRs into plenty of large enterprises for critical roles.
In one specific example, we put 4 MikroTik routers into the flagship data center of a 19 billion dollar publicly traded company
This does not mean anything other than that you like to take risks (and you're just lucky).
Yes, you can build a "server" on Core i3 and chipset raid array on WD Green, but it will not become a real server from this.

RohanAJoshi · Sun Dec 12, 2021 11:44 am

It's done now.
Rebooted even on 7.1 stable version with 6 days uptime.
Same like before, even with given fixed default-ethernet settings etc.
No hopes guys.

Splash · Mon Dec 13, 2021 9:01 am

Thats a little worrying....

You updated the firmware too?

Splash · Mon Dec 13, 2021 10:33 pm

Anyone seen an issue with the CCR2004 on ROS V6.49.1 where there is packet loss each time BGP needs to converge its routes?

If the routes from the remote peer are stable, then there is no packet loss across any interfaces.
As soon as the CCR2004 needs to converge received or withdrawn routes, CPU goes up and packet loss appears on all interfaces traffic passes through. Even running a Timeout of 100ms over a point to point IP shows packet loss.

Are there any work arounds?

emunt6 · Tue Dec 14, 2021 2:06 am

CCR2004 trash hardware not usable in a professional network.
What's the alternative? The equivalent Cisco would cost 100 times as much.

It depends on use cases:
> BGP/OSPF with VRF: i will definetly go with ARISTA ( example: ARISTA DCS-7050QX-32S )
> 10G NAT: hard question ( example: Cisco 6504E + 2x Sup2T; Juniper MX240/MX480; HPE HSR6804 )
> 1k PPPOE BRAS - no idea

You can find companies who offer used/refurbished hardware with support/spare-parts services.

neg2led · Tue Dec 14, 2021 3:21 am

What's the alternative? The equivalent Cisco would cost 100 times as much.
It depends on use cases:
> BGP/OSPF with VRF: i will definetly go with ARISTA ( example: ARISTA DCS-7050QX-32S )
> 10G NAT: hard question ( example: Cisco 6504E + 2x Sup2T; Juniper MX240/MX480; HPE HSR6804 )
> 1k PPPOE BRAS - no idea

You can find companies who offer used/refurbished hardware with support/spare-parts services.

Even refurbished hardware costs 10x what these do, and the stuff you've suggested consumes a ludicrous amount of power (by comparison) - datacenter power ain't cheap!

The closest thing you could find in a somewhat comparable pricepoint would be running VyOS or TNSR on a Xeon D-15xx/21xx network appliance box from Supermicro or something along those lines; even then, if you want support, VyOS and TNSR licenses are half the cost of a 2004 annually all by themselves.

There is pretty much nothing else on the market that can handle the same level of throughput, at a comparable pricepoint, with a comparable feature set - even used. The closest you can get with used gear requires something that consumes 200-300W+ and still costs more to buy, to say nothing of the 3rd-party hardware maintenance contract.

I've spent the last year or so trying to find something that meets these requirements, and it just doesn't exist. The major vendors are all moving to running on top of x86 virtualization appliances for the lower-end devices, with prohibitively high licensing costs, so they've no interest in making hardware to fit the bill. You could maybe consider the EdgeRouter Infinity, but, well, that's got its own problems.

I may have missed something - if so, I'm all ears - but I suspect most of the reason there are so many annoyed/upset/angry people in this thread is because there are very few other options than these devices for a lot of their intended use-cases. That's probably a significant part of the reason why MikroTik made them in the first place - there is a massive market vacuum for "low-to-medium-cost router with multiple 10G ports and capacity to handle full routing tables / etc".

Hell, look at the CRS2116 - that box is basically everything I could've asked for in a $1000 router, MT could've charged almost 2x as much and I'd still have a preorder with our distributor.

Paternot · Tue Dec 14, 2021 12:03 pm

Hell, look at the CRS2116 - that box is basically everything I could've asked for in a $1000 router, MT could've charged almost 2x as much and I'd still have a preorder with our distributor.

Shhh... Don't give them ideas...

issme · Tue Dec 14, 2021 6:29 pm

In case any of you are experiencing issue with random reboots on 2004 and 6.49.1/7.1rc7
Please, contact us (support@mikrotik.com) with the information and attached support output files.

Reboot on 6.48.4 after 76 days uptime (was latest available firmware at the time, supposedly had 'fixes' for this type of issue.) SUP-68916
Supout was not automatically generated, had to manually generate.

Using as internal IPv4 router only - running OSPF, along with some very light bridging for a couple VLANs. Not doing any NAT/BGP/MPLS/IPv6/PPP/queues/mangling etc. Have disabled hotspot, ipv6, mpls, ppp, and wireless packages. Overall config is very basic.

Only two SFP+ interfaces are populated. Using 10G LR and DAC SFP+ transceivers.

Have since updated to 6.49.2, but am not confident the issue is resolved seeing how multiple other forum users are still reporting the same issue.

Have a large batch of 2004s on hand that am waiting to deploy at our primary sites, but may have to return these as the three trial units we are using at some smaller locations have been affected by this bug.

diegofranca · Wed Dec 22, 2021 11:05 pm

Dear ones, I have the same problem as you!
I used the 2004 for approximately 11 months just running ospf with an average of 600Mbps per second and no restart.
15 days ago I put the box to make partial bgp router with 3 operators + 1 PTT, the crashes and crashes started.
is the problem in the bgp?
do we have to throw this ccr in the trash?

Splash · Fri Dec 24, 2021 12:13 pm

I would say a say as a rule of thumb.... If you have a CCR2004, then run RouterOS 7.1.x

The CPU is 64bit based architecture and the RouterOS V6 is a 32 bit operating system.

After upgrading to 7.1.x you will also see 4GB of RAM vs 2GB. It seems with V6 on 2004, if the CPU is hit, then it interrupts traffic as well as other services it runs.
Before upgrading to V7, prepare a manual conversion of your Routing Filters as it will not always convert correctly.

*I have been running 7.1.1 on a device that used to reboot every few days and it has been far more stable.... (no reboots since upgrade)

mhugo · Tue Dec 28, 2021 3:15 am

*I have been running 7.1.1 on a device that used to reboot every few days and it has been far more stable.... (no reboots since upgrade)

I have 19 2004s running 7.1.x - no reboots.

Only thing we see is sometimes during upgrade we need to reboot the unit 3-4 times. Something in l2 get wierd and even Mac ping is loosing packets. After 3-4 reboots ita stable but accessing with heavy packet losses takes time.

Happens only in 2004s.

Reported to mikrotik multiple times but ignored but once when they said they could not reproduce.

/M

server8 · Tue Dec 28, 2021 12:57 pm

Other users reports that problem is solved with ros 7.1.x?

thank you

StubArea51 · Tue Dec 28, 2021 9:14 pm

What's the alternative? The equivalent Cisco would cost 100 times as much.
It depends on use cases:
> BGP/OSPF with VRF: i will definetly go with ARISTA ( example: ARISTA DCS-7050QX-32S )
> 10G NAT: hard question ( example: Cisco 6504E + 2x Sup2T; Juniper MX240/MX480; HPE HSR6804 )
> 1k PPPOE BRAS - no idea

You can find companies who offer used/refurbished hardware with support/spare-parts services.

Arista doesn't support IPv6 BGP Route Reflection or VPLS so while I generally like them, there are some protocol gaps they are still trying to fill in as they move towards service provider.

Used/refurb hardware used to be a solid play but has become more challenging for a few reasons:

1) Supply chain integrity - increasingly, there are either perfect copies of legit gear with malware embedded on the gray market or legit hardware is being embedded with malware and resold. Either way, it's increasingly a security risk.

2) Availability - the supply chain issues and chip shortage have created a premium on the graymarket

3) Software/support - You either need to pay Cisco/Juniper to recertify it for TAC support which is a PITA or scour the internet for a recent version of the NOS you need - which loops back around to security - you have no idea if the software you're downloading from some random location on the Internet is embedded with some kind of malware to use the device as command and control, DDoS traffic generation, etc.

Coming back to the main point - now that ROSv7 is starting to work towards stability with new feature sets, it's hard to ignore the ROI of something like a CCR2004 vs. ASR/MX or Arista gear

mhugo · Wed Dec 29, 2021 12:03 am

Coming back to the main point - now that ROSv7 is starting to work towards stability with new feature sets, it's hard to ignore the ROI of something like a CCR2004 vs. ASR/MX or Arista gear

We couldent operate our network with Juniper MX204s and still make money. We have 18 main sites just for the main site and 300 subnodes running CRS317.

Now we just need some real CGNAT in ROS7 as it would be great benefit to have this in many places for us, but guessing that will be ROS8 feature best case. Not sure using a container would be great.

The new CRS2116 looks great but we cant use anything with copperports so hoping a version with sfp+ will come (or all 25G ports perhaps). For now 2004s are used.

/M

StubArea51 · Wed Dec 29, 2021 7:40 pm

IPANetEngineer
Have to disagree, I've put MIkroTik CCRs into plenty of large enterprises for critical roles.
In one specific example, we put 4 MikroTik routers into the flagship data center of a 19 billion dollar publicly traded company
This does not mean anything other than that you like to take risks (and you're just lucky).
Yes, you can build a "server" on Core i3 and chipset raid array on WD Green, but it will not become a real server from this.

Sorry, that's just not true. We run the largest MikroTik consulting company globally. Most of the publicly traded companies and large ISPs/DCs that want to use MikroTik in some way come to us for integration help and guidance.

In the last decade we've put plenty of them into critical roles and been successful.

CZFan · Wed Dec 29, 2021 8:29 pm

Anyone seen an issue with the CCR2004 on ROS V6.49.1 where there is packet loss each time BGP needs to converge its routes?

If the routes from the remote peer are stable, then there is no packet loss across any interfaces.
As soon as the CCR2004 needs to converge received or withdrawn routes, CPU goes up and packet loss appears on all interfaces traffic passes through. Even running a Timeout of 100ms over a point to point IP shows packet loss.

Are there any work arounds?

I experienced the same in Teraco data centre peering with NAP Africa, so my workaround was to make the 2004 the PPPoE AC and put a 1016 in place as edge/BGP/FW device

RohanAJoshi · Thu Dec 30, 2021 12:53 pm

Other users reports that problem is solved with ros 7.1.x?

thank you

No, still weekly reboots on ccr2004 with stable v7.1.1

server8 · Thu Dec 30, 2021 7:34 pm

Other users reports that problem is solved with ros 7.1.x?

thank you
No, still weekly reboots on ccr2004 with stable v7.1.1

Thank you I hope to have more positive reports

mhugo · Thu Dec 30, 2021 9:06 pm

No, still weekly reboots on ccr2004 with stable v7.1.1
Thank you I hope to have more positive reports

Im very curious to understand what things are causing this.

We run pure ospf and bgp with multiple full feeds. We don't have any reboots but had loads before.

/M

Splash · Fri Dec 31, 2021 9:52 am

Anyone seen an issue with the CCR2004 on ROS V6.49.1 where there is packet loss each time BGP needs to converge its routes?

If the routes from the remote peer are stable, then there is no packet loss across any interfaces.
As soon as the CCR2004 needs to converge received or withdrawn routes, CPU goes up and packet loss appears on all interfaces traffic passes through. Even running a Timeout of 100ms over a point to point IP shows packet loss.

Are there any work arounds?

I experienced the same in Teraco data centre peering with NAP Africa, so my workaround was to make the 2004 the PPPoE AC and put a 1016 in place as edge/BGP/FW device

Migrating to V7.1 resolved this for future FYI

RohanAJoshi · Fri Dec 31, 2021 12:26 pm

Thank you I hope to have more positive reports

Im very curious to understand what things are causing this.

We run pure ospf and bgp with multiple full feeds. We don't have any reboots but had loads before.

/M

Are you using this router as ppp server ?
I am 200+ ppp clients.
.
For test, I only one ppp client on this ccr, no reboot for 30 days, then i gave full load of 200+ users, and it's started to reboot again. 7 days to 15 days gap.

mhugo · Fri Dec 31, 2021 1:44 pm

Are you using this router as ppp server ?
I am 200+ ppp clients.
.
For test, I only one ppp client on this ccr, no reboot for 30 days, then i gave full load of 200+ users, and it's started to reboot again. 7 days to 15 days gap.

I have no PPP users and no reboots.

/M

lele · Tue Jan 04, 2022 10:46 am

We have about 40 CCR2004s in production. Only two of them are rebooting regularily. Plus a third that has rebooted only once. Still, we can't determine what is making these two reboot. They aren't the most loaded device in the network or the most articulated configuration.
All they do is:
- an handful VRRPs
- BGP, announcing a couple handfuls of routes to reflectors, and receiving the default.
- basic firewall, with a dozen ACLs

load is consistently below 5%, average 1%. Aggregate traffic is in the 100Mb/s range.
Ironically enough these two are the most critical devices in the network. We went with the 2004s after having no issues with the rest of them.

server8 · Tue Jan 04, 2022 6:11 pm

We have about 40 CCR2004s in production. Only two of them are rebooting regularily. Plus a third that has rebooted only once. Still, we can't determine what is making these two reboot. They aren't the most loaded device in the network or the most articulated configuration.
All they do is:
- an handful VRRPs
- BGP, announcing a couple handfuls of routes to reflectors, and receiving the default.
- basic firewall, with a dozen ACLs

load is consistently below 5%, average 1%. Aggregate traffic is in the 100Mb/s range.
Ironically enough these two are the most critical devices in the network. We went with the 2004s after having no issues with the rest of them.

Ros7?

lele · Mon Jan 10, 2022 11:35 am

Ros7?

6.49.2

emunt6 · Sat Jan 15, 2022 6:31 pm

It depends on use cases:
> BGP/OSPF with VRF: i will definetly go with ARISTA ( example: ARISTA DCS-7050QX-32S )
> 10G NAT: hard question ( example: Cisco 6504E + 2x Sup2T; Juniper MX240/MX480; HPE HSR6804 )
> 1k PPPOE BRAS - no idea

You can find companies who offer used/refurbished hardware with support/spare-parts services.
Even refurbished hardware costs 10x what these do, and the stuff you've suggested consumes a ludicrous amount of power (by comparison) - datacenter power ain't cheap!

The closest thing you could find in a somewhat comparable pricepoint would be running VyOS or TNSR on a Xeon D-15xx/21xx network appliance box from Supermicro or something along those lines; even then, if you want support, VyOS and TNSR licenses are half the cost of a 2004 annually all by themselves.

There is pretty much nothing else on the market that can handle the same level of throughput, at a comparable pricepoint, with a comparable feature set - even used. The closest you can get with used gear requires something that consumes 200-300W+ and still costs more to buy, to say nothing of the 3rd-party hardware maintenance contract.

I've spent the last year or so trying to find something that meets these requirements, and it just doesn't exist. The major vendors are all moving to running on top of x86 virtualization appliances for the lower-end devices, with prohibitively high licensing costs, so they've no interest in making hardware to fit the bill. You could maybe consider the EdgeRouter Infinity, but, well, that's got its own problems.

I may have missed something - if so, I'm all ears - but I suspect most of the reason there are so many annoyed/upset/angry people in this thread is because there are very few other options than these devices for a lot of their intended use-cases. That's probably a significant part of the reason why MikroTik made them in the first place - there is a massive market vacuum for "low-to-medium-cost router with multiple 10G ports and capacity to handle full routing tables / etc".

Hell, look at the CRS2116 - that box is basically everything I could've asked for in a $1000 router, MT could've charged almost 2x as much and I'd still have a preorder with our distributor.

You can go with "Raisecom" ( chinese vendor ) - cost-efficient solution, i've heard good feedbacks from other operators.

RohanAJoshi · Sun Jan 16, 2022 2:04 pm

Even refurbished hardware costs 10x what these do, and the stuff you've suggested consumes a ludicrous amount of power (by comparison) - datacenter power ain't cheap!

The closest thing you could find in a somewhat comparable pricepoint would be running VyOS or TNSR on a Xeon D-15xx/21xx network appliance box from Supermicro or something along those lines; even then, if you want support, VyOS and TNSR licenses are half the cost of a 2004 annually all by themselves.

There is pretty much nothing else on the market that can handle the same level of throughput, at a comparable pricepoint, with a comparable feature set - even used. The closest you can get with used gear requires something that consumes 200-300W+ and still costs more to buy, to say nothing of the 3rd-party hardware maintenance contract.

I've spent the last year or so trying to find something that meets these requirements, and it just doesn't exist. The major vendors are all moving to running on top of x86 virtualization appliances for the lower-end devices, with prohibitively high licensing costs, so they've no interest in making hardware to fit the bill. You could maybe consider the EdgeRouter Infinity, but, well, that's got its own problems.

I may have missed something - if so, I'm all ears - but I suspect most of the reason there are so many annoyed/upset/angry people in this thread is because there are very few other options than these devices for a lot of their intended use-cases. That's probably a significant part of the reason why MikroTik made them in the first place - there is a massive market vacuum for "low-to-medium-cost router with multiple 10G ports and capacity to handle full routing tables / etc".

Hell, look at the CRS2116 - that box is basically everything I could've asked for in a $1000 router, MT could've charged almost 2x as much and I'd still have a preorder with our distributor.

You can go with "Raisecom" ( chinese vendor ) - cost-efficient solution, i've heard good feedbacks from other operators.

Which model specifically ? Compared to ccr2004 or ccr216/ccr1036 ?

RohanAJoshi · Thu Jan 20, 2022 8:53 pm

Any update guys ?
Does anyone getting reboots with vi stable ?
For me, it's still rebooting with v7.1.1

server8 · Thu Jan 20, 2022 10:42 pm

Guys it's hardware problem +18 months without solving can't be the software!!!

mducharme · Thu Jan 20, 2022 11:54 pm

Any update guys ?
Does anyone getting reboots with vi stable ?
For me, it's still rebooting with v7.1.1

Everbody I know with this issue has had it gone entirely since 7.1.1, except for you. Probably there are two reasons for the reboots, one is fixed by 7.1.1 and the other is possibly configuration specific. If you share your config for the device, we might be able to identify the issue.

edrichdelange · Fri Jan 21, 2022 5:03 am

I can confirm I have reboots on my one router with 7.1.1. And the it’s sister router which has same config Save the ips has been up for 147 days. On a v6.48.3

markos222 · Sat Jan 22, 2022 11:50 am

Hi,

We have changed 3 CCR1009 , for CCR2004, same configuration, this week,

On two CCR2004 we have experimented two watchdog reboot, all have 6.49.2,

VLAN, AND ROUTING, BASIC FIREWALL,

Any ideas , hardware problem?

Thank

server8 · Sat Jan 22, 2022 1:51 pm

Yes at this point 'd be an hardware problem not solved after +18 months, reboot occurs also with ros 7x

mducharme · Sat Jan 22, 2022 7:53 pm

On two CCR2004 we have experimented two watchdog reboot, all have 6.49.2,

If you are doing VLAN, routing, basic firewall, you can probably use 7.1.1 on them to fix the reboots.

RohanAJoshi · Sun Jan 23, 2022 8:38 am

On two CCR2004 we have experimented two watchdog reboot, all have 6.49.2,
If you are doing VLAN, routing, basic firewall, you can probably use 7.1.1 on them to fix the reboots.

No, for your information, 7.1.1 also does not fix reboot issue for ccr2004 but also introduces reboots on other ccr like 1009.

I think they pushed these releases without proper testing and in hurry.
Maybe that's why they slowed down now. No any release from December

lele · Fri Jan 28, 2022 7:09 pm

Any update guys ?
Does anyone getting reboots with vi stable ?
For me, it's still rebooting with v7.1.1
Everbody I know with this issue has had it gone entirely since 7.1.1, except for you. Probably there are two reasons for the reboots, one is fixed by 7.1.1 and the other is possibly configuration specific. If you share your config for the device, we might be able to identify the issue.

Count me in too. We have seen reboots on 7.1.1 too, but we're not using 7.x in production.

RohanAJoshi · Sat Jan 29, 2022 7:10 am

Everbody I know with this issue has had it gone entirely since 7.1.1, except for you. Probably there are two reasons for the reboots, one is fixed by 7.1.1 and the other is possibly configuration specific. If you share your config for the device, we might be able to identify the issue.
Count me in too. We have seen reboots on 7.1.1 too, but we're not using 7.x in production.

Yes, issue is not solved at all.
But there are lower reports than before because lot users moved from it like me ( kept it in shelf, using 1009 now ). And they are not recommending ccr2004 to anyone.

mducharme · Sat Jan 29, 2022 6:40 pm

I know some people whose reboots have completely gone away since the upgrade to 7.x (3 months+ uptime vs only days previously), so I can say it is not true for everybody that it has not fixed things.

brunoforcajovem · Sat Jan 29, 2022 8:32 pm

If its just doing ospf you may consider 7.1rc4. Been pretty stable for mine as a firewall. 30 days tops with any 6 release. There's a slight configuration change but it seems good.
I have another 2004, we already put the V7 on it, the only problem presented was the IPv6 did not close adjacent, as behind it there is a BRAS for pppoe, I'm thinking of making a tunnel and taking IPv6 to the edges, the only solution I found to use to 2004 with the V7. Next week I'm going to change this 2004 and put the other one with the V7. Let us pray!!!

My best scenario is with V7, I only use it here doing ospf going from 3 to 4G, I use a reboot script that over time I increase a few days, so since I made the change I'm validating it, today I reboot every 15 days at 03:30 AM. I didn't have the courage to leave 2004 straight without this script.

hubber · Tue Feb 01, 2022 11:33 am

system,error,critical router was rebooted without proper shutdown by watchdog timer
MikroTik RouterOS 6.49.2
version: 6.49.2 (stable)
build-time: Dec/03/2021 14:53:53
factory-software: 6.47.8
free-memory: 1715.1MiB
total-memory: 1792.0MiB
cpu: ARMv7
cpu-count: 4
cpu-frequency: 1700MHz
cpu-load: 0%
free-hdd-space: 91.5MiB
total-hdd-space: 129.0MiB
architecture-name: arm64
board-name: CCR2004-1G-12S+2XS
platform: MikroTik

config - main router, ppp, firewall, nat, 1 vrrp, 5 vlans on bridge vlan-filtering=yes, dhcp-server, 1 simple queue 1 ip/32, 5 active sfp+ ports and 1 wan

stefan4242 · Wed Feb 02, 2022 12:26 pm

Please check out the comments on "MikroTik product news: CCR2116-12G-4S+" on youtube https://youtu.be/TVZG7TvUxXY. Someone named Mikrotik says "CCR2004 working great, did you upgrade to the latest version that has plenty of fixes for stability?" ...

Rudiger420 · Thu Feb 03, 2022 7:49 pm

Does anyone else get one CPU core constantly pegged at 95.5 - 100%?

Just put my last remaining CCR2004 on RouterOS v7.1.1 (I ripped out all of my CCR2004's about 1 year ago).

server8 · Thu Feb 03, 2022 9:43 pm

Running 6.48.4 RouterOS and firmware on all three units, no reboots yet since each was installed... but definitely watching for a reboot to occur.
Uptime: 37d 12:50:20
Uptime: 27d 12:31:48
Uptime: 12d 16:13:14

Same ros version and similar config i have reboot around every 50/60 days

issme · Thu Feb 03, 2022 10:33 pm

Same ros version and similar config i have reboot around every 50/60 days

These wound up rebooting on 6.48.4. Have since upgraded to 6.49.2 per Mikrotik support recommendation, at 55d uptime for all three devices. However am not confident that the reboot issue is resolved on this code, waiting for it to hit again. Once ROS 7.x is stable enough with OSPF etc will try that release, but probably will have to wait until mid 2022 at the earliest for the code base to be stable enough.

RohanAJoshi · Sat Feb 05, 2022 7:40 am

system,error,critical router was rebooted without proper shutdown by watchdog timer
MikroTik RouterOS 6.49.2
version: 6.49.2 (stable)
build-time: Dec/03/2021 14:53:53
factory-software: 6.47.8
free-memory: 1715.1MiB
total-memory: 1792.0MiB
cpu: ARMv7
cpu-count: 4
cpu-frequency: 1700MHz
cpu-load: 0%
free-hdd-space: 91.5MiB
total-hdd-space: 129.0MiB
architecture-name: arm64
board-name: CCR2004-1G-12S+2XS
platform: MikroTik

config - main router, ppp, firewall, nat, 1 vrrp, 5 vlans on bridge vlan-filtering=yes, dhcp-server, 1 simple queue 1 ip/32, 5 active sfp+ ports and 1 wan

Can you try latest test build v7.2rc3 ?

RohanAJoshi · Sat Feb 12, 2022 5:26 pm

Anyone ca give update about latest RC or stable build about reboot issue ?

Leok · Fri Feb 18, 2022 4:44 pm

CCR2004 on v6.49.3 uptime 17hrs 18minutes and going strong

server8 · Fri Feb 18, 2022 5:06 pm

I have had 48 days before reboot 6.48.4 now I am at 23 days since last

issme · Fri Feb 18, 2022 11:25 pm

Currently at 70 days uptime on 3 CCR2004s running v6.49.2.

Last unexpected reboot on these units hit around the 76 day mark (was then on v6.48.4), so will see if these make it the same or longer on the newer code track.

vasa85 · Wed Mar 02, 2022 9:52 pm

We are still experiencing reboots. Just sent in a supout.

CCR2004_uptime.png

VoIPYoungster · Wed Mar 02, 2022 10:18 pm

Hi, on which firmware runs your CCR2004?

guipoletto · Thu Mar 03, 2022 2:03 am

We are still experiencing reboots. Just sent in a supout.
CCR2004_uptime.png

347DAYS?

i maxed out here @60days, but the graph is the same, same shape: sometimes we get 60days of quiet, then reboot every 2days 2-3times, then 60days of quiet again. repeat for >1year

at least they boot fast!

server8 · Thu Mar 03, 2022 9:27 am

CCR2004 is an expensive paperweight, the rebbot issue without solution after 2 years on the market means that it's probably an hardware issue.

We have the last one in production and we have the same shape too

RohanAJoshi · Thu Mar 03, 2022 11:09 am

Issue is fixed in latest private test build.
Will be added to upcomming 7.2rc builds.
For me, maximum uptime was 15 days on v6 and 3 days uptime on v7
, but now I am with 19days + uptime.

djdrastic · Fri Mar 04, 2022 8:34 am

Really hope we can finally see something positive of note happening with these line of routers.
I have a series of CCR1016's I have to decomm and they just work.If not for security patches we roll out we reckon we'd probably achieve 3-5 year uptimes on them.
Not looking forward to random crashing boxes.

RohanAJoshi · Mon Mar 07, 2022 6:37 am

22 Days Uptime on private beta build which includes fixes for ccr2004.
This is maximum uptime for me.
I think they fixed issue.
These changes will be added to next rc release.

server8 · Mon Mar 07, 2022 8:49 am

22 Days Uptime on private beta build which includes fixes for ccr2004.
This is maximum uptime for me.
I think they fixed issue.
These changes will be added to next rc release.

I have 40 max 50 days uptime before reboot it's early to call for a victory

but I hope you are true

mkx · Mon Mar 07, 2022 8:58 am

it's early to call for a victory but I hope you are true

Let's keep calling this a victory ... until we're proven wrong

I sincerely hope we won't be.

mducharme · Mon Mar 07, 2022 9:10 am

I hope so too. Most people I know have had the remaining issues disappear in RouterOS v7. I suspect the remaining issues are instead more general bugs and not platform specific.

We have a bunch of CCR2004's, never deployed, collecting dust. I would like to be able to deploy them.

RohanAJoshi · Mon Mar 07, 2022 1:53 pm

22 Days Uptime on private beta build which includes fixes for ccr2004.
This is maximum uptime for me.
I think they fixed issue.
These changes will be added to next rc release.
I have 40 max 50 days uptime before reboot it's early to call for a victory but I hope you are true

It's your maximum uptime, I never gone that far.
My maximum was 18 days on by with given Changes by support.
And I think reboots was configuration specific.
Mostly for those who use it for ppp server like me.
Many users didn't get reboots at all.
Maybe your configuration was least affected and my configuration was most affected for this reboot cause.
But 23 days is victory for me.
And I am sure they found proper cause and fixed it.

pablorib · Mon Mar 14, 2022 4:43 pm

We are still experiencing reboots. Just sent in a supout.
CCR2004_uptime.png

@vasa85 what version of ROS are you using?

RohanAJoshi · Tue Mar 15, 2022 7:53 am

30 days uptime and going on.
With private beta build v7.3beta15

mk47 · Wed Mar 16, 2022 7:05 pm

I've been using CCR2004+ at home for 6 weeks now and haven't had a single reboot since I bought it. I am on v7.1.2. Current uptime 32d following an upgrage from 7.1.1 (I think)

sirbryan · Thu Mar 17, 2022 1:26 am

I have two CCR2004-1G-12X+2XS routers. One is in a datacenter acting as my border router to one ISP, the second is acting as my home router.
I also have a CCR2004-16G-2S+ terminating a VPN tunnel from the datacenter 2004 to my shop.

The datacenter router is running 6.47.10 and has an uptime of 127 days 12 hours. It has been in production the whole time handling traffic for a dozen servers and hundreds of customers.

The shop router (7.1.3 now) only has 23 days of uptime because I was testing 7.2.rc4 and the 7.1.x series.
The home router (7.1.3) has 21 days for the same reason.

Except for some of the 7.x testing I've done on the home router, none of them have spontaneously rebooted on me.

arpiska · Thu Mar 17, 2022 2:20 pm

Hello,

I want to buy a router and switch for a medium processing plant.
We would use it for a backbone network, which runs entirely on optics.

I have selected two device:
- for router: CCR2004-1G-12S+2XS - for: routing, vlan, firewall, vpn, qos
- for switch: CRS326-24S+2Q+RM - for: backbone network, vlan

I liked this router because sfp ports are universal, but two things make me uncertain:
1. problem with reboot, is this problem still present with v7 ?
2. similar price range device at least 2 SFP+ ports is not available

For a switch, the point is to have at least 16 SFP ports in it for our needs and reserve, and I'm thinking in SFP+ for the future.

I asking for help to make a choice, are these devices satisfying for me or would you recommend another one?

Thank you!

markonen · Fri Mar 18, 2022 8:51 pm

I went to apply some overdue upgrades on one of our workhorse CCR2004s and remembered this thread. This is with full BGP tables and 2-10G of 24/7 web traffic.

francescoc · Mon Mar 21, 2022 2:09 pm

Hi, I'm Italian ISP, and we also have 2 CCR2004 that reboot for no reason.
I heard someone talking that all this did not happen with OS7 because it is specific for these devices. Anyone have experience?

pablorib · Tue Mar 22, 2022 3:44 pm

for version 6.49.1 (stable)
sergejs b. indicates the following

There are two potential fixes for the particular problem.

1) Soft fix to modify queues,

/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default

2) There are few important fixes in v7 regarding that matter, we can provide you with 7.3beta packages, if you have an option to use beta packages on your device.

RohanAJoshi · Wed Mar 23, 2022 5:40 am

for version 6.49.1 (stable)
sergejs b. indicates the following

There are two potential fixes for the particular problem.

1) Soft fix to modify queues,

/queue type set ethernet-default pfifo-limit=300
/queue interface set [find where queue!=no-queue] queue=ethernet-default

2) There are few important fixes in v7 regarding that matter, we can provide you with 7.3beta packages, if you have an option to use beta packages on your device.

Those soft fixes about que related changes fixes nothing. You still get reboots.
And yes, 7.3beta15 fixes this reboot issues permanently and it will soon get on stable 7 series, they are not sure about including it in v6.
I am on 38days uptime.

jamescr · Wed Mar 23, 2022 5:13 pm

It looks like RouterOS 7.2 RC5 that came out today includes the CCR2004 fixes. I've not had a chance to update our CCR2004s just yet to see if it helps

RohanAJoshi · Sat Apr 02, 2022 7:18 pm

I am with 49 days uptime with v7.3beta15.
So, this fixes my issue permanently.
Those changes are added to latest rc which is v7.2rc7 and will be added to stable v7.2 release soon.
But support is not sure when these fixes will be added to v6 branch.
So guys, try latest rc.

Wed Apr 06, 2022 4:55 pm

Thank you very much for everyone, who helped us with the provided information, feedback and for the patience.
It took us unusual effort to nail this one, and hopefully biggest part of mysterious reboots should be fix with this commit,

What's new in 7.2 (2022-Mar-31 12:11):

*) ccr2004 - improved system stability on CCR2004-12S+2XS;

P.S.
As v7.x is a little different than v6, currently fix backporting might take unreasonable amount of time, and it should not be expected at v6 at the particular moment.

server8 · Thu Apr 07, 2022 10:15 am

I know that it's early but someone else can report that the issue is solved with 7.2?

Thank

lele · Thu Apr 07, 2022 2:44 pm

As v7.x is a little different than v6, currently fix backporting might take unreasonable amount of time, and it should not be expected at v6 at the particular moment.

Ah yes "long term support".
So correct me if I am wrong. We're left to choose if we want to keep rebooting our production routers every week, or upgrade to a beta quality release with missing/incomplete features and no upgrade path from 6 that doesn't end into to a broken configuration?

Thu Apr 07, 2022 2:50 pm

As v7.x is a little different than v6, currently fix backporting might take unreasonable amount of time, and it should not be expected at v6 at the particular moment.
Ah yes "long term support".
So correct me if I am wrong. We're left to choose if we want to keep rebooting our production routers every week, or upgrade to a beta quality release with missing/incomplete features and no upgrade path from 6 that doesn't end into to a broken configuration?

Please let us know if you are aware of any issues with current 7.x version on your setups.

IanICTSG · Thu Apr 07, 2022 2:58 pm

Hey guys,

Just wanted to share:

No issues whatsoever here on v7.0.4

lele · Thu Apr 07, 2022 3:22 pm

Please let us know if you are aware of any issues with current 7.x version on your setups.

Just to name an handful that comes to mind:

- So far (admittedly before 7.2, we haven't been trying that yet) we haven't had a single 6 > 7 upgrade that did not need manual adjustments to work as intended. Mainly in the router filters/bgp area
- OSPF has plenty of instabilities compared to 6.x (or it has in a network with mixed 6.x, 7.x devices)
- BGP had dozen of issues we ran into, missing features too, sometimes it even just doesn't advertise prefixes unless you manually flap the peer
- We had a number of instances where a learned route was withdrawn and not shown in the UX/CLI but was clearly still in the kernel routing table and kept active until we flapped the relevant interfaces. This happens in 6.x very occasionally, but started to happen regularily in in the upgraded 7.x boxes
- connecting to a 7.2 crashes older winbox (all of them, if you have 32 windows open)
- not to mention, configurations are incompatible, so we need to migrate our internal provisioning tools

We have 500+ mikrotik devices in the internal network plus some 3500 as CPEs, can't just click a button and hope for the best.

Thu Apr 07, 2022 3:38 pm

1) Could you specify such setup, where adjustment is required? So we can fix it shortly. (based on my experience filter converting works on 99.99% even with 7.1).
2)3) Once again OSPF is rock solid at 7.2. Regarding BGP/OSPF, make sure you are not redistributing full feed later by OSPF.
4) We didn't notice reported things in the reports. It would be nice to get more information (support@mikrotik.com).
5) Winbox has been dramatically improved, newest version required for it.

clambert · Thu Apr 07, 2022 5:03 pm

Ah yes "long term support".
So correct me if I am wrong. We're left to choose if we want to keep rebooting our production routers every week, or upgrade to a beta quality release with missing/incomplete features and no upgrade path from 6 that doesn't end into to a broken configuration?
Please let us know if you are aware of any issues with current 7.x version on your setups.

@sergejs, as TheNetworkBerg indicates in the following post: viewtopic.php?t=184778#p924179, BGP/MPLS Layer 3 VPN is not working correctly in ROSV7.2. This situation prevents us from using this version in our CCR2004.

RohanAJoshi · Fri Apr 08, 2022 8:21 am

I know that it's early but someone else can report that the issue is solved with 7.2?

Thank

Yes, I can confirm it.
Running 54 days and no reboot. ( With private built of v7.3rc15 which contains same fix as v7.2 contains. )
Before, only 15 days was max for me.

guipoletto · Fri Apr 08, 2022 8:48 am

V7.1.5 looks alot better than v6, even before the purported v7.2 fixes:

flaviormedeirosjr · Thu May 26, 2022 6:40 am

Now I realized that the problem is only BGP/MPLS, correct?
Will there be any update forecast with this fix?

murrayis · Fri Jun 17, 2022 7:26 am

Still can't use V7 without BFD on our 2004's so they're rebooting a couple times a year only luckily

aluputi · Fri Jul 15, 2022 12:27 pm

We have two new CCR2004-1G-12S+2XS, coming from different batches (one came with ROS6 and the other with ROS7).
We run a BGP connection with 2 providers but with multiple connections (main / backup).

We are seeing a memory leak, the routers are eating around 80MB of memory each day.
The issue is that because of this, when the memory usage reaches around 2GB the routers get stuck and must be rebooted.

There is nothing else configured, only the BGP and some drop rules in the Firewall. We also tried with SNMP enabled / disabled, makes no difference.
This happens on ROS 7.2 as well as on the 7.3.1. Downgrading to ROS 6 isn't an option since one of them came with 7.

For now we're rebooting every 2 weeks, during the night, to avoid downtime... but this isn't a long term solution.

Any help would be highly appreciated!

EdgarYans · Mon Jul 25, 2022 4:47 pm

Already registered, from incomprehensibility.
I have two CCR2004-1G-12S+2XS.
One was recently installed.
Both work in "cluster", 100vlans are up on the interfaces. About 100 vrrp int. DHCP in 30 vlans, and ospf.

And here is one of them, which was taken not so long ago over the past 2 weeks, has already rebooted 1 time with the watchdog log.
After the first case, I decided to disable watchdog. Now, 4 days later, it's stuck again. But already without a reboot, because. apparently watchdog - still worth turning on.
At the same time, the router that is in stand-by status is rebooted, the traffic on it goes up to 300 meters.

And on the main gigabit and rare peaks up to 3 gigabit. The firmware on the first is 7.3.1, on the second it has already been updated to 7.4

And also an incomprehensible story with SFP +:

There is Cisco Nexus3500 C3548P-10G and Mikrotik CCR2004-1G-12S+2XS.
There are SFP+ modules (10G) mikrotik, fang hang, ubiquiti.
I'm trying to raise a link between two pieces of iron between them, and it does not rise.
Tried a couple of different sfp modules. Mess around with negotion. Change patch cords.
Thus for example between Nexus - and other cisco the link rises without problems. Mikrotik to Mikrotik - no problem either.
Signal levels on both sides are shown as normal. (All within one rack)

And as a workaround, I temporarily solved the problem like this:
I took CCR2004-1G-12S+2XS.
I connected one sfp module to it. I connected the second sfp module to the neighboring interface, raised the link between them without any problems.
After one of these sfp modules I quickly switch to nexus, connect optics and everything starts up. Everything is working. Until the link falls, or any piece of hardware is rebooted.

This raises the question of what is wrong and how to live with it.

magnavox · Fri Jul 29, 2022 5:41 pm

Hi all,
so, this issue, CCR2004 rebooting, still not solved?

RohanAJoshi · Fri Jul 29, 2022 5:49 pm

Hi all,
so, this issue, CCR2004 rebooting, still not solved?

It's totally resolved after 7.3 update.
But some users having issue in BGP.

magnavox · Fri Jul 29, 2022 6:43 pm

Hi all,
so, this issue, CCR2004 rebooting, still not solved?

Oh very very nice!!!

But some users having issue in BGP.

Like that issue?

magnavox · Fri Jul 29, 2022 6:53 pm

Hi all,
so, this issue, CCR2004 rebooting, still not solved?
It's totally resolved after 7.3 update.

This is for alle CCR2004?

CCR2004-1G-12S+2XS
CCR2004-16G-2S+

kraljek · Tue Aug 16, 2022 5:31 pm

Hi all,
so, this issue, CCR2004 rebooting, still not solved?
It's totally resolved after 7.3 update.
But some users having issue in BGP.

Is this 100% info that reboot is resolved?

herger · Tue Aug 16, 2022 8:50 pm

Please let us know if you are aware of any issues with current 7.x version on your setups.

For us v7 is a no go as we cannot use ipipv6 tunnels with ospfv3 -- SUP-70226 open since Dec/21; no fix on the horizon so far.

korsakik · Tue Jan 10, 2023 7:51 am

Uptime 153d 01:16:02
CCR2004-1G-12S+2XS
7.4.1 (stable)

But I have problem, when I remove static route, they goes to deleted, but routes still working...
Via /export or /route print they not presented.

carl0s · Thu Jan 12, 2023 12:08 pm

So are these safe to buy now?

(when stock available !)

icttech · Mon Jan 16, 2023 12:22 am

Anyone still seeing same reboots with BGP on v7.5 or 7.6

DelwinSand · Mon Jan 30, 2023 10:44 am

Anyone still seeing same reboots with BGP on v7.5 or 7.6

Yes! Still has a problem on 7.5 7.6 and 7.7 version, but I have no BGP, I'm using this model as regular main office router (vlan, gateway, firewall, dhcp, capsman and vpn server). It has not much load, but reboot one time a week or five times a day. On 7.1.5 or 7.2 I don't have this issue.

ericsooter · Thu Feb 02, 2023 6:22 am

Disabling connection tracking was my fix. There is some sort of bug currently affecting 2004 and 2116 models on 7.6, 7.7 and 7.8beta2 in certain configurations. I tried all combos with both models and was rebooting 2-5 times daily.

Currently being discussed in this thread: viewtopic.php?p=981476#p981476

Stefan10G · Mon Apr 17, 2023 5:36 pm

Did anyone already install the latest firmware 7.8 stable on his CCR2004-1G-12S+2XS?

Is the device stable now also in NAT setups?

This one here: https://mikrotik.com/download/changelogs

I am heavily thinking about buying a MikroTik CCR2004-1G-12S+2XS as a quiet powerful home router in a 10GBit network with an 1 GBit Fiber Uplink (so NAT and Connection Tracking is urgently required; and that seems to be one of the origins of rebooting) as I think it would quite well match my requirements, but of course I don't want to get into the deep trap of reboot hell.

Stefan10G · Sat Apr 22, 2023 10:21 pm

No one experience with 7.8 on CCR2004 with all SFP+ interfaces?

RohanAJoshi · Sun Apr 30, 2023 1:16 pm

In last 2 weeks I changed my radius provider to Icon Radius.
They Use SNMP and API services for their purpose to utilise function in portal.
It's causing memory filling so fast that in 1 week it's grabs 700mb memory and when free memory reaches 3gb mark ( when restarted it's 3.8gb free ) this ccr get reboot.
Log shows reboot by watchdog timer, raised ticket to support but not a single reply in month.
Also, since then, I was getting syn flood attack on my public ip pool.
Then I blocked ( chain input and action tarpit ) all external traffic other than my allowed pool.
.
I don't know what's causing this.
Radius support says configuration is ok, there no issue.

j081 · Mon May 29, 2023 2:31 am

Is anyone else having winbox issues with the CCR2004's? We have two of these routers on our network, one is running v7.9 and another is running v7.9.1, however winbox v3.38 crashes after logging in to our 2004's. Winbox v3.35 seems to work fine, and winbox v3.38 works when accessing one of our RB4011's running v7.9.1 so I am wondering if the issue is hardware related.

Network5 · Mon May 29, 2023 12:26 pm

We have several CCR2004 in lab and in production, running 7.8, 7.9 and 7.10rc, no issues with winbox.

j081 · Tue May 30, 2023 1:36 am

We have several CCR2004 in lab and in production, running 7.8, 7.9 and 7.10rc, no issues with winbox.

Thanks, it seems my issue was actually to do with the local winbox session folder on my PC. Deleting the sessions folder under Winbox>Tools and clearing my winbox cache fixed the issue.

ranvijay · Wed Sep 13, 2023 9:28 pm

As I overserved there is fan issue with CCR2004. I have two CCR2004. One is working perfectly whereas second one has random rebooting issue.
Which one is working fine his fan run periodically.

system health print
Problematic CCR2004
system health print
cpu-temperature: 56C
board-temperature1: 40C
board-temperature2: 41C
psu1-state: ok
psu2-state: fail
fan1-speed: 0RPM
fan2-speed: 0RPM

CCR2004 good condition.
temperature: 57C
cpu-temperature: 53C
board-temperature1: 46C
board-temperature2: 50C
psu1-state: ok
psu2-state: fail
fan1-speed: 3915RPM
fan2-speed: 3960RPM

Anyone have any idea how to fix fan issue?

ZPrime · Thu Oct 12, 2023 2:01 am

ccr2004-2years-uptime.png

This is running 6.48.6, and has no dynamic routing at all, all route entries are static.
It is acting as a firewall & NAT device in front of some servers.

We're lucky to see more than 60 days of uptime without a restart.
It is in a climate controlled datacenter with stable & clean power (from UPS and generator), so "environmental concerns" are not part of the problem.

All of the reboots throw up log messages that the device was rebooted "without proper shutdown by watchdog timer" (we have no watchdog settings configured, it is at default, so it is not trying to ping anything, although it does generate an automatic supout.rif).

Here's another 2004, this one is running dynamic routing (both iBGP and OSPF), same LTS version of RouterOS. This one is "in the field" at a site without protected power or any extensive cooling, but it doesn't lose power very frequently and the temperatures seem tolerable ("cpu-temperature" sitting around 40C most of the time)

other-ccr2004-2years-uptime.png

We have not moved either of these to v7.x yet due to concerns with incomplete features / etc over there...

Thu Oct 12, 2023 1:00 pm

This is for alle CCR2004?

CCR2004-1G-12S+2XS
CCR2004-16G-2S+

No, only the CCR2004-1G-12S+2XS

amt · Thu Dec 14, 2023 5:30 pm

We just had a 2004 have all interfaces go down and up again. Also seems OSPF dident fully recover, so in the end we rebooted it.

/Mikael

2023 is almost over and the problem still persists, is any one test and get good results with v7.12.1 ?