CCR2004 - random crashes watchdog no IP address

jweidman · Sun Jun 05, 2022 2:04 am

CCR2004-1G-12S+2X - random crashes watchdog no ip address

i have had this issue for over a year. It seemed to go away for 4 months or so with little to no reboots then bam its back and 3 do it within 5 hours. all routers are ccr 2004, 1 is brand new updated to 6.49, others are on 6.48. the only thing i was linking this to was OSPF bug but that was just based on reading i was doing and no proof. the only thing in the logs is (system,error,critical ) router was improperly shutdown by watchdog timer. like i said sometimes 2 or 3 times in a day then not for a month. but this is just wreaking havoc on my uptime.

i'm at a loss does anyone have any ideas? I want to get better logs by hooking up to console can we enable advanced logging to dump what is going on? i cant shut the watchdog off and let it hang until i can get to it that's insane.

jweidman · Sun Jun 05, 2022 2:41 am

I posted this once but it didnt show up for some reason. I am getting reboots the only log i get is critical, error, system improper shutdown improperly by watchdog timer. watchdog is enabled with default settings with no added ip address. based on reading i was wondering if this was an ospf bug but not sure. I have tried 6.49 code and 48,old hardware and new hardware. what is weird is the problem seems to go away, then comes back with a vengeance and will hit 2 to 3 units within a day. sometimes no reboots for 30 days sometimes 1 device in 3 days.

this is driving me insane. how do i get more logs via console? is there advanced logging, is there a solution? this is wreaking havoc on my uptime! if anyone can help please let me know

mkx · Sun Jun 05, 2022 3:01 pm

First thing is to configure other logging destinations ... memory gets cleared with reboot and any potential log entries from before reboot which might shed some light are gone.

jweidman · Sun Jun 05, 2022 11:43 pm

that was why i had asked about console logging - can i get it to dump to physical console port and do i need to enable any advanced logging or extend timeouts on the console? this issue is rampant all over the place with tons of people having same issue. Has there been a core issue for the ccr2004 reboots from the watchdog?

First thing is to configure other logging destinations ... memory gets cleared with reboot and any potential log entries from before reboot which might shed some light are gone.

mkx · Mon Jun 06, 2022 8:37 am

Not sure about console logging ... I'd go with logs written to disk files. With 10 log files (rotated as they fill in) and 1000 lines per log file logs will consume around 1MB of precious disk space in total (100kB per file). Depending on number of events logged such setup might contain more than one month worth of logs entries.

Logging to internal disk does add some disk writes, but with your rate of reboots you'll be running extensive disk logging only for a few days at most. I guess.

Cris@usai.net · Sun Nov 20, 2022 7:15 pm

Any more info on this... Having a Similar problem with a CCR2116-4S+
Reboot from watchdog timer.

did it about 8 times in a row.. and then stabilized .

jweidman · Sun Dec 25, 2022 1:00 am

got so bad i had to rip and replace - now i'm getting a reboot " probably power related" on the 2216. makes zero sense -its just the core router something isnt stable and they dont know how to log it.

Any more info on this... Having a Similar problem with a CCR2116-4S+
Reboot from watchdog timer.

did it about 8 times in a row.. and then stabilized .

ericsooter · Wed Jan 18, 2023 6:03 pm

We also had a CCR2004 that also would do random chain reboots everyone few weeks with the "watchdog" error message. Upgraded it to 7.7 and problem seemed to worsen (watchdog reboots every few hours). So I replaced router with a CCR2116-12G-4S+ running version 7.7. Now it's not chain rebooting; but rebooting once or twice daily with same "watchdog" error. My configuration uses a Bonding interface with two SFP+ ports as the members. Bonding is 802.3ad LAGing over two stacked switches. And then we run vlans (about 12) under the bond interface. Other than that, nothing too crazy (ospf and some light natting). Anyone else experiencing this problem and running bonding? I've had a few other sites with the exact configuration and ended up downgrading to CCR1036's and problem solved. I'm just disabled one of the slave interfaces; just to see if everything starts to go stable. I've purchased quite a few CCR2116 and even some CCR2216's for my edge. Lots of nice power with these newer routers. But I'm terrified to use them in core/edge infrastructure until I know this problem is resolved.

Cris@usai.net · Wed Jan 18, 2023 6:08 pm

Did anyone attempt disabling Connection tracking in the FW ?
As long as your not using anything requires it : Nat... etc.. etc..
Seems that my router is tracking all the connections that it really has no need to watch for.. I'm just using this to route public traffic.
I see it watching 40-50k sip connections all the time. I just want it to pass the packets, no need to look at them.. let the destination devices firewall deal with it..

I'm going to disable it and see if it makes a difference..

Thoughts.

hbjlee17 · Wed Jan 18, 2023 7:18 pm

we have had CCR2116-12G-4S+ and the same issue
the router reboots itself due to watchdog timer.
I have updated the software and firmware all the way up to 7.7 and still experience the same issue. CPU cores goes to 100% then watchdog reboots the router.

Cris@usai.net · Wed Jan 18, 2023 7:51 pm

did you disable connection tracking?

ffries · Wed Jan 18, 2023 7:56 pm

When I reinstalled on of my routers I had the same problem and after netboot reinstallation (took me lot of time) this watchdog problem was gone. You may also disable watchdog.

You can enable ping Watchdog by specifying an IP address and you can disable the software Watchdog by unsetting the Watchdog Timer option.

In the log it shows that there is no DNS resolution. It could be a wrong DNS, a wrong IP or no gateway or any networking problem.

If there is not IP on your router and no valid gateway it cannot ping and will reboot. You need to unset watchdog timer and set IP+gateway and fix your network problems. When this is fixed, set watchdog timer again.

Hope this helps.

hbjlee17 · Wed Jan 18, 2023 8:07 pm

did you disable connection tracking?

at this point I am trying everything, so I currently have ip > firewall > connection > tracking disabled

ffries · Wed Jan 18, 2023 8:08 pm

Setting watchdog timer to zero will disable watchdog. Then fix your networking problems.

hbjlee17 · Wed Jan 18, 2023 8:27 pm

Setting watchdog timer to zero will disable watchdog. Then fix your networking problems.

There are no networking issues.

The watchdog is resetting the router because the router software glitches. and becomes entirely unresponsive, multiple CPU cores at 100%.
I say software glitches because I had already tried swapping the router with another identical model and the issue persisted.

If the watchdog is disabled, it will leave the router in the hung state; and WILL require you to manually reset the router. so I strongly advice against disabling watchdog.

ericsooter · Fri Jan 20, 2023 4:36 pm

I tried disabling the bonding port and it didn't seem to make a difference. So I downgraded the CCR2116 back to 7.6 and so far no reboots for a few days. On version 7.7, it was rebooting 2-3 times daily and sometimes chain-rebooting (2-3 times in a row). So 7.7 made rebooting MUCH worse. If 7.6 is anything like with the CCR2004, I suspect it will start rebooting much less frequently. So far uptime of almost 2 days with no "watchdog reboots". I haven't tried disabling the watchdog timer; worried about locking the router up indefinitely. Also someone mentioned disabling connection tracking; unfortunately, I am doing some light NATing on this router. So I have to leave it on.

gmsmstr · Fri Jan 20, 2023 7:15 pm

Be sure the router is secured as well.. Just a FYI..

We have seen these stop rebooting once we disable connection tracking.

cwachs · Tue Jan 24, 2023 12:03 am

Following. Brand new 2216 with 7.7 and it reboots multiple times. I do have connection tracking on since the router has some NAT on it. The config is a clone of a CCR1036 that has never rebooted in 2 years. As soon as that config was put on the 2216, reboots started within a couple hours.

shdwmstff · Tue Jan 24, 2023 1:56 am

Same problem.. had a CCR1036 working fine.. same config on a CCR2116 and it reboots...
Disabled connection tracking and it has been stable for 3 days.

Is it possible that the ROS is having a problem with the ARM64 processor
The CCR1036 is a Tile processor.. maybe that is where Mikrotik should start their debugging.
I don't believe that there is a "CONFIG" problem. I Have been emailing SUPOUT files to Support for 2 monthes.. and they keep telling me to upgrade to the newest firmware.
So I updated to 7.7 and it still rebooted till I disabled Connection tracking.. Crossing fingers that it stays stable.

cwachs · Tue Jan 24, 2023 5:43 am

This is the same exact issue we saw two years ago when the original CCR2004 came out. Those would reboot randomly as well. People said the same thing then... Turn off connection tracking. We sent file after file to Mikrotik. Even shipped two routers back to them. Nothing fixed it until one day a new firmware came out and it stopped rebooting. No official word that I ever saw. Our 2004 went a year without a reboot.

I expect the same thing here. Not all 2216s have this issue so trying to figure out exactly what is causing it is very hard. But there is an issue. Too many of us have the same reboot problem. It might be arm64 related but it's not happening on a 2004 with a nearly identical config. This is specific to the 2216.

hbjlee17 · Tue Jan 24, 2023 11:21 pm

Same problem.. had a CCR1036 working fine.. same config on a CCR2116 and it reboots...
Disabled connection tracking and it has been stable for 3 days.

Is it possible that the ROS is having a problem with the ARM64 processor
The CCR1036 is a Tile processor.. maybe that is where Mikrotik should start their debugging.
I don't believe that there is a "CONFIG" problem. I Have been emailing SUPOUT files to Support for 2 monthes.. and they keep telling me to upgrade to the newest firmware.
So I updated to 7.7 and it still rebooted till I disabled Connection tracking.. Crossing fingers that it stays stable.

Same thing, I had CCR1036s without issue; since upgrading (if you want to call it that) to CCR2116s the rebooting issue has persisted through all the stable branch of firmware, all the way up to 7.7.

I also have connection tracking disabled and it has not rebooted for a week. If it reboots again I am either switching back to CCR1036, or a different vender all together.

DaviV · Wed Jan 25, 2023 12:44 am

I tried disabling the bonding port and it didn't seem to make a difference. So I downgraded the CCR2116 back to 7.6 and so far no reboots for a few days. On version 7.7, it was rebooting 2-3 times daily and sometimes chain-rebooting (2-3 times in a row). So 7.7 made rebooting MUCH worse. If 7.6 is anything like with the CCR2004, I suspect it will start rebooting much less frequently. So far uptime of almost 2 days with no "watchdog reboots". I haven't tried disabling the watchdog timer; worried about locking the router up indefinitely. Also someone mentioned disabling connection tracking; unfortunately, I am doing some light NATing on this router. So I have to leave it on.

Hi, using pair of 2116 with 2x bonding 802.3ad on copper. one on 7.5 one on 7.6. not a single issue since they were installed ( 116 days/90days)
basically doing just NAT there.
2004 also pair in production apart form having strange bug that remote syslog is hitting drop all rule by going into localhost... all good

I am really interested in this topic as 2004/2116 is said to be replacement for 1009/1036.
Distis are saying no more 1009/1036. ( we have around 1k of them) and I am noticing constant compains about its stability. Maybe its CHR time....

gmsmstr · Wed Jan 25, 2023 4:37 pm

We have about 30 1009s still left. Just a FYI.

ericsooter · Thu Jan 26, 2023 4:28 pm

CCR2116 started chain rebooting yesterday with watchdog error. Just to confirm what has already been mentioned. I did pull up the CPU load and watched 4 cores slowly max out to 100pct and then it rebooted. So this tends to be the pattern. I even tried the latest 7.8beta2 just to see if it changed any; it's rebooted 3 times in 24 hours since then; so no fix yet. I think I'll try what Dennis mentioned and move my NATs to another router and turn off connection tracking.

DaviV · Mon Jan 30, 2023 4:18 pm

We have about 30 1009s still left. Just a FYI.

Thanks, but I guess its not worth to ship them into EU

gmsmstr · Mon Jan 30, 2023 4:40 pm

We have about 30 1009s still left. Just a FYI.
Thanks, but I guess its not worth to ship them into EU

Maybe not. . Up to you though.

ericsooter · Mon Jan 30, 2023 5:09 pm

So I moved my NATs to another router and completely disabled "Connection Tracking". So far, been stable for about 4-5 days (instead of 1-2 reboots per day); longest run so far. This router was in front of our server network which was handling our private servers and some office network. So when NATing was on, it tended to have a very large amount of connections to track. I'm thinking the problem with the CCR2xxx is with heavy connection tracking.

gmsmstr · Mon Jan 30, 2023 5:43 pm

So I moved my NATs to another router and completely disabled "Connection Tracking". So far, been stable for about 4-5 days (instead of 1-2 reboots per day); longest run so far. This router was in front of our server network which was handling our private servers and some office network. So when NATing was on, it tended to have a very large amount of connections to track. I'm thinking the problem with the CCR2xxx is with heavy connection tracking.

Yep, we have turned off connection tracking on a number of customer routers, stopped the reboots. the ones we turned it off on did not have "heavy' NAT usage, really just for management devices, but still. I hope MT is following this tread to help out.

ericsooter · Mon Jan 30, 2023 6:05 pm

I've sent them quite a few supout files. So hopefully they are taking a look.

sirbryan · Tue Jan 31, 2023 9:58 am

None of my 2004's (had two for a while) and 2116's (five in production) exhibited this behavior. I'll share what has worked for me in hopes that it helps others.

My CRS310's randomly rebooted if I had an IP address in the watchdog settings, even though watchdog timer was set to "no". I cleared out the watch address and they stopped rebooting. (Interestingly, both my border 2116's have it set to "yes" with no address.)

Both 2004's and 2116's use IP's assigned to VLAN's on the main bridge, with a couple of exceptions where the VLAN is assigned to specific interfaces. Bridge VLAN filtering is enabled. Three transport 2116 routers have L3HW offload enabled and are running 7.4.1. Two NAT 2116's have worked fine on 7.4.1, 7.6, 7.7, and are now running 7.8b2 for testing the new ROSE features. L3HW offload is off for the NAT boxes (it breaks more things than it helps). A 2004 has been configured for a new site, but has no traffic flowing.

HTTPS is configured and HTTP, FTP, Telnet, other unused services are disabled. All services in the NAT section (that can be) are also disabled.

Tracking is enabled for both NAT routers and the border routers. The NAT routers handle websites, streaming cameras, VoIP, Speedtest.net server, UISP/UniFi polling, and church video broadcasts for a few hundred users each weekend. So not a lot of bandwidth, but plenty of connections over the course of 7 days.

On the border routers, mangle rules are in place to not track anything but the routers' own generated traffic (for firewall connection state purposes).

Here's my watchdog and connection tracking settings for the 2116's.

 /system/watchdog> print
    watch-address: none
         watchdog-timer: yes
  ping-start-after-boot: 5m
           ping-timeout: 1m
       automatic-supout: yes
       auto-send-supout: no

/ip/firewall/connection/tracking> print
 enabled: auto
      tcp-syn-sent-timeout: 5s
  tcp-syn-received-timeout: 5s
   tcp-established-timeout: 1d
      tcp-fin-wait-timeout: 10s
    tcp-close-wait-timeout: 10s
      tcp-last-ack-timeout: 10s
     tcp-time-wait-timeout: 10s
         tcp-close-timeout: 10s
   tcp-max-retrans-timeout: 5m
       tcp-unacked-timeout: 5m
        loose-tcp-tracking: yes
               udp-timeout: 10s
        udp-stream-timeout: 3m
              icmp-timeout: 10s
           generic-timeout: 10m
               max-entries: 1048576
             total-entries: 1102

ericsooter · Tue Jan 31, 2023 7:05 pm

Sirbryan, thanks for that information. I did try disabling the watchdog timer with a blank watch address; but that just caused it to eternally lock up when the time came (had to physically reboot it). It's always the crazy 100pct on 4 cores before it reboots or locks. My router is still stable so far after disabling connection tracking. We are also lagging 2 SFP+ ports into a single bond interface and running vlans on the 802.3ad bond. Maybe it's the combination of bonding and connection tracking that is giving us the problem.

cwachs · Tue Jan 31, 2023 7:14 pm

We had our 2216 go into a reboot loop twice now. They last about 15 minutes and will reboot around 20 times in a row before stopping. We *think* we were able to stop it by disabling OSPFv3 (we do run IPv6 on ours) but that might have been a coincidence with it just stopping on its own.

Since then, we replaced the 2216 with a CCR1036 running the same ROS 7.6 firmware and the identical config from the 2216 (with OSPFv3 enabled). No reboots. This is 100% hardware related. Still waiting to hear back from Mikrotik support on our supout.rif file.

gmsmstr · Tue Jan 31, 2023 8:06 pm

Sirbryan, thanks for that information. I did try disabling the watchdog timer with a blank watch address; but that just caused it to eternally lock up when the time came (had to physically reboot it). It's always the crazy 100pct on 4 cores before it reboots or locks. My router is still stable so far after disabling connection tracking. We are also lagging 2 SFP+ ports into a single bond interface and running vlans on the 802.3ad bond. Maybe it's the combination of bonding and connection tracking that is giving us the problem.

Yes, disabling the watchdog is good if you want to capture anything from the console, but will require a Powe cycle to get it going again. This is the bios watchdog, if the bios does not get a responce from the OS, it will reboot, this is what normally happens. Note, that it should not be a 100% cpu thing, i.e. if the router goes to 100% cpu, it should not cause this, but i could be wrong.

sirbryan · Wed Feb 01, 2023 7:47 am

It would certainly be interesting to compare configs.. figure out a way to sanitize them, then diff them and see what different people are doing.

ericsooter · Wed Feb 01, 2023 4:38 pm

I'm still stable since disabling connection tracking and moving NATs. If you want to compare configs. I did include our sanitized configuration (blanked out public ips and passwords).

With NAT and connection tracking on, both 2004 and 2116 reboot 2-3 times a day. Sometimes they chain reboot several times in a row. (this was tested on 7.6, 7.7 and 7.8beta2)

# feb/01/2023 06:17:25 by RouterOS 7.8beta2
# software id = KMRG-539B
#
# model = CCR2116-12G-4S+
# serial number = HCX081Q7RWX
/interface ethernet
set [ find default-name=ether1 ] 
set [ find default-name=sfp-sfpplus1 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full name=\
    sfp-sfpplus1-coreswitch1 speed=1Gbps
set [ find default-name=sfp-sfpplus2 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full name=\
    sfp-sfpplus2-coreswitch2 speed=1Gbps
set [ find default-name=sfp-sfpplus3 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full speed=\
    1Gbps
set [ find default-name=sfp-sfpplus4 ] advertise=10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full speed=\
    1Gbps
/interface bonding
add mode=802.3ad name=Po1 slaves=sfp-sfpplus1-coreswitch1,sfp-sfpplus2-coreswitch2 transmit-hash-policy=\
    layer-2-and-3
/interface vlan
add interface=Po1 name=vlan2-old-core-stub vlan-id=2
add interface=Po1 name=vlan3-noc-core-stub vlan-id=3
add interface=Po1 name=vlan5-public-servers vlan-id=5
add interface=Po1 name=vlan6-virtual-sites vlan-id=6
add interface=Po1 name=vlan7-public-workstation vlan-id=7
add interface=Po1 name=vlan8-private-servers vlan-id=8
add interface=Po1 name=vlan9-private-workstations vlan-id=9
add interface=Po1 name=vlan11-storage vlan-id=11
add interface=Po1 name=vlan12-positronix-servers vlan-id=12
add interface=Po1 name=vlan13-positronix-hosted vlan-id=13
add interface=Po1 name=vlan14-positronix-private vlan-id=14
add interface=Po1 name=vlan20-guest vlan-id=20
add interface=Po1 name=vlan101-voip vlan-id=101
add interface=Po1 name=vlan113-mgmnt vlan-id=113
add interface=Po1 name=vlan250-cnwave-ipv6 vlan-id=250
/ip pool
add name=dhcp_pool-mgmnt ranges=10.30.100.20-10.30.100.254
add name=dhcp_pool-voip ranges=10.81.3.50-10.81.3.254
add name=dhcp_pool-office ranges=10.80.10.50-10.80.10.254
add name=dhcp_pool-guest ranges=10.81.9.20-10.81.9.254
/ip dhcp-server
add address-pool=dhcp_pool-guest interface=vlan20-guest name=dhcp1-guest
add address-pool=dhcp_pool-mgmnt interface=vlan113-mgmnt name=dhcp2-mgmnt
add address-pool=dhcp_pool-office interface=vlan9-private-workstations name=dhcp3-office
add address-pool=dhcp_pool-voip interface=vlan101-voip name=dhcp4-voip
/port
set 0 name=serial0
/routing id
add disabled=no id=xxx name=id-1 select-dynamic-id=""
/routing ospf instance
add disabled=no name=default-v2 router-id=id-1
/routing ospf area
add disabled=no instance=default-v2 name=backbone-v2
/user group
add name=backups policy="ftp,read,sensitive,!local,!telnet,!ssh,!reboot,!write,!policy,!test,!winbox,!password,!web,\
    !sniff,!api,!romon,!rest-api"
/ip firewall connection tracking
set enabled=no
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip address
add address=xxx interface=vlan3-noc-core-stub network=xxx
add address=xxx interface=vlan6-virtual-sites network=xxx
add address=10.30.0.1/16 comment=Mgmnt interface=vlan113-mgmnt network=10.30.0.0
add address=10.81.1.1/24 comment="Workstations(private) - Moved subnet" disabled=yes interface=\
    vlan9-private-workstations network=10.81.1.0
add address=xxx comment="Workstations(public)" interface=vlan7-public-workstation network=xxx
add address=xxx comment=Voip interface=vlan101-voip network=xxx
add address=xxx comment="Servers(private)" interface=vlan8-private-servers network=xxx
add address=10.50.1.1/24 comment="Storage - Do not advertise" interface=vlan11-storage network=10.50.1.0
add address=xxx comment="Servers(public)" interface=vlan5-public-servers network=xxx
add address=xxx comment="Positronix Servers" interface=vlan12-positronix-servers network=xxx
add address=xxx comment="Positronix Hosted" interface=vlan13-positronix-hosted network=xxx
add address=xxx comment="Positronix Private Servers" interface=vlan14-positronix-private network=xxx
add address=xxx comment="Old Core Stub - Removing Later" interface=vlan2-old-core-stub network=\
    xxx
add address=10.185.1.1/24 comment="CNWave IPv4 Mgmnt Ruby" disabled=yes interface=vlan250-cnwave-ipv6 network=\
    10.185.1.0
add address=10.80.10.1/24 comment="Workstations(private)" interface=vlan9-private-workstations network=10.80.10.0
add address=10.81.9.1/24 comment="Guest Network" interface=vlan20-guest network=10.81.9.0
/ip dhcp-server lease
add address=10.81.1.51 client-id=ff:54:41:dc:74:0:1:0:1:2a:fb:e5:2e:70:f7:54:41:dc:74 mac-address=70:F7:54:41:DC:74 \
    server=dhcp3-office
/ip dhcp-server network
add address=10.30.0.0/16 gateway=10.30.0.1
add address=10.80.10.0/24 comment="Office Workstations" dns-server=xxx gateway=10.80.10.1
add address=10.81.3.0/24 gateway=10.81.3.1
add address=10.81.9.0/24 gateway=10.81.9.1
/ip dns
set servers=xxx,1.1.1.1
/ip firewall address-list
add address=xxx list=secure
(cut out all my ip lists)
/ip firewall filter
add action=accept chain=forward comment="Allow radius requests" dst-address=xxx dst-port=1812-1813 \
    protocol=udp
add action=drop chain=input comment="Secure router" protocol=tcp src-address-list=!high-secure
add action=accept chain=forward comment="Allowed public webservers" dst-address-list=allwed-public-webservers \
    dst-port=80-88,443 protocol=tcp
add action=accept chain=forward comment="Allowed sip trunks" src-address-list=voip-trunks
add action=accept chain=forward comment="Allow trusted external ips" src-address-list=trusted-external-ips
add action=drop chain=forward comment="Securing Servers" dst-address-list=servers dst-port=\
    21-23,80-88,389,443,3389,10000,5566,5900-5910 protocol=tcp src-address-list=!secure
add action=drop chain=forward comment="Securing certain web servers (outside of normal range)" dst-address-list=\
    web-blocked dst-port=21-23,80-88,389,443,3389,10000,5566,5900-5910 protocol=tcp src-address-list=!secure
add action=drop chain=forward comment="Block all inbound to ad, printers, etc" dst-address-list=fully-blocked-hosts \
    src-address-list=!secure
add action=drop chain=forward comment="Drop unsecure SSH, webmin and zimbra policyd to Positronix network" \
    dst-address-list=positronix-networks dst-port=22,10000,7780 protocol=tcp src-address-list=!secure
/ip firewall nat
((These were the NATs we had to move to another router)))
add action=accept chain=srcnat comment="NAT bypass for mgmnt to server network" disabled=yes dst-address=\
    10.80.1.0/24 src-address=10.30.0.0/16
add action=src-nat chain=srcnat comment="Guest Network NAT" disabled=yes src-address=192.168.75.0/24 to-addresses=\
    xxx
add action=src-nat chain=srcnat comment="Mgmnt NAT" disabled=yes src-address=10.30.0.0/16 to-addresses=xxx
add action=src-nat chain=srcnat comment="Office Workstation(private) NAT" disabled=yes src-address=10.81.1.0/24 \
    to-addresses=xxx
add action=src-nat chain=srcnat comment="Voip NAT" disabled=yes src-address=10.81.3.0/24 to-addresses=xxx
add action=src-nat chain=srcnat comment="Storage NAT - NO NATTING AT ALL " disabled=yes src-address=10.50.1.0/24 \
    to-addresses=xxx
/ip firewall service-port
set sip disabled=yes
/ip route
add comment="VPN Static - Routing xxx VPN through Internet Connection" disabled=no dst-address=xxxx \
    gateway=xxx
/routing ospf interface-template
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxx cost=10 disabled=no interfaces=vlan3-noc-core-stub \
    networks=xxx priority=1
add area=backbone-v2 cost=10 disabled=no interfaces=vlan6-virtual-sites networks=xxx passive priority=1
add area=backbone-v2 cost=10 disabled=no interfaces=vlan113-mgmnt networks=10.30.0.0/16 passive priority=1
add area=backbone-v2 cost=10 disabled=no interfaces=vlan20-guest networks=10.81.9.0/24 passive priority=1
add area=backbone-v2 cost=10 disabled=no interfaces=vlan7-public-workstation networks=xxx passive \
    priority=1
add area=backbone-v2 cost=10 disabled=no interfaces=vlan101-voip networks=10.81.3.0/24 passive priority=1
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxx cost=10 disabled=no interfaces=vlan8-private-servers \
    networks=10.80.1.0/24 priority=50
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxx cost=10 disabled=no interfaces=vlan5-public-servers \
    networks=xxx priority=1
add area=backbone-v2 cost=10 disabled=no interfaces=vlan12-positronix-servers networks=xxx priority=1
add area=backbone-v2 cost=10 disabled=no interfaces=vlan13-positronix-hosted networks=xxx passive \
    priority=1
add area=backbone-v2 cost=10 disabled=no interfaces=vlan14-positronix-private networks=10.80.2.0/24 passive \
    priority=1
add area=backbone-v2 auth=md5 auth-id=1 auth-key=xxx cost=10 disabled=no interfaces=vlan2-old-core-stub \
    networks=xxx priority=1
add area=backbone-v2 comment="For IPV4 CNwave Mgmnt" disabled=yes interfaces=vlan250-cnwave-ipv6 networks=\
    10.185.1.0/24 passive
add area=backbone-v2 disabled=no interfaces=vlan9-private-workstations networks=xxx passive
/system clock
set time-zone-name=America/Chicago
/system identity
set name=NOC-ServerOfficeRouter
/system ntp client
set enabled=yes
/system ntp client servers
add address=10.80.1.69
/tool romon
set enabled=yes

stuartpbb · Fri Feb 10, 2023 9:41 pm

FWIW, we are also experiencing weekly (or more frequent) reboots on our CCR2004-1G-12S+2XS running ROS v7.7. I've had a support case open (SUP-105519) for a few weeks with no response from Mikrotik.

The most recent reboot this morning looks to be the same issue that others are reporting:
07:12:16 router was rebooted without proper shutdown by watchdog timer

At one point in the past we were seeing a bit more information:
05:26:15 router was rebooted without proper shutdown by watchdog timer
05:26:15 kernel failure in previous boot
05:26:15 out of memory condition was detected

We cannot disable connection tracking as this router handles a lot of NAT (CGNAT) for customers using PPPoE. Tracking RAM usage, it does appear that the usage keeps increasing, so seems like there may be a memory leak somewhere?

ccr2004-memory.png

I'm unsure what can be done to stabilize this??? Downgrading RouterOS to 7.5 or earlier?

cwachs · Sat Feb 11, 2023 12:25 am

We have had reboots on 7.5 to 7.8beta3. All autosupout.rif files sent to Mikrotik. No response at all to the support file and it's been open for two weeks. They have 6 supout files from us from 5 different firmware. We've moved on and put these routes on a Juniper.

JimBouse · Mon Feb 13, 2023 10:40 pm

We were also seeing reboot loops on our 2216 until we disabled all FW and NAT rules. Once that was done, it has been stable for nearly a month.
It was rebooting every 1:40 (200 seconds).

I believe it is a memory leak issue but am not sure.
I sent in SUP files as well.

stuartpbb · Fri Feb 17, 2023 7:51 pm

Mikrotik Support finally responded this morning stating that "This issue is already fixed in the latest version of RouterOS 7.8rc2..". Dealing with the reboots for 3.5+ weeks to be told to try a release candidate they pushed three days prior feels like pretty poor support.

From the discussion above, I see references to others trying 7.8b2 and 7.8b3 but no mention of anyone trying 7.8rc2 ... has anyone tried this yet?

stuartpbb · Sat Feb 18, 2023 9:52 pm

Update attempting to upgrade to 7.8rc2 ... the firmware bricked the router, tried a netinstall with no luck. Will be later next week before we can investigate further to see if any of the CCR2004 is recoverable.

lanhampr · Wed Mar 01, 2023 5:53 pm

We are on 7.7 and having random reboots. (watchdog) High NAT traffic. Is 7.8.rc3 the fix? Haven't seen anybody post in a few weeks.

sparky · Sun Apr 16, 2023 8:35 am

Bumping this to see if 7.8 solved the boot loop issue for anyone?

stuartpbb · Sun Apr 16, 2023 9:51 am

v7.8 does appear to have resolved the frequent out of memory crashes we were experiencing. Tracking memory usage over time shows flat usage with v7.8 while in the past usage would continue increasing until the router crashed.

Now, we have seen one issue after ~2 weeks of uptime where our PPPoE customers had been dropped and the sessions would not reconnect, logging "pppoe terminating... - could not add queue: already have such name (6)”. There are reports in the forums of people experiencing this error back to 2005 with no resolution that I could find. Mikrotik Support is being rather unhelpful with this and is requesting we upgrade to the latest prerelease firmware even though they can't provide any information that the PPPoE or Queue subsystems have been updated.

RohanAJoshi · Sun Apr 30, 2023 1:17 pm

In last 2 weeks I changed my radius provider to Icon Radius.
They Use SNMP and API services for their purpose to utilise function in portal.
It's causing memory filling so fast that in 1 week it's grabs 700mb memory and when free memory reaches 3gb mark ( when restarted it's 3.8gb free ) this ccr get reboot.
Log shows reboot by watchdog timer, raised ticket to support but not a single reply in month.
Also, since then, I was getting syn flood attack on my public ip pool.
Then I blocked ( chain input and action tarpit ) all external traffic other than my allowed pool.
.
I don't know what's causing this.
Radius support says configuration is ok, there no issue.
,
ROuterOs v7.8 stable

Who is online