I am writing from my mobile phone since I still didn't revive all my network. Suddenly, soon after 00:00 UTC (maybe between 00:00 and 00:15) all the CCR routers in my network crashed hard at the same time. The only way to make them working again is to physically disconnect power.

It happened on ~60 routers at the same time, various models, the only thing in common: they are all Ccr.

I read quickly online that an unexpected leap second caused troubles to various network equipments. I don't know yet if this is related.

Is there any info/advice to avoid it happening again? Our network is big and there are routers installed far apart.

It is the 1'st April's joke or IMHO there was an attack on your equipment.
I could not imagine that different CCR models, from different production batches, with different uptimes, with different power source, with ....... are crashing all at same time. There should be external reason.

Brace yourself

It's not an april fool. It just happened in the wrong day of the year. Unfortunately I spent all the night back and forth all my nodes to restart the routers.

I would exclude an attack because it would have been made starting from the ending nodes backwards in order to reach all the routers, and who did it would have known the exact topology of the network. This happened to exactly ALL the CCR at the same time.

If it was a DOS attack, anyway, it means that every CCR router is in danger and can be crashed anytime.

Or it could be something caused by a bug in the IGP routing, (OSPF or BGP). It's the only thing all the router have in common.

Yours CCRs could be hacked earlier and poisoned today.

Yours CCRs could be hacked earlier and poisoned today.

It could be everything. If this happened only to me and nobody else, then it's either an attack directed to my network or something triggered by my setup.

All the other routers, which include RB1100 (various models), RB1200, RB2011 (various models), RB750, RB951, CRS (various models), Netmetal, Netbox, SXT, QRT were not affected. I may think this trouble is related to the TILE architecture.

Since it has been mentioned, there *is* something weird concerning leap seconds that *could* be related.

We had a number of Linux hosts incorrectly adding a leap second last night:
we are still investigating the cause. I also heard rumors of unrelated (ie: not MT) network devices having issues at midnight UTC because of spurious NTP leap seconds insertions, but can't comment much on those, will ask for more details.

not sure whether this will apply to each device running linux, but there are vendors out there pretty much concerned about the "leap seconds bug"
https://access.redhat.com/articles/15145

There is at least a report of spurious leap seconds observed in Italy on march 31, 23.59.60, with a possible explaination:

http://lists.ntp.org/pipermail/pool/201 ... 07338.html

Given what happened. I would try and simulate the addition of a leap second on the CCRs well before june 30.

cheers,
L.

Any news on this ?

Any news on this ?

No news. I wrote a script that connects and removes NTP servers from the configuration of every CCR router on the last hour / last day of the month and adds it back an hour after the first day of the month.

I don't have enough resources to mock a NTP server and introduce a fake leap second to reproduce the bug.

There is an open ticket [Ticket#2015040166000161] where I provided all the details, supouts, and everything I could document about this.

Tonight a leap second will be introduced by NTP servers from all the world. I disabled NTP on all CCRs, excluding a few ones that I have at hand. Let's see how it is going this time.

Tonight a leap second will be introduced by NTP servers from all the world. I disabled NTP on all CCRs, excluding a few ones that I have at hand. Let's see how it is going this time.

Well at about 00:00 UTC we had pretty much all of our CCRs crash and lock solid .. at the same time

Thankfully we've already removed (and ebay'd) most of the CCRs so these were just a few edge cases and not the whole network

Still a pita

I confirm that the bug is due to the leap second. All the CCRs where the NTP configuration has not been removed have crashed very hard.

Last time it happened because of a bug in a few italian NTP servers, now that the leap second was introduced officially, it might have happened worldwide.

There is a ticket opened for this issue but it hasn't been taken too much in consideration: [Ticket#2015040166000161] .

I think it should be investigated and solved because it's critical and dangerous. It happens only on CCR routers and not on MIPSBE / PPC.

I also had all CCRs reboot themselves by watchdog at 17:00 PST (probably after locking up). Even the one running 6.29.1 crashed.

Really disappointing.

Same here

3x CCR1009's all hard locked no winbox/serial. Had to go on-site and power cycle.

2x Running 6.29.1 + NTP package (whether that makes any diff). And 1x running latest rc22.

I can confirm all my border CCR crashed at 01:00BST. The common factor was BGP and NTP server. All other CCR in my network were just using OSPF and NTP client. What a pile of shite - seriously!!!!

All running v6.27 and were CCR1036-8G-1S

CCR1016-12G
RouterOS 6.27
Rouberboot 3.19
NTP Server running

At 0:00 UTC (2:00 Spain local time), it died

Not reachable by MAC ping, or ping. All interfaces working and blinking. Can't see it on IP > Neighbors from other devices.

After rebooting, all working again.

We've 3 more CCR1016, but they're 12S-S+. All them remained working fine (they had no NTP package installed)

Any explanation from Mikrotik?

WTF!?

I had 7 CCR1009's crash at 5:00PST!

What is going on?

Can people please edit their posts to include ROS version and if BGP, NTP server etc was running?

I have 28 tile devices running NTP (and no routing protocols), versions are between 6.22 and 6.29.1. All crashed, though 25 of them rebooted themselves via watchdog timer. 3 needed power cycling.

Have three CCRs, and they all crashed. They were each running a different version (6.26, 6.28, and 6.29)

Have a customer that has a CCR that didn't lockup, but don't know if it is poling ntp. It could also be that the others caused it to not be able to update it's clock, so it didn't get the time 23:59:60.

I just got back from the trip around my network nodes (it took about 2 hours by car. Fortunately I disabled in time the NTP server on most of the routers, I have more than 60 installed and some are very far)...

Same here
3x CCR1009's all hard locked no winbox/serial. Had to go on-site and power cycle.
2x Running 6.29.1 + NTP package (whether that makes any diff). And 1x running latest rc22.

My routers have 6.26, 6.27, 6.28, 6.29, 6.29.1 and they all crashed. All kinds of CCR (1009, 1016, 1032 on various flavors).

They have all BGP routing protocol, only a few have the full routing table, while most of them have only my internal routes.

They have all the clock synchronized through NTP.

It happened both to routers with and without NTP package, at 00:00 UTC (02:00 local time).


The funny thing is that by chance it happened for the first time on the 1st of April and other forum users thought that it was an april fool's joke.

Mikrotik didn't take the ticket seriously because it said it happened only to me. (As I stated it was by chance, due to a driver bug on public italian NTP services, and it hasn't been easy to find out!).

20+ CCRs all dead.


Time to replace all my mikrotiks.

I have two CCR with NTP package installed. Both crashed today.

No NTP package - no crash.

All our CCR's locked up right at 7pm CDT. 3 x86 units did not, MT493 units did not.
This is very strange.

Same here, almost all CCR down, and the only way was to power reboot. NTP, BGP, OSPF, var 6.29.1, OSPF 3,

does setting up watchdog helps ?

all CCR down here as well, until manual power cycle. 6.29, no BGP, static routing, NTP as client from 192.53.103.104/192.53.103.108.

i got 4 ccr1036 v6.27, 4 of them crashed. the others ccr is safe, among them using v6.29. and v6.19.

OK, first of all sorry about previous joke poster, cause looks like i had similar problem, but only on few CCRs.

Am i right to say that only CCRs with NTP package installed and configured was affected?

2 CCR crashed, i hope the bug will get fixed soon.

More details:

• - CCR1036-12G-4S
• - version 6.27
• - NTP package installed. NTP client configured to a public server

yes, it seems so, i got v6.27 which NTP package installed and the CCR was crashed.
but the others which have v6.27 without NTP package installed, they are normal.
but some of my CCR using v6.19 with NTP package installed are not crashed.

OK, first of all sorry about previous joke poster, cause looks like i had similar problem, but only on few CCRs.

Am i right to say that only CCRs with NTP package installed and configured was affected?

yes, it seems so, i got v6.27 which NTP package installed and the CCR was crashed.
but the others which have v6.27 without NTP package installed, they are normal.
but some of my CCR using v6.19 with NTP package installed are not crashed.

And all were synchronized (configured) to public server?

Given what happened. I would try and simulate the addition of a leap second on the CCRs well before june 30.

Now, tell me I did not warn you.

Am i right to say that only CCRs with NTP package installed and configured was affected?

Most of my CCRs don't have the NTP package installed, and they all crashed badly.

I have one ccr (ccr1036-8g-2s+) running BGP, OSPF, OSPF3 on v6.19 (uptime 200+ days)

Yesterday, I saw this thread while searching for possible issues with the upcoming leap second,
disabled the ntp client on time.

It did not crash.

Maybe the time on the router is not that important,
maybe, we could let it disabled?
Could a time drift cause any problems for example BGP?

I have one ccr (ccr1036-8g-2s+) running BGP, OSPF, OSPF3 on v6.19 (uptime 200+ days)

Yesterday, I saw this thread while searching for possible issues with the upcoming leap second,
disabled the ntp client on time.

It did not crash.

Maybe the time on the router is not that important,
maybe, we could let it disabled?
Could a time drift cause any problems for example BGP?

you can also use "IP Cloud" automatic time. It will not be as precise, within 1-2 seconds usually.

Having the time synchronized is a *must*, because otherwise it's difficult (if not impossible) compare logs of multiple devices, and in a complex network it helps debug troubles quickly.

Normis ip cloud time synchronization is not an option for many reasons, and it's not always applicable.

Having the time synchronized is a *must*, because otherwise it's difficult (if not impossible) compare logs of multiple devices, and in a complex network it helps debug troubles quickly.

Normis ip cloud time synchronization is not an option for many reasons, and it's not always applicable.

of course. but better than no NTP at all

Let's move this discussion to more specific title: http://forum.mikrotik.com/viewtopic.php?f=2&t=98138