HI All

Im hoping anyone else experienced this and can answer me. We have a rather large network and regularly use ROMON to access our network. Our basic network architecture is about 4 levels deep, Core network, Broad wireless network, Client CPE's, Client switches.

On Friday 24/01 at around 11:20 SAST we had a synchronous network drop for about 20-40 seconds. 80% of of our core network, 60% of our wireless network, and about half our clients shows on our monitoring system that the devices disconnected. Some of our devices reconnected in a "hanged" state and needed to be power cycled to restore functionality. THe issue was experienced on a few CCR1036's, a bunch of LHG radios, some RB2011's and a whole bunch of CRS 326's. Firmware on all these devices vary from 6.42.x to 6.44.x (either LTR or Stable versions, no beta or dev versions)

The only thing we can see that all these devices had in common was Romon was enabled an ALL effected devices. THe devices we have on the network that does not show the drop, did not have ROMON enabled.

Has anyone else experienced something in this effect, or can shed light on how or where to start looking for the cause of this issue.

Logs only report BGP failure or port disconnects. Router that were in hanged state, simply report power failure. Services like watchdog also did not run to autoboot the routers.

Please help?

Since it is a large network it would be better if you used VPN instead of Romon to access all the network nodes...

Hi

We have VPN's, L2 access and L3 access to the network, but we also used Romon as a backup. Since we have never heard or seen any attack on romon, we didnt disable it.
We can still access all our equipment by other means, as we have since disabled romon on all our routers.

My concern is, has anyone else ever had this occur, or was this a malicious attack on using romon?

We had the same issue that lasted 2 years.
Nearly brought our whole business to its knees.
Our network is fully routed with no bridges.

Support couldn't find the issue and gave us ZERO help.
We tried everything from net installing all routers on our network to putting extreme firewall policies on place (thinking it was a hack of some sort)
Eventually we caught the issue on a pcap after putting inline packet capturing devices over links.

We saw a massive amount of STP packets seconds before we had the symptoms which, depending on the model router were-

• All interfaces flapping uncontrollably for 5 mins

• Router lockup

• Router kernel panic and reboot

Romon makes use of the same flooding mechanism as STP
We disabled ROMON network wide and the issue went away.
We have since enabled ROMON on select routers, this allows us to mac telnet into a router on the next hop so we can enable ROMON if we need access.
We always keep active ROMON routers to a minimum.
It has transpired that the flood of packets would cause the ethernet drivers on the routers to crash. Mikrotik support said that they were making changes.
My advice is DISABLE ROMON.

Good Day,

thank you for this Topic, we have encountered the same symptoms as mentioned below. i haven't disable ROMON yet, is this advisable to disabled romon and does it have a massive impacts.
we have about 400 Wireless towers and once a month maybe 2 our whole network goes down. i have set the ospf to PTP for not broadcasting anything but still this happens.

any advice.