Unicast Flood Prevention

elsev7 · Mon Aug 17, 2015 11:02 am

Hi

We are using a CRS125-24G-1S as a switch (no CRS switching feautures used). Occasionally we are experiencing unicast floods while our backup software (Veeam) is running. The number of backup hosts is half a dozen, traffic is appr. 5Mbps.

/ip settings print
              ip-forward: yes
          send-redirects: yes
     accept-source-route: no
        accept-redirects: no
        secure-redirects: yes
               rp-filter: no
          tcp-syncookies: no
         max-arp-entries: 16384
             arp-timeout: 5m
         icmp-rate-limit: 10
          icmp-rate-mask: 0x1818
         allow-fast-path: yes
   ipv4-fast-path-active: yes
  ipv4-fast-path-packets: 0
    ipv4-fast-path-bytes: 0
   ipv4-fasttrack-active: no
  ipv4-fasttrack-packets: 0
    ipv4-fasttrack-bytes: 0

If the backup traffic starts, we see the same outgoing traffic pattern (RRD interface graphs) on all connected switch ports.

storage2.png

Any ideas how we could avoid that?

elsev7 · Wed Aug 19, 2015 4:53 pm

Running on 6.29.1 Version.

Anything else, that would help to get a response from this forum?

/system routerboard> print 
    routerboard: yes
    model: CRS125-24G-1S
    serial-number: 624405C060FA
    current-firmware: 3.22
    upgrade-firmware: 3.22

elsev7 · Thu Aug 20, 2015 4:44 pm

Refering to http://serverfault.com/questions/297035 ... be-unicast

Is there a way to configure the CAM timers on RouterOS?

Ape · Fri Aug 21, 2015 1:04 pm

Hi,

I'm not completly sure, but maybe

/interface ethernet switch set unicast-fdb-timeout=300

is what you are looking for?

If you're interested in further reading:
http://wiki.mikrotik.com/wiki/Manual:CR ... nicast_FDB

Ape

elsev7 · Fri Aug 21, 2015 3:30 pm

Hi Ape

Thanks for replying! This sounds promising! We will keep an eye on that table entries and see if there is a correlation between the event and the table contents.

What happens if an entry disappears from the unicast-fdb (5min timeout), but is still in the arp table (4h timeout)?

Sven

Ape · Fri Aug 21, 2015 3:37 pm

Hi,

as far as I understand this, ARP has nothing to do with the FDB.
I also have a CRS125-24G-1S, the only entry in my ARP table is the MAC from the computer I'm accessing the switch for management.
So, AFAIK the entries in ARP table and FDB table are not related.

Ape

elsev7 · Fri Aug 21, 2015 4:03 pm

Mmm... that makes it even weird. Finally I'm trying to figure out, what causes the switch to send this unicast flood and how we could prevent that situation.

As a first action we are going to upgrade to the latest stable and see, if this still occures. If it happens again, we need to trigger a support request to Mikrotik support, I guess.

Sven

elsev7 · Fri Aug 21, 2015 4:10 pm

what do you see in your FDB table? I don't see any entry here. Maybe because we are not using CSR features?! The reason is, that we can't have STP working in CSR.

# aug/21/2015 15:07:27 by RouterOS 6.29.1
# software id = 8UYE-K7ZV
#
/interface bridge
add name=bridge1 priority=0x7000
/interface bridge port
add bridge=bridge1 edge=no interface=ether1
add bridge=bridge1 interface=sfp1
add bridge=bridge1 interface=ether2
add bridge=bridge1 interface=ether3
add bridge=bridge1 interface=ether4
add bridge=bridge1 interface=ether5
add bridge=bridge1 interface=ether6
add bridge=bridge1 interface=ether7
add bridge=bridge1 interface=ether8
add bridge=bridge1 interface=ether9
add bridge=bridge1 interface=ether10
add bridge=bridge1 interface=ether11
add bridge=bridge1 interface=ether12
add bridge=bridge1 interface=ether13
add bridge=bridge1 interface=ether14
add bridge=bridge1 interface=ether15
add bridge=bridge1 interface=ether16
add bridge=bridge1 interface=ether17
add bridge=bridge1 interface=ether18
add bridge=bridge1 interface=ether19
add bridge=bridge1 interface=ether20
add bridge=bridge1 interface=ether21
add bridge=bridge1 interface=ether22
add bridge=bridge1 interface=ether23
add bridge=bridge1 interface=ether24
/ip settings
set arp-timeout=4h max-arp-entries=16384

Ape · Fri Aug 21, 2015 4:13 pm

Ah!

You are bridging all ports - not using the switch chip.
I use the "real" switch feature, in my FDB table there are entries for every port/MAC association known to the switch.

Why are you bridging all your ports instead of using the switch chip?

Ape

elsev7 · Fri Aug 21, 2015 4:20 pm

Thanks for sharing!

We use traditional bridging, because of the missing STP feature in CRS chip. But at least, your setup is consistent - you see entries in the FDB. So, back to start

What could trigger this behaviour on the switch?

Ape · Fri Aug 21, 2015 4:35 pm

Okay,

some causes could be:

ARP timers on hosts are longer than the switch's address cache - so the switch basically forgets what MAC is associated to which port, resulting in flooding. But this should not be a matter of hours of unicast flooding like in your case.

MAC table of switch/bridge is full, so fallback is flooding to all ports.

Also, subsequent topolgy changes can lead to continues unicast flooding because the switch/bridge is in learning mode after every topolgy change (when STP/RSTP is used).

Last thing I can think of are redundant L2 paths which can cause assymetric L2 traffic flow, resulting in the switch never learning the MAC/port association for some paths.

If you can rule out all these points you should open a ticket with attached supout.rif file.

Ape

Update: I also did some googling and found this in wikipedia: https://en.wikipedia.org/wiki/Unicast_flood#Causes, maybe it is helpful to you.

elsev7 · Fri Aug 21, 2015 4:52 pm

Good points, Ape!

I guess I need to go for the support in case it will happen again. Just for completeness the topology:

Veeam Backup --> Windows --> vswitch0 --> active/standby netadapter --> internal switch --> Mikrotik --> Linux Server

The frustrating point is, that it is not reproducable.

Ape · Fri Aug 21, 2015 6:59 pm

Are you sure, the switch is the origin of the unicast flood?
Is your backup software possibly using (excessive) broadcasts / weird multicasts?

elsev7 · Sat Aug 22, 2015 12:05 pm

We havn't seen any broadcast/multicast in the tcpdumps, just unicast packets. Excluding the causes for a unicast flood, I think that some weird layer 2 topology or topology changes (RSTP) could be the root cause. But I need to proove this first.

Ape · Mon Aug 24, 2015 2:15 pm

Hi,

I just read the changelog (http://forum.mikrotik.com/viewtopic.php?f=21&t=99531) for RouterOS 6.31 and found this line:

What's new in 6.31 (2015-Aug-14 15:42):
[...]
*) bridge fastpath - fixed updating bridge FDB on receive (could cause TX traffic flooding on all bridge ports)

Maybe this is related to your flooding problem. You are using the brigde features and fast path is set to "allowed" as default.

Ape

elsev7 · Wed Aug 26, 2015 4:04 pm

Hi Ape

This is exactly what I was looking for! Great. So hopefully the upgrade to 6.31 will fix it.

Thanks for your help!

elsev7 · Mon Aug 31, 2015 10:10 am

Hi Ape

The upgrade to 6.31 didn't help. The flood occurs now on a regular base, which makes it easier to track. So possibly a layer 2 issue in the network, not directly related to the Mikrotik.

Thanks for your support

Ape · Mon Aug 31, 2015 11:28 am

Hi elsev7,

I'm sorry to hear that.

Yes, I agree, you have to exclude external L2 issues.
In the hope we're dealing with deterministic devices, there must a be chain of causality.

If you need further assistance, feel free to ask.

Ape

Unicast Flood Prevention

Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Re: Unicast Flood Prevention

Who is online