Hi all -

Running ROS 3.24 on RB1000 in a data center setting, routing traffic to/from client VLANs via a network of L2 switches. I have a couple of clients in my network who occasionally seem to create or receive little "packet storms": brief bursts of traffic that are moderate in terms of Mbps but enormous in terms of the PPS increase through the router. For instance, one of the clients today had a brief burst that registered under 30Mbps data rate, but pushed the router's PPS from it's typical 10K to nearly 100K, and as you can imagine it put quite a load on the CPU as well. This particular router is typically moving 50-100Mbps in both directions and registers a PPS of about 8K to maybe 12K to do it, so I can only surmise that these burst represent a brief "storm" of *very* small packets that really tax my router. Here's the graphs of his traffic, PPS on the router, and CPU on the router:
Unfortunately, when I contact the client to ask if he has any idea what it was all about, he just shrugs. The other one is of the type where I would not even waste my breath asking. So as a mitigating measure, is there something I can do to tell the router to just drop/ignore any packets that are less then a specified size? Failing that, how about limting the maximum packet rate in some way? The last time this happened with this client, it taxed the router severely enough that I experience bad latency throughout my network and clients complained...

TIA!
Ed

How do you know that it was those particular clients? If you control/have access to CPE exquipment it's best to shape traffic there. If each client is connected to particular interface you could limit the rate by changing size of the queue on the interface.

How do you know that it was those particular clients? If you control/have access to CPE exquipment it's best to shape traffic there. If each client is connected to particular interface you could limit the rate by changing size of the queue on the interface.

The traffic spike which corresponds to the CPU and PPS spikes shows up only on the one client's interface (which is a VLAN, by the way) - they are very clearly related. I do have a queue set up for that client, and interestingly, it's set to 20Mbps max limit up and down, yet the peak on his interface got closer to 28Mbps. I'm not sure how that's possible but in any case, the evidence indicates that the cpu and pps spike was due to traffic on his VLAN.

The queues in ROS only seem to work with bits per second, not packets per second. Packets per second limiting - or packet size filtering - is what I need to set for this client. Anyone else have any ideas?

Ed

Lower fifo queue size for offending VLAN

Lower fifo queue size for offending VLAN

I'm already using default-small: 10 pfifo packets. How much smaller can one go?

my guess is that traffic didnt really exist. there was a misread of a few timeouts of SNMP polls, and when mrtg polled the next time it compiled all the data into a single blip. ive seen this hundreds of times. it also happens when the counters roll over.

I've seen false spikes too, but that's not the case here. Not only do the traffic, PPS and CPU spikes line up exactly, but there were also corresponding spikes on the router trunk ethernet port, router WAN ethernet port, core switch router WAN and main uplink ports, and on the master traffic graphs for my router port from the data center - and the combined monitoring of all those points is handled by three different systems. This traffic was real.

I'd like to refocus this thread away from speculation as to whether or not the traffic really happened. The overwhelming evidence is that it did. Perhaps I should have left all the detail out. The topic of this post is: can ROS be set to ignore (drop) packets below a specified size? Or can ROS rate-limit pps the was it rate-limits bps? If anyone has any suggestions on how to go about either of those two things, please post - but please don't suggest limiting bps, that's not going to work for me.

Thanks!

Ed

"/ip firewall filter" has a matcher for packet-size, so you could drop those packets if you wanted to.

However, that's largely useless. It doesn't make any sense to deal with an attack once it's already at your router, because, well, it's already at your router. The main effect that 10x the usual amount of packets have on your router is devastating unless your router is completely overspec'd for normal operations - dropping them may save you some resources, but just filtering them out and dropping them is probably already going to kill you.

You need to filter this upstream, at the CPE.

Thanks very much for your response - apologies for my ignorance here, but what is the 'CPE' in my context - what does that stand for?

CPE = customer premises equipment. It refers to the gear at the customer's site, you should manage the traffic there so that the customer can only affect themselves, and not the core router at the data center where it has an impact on all customers.

Unfortunately this does not apply to me. All my equipment: the router and the L2 switch network which distributes bandwidth to clients is in the same data center as the client's equipment. The L2 switch this client is connected to has no means of managing traffic by packet size, I can rate-limit by bps but that's about it - and I don't have the option to choke him down to something really small, his allotment is 20Mbps and he's rate-limited.

Besides that, the traffic spike with the big PPS burst in question was inbound to his equipment from the uplink I provide him, which means that it originated on at the top of my network on the WAN side, passed through the router *first* and then went down the L2 switch connection to him. So as you can see, I need to be able to control it on my side as well.

I guess I'm doing something different than most people here, routing a data center network for many clients INSIDE the data center, with the ROS router between all of them and the Internet at large.

Any other ideas out there?

In that case it would be beneficial to talk to your uplink. Their router is presumably more powerful than yours, so they'd be in a better position to deal with the packet flood.

The main problem with just dropping all small packets is that they could very well be legitimate traffic. ACKs during an FTP upload, for example, are very small - the client is sending all the date, the server is just confirming it received it. I can't think of a way in RouterOS to drop above a PPS threshold, maybe someone else can.

+1 to fewi post..
however, you might try to put to a temporary address-list the src-address of the small-packets storm and tarpit all the connection connections from that address..
Maybe make a script that logs the cpu % when it happens. And log the src-address and notify the problems to the sysadmin of it.
It happened something similar to me, a stupid DOS attack freezed my whole network for 10 mins. I contacted the sysadmin of that ip (it was a server farm) and he locked the account of the attacker.
After few days i met on IRC the author of the attack (yeah i look for him, just for fun) and told him that i know his name and i was going to call the Police.. He was a young wannabehacker..

also, it should be possible to use 'limit' or even 'dst-limit' matcher to detect this type of attack. then you can add the source of attack to some address list - and drop small packets from the hosts in that list =)

do you use shaping? I think, if you drop the packet before it goes into the queue, you're lowering CPU usage a bit...