Hi all -

Running RoS 3.24 on a RB1000. Last night one of my colocation clients downstream from the router got hit with a DDoS. The data center NOC said is was a SYN flood and ended up null-routing the target IP address (he stopped counting when source IP entries approached 5000). All is well now, but I have to wonder what was happening inside the RB. The packet count going coming into the router from upstream was not particularly high:
and the bitrate was nowhere near out of range for what the router normally handles:
But the router's CPU loading was pegged at the ceiling the whole time and there was severe latency throughout my entire client network as a result:
So a rather moderate increase in packets and bits brought the RB to it's knees. Why is this so? My config inside the RB is minimal; I'm routing traffic to about 5 dozen client VLANs, I have about six simple queues, no firewall rules and no connection tracking or logging. It seems like the RB should have been able to handle this, but it didn't. Anyone have any ideas why? The info I have from the data center network admins:

I took a look at the flowstats and saw a ton of incoming SYN packets to port 80 on that IP from either ports 1024 or 3072 from the DDoS sources. I tried to ACL it out, but there were probably additional items inbound to it that flows wasn't showing me in the top 100 sources. At that point it just made sense to drop everything than keep looking for more.

Can anyone suggest firewall rules that would help in this particular situation?

Thanks!
Ed

are you logging all those drops? logging will kill a router under any type of heavy load.

are you logging all those drops? logging will kill a router under any type of heavy load.

no firewall rules, no connection tracking, no sniffing or logging.

It seems that I should consider adding some global firewall rules to the router to help mitigate attacks. I found this thread:
http://forum.mikrotik.com/viewtopic.php ... os#p178844

Where someone posted these rules:

/ip firewall filter add chain=forward protocol=tcp tcp-flags=syn connection-state=new action=jump jump-target=SYN-Protect comment="SYN Flood protect" disabled=yes
/ip firewall filter add chain=SYN-Protect protocol=tcp tcp-flags=syn limit=400,5 connection-state=new action=accept comment="" disabled=no
/ip firewall filter add chain=SYN-Protect protocol=tcp tcp-flags=syn connection-state=new action=drop comment="" disabled=no

"syn limit=400" is a threshold, just enable rule in forward for syn packets to get dropped (for excessive amount of new connection)

The Mikrotik wiki topic "Protecting your Customers" also suggests these rules:

/ip firewall filter
add chain=forward connection-state=established comment="allow established connections"
add chain=forward connection-state=related comment="allow related connections"
add chain=forward connection-state=invalid action=drop comment="drop invalid connections"

Would all of the above be appropriate for my situation? Also, I assume the first group the deals with syn and can also be modified to deal with udp applies to the total number of connections attempted to *all* clients on the downstream side of the router at any given time, not just per individual client? And how can I determine an appropriate value for limit?

if your not logging anything, or have anything in the your firewall filters, maybe the attack was against your SSH port, or something directed at the router that would cause it to be busy? Do you know what they were attacking? Normally you could pass that amount of traffic if your not doing anything with it except routing it.

two more quick comments (gotta get my son to bed):

invalid rule should be above established / related, not below.

on the syn rules, instead of connection-state=NEW it should be tcp-flags=SYN ... otherwise you have to check the connection-tracking table for every single one of those, which is more work than just looking at the TCP flags in the packet. if no one posts them I will try tomorrow morning.

if your not logging anything, or have anything in the your firewall filters, maybe the attack was against your SSH port, or something directed at the router that would cause it to be busy? Do you know what they were attacking? Normally you could pass that amount of traffic if your not doing anything with it except routing it.

It's possible. What the upstream network saw going into the router was a massive amount of new SYNs - and they were directed at one of my clients on the downstream side of the router, not the router itself. That's not to say that the router was not also getting hit, but if it was, it was not near enough to get the netadmin's attention. They just looked at flows and saw what percolated to the top, and overwhelmingly it was the SYN flood directed at my client.

I did see a bunch of rules for protecting the router in the mikrotik wiki and I can implement those. I'm a firewall newbie, so just to confirm: chain=input would be used for the router itself, chain=forward would be used for my clients (who are on the downstream side of the router), right?

I appreciate anything you want to contribute. The SYN protect filters were taken right off the Mikrotik wiki, but if they can be improved, great. One concern I have is that the appear to work with cummulative SYN traffic. So if the overall SYN traffic perks above the limit I've set, the firewall will start dropping SYN packets, both good and bad. In a way, this is kind of like assisting the attacker! I'd much rather have rules that evaluate the new SYN packet rate per-client, an only drop packets for individual clients who exceed the limit. That would really minimize the collateral damage. Any idea how I would go about that?

Thanks,

Ed

...on the syn rules, instead of connection-state=NEW it should be tcp-flags=SYN ... otherwise you have to check the connection-tracking table for every single one of those, which is more work than just looking at the TCP flags in the packet. if no one posts them I will try tomorrow morning.

The rules from the wiki actually have both: connection-state=NEW *and* tcp-flags=SYN. I guess it's to hone it down to just SYN packest requesting new TCP connections. Think it's OK, or still too much processing? I'm really mostly concerned that all this SYN protect code is going to do is essentially create denial-of-service via firewall filter instead of via CPU overload, for the reasons I explained above. Opinions?

TX,
Ed

I have recently similar incident on my network and main router which is only RB433AH simply stop to respond as it was overloaded by SYN packet traffic. Attack was directed to one of hosted server inside network.

So I did some investigation and realized that there is no way I can handle DoS attack which leads to overloading my router by setting rules in same router. What I can do is to limit somehow impact of attack to inner network.

Some hint are here: http://wiki.mikrotik.com/wiki/DoS_attack_protection

So the best way to handle these situations is get more powerfull edge router and clear all not needed rules and functions which can lead to more load per packet. In fact if TCP SYN flood attack is directed to some server inside, for main router without connection tracking it is only too much little packets. And count of packet can be limited.
But if we need additional operations on router the best way could be to have two routers. First on edge to have powerfull router for basic stateless without filtering to carry huge amount of packets and aside it have a router with all other functions like connection-tracking, logging using traffic-flow, shaping using queues, NATing, additional firewalling, etc.

But as small ISP I can't handle DDoS attack incoming from different addresses. This should be traced to source and blocked by upstream provider.

So in general you need more powerfull router.

Hi Chronos -

I completely agree. And my humble opinion is that the RouterBoard products work great - if everything runs perfectly all the time, which of course it does not. The RB1000's worked great for me for most of last year and handled a surprising amount of *legitimate* traffic, in excess of 500Mbps in some cases. But these malicious floods of packets, which started to hit me regularly about 2-1/2 months ago, just render the RB1000 dead in the water. I primarily use the RB1000 for inter-vlan routing and disable most other features. My experience is that packet rates of 70kps or thereabouts put the CPU at around 100% and then everyone starts getting severe latency and packet drops.

I just replaced one of the RB1000's with a Powerouter732, and I'm going to be shopping for even more powerful hardware soon. If anyone is interested in picking up some gently used RB1000's at a good price, let me know.

Ed