Why source-based blackhole instead of firewall drop

Murmaider · Tue Nov 15, 2016 10:55 am

EDIT : There was an initial flaw in my testing, jump to this post viewtopic.php?f=2&t=114664#p599689 which shows that using the firewall RAW DROP has the same efficiency as source-based blackhole + route cache disabled - thanks mrz & cha0s

Disclaimer:
I am by no means a network expert and this post is intended for those with smaller public networks and / or beginners.
I welcome any experts recommendation if any of my information below is incorrect

Background
I decided to post this after a post I did in this topic - http://forum.mikrotik.com/viewtopic.php ... 93#p567454

Firewalling can be bad mmmkay
Firewalling is great and everyone should use it, but try to do this as close to the edge (as close to the customer / server) of your network as possible.
Do not firewall on your border routers (your routers between your network and the internet) unless you absolutely have to (eg: NAT).
When making a decision to firewall, expect there to be a throughput impact in high packet-per-second (pps) attacks.
The more firewall rules you have, the more you affect the throughput.This is because every packet has to be matched against the firewall rules until one condition is met.

This is shown clearly in Mikrotik's own performance tests, here is an example from the CCR-1072 router tests:

Without any firewalling, 44Gbits and 86 million pps are possible through the CCR router (64byte).
Simply add 25 firewall rules and your throughput drops dramatically to 3Gbits and 5.8 million pps.
That is a 93% drop in throughput by adding firewall rules to your router.

Using the CCR-1072 as an example, If you have a 5Gb attack on your network and you are firewalling on your border router, the moment 3Gbits of traffic is exceeded in the attack, everything behind your border router is affected. Regardless of having a 10Gb link.

If however you have no firewall on your border router and rather firewall close to the edge, the CCR will withstand this attack and the only part of your network that will be affected is the area behind the firewall on the edge of your network. This means the rest of your network behind the border router is still accessible. This will limit the impact of what is affected by the attack on your network.

The opposite also applies, if an attack is initiated from an edge device on your network, you can stop it as close to the device as possible to prevent it affecting the rest of your network.

Cool story bro, but how do I now block traffic at the border of the network?
There are a number of ways like Remote Trigger Blackhole with your ISP, but this blackholes the target IP on the network upstream.
You essentially giving an attacker what he wants, a host that's down. You are saving the rest of your network at the expense of the target IP being inaccessible.
Sometimes this is absolutely necessary, but sometimes you might want to just block the source addresses attacking the target to keep the target up to legitimate traffic.
In this case, a source-based blackhole routing can be used to achieve blocking, instead of a firewall rules.
Keep in mind that routes do not slow down the performance of throughput on your router as much as firewall rules do.

Here is an example I did to illistrate how effective source-based blackholes can be over a Firewall Drop rule.(a youtube video):
https://youtu.be/SHjBbbF8Ek4

I setup a test server running hping3 in flood mode with 64b packets to simulate a DDoS attack.
The test server sends packets to the mikrotik router, which then routes the traffic to the target IP.

The initial attack puts the CHR at 40% CPU usage, when the firewall DROP rule is activated, the CPU averages out around 18% usage, but you can still see the traffic being received on the interface (both in RX bandwidth and in RX packets-per-second.) Yes the drop rule prevents the traffic from reaching the target IP, but doesn't solve the load on your mikrotik or the traffic on the interface.

I then disable the firewall rule, which allows the DDoS traffic to flow freely again, except this time I enable the source-based blackhole route for the DDoS source IP address.
Almost immediately you can see it perform the exact same action as the firewall DROP rule, except after a number of seconds you will see the traffic drop off entirely from the interface and the mikrotik CPU return to 0% usage.

One thing to note when doing the above is having RP Filter set to loose (or strict, but strict can cause some routing issues in asymmetric routing).
Loose RP Filter checks the packet and forwards it if there is a route entry for the source IP of the incoming packet in the router FIB. If the router
does not have an FIB entry for the source IP address, or if the entry points to blackhole, the Reverse Path Forwarding (RPF) check fails, and the
packet is dropped

A source-based blackhole achieves the same result as a firewall drop, but more efficiently and without impacting the throughput of your device.
You can also use source-based blackholes in conjunction with your normal firewall rules as a means of dropping IP traffic.

There are many tools out there (wanguard, fastnetmon, etc) which automate the detection of a DDoS and can call the mikrotik API (or through BGP) to automatically create these source-based blackhole rules.

Regards,
Dave

Tue Nov 15, 2016 11:52 am

Nice. What about to drop in raw table according to the blacklist while an attacking ip address was identified by firewall rules and put on the list?

mpreissner · Tue Nov 15, 2016 12:59 pm

I think what this post really says is DON'T USE A MIKROTIK FOR FIREWALLING.

Murmaider · Tue Nov 15, 2016 1:15 pm

Nice. What about to drop in raw table according to the blacklist while an attacking ip address was identified by firewall rules and put on the list?

By this I assume you mean what is referred to in this post - http://forum.mikrotik.com/viewtopic.php ... 50#p463532 which refer's to this code:

/ip firewall filter
add action=jump chain=forward connection-state=new jump-target=detect-ddos
add action=return chain=detect-ddos dst-limit=32,32,src-and-dst-addresses/10s
add action=add-dst-to-address-list address-list=ddosed address-list-timeout=10m chain=detect-ddos
add action=add-src-to-address-list address-list=ddoser address-list-timeout=10m chain=detect-ddos
/ip firewall mangle
add action=mark-routing chain=prerouting dst-address-list=ddosed new-routing-mark=ddoser-route-mark passthrough=no src-address-list=ddoser
/ip route
add distance=1 routing-mark=ddoser-route-mark type=blackhole

There is still firewall interaction as the packets need to be marked here and it yields the same results as a firewall drop policy.
Here's the test - https://youtu.be/9xoO_tUBYEA

Tue Nov 15, 2016 1:34 pm

No. I do not refer to mangling in prerouting and dropping connection by marks but to direct packet drop in raw firewall table without any other interactions.

maznu · Tue Nov 15, 2016 1:55 pm

Firewalling can be bad mmmkay
Firewalling is great and everyone should use it, but try to do this as close to the edge (as close to the customer / server) of your network as possible.
Do not firewall on your border routers (your routers between your network and the internet) unless you absolutely have to (eg: NAT).
When making a decision to firewall, expect there to be a throughput impact in high packet-per-second (pps) attacks.
The more firewall rules you have, the more you affect the throughput.This is because every packet has to be matched against the firewall rules until one condition is met.

While I agree that having lots of "/ip firewall filter" entries will slow your router down (CPU has to go through each rule in sequence), using an address-list should consolidate a lot of blocked IPs into one firewall rule. I believe address-list is implemented as the Linux kernel's IPSET feature for iptables, and so it will work in pretty much the same amount of time even if there are thousands of entries in the list (it's more like a hash-table than a linear list).

We have no problem blocking traffic at our provider edge using this technique. I gave a talk about this at the UK MUM yesterday, actually. We use an /ip route rule, packet marking in /ip firewall mangle, and an /ip firewall address-list to move abusive traffic away from our customers and to special VMs which are honeypots. However, if you just want to block traffic, it'd be fine to use a filter instead of /ip firewall mangle.

PDF of Presentation: https://faelix.link/mum16

MUM Video: https://faelix.link/mum16video

Murmaider · Tue Nov 15, 2016 3:00 pm

While I agree that having lots of "/ip firewall filter" entries will slow your router down (CPU has to go through each rule in sequence), using an address-list should consolidate a lot of blocked IPs into one firewall rule. I believe address-list is implemented as the Linux kernel's IPSET feature for iptables, and so it will work in pretty much the same amount of time even if there are thousands of entries in the list (it's more like a hash-table than a linear list).

100% correct. I'm more illistrating that the moment firewalling is involved in anyway with the border router, it will cause CPU load under attack as all the packets still need to be filtered.
The source-based blackhole seems more efficient at achieving the same result.

No. I do not refer to mangling in prerouting and dropping connection by marks but to direct packet drop in raw firewall table without any other interactions.

Ah yes I see what you mean, I did another test with running the hping3 on a 3 minute test and took a graph from the network switch.
One for a normal firewall drop rule, one for source-based blackhole and one with a firewall RAW table drop rule.

The firewall raw table drop rule worked very slightly better than a normal firewall drop rule, with the CPU sitting at 14% instead of 18%.

We have no problem blocking traffic at our provider edge using this technique. I gave a talk about this at the UK MUM yesterday, actually. We use an /ip route rule, packet marking in /ip firewall mangle, and an /ip firewall address-list to move abusive traffic away from our customers and to special VMs which are honeypots. However, if you just want to block traffic, it'd be fine to use a filter instead of /ip firewall mangle.

Do you not lose fastpath by doing this though?

Tue Nov 15, 2016 3:45 pm

Hard data prooves... I have expected better results. Good job.

Murmaider · Tue Nov 15, 2016 5:20 pm

Hard data prooves... I have expected better results. Good job.

Just to be sure I've tested a few different attack variations with the initial tests being syn flood attacks.
I've now tested:
- udp flood attacks
- syn attack with random source and spoofed source
- udp attack with random source and spoofed source
- TCP Connect attack with 200 000 pps directed to an open port

All show the exact same great result with using RP Filter (loose) and source-based blackhole.
It's basically tricking the RP Filter into dropping the source address.

maznu · Tue Nov 15, 2016 7:40 pm

Do you not lose fastpath by doing this though?

Yes, but in our use case that isn't something I'm worried about. I need to use an address-list (so lose fastpath) so that I can redirect abusive traffic:

1) to display a "stop typing your password wrong!" message to customers failing to log themselves in to websites or whatever
2) to direct hacking traffic away from customer servers
3) to take the attack traffic to our honeypot for further analysis so that we can see what things they're scanning for (to make more rules to detect them, etc)

For us it's better that our edge CCRs took the CPU hit and used ~30W more power than, say, a rack of servers hit a loadavg of 60+ due to attack traffic. With only one mangle rule in the firewall, we're not going to choke the CPU as much as the "25 filter rules" statistics quoted on the routerboard.com website. And so long as we've chosen CCRs that will cope until our upstream links are saturated, that's fine - after that, it's not the CCRs' fault ;-)

Murmaider · Wed Nov 16, 2016 6:48 am

Do you not lose fastpath by doing this though?
Yes, but in our use case that isn't something I'm worried about. I need to use an address-list (so lose fastpath) so that I can redirect abusive traffic:

1) to display a "stop typing your password wrong!" message to customers failing to log themselves in to websites or whatever
2) to direct hacking traffic away from customer servers
3) to take the attack traffic to our honeypot for further analysis so that we can see what things they're scanning for (to make more rules to detect them, etc)

We had a similar use case on our side, but a slightly different approach (as we don't want to lose fastpath).

1) In our code of our portals we give the "stop trying your password is wrong" messages and after X brute-force attempts the code on the portal calls the mikrotik API to blackhole the IP for XX period of time. It starts with a few minutes and then removes the blackhole. Each time it picked up subsequent brute-force attempts it increases the time they blackholed.

2 & 3 - We do this using Wanguard. It detects the hacking / attack traffic and informs the filter (honeypot in your case) to inject a route into the mikrotik to redirect this traffic to itself where it can be captured for analysis. This is also filtered and the clean traffic can then be injected back into the network.

maznu · Wed Nov 16, 2016 9:19 am

2 & 3 - We do this using Wanguard. It detects the hacking / attack traffic and informs the filter (honeypot in your case) to inject a route into the mikrotik to redirect this traffic to itself where it can be captured for analysis. This is also filtered and the clean traffic can then be injected back into the network.

Ah interesting. How does the Wanguard get the traffic in the first place to determine what is an attack? Is it working off NetFlow/IPFIX data? Or is it connected to a mirror port or something?

In our case, our main abuse detection - besides fastnetmon etc - is because the end servers are running some inspection (e.g. mod_security, fail2ban, etc).

Murmaider · Wed Nov 16, 2016 9:52 am

Ah interesting. How does the Wanguard get the traffic in the first place to determine what is an attack? Is it working off NetFlow/IPFIX data? Or is it connected to a mirror port or something?

It supports both Flow Sensors or Packet Sensor. We use the Packet Sensor (create a mirror port on your switch) as it can pick up an attack in less than 1 second.
A flow sensor can detect the an attack in "flow export time + 5 seconds".

This explains it better than I can : https://www.andrisoft.com/software/wang ... protection

In our case, our main abuse detection - besides fastnetmon etc - is because the end servers are running some inspection (e.g. mod_security, fail2ban, etc).

You can still use these, just have them trigger a script that speaks to the mikrotik API, for example:

# /usr/bin/blackhole.php <attacker ip> <block time>

It calls the mikrotik api and then stores the block time in a db / flat file.
then have a cron that runs every minute that looks in the db / flat file for items which are "expired based on time" and calls the CCR to remove the blackhole.

maznu · Wed Nov 16, 2016 9:52 am

You can still use these, just have them trigger a script that speaks to the mikrotik API, for example:

# /usr/bin/blackhole.php <attacker ip> <time>

It calls the mikrotik api and stores this in a db or flat file.
then have a cron that runs every minute that looks in the db / flat file for items which are "expired" and calls a script to remove the blackhole from the CCR.

Oh, our fail2ban/etc boxes don't have access to the edge routers' API. They send attack information to a process we run which stores IP reputation data, and that decides whether to block addresses at the edge. It does that with an address-list entry, with an appropriate timer. Again, like yours, we block for longer periods of time for longer attacks. And our code expands the blocked range from a /32 gradually up towards a /24 if neighbouring IPs are implicated or are attacking other hosts in our network.

More details in the PDF of the talk I linked.

dadaniel · Wed Nov 16, 2016 11:28 am

Can you please share the code for this source based blackhole? From above I can see that I have to packet mark so filter is still involved?!

maznu · Wed Nov 16, 2016 11:40 am

Can you please share the code for this source based blackhole? From above I can see that I have to packet mark so filter is still involved?!

Two different people debating different approaches.

Murmaider's using a loose RP filter and route injection to trick the router into dropping packets.

I'm using a combination of /ip route rule with /ip firewall mangle and an /ip firewall address-list to redirect traffic from our border routers (i.e. as soon as it comes in from our upstream providers) straight to a honeypot.

Murmaider · Wed Nov 16, 2016 11:45 am

Can you please share the code for this source based blackhole? From above I can see that I have to packet mark so filter is still involved?!

For the mikrotik, enable RP Filter under IP => Settings (set to loose)
Then to add a blackhole use:

/ip route
add distance=1 dst-address=<source address> type=blackhole

Source address can be an individual ip or a network range.

maznu · Wed Nov 16, 2016 11:50 am

I think the last question I have about this for Murmaider is:

Will your RP-based blackhole approach work where there are multiple valid routes (learned by BGP) to external addresses, but only one is "active" in the RouterOS routing table? Because we see traffic come in on interfaces which aren't necessarily the best path that we would have chosen for sending traffic out. Presumably this is why you're "firewalling" as close to the destination as possible, whereas we're redirecting attack traffic as close as possible to its ingress point to our network (which means on routers where there could be multiple possible routes available).

Murmaider · Wed Nov 16, 2016 12:20 pm

I think the last question I have about this for Murmaider is:
Will your RP-based blackhole approach work where there are multiple valid routes (learned by BGP) to external addresses, but only one is "active" in the RouterOS routing table?

With RP Filter set to loose it should. Loose RP Filter checks the packet and forwards it if there is a route entry for the source IP of the incoming packet in the router FIB. This includes a default route or any route you have learned via BGP. If however an entry for the source ip points to blackhole, the Reverse Path Forwarding (RPF) check fails, and the packet is dropped.

This approach has been around for a while:
http://packetlife.net/blog/2010/aug/23/ ... ased-rtbh/
http://packetlife.net/blog/2009/jul/20/ ... s-attacks/
https://anetworkerblog.com/2009/08/25/rtbf/

maznu · Wed Nov 16, 2016 12:23 pm

With RP Filter set to loose it should.

Ahhhh, that's the bit I didn't grok. I'd only ever encountered reverse path filtering in a strict sense as part of a poor-man's BCP38. Ok, that's interesting.

Thanks - I learned something today.

dadaniel · Wed Nov 16, 2016 12:50 pm

Source address can be an individual ip or a network range.

Oh, so is there an easy way to do this for all IPs in a address-list without using mangle/filter/etc before?

Murmaider · Wed Nov 16, 2016 1:09 pm

Source address can be an individual ip or a network range.
Oh, so is there an easy way to do this for all IPs in a address-list without using mangle/filter/etc before?

To use the address list, the interface has to accept the packet first in order to mangle it (by using the firewall).

What I mean is that you can do single IP.

/ip route
add distance=1 dst-address=192.168.1.10 type=blackhole

or network range

/ip route
add distance=1 dst-address=192.168.1.0/24 type=blackhole

What we do is we set a custom comment on the blackhole route which we link to an incident ID, then when we need to remove the blackholes, we delete all routes that match the comment of the incident ID.

msatter · Thu Nov 17, 2016 10:49 am

When this is so efficient then it is may interesting to transparent for the users to use a different screen to input this. Since short we have the RAW option that should be more efficient but is not that much of a difference.

So a user could enter this in the RAW section and an added "action" would be "blackhole" and when "blackhole" is selected then a route is added to the routing lines.
When a RAW line is changed, with action "blackhole", the route is changed accordingly. If a change is so that setting that routing is not a solution any more then the line in routing is removed and only the RAW rule will be present and used this will also be the case if the cation is changed to something else than "blackhole".

Now the problem is when a user is going to edit directly in the routing section which interferes with the settings in RAW. Proposal is that the routing line is show in routing but can't be changed in any way on the routing page. The comment is beginning with "RAW filter: " followed by the found text that the user has entered or not in the RAW section. It would also be preferable that the letters in that line is not black but gray to indicate a deferred rule.

Murmaider · Thu Nov 17, 2016 11:05 am

When this is so efficient then it is may interesting to transparent for the users to use a different screen to input this. Since short we have the RAW option that should be more efficient but is not that much of a difference.

So a user could enter this in the RAW section and an added "action" would be "blackhole" and when "blackhole" is selected then a route is added to the routing lines.
When a RAW line is changed, with action "blackhole", the route is changed accordingly. If a change is so that setting that routing is not a solution any more then the line in routing is removed and only the RAW rule will be present and used this will also be the case if the cation is changed to something else than "blackhole".

Now the problem is when a user is going to edit directly in the routing section which interferes with the settings in RAW. Proposal is that the routing line is show in routing but can't be changed in any way on the routing page. The comment is beginning with "RAW filter: " followed by the found text that the user has entered or not in the RAW section. It would also be preferable that the letters in that line is not black but gray to indicate a deferred rule.

You could simplify this by setting up a box running exabgp or quagga and have a BGP connection to the mikrotik(s).
setup a BGP community of say 666:666 and set the type to blackhole.
Then you can push these blackhole routes or retract them from all your border/edge routers from a central point.

msatter · Thu Nov 17, 2016 12:03 pm

When this is so efficient then it is may interesting to transparent for the users to use a different screen to input this. Since short we have the RAW option that should be more efficient but is not that much of a difference.
.
.
.....the problem is when a user is going to edit directly in the routing section which interferes with the settings in RAW. Proposal is that the routing line is show in routing but can't be changed in any way on the routing page. The comment is beginning with "RAW filter: " followed by the found text that the user has entered or not in the RAW section. It would also be preferable that the letters in that line is not black but gray to indicate a deferred rule.

You could simplify this by setting up a box running exabgp or quagga and have a BGP connection to the mikrotik(s).
setup a BGP community of say 666:666 and set the type to blackhole.
Then you can push these blackhole routes or retract them from all your border/edge routers from a central point.

That would be a wet dream and maybe start a bit more down to earth and have a list supplied by IntrusDave: http://forum.mikrotik.com/viewtopic.php?f=9&t=98804

fabiandr · Mon Apr 10, 2017 4:05 am

When this is so efficient then it is may interesting to transparent for the users to use a different screen to input this. Since short we have the RAW option that should be more efficient but is not that much of a difference.

So a user could enter this in the RAW section and an added "action" would be "blackhole" and when "blackhole" is selected then a route is added to the routing lines.
When a RAW line is changed, with action "blackhole", the route is changed accordingly. If a change is so that setting that routing is not a solution any more then the line in routing is removed and only the RAW rule will be present and used this will also be the case if the cation is changed to something else than "blackhole".

Now the problem is when a user is going to edit directly in the routing section which interferes with the settings in RAW. Proposal is that the routing line is show in routing but can't be changed in any way on the routing page. The comment is beginning with "RAW filter: " followed by the found text that the user has entered or not in the RAW section. It would also be preferable that the letters in that line is not black but gray to indicate a deferred rule.

You could simplify this by setting up a box running exabgp or quagga and have a BGP connection to the mikrotik(s).
setup a BGP community of say 666:666 and set the type to blackhole.
Then you can push these blackhole routes or retract them from all your border/edge routers from a central point.

Hello,

can you help me?

Fastnetmon identifies the attacked IP, but I can not get exabgp to advertise to mikrotik.

Murmaider · Mon Apr 10, 2017 7:31 am

Hello,

can you help me?

Fastnetmon identifies the attacked IP, but I can not get exabgp to advertise to mikrotik.

Hi,

There are 2 ways to do this:

1) Use fastnetmon's mikrotik plugin - https://github.com/pavel-odintsov/fastn ... tik_plugin
It speaks to the mikrotik API and handles the automatic blackholing.

2) PM me your exabpg config and the portion of your mikrotik config relating to the exabgp peer and any filters you have for it.

Jacka · Mon May 22, 2017 11:34 am

Can someone post the complete rules for fighting DDoS using "blackhole" ?
The rules below are ok ? Smb was saying there should be any mangle rule so it work better.

 /ip firewall filter
add action=jump chain=forward connection-state=new jump-target=detect-ddos
add action=return chain=detect-ddos dst-limit=32,32,src-and-dst-addresses/10s
add action=add-dst-to-address-list address-list=ddosed address-list-timeout=10m chain=detect-ddos
add action=add-src-to-address-list address-list=ddoser address-list-timeout=10m chain=detect-ddos
/ip firewall mangle
add action=mark-routing chain=prerouting dst-address-list=ddosed new-routing-mark=ddoser-route-mark passthrough=no src-address-list=ddoser
/ip route
add distance=1 routing-mark=ddoser-route-mark type=blackhole

Murmaider · Tue May 23, 2017 12:37 pm

Can someone post the complete rules for fighting DDoS using "blackhole" ?
The rules below are ok ? Smb was saying there should be any mangle rule so it work better.

 /ip firewall filter
add action=jump chain=forward connection-state=new jump-target=detect-ddos
add action=return chain=detect-ddos dst-limit=32,32,src-and-dst-addresses/10s
add action=add-dst-to-address-list address-list=ddosed address-list-timeout=10m chain=detect-ddos
add action=add-src-to-address-list address-list=ddoser address-list-timeout=10m chain=detect-ddos
/ip firewall mangle
add action=mark-routing chain=prerouting dst-address-list=ddosed new-routing-mark=ddoser-route-mark passthrough=no src-address-list=ddoser
/ip route
add distance=1 routing-mark=ddoser-route-mark type=blackhole

These rules yield the same result as a normal firewall DROP rule. Every packet still needs to be inspected by the firewall before marking them to be blackholed, even after the address exists on the ddoser and ddosed list.

mrz · Tue May 23, 2017 12:42 pm

I would suggest to use raw firewall rules to drop DDOS packets.

Murmaider · Tue May 23, 2017 1:22 pm

I would suggest to use raw firewall rules to drop DDOS packets.

Hi mrz,

I tested the RAW firewall rules vs normal firewall rules above in this thread (viewtopic.php?f=2&t=114664&p=599485#p568217), and the performance is almost identical.

mrz · Tue May 23, 2017 2:27 pm

It was meant for an example few posts above, drop in RAW will be significantly faster than routing marks and connection marks.
Even in your test compared 18% to 14% it is roughly 30% faster, which is exactly how RAW was advertised and it does not overload connection tracking.

Of course blackhole is fastest one and more preferred, there is no argue about that.

Murmaider · Tue May 23, 2017 2:40 pm

It was meant for an example few posts above, drop in RAW will be significantly faster than routing marks and connection marks.
Even in your test compared 18% to 14% it is roughly 30% faster, which is exactly how RAW was advertised and it does not overload connection tracking.

Of course blackhole is fastest one and more preferred, there is no argue about that.

I'm not sure how you are getting the 30% faster...

The 18% to 14% is the CPU usage difference between a normal firewall and RAW. (so 4% cpu gain based on 100% cpu usage or 23% more cpu efficient based on dropping from 18% to 14% - pinch of salt needed ).
If we look at packets processed, the normal firewall did 29 870 725 packets compared to RAW which did 30 001 968 packets, which is a difference of 131 243 packets over the same period. This puts RAW 0,44% faster than the normal firewall.

Jacka · Tue May 23, 2017 4:40 pm

These rules yield the same result as a normal firewall DROP rule. Every packet still needs to be inspected by the firewall before marking them to be blackholed, even after the address exists on the ddoser and ddosed list.

Could you please share the rules without using the firewall? Thank you

mrz · Tue May 23, 2017 6:05 pm

@Murmaider there seems to be some flaw in your tests.

Lets make a real load on the router
CCR1036 with 10G link DDOS attack from x.0.0.0/8 different addresses 64byte packet size

Attack to routers address (connected route no firewall) = 1.01Mil pps with 100% CPU load
Blackhole route = 1.6Milion pps with 100% CPU load
RAW rule with action no-track = 1Mil pps with 100% CPU load (this just indicates how fast RAW actually is when packet is not sent to connection tracking)
RAW rule action drop = 4.1mil pps with 100% CPU load
1 mangle rule to accept all traffic in prerouting = 94000 pps with 100% CPU load

So according to test it turns out that RAW drop is actually more than twice faster than blackhole.

Murmaider · Tue May 23, 2017 6:19 pm

@Murmaider there seems to be some flaw in your tests.

Lets make a real load on the router
CCR1072 with 10G link DDOS attack from x.0.0.0/8 different addresses 64byte packet size

Attack to routers address (connected route no firewall) = 1.01Mil pps with 100% CPU load
Blackhole route = 1.6Milion pps with 100% CPU load
RAW rule with action no-track = 1Mil pps with 100% CPU load (this just indicates how fast RAW actually is when packet is not sent to connection tracking)
RAW rule action drop = 4.1mil pps with 100% CPU load
1 mangle rule to accept all traffic in prerouting = 94000 pps with 100% CPU load

So according to test it turns out that RAW drop is actually more than twice faster than blackhole.

Blackhole should be quicker, I don't think you taking reverse path filtering ( RP Filter ) set to loose into account. When a route is blackholed and RPF is set to loose, RP Filter checks the packet,if the route entry points to blackhole, the Reverse Path Forwarding (RPF) check fails, and the packet is dropped. It's my understanding that this happens before any route decissions are made (be it blackhole or forward). This should drop far more than 4.1mpps.

I would like to see this done in real load, keeping in mind the uRPF will only kick in after a few seconds as seen in the video i made - https://www.youtube.com/watch?v=SHjBbbF8Ek4

mrz · Wed May 24, 2017 1:00 pm

RP filter in this case doesn't make any difference. But route cache does.
With disabled route cache:
blackhole = 4.45Mil pps
raw drop = 4.2Mil pps

Murmaider · Wed May 24, 2017 2:37 pm

RP filter in this case doesn't make any difference. But route cache does.
With disabled route cache:
blackhole = 4.45Mil pps
raw drop = 4.2Mil pps

Why does disabling the route cache increase the pps?

What CPU was observered at blackhole 4.45Mpps?

mrz · Wed May 24, 2017 2:58 pm

100% CPU in all tests.
Route cache is just an additional load on CPU. Whenever new packet arrives, it goes through route cache table to find matching hash, if there is no match new src/dst hash entry is generated.

Murmaider · Wed May 24, 2017 3:16 pm

100% CPU in all tests.
Route cache is just an additional load on CPU. Whenever new packet arrives, it goes through route cache table to find matching hash, if there is no match new src/dst hash entry is generated.

Interesting, any idea why in my test and video the CPU usage drops to 0% with the blackhole + RP Filter, but with raw the CPU stays at 14% ?
Have you done a test on your side with the RP filter set to loose? I understand you say it would not make a difference, but I'm interested to see if you observed the same thing on yourside as in my video where the DDoS packets, bandwidth and CPU usage drop to zero while the attack continues.

Cha0s · Wed May 24, 2017 6:00 pm

Interesting, any idea why in my test and video the CPU usage drops to 0% with the blackhole + RP Filter, but with raw the CPU stays at 14% ?

I am not sure if your test is correct. If you are getting DDoSed you cannot stop the incoming packets from reaching the router regardless of how you drop them (firewall, blackhole, whatever) on the router itself. The only real way of blocking the attack is before it reaches you. ie: you should block the attack upstream (usually using BGP community blackholing) or employ an anti DDoS technology which can identify the attack flows and blackhole them (again in upstream) leaving the legit flows pass through.

What you are showing in your video is the attack stopping altogether a little after you enable the blackhole route. Which doesn't make sense.
The router should have still received the attack packets but simply not forward them anywhere as with firewall method.

Just to make sure I tried to recreate your test on my lab and I couldn't repeat the behavior shown in your video.
When I enable the blackhole route the attack stops being forwarded but the packets still reach the router before being dropped.

What I suspect (and I may be wrong!) is happening in your video is once you enable the blackhole route, when the ARP entry on your attack machine expires, it tries to send a new arp broadcast to find out which mac address has the IP you are sending packets to and since the router cannot send packets to the source IP, your attack box never receives the arp reply and effectively cannot send the traffic anywhere thus your router stops receiving any attack packets. But this is not a real world scenario of course.

Regardless, I don't think it's a viable option for a true DDoS attack. On the one hand it's more likely your uplink will get congested even before the packets reach the router (ie: you have 10gbit uplink but the attack is 15gbit, so even if you drop them one way or another, nothing changes really since it's your physical medium that's congested not only the router).
On the other hand how will you automatically block thousands of IPs (which is usually the case in a true DDoS attack)? How will you classify which source IP is bad and which is good?

Personally I've settled on Fastnetmon and BGP community blackholing on my upstreams.
Sure, it will block the IP being attacked but at least the rest of my network will not be affected. Most datacenters nowadays use this method to automatically block incoming attacks. And some datacenters are beginning to deploy more complete solutions like Arbor to scrub the attack traffic and only let the legit traffic pass through. But Arbor is extremely expensive...

Just for the record, I did try the raw firewall drop rule and from 50% cpu usage (100% on one core) it dropped immediatelly to 18% cpu usage (on an RB3011).
What's interesting is that doing a drop the cpu drops immediatelly. But doing a no-track the cpu usage is at 100% (on a single core) but it will push more mbit to the outgoing interface with it than without it.

Murmaider · Wed May 24, 2017 7:03 pm

Interesting, any idea why in my test and video the CPU usage drops to 0% with the blackhole + RP Filter, but with raw the CPU stays at 14% ?
I am not sure if your test is correct. If you are getting DDoSed you cannot stop the incoming packets from reaching the router regardless of how you drop them (firewall, blackhole, whatever) on the router itself. The only real way of blocking the attack is before it reaches you. ie: you should block the attack upstream (usually using BGP community blackholing) or employ an anti DDoS technology which can identify the attack flows and blackhole them (again in upstream) leaving the legit flows pass through.

I agree that in a massive DDoS, your uplink is still going to get saturated and blackholing upstream or external mitigation is the winner for this, the topic at hand is more about firewall drop vs RP Filter blackhole and whether or not sourc-based blackholing is more efficient than firewall drop.

What you are showing in your video is the attack stopping altogether a little after you enable the blackhole route. Which doesn't make sense.
The router should have still received the attack packets but simply not forward them anywhere as with firewall method.

Just to make sure I tried to recreate your test on my lab and I couldn't repeat the behavior shown in your video.
When I enable the blackhole route the attack stops being forwarded but the packets still reach the router before being dropped.

What I suspect (and I may be wrong!) is happening in your video is once you enable the blackhole route, when the ARP entry on your attack machine expires, it tries to send a new arp broadcast to find out which mac address has the IP you are sending packets to and since the router cannot send packets to the source IP, your attack box never receives the arp reply and effectively cannot send the traffic anywhere thus your router stops receiving any attack packets. But this is not a real world scenario of course.

I know why it takes a while before the "mitigation" kicks in and this is mostly likely because of the route cache, the blackhole route does not become active immediately.
ARP may be an issue, however I don't see the arp entry expiring within 10 seconds of it's last successful message sent. The default timeout on linux is 60 seconds, I will setup my testing again and increase the arp timeout to 5 minutes, this should give an answer to this.

The packets should still be hitting the interface of the mikrotik, however I don't think the mikrotik is counting these discard packets or traffic on their interface.
The CPU drop is possibly due to the way the packet is inspected with rp filter set to loose, compared to a firewall inspect of the packet.

Regardless, I don't think it's a viable option for a true DDoS attack. On the one hand it's more likely your uplink will get congested even before the packets reach the router (ie: you have 10gbit uplink but the attack is 15gbit, so even if you drop them one way or another, nothing changes really since it's your physical medium that's congested not only the router).
On the other hand how will you automatically block thousands of IPs (which is usually the case in a true DDoS attack)? How will you classify which source IP is bad and which is good?

Personally I've settled on Fastnetmon and BGP community blackholing on my upstreams.
Sure, it will block the IP being attacked but at least the rest of my network will not be affected. Most datacenters nowadays use this method to automatically block incoming attacks. And some datacenters are beginning to deploy more complete solutions like Arbor to scrub the attack traffic and only let the legit traffic pass through. But Arbor is extremely expensive...

Blocking the source-ip's isn't difficult, software like wanguard and many others do this out the box and pushes the ip's to exabgp which you just feed into mikrotik.
BGP Flowspec is just an enhancement of this.
I completely agree that, any DDoS that is larger than your available bandwidth and saturates your links is going to require upstream devices to assist in mitigating it. Be it RTBH, arbor, etc.

Cha0s · Thu May 25, 2017 6:17 am

I run the tests myself and here are my results.

The topology is

Attacker VM <--> CHR <--> Victim VM

Tests where done using CHR v6.39.1 on ESXi Hypervisor with i7-6700 CPU.
The 'Attacker' and 'Victim' VMs were a minimal debian installations.
The test was CPU bound at times (on the hypervisor level due to other vms) so the results might not be very accurate.
That's why I didn't try to reach as much kpps as possible (with multiple hping3 instances) and max out the cpu, but I rather wanted see which process takes more cpu depending on the test before congestion starts.

Command used to simulate a UDP flood attack on port 53 with packet size 64

hping3 -d 64 -2 -p 53 --flood 192.168.2.2

Test 1

No firewall rules
No Blackhole routes
No RP Filter
No Route Cache

test1.png

Test 2

No firewall rules
No Blackhole routes
No RP Filter
Route Cache Enabled

test2.png

Test 3

No firewall rules
No Blackhole routes
RP Filter Enabled
Route Cache Enabled

test3.png

Cha0s · Thu May 25, 2017 6:18 am

Test 4

Filter Firewall rule - drop
No Blackhole routes
RP Filter Enabled
Route Cache Enabled

test4.png

Test 5

Filter Firewall rule - allow
No Blackhole routes
RP Filter Enabled
Route Cache Enabled

test5.png

Test 6

Raw Firewall rule - no-track
No Blackhole routes
RP Filter Enabled
Route Cache Enabled

test6.png

Cha0s · Thu May 25, 2017 6:19 am

Test 7

Raw Firewall rule - drop
No Blackhole routes
RP Filter Enabled
Route Cache Enabled

test7.png

Test 8

No firewall rules
Blackhole Route
RP Filter Enabled
Route Cache Enabled

test8.png

The interesting thing with the blackhole route is that there's more cpu usage than with a raw drop firewall rule. It seems that the blackhole route is causing the 'routing' process to use cpu.

Also while testing I reproduced the behavior you recorded on your video.
It is in fact an arp time out thing. Maybe due to VMware's vSwitch, I didn't look into it...
The moment I add the blackhole route after a few seconds the attacker stops sending traffic to the router.
Checking the arp table on the attacker show the gateway IP (192.168.1.1) as incomplete. Once I statically added the mac address to the arp table then the traffic started going to the router as shown on the screenshot on test8.

So adding source-based blackhole routes definetaly does not stop the attack (in terms of resources both bandwidth and cpu) as shown on your video nor does it make mikrotik not count the incoming packets. It simply made your 'attack' machine to stop sending out packets because it didn't know where to send them since it was on the same broadcast domain as the router. Packets coming from your uplinks will reach the router no matter what (as shown on all the screenshots that drop the traffic one way or another).

In terms of performance I also thought that blackhole routes were more lightweight tbh. But the test above showing the routing process taking more cpu with the blackhole route got me thinking. There may be a flaw in my test process or something else I missed. More tests are needed for sure

Murmaider · Thu May 25, 2017 9:09 am

I've setup my test environment again with the static arp and you are 100% correct - I'll update the first post of this topic to reflect this.

In my tests I could only get the CPU up to 88%, doing RAW drop managed to get the cpu down to 18%.
When doing blackhole with route cache disabled, I managed to get the cpu down to 15% - this seems in line with what mrz mentioned regarding disabling the route cache.

Thanks for the correction in the results!

Murmaider · Thu May 25, 2017 9:26 am

Doesn't the firewall RAW drop disable ipv4 fastpath? If it does, then the blackhole would appear to be better.

amt · Wed Jun 07, 2017 2:35 pm

hi,

Im new on mikrotik and read this topic. I would like to ask for sending ip to upstream blackholing, what should i do ? do i need any routing filter to do this ? I have BGP community of upstream provider. where should i use it ?
just adding this codes as below enough ? no need any routing filter ?

/ip settings set rp-filter=loose

add bgp-communities=666:666 distance=1 dst-address=192.168.1.0/24 type=blackhole

Thanks.

berlo · Sun Jul 02, 2017 4:26 am

Blackholing is better because we can use it with fastpath active. Disabling route cache is not an option because this action disable fastpath

As DDoS Protected company we're tuning ccr1072 mikrotik devices that we have on IXs and we have this configuration:

- Fastpath active (i see if we disable route cache fastpath is automaticaly disabled)
- Connection tracking disabled
- syncookie active and rp filter in loose
- Most know involved ip/networks in BH route
- Traffic divert setup in place
- If attack start our monitor will divert target ip to our filters

We experienced during TCP flood > 500k pps packet loss for around first 1-2 minutes, someone else experiencing the same?

With that configuration on CCR1072 we have about 10% CPU usage with 8Gbps troughput, so the issue shouldn't be on cpu and i not understand why it happen only for 1-2 minutes (no matter the attack lengh)

jtuttle · Thu Apr 11, 2019 3:47 am

I think what this post really says is DON'T USE A MIKROTIK FOR FIREWALLING.

Then why are you here?

mistyrhythm · Thu Jul 23, 2020 8:10 pm

I think what this post really says is DON'T USE A MIKROTIK FOR FIREWALLING.
Then why are you here?

Do you have any solution ?

Murmaider · Thu Jul 23, 2020 8:16 pm

I think what this post really says is DON'T USE A MIKROTIK FOR FIREWALLING.
Then why are you here?

Do you have any solution ?

Honestly, we migrating all our hardware to supermicro Embedded superservers running 6wind turbo gate. Their dpdk implementation is just on a whole other level and stands up in a ddos like a beast, their bgp flowspec paired with wanguard bats ddos like a champ.

Who is online