Fetch & update AV OTX IPv4 blacklist list for auto IoC IPS firewalling
Posted: Wed Jun 09, 2021 10:05 pm
Hello
Today I had a slow day so made a script which fetches all the current days Open Threat Exchange (OTX) Indicators of Compromise (IoC) IPv4 addresses and adds them to a firewall address-list (if they don't already exist). This way you can automatically block the naughty IPv4 addresses where malware and exfiltration has been found by others.. I'd call it a free IPS for IPv4 ::)
The addresses are added as AVOTX_IndicatorAddresses so use that in firewall rules.
REQUIREMENTS:
.
.
.
WARNING
Today I had a slow day so made a script which fetches all the current days Open Threat Exchange (OTX) Indicators of Compromise (IoC) IPv4 addresses and adds them to a firewall address-list (if they don't already exist). This way you can automatically block the naughty IPv4 addresses where malware and exfiltration has been found by others.. I'd call it a free IPS for IPv4 ::)
The addresses are added as AVOTX_IndicatorAddresses so use that in firewall rules.
REQUIREMENTS:
- You need to get yourself an AV OTX API key from here: AlienVault OpenThreatExchange (it's free)
- Open my "syncOTXIndicators_v3" script in a text editor, change the "PUT_YOUR_API_KEY_HERE" to your actual API Key
- Install my "syncOTXIndicators_v3" script which you edited (/import syncOTXIndicators_v3)
- Set up a scheduled task to run the script once per day (at 23:55 UTC for example, IoC's resets at 0:00 UTC)
NOTE! If you run it early in day (UTC time) there may not be any, or any all day! Uncomment line 12 to set custom start date - Add two raw firewall rules to drop & log all src and dst traffic using the "AVOTX_IndicatorAddresses" address-list
.
Code: Select all
# Fetch IP indicators on OTX from past day to update local blacklist address-list
#
# PUT API KEY HERE:
:local otxApiKey "X-OTX-API-KEY:PUT_YOUR_API_KEY_HERE";
#
:local today [/system clock get date];
:local montonum {"jan"="01";"feb"="02";"mar"="03";"apr"="04";"may"="05";"jun"="06";"jul"="07";"aug"="08";"sep"="09";"oct"="10";"nov"="11";"dec"="12"};
:local otxDate ([:tostr [:pick $today 7 11]]."-".(:$montonum->[:pick $today 0 3])."-".[:tostr [:pick $today 4 6]]."T00:00:00+00:00");
# Uncomment below to set custom start date
#:set otxDate "2021-05-09T00:00:00+00:00"
#
# Ask OTX for all IP indicators
:local otxUrl ("https://otx.alienvault.com/api/v1/indicators/export?types=IPv4&modified_since=".$otxDate);
:log info "AVOTX: Syncing OTX indicators from date $otxDate";
:local otxResponce [:tostr [/tool fetch url=$otxUrl http-header-field=$otxApiKey mode=https output=user as-value]];
#
# Loop for addresses in OTX responce JSON
:local start ([:find $otxResponce "indicator\": "]+13);
:local end ([:find $otxResponce "\"type"]-3);
:local caughtIp [:pick $otxResponce $start $end];
/ip firewall address-list
:while ($end >0) do={
:if ([:len [find where address=$caughtIp and list="AVOTX_IndicatorAddresses"]] = 0) do={
add address=$caughtIp list="AVOTX_IndicatorAddresses";
:log info "AVOTX: Added $caughtIp to blacklist";
} else={ :log info "AVOTX: $caughtIp already in blacklist" }
:set start ([:find $otxResponce "indicator\": " $start]+13);
:set end ([:find $otxResponce "\"type\":" ($end+3)]-3);
:set caughtIp [:pick $otxResponce $start $end];
}
:log info "AVOTX: Sync completed";
.
WARNING
- The script is CPU intensive, it will max out a single core to 100% until it finishes, this will harm performance on single core or already high load routers for a few minutes.
- The OTX lists refresh once per day, so use this late at night (UTC time!!) to get the full days list.
- Over time the IPs will be ceased and possibly be returned to good-guy addresses, after a while if you start getting false-positives this may be why, investigate each log!
- If you want to run the script with a custom start date, for example, 1 month behind to catch up; uncomment line 11 with your own UTC datetime. I'd run this script once with this line saying 1 month behind, then comment it out again.. it will take a while to parse.
- Yes you could use the russian MT JSON parser instead but I didn't want that on production units.