Page 1 of 1
:( ICMP Destination Unreachable Storms Killing Me!
Posted: Wed Mar 22, 2006 1:40 am
by IntraLink
I've got a handfull of users crappy routers spewing ICMP Destination Unreachable packets all over the place.
Is there a way in MT to just kill these all together?
The only thing helping keep this under control is my managed switch storm packet settings. But they have a minimum threshold that is too high and doesn't stop repeated attacks.
Posted: Wed Mar 22, 2006 3:17 pm
by dot-bot
What would I consider:
1. Help these users config their routers and hosts.
2. Try to see a patternt in most of the packets and drop the packets by this pattern.
3. Disconnect users that damage the network and tell them "fix your sh*t and you're back online"
note: I would consider these, not go and execute them right away...
Posted: Wed Mar 22, 2006 4:00 pm
by IntraLink
Here is an example of what the packets look like:
0000 ff ff ff ff ff ff 00 0f b5 a4 cb 23 08 00 45 00 ........ ...#..E.
0010 00 3c 3e c3 00 00 06 01 b4 58 c0 a8 00 fe 00 00 .<>..... .X......
0020 00 00 03 02 a9 d1 00 00 00 00 46 00 00 28 3e c3 ........ ..F..(>.
0030 00 00 01 02 05 f7 00 00 00 00 e0 00 00 16 94 04 ........ ........
0040 00 00 22 00 eb 03 46 00 00 28 .."...F. .(
Ethereal decodes it as a line item:
source: 192.168.0.254
Destination: 0.0.0.0
Protocol: ICMP
Info: Destination unreachable (Protocol unreachable)
Drilling down into the IGMP information it says that the Header Checksum: 0xeb03 [incorrect, should be 0x97d7]
Some things to note:
We don't have 192.168.0.0/24 on our own network or on any of our routers.
Posted: Wed Mar 22, 2006 4:05 pm
by IntraLink
What would I consider:
1. Help these users config their routers and hosts.
2. Try to see a patternt in most of the packets and drop the packets by this pattern.
3. Disconnect users that damage the network and tell them "fix your sh*t and you're back online"
note: I would consider these, not go and execute them right away...
1. What is misconfigured with these customers routers that causes this?
2. I think I have the pattern, but I can't figure out what to put in my MT to block it.
3. I can't just disconnect them because like most users they don't have a clue and wouldn't know how to fix this let alone even begin to understand what the problem is.
I guess if I knew the real cause or root of this problem and there was a fix from the consumer router end I could visit each site and fix it individually.
But sooner or later it will crop up again and it does appear to affect more than just a couple of devices. So it would be nice to be able to stop it from the radio or router level (we use MT and Canopy).
Posted: Wed Mar 22, 2006 6:18 pm
by UniKyrn
We don't have 192.168.0.0/24 on our own network or on any of our routers.
Then put a firewall rule at the router the customers are talking to that drops all 192.168.0.0/16 (fakenet) traffic. That's one of the first things I do on a new AP, just to keep stupid customer routers from flooding our network with that junk. If I'm in a bad mood, I also add a dummy dhcp server on the customer connection that hands out 127.0.0.X IP's to dhcp requests that leak from their network into ours.
Posted: Wed Mar 22, 2006 6:41 pm
by airtech
We have had the same problem on our network and what we are doing to fix it is we are putting every customer on their own VLAN with their own DHCP server to them. There have been a couple things we have seen cause these storms. One of them is we have had quite a few "brilliant" customers plug our connection into the LAN side of their routers. Also, we have noticed, primarily with Linksys routers, that when they hold the reset button in with our connection still plugged in, it causes these storms. So, our solution to this has been to put customers on their own VLANs so that if one customer is causing an ICMP storm, it only effects their connection.
Posted: Wed Mar 22, 2006 6:43 pm
by IntraLink
I like your fake DHCP for 127.0.0.0! LOL
I thought about putting a rule in for the 192.168.0.0/24 network, but even though it's listing that as a source, these are broadcast ICMP packets. So I don't think an IP rule is going to catch that.
Posted: Wed Mar 22, 2006 7:33 pm
by UniKyrn
I like your fake DHCP for 127.0.0.0! LOL
I thought about putting a rule in for the 192.168.0.0/24 network, but even though it's listing that as a source, these are broadcast ICMP packets. So I don't think an IP rule is going to catch that.
You can also put an ICMP specific rule in the firewall forwarding table to block those packets, pick the source interface that makes sense.
Posted: Wed Mar 22, 2006 11:05 pm
by jp
We've had broadcast/multicast problems due to faulty customer routers. Two years ago, Linksys bfsr41's and their like were causing a problem. We worked with Cisco and they made a fix for it that they incorporated into the next firmware. More recently we had the problem with gigafast routers. Gigafast said I could return them for exchange, (which I didn't want), took my details, and never did anything about it. We've just yanked every gigafast that participates in the traffic storm. I've used wrt54g and Airnet routers in their place.
We graph every customer radio with mrtg, so we know who is broadcasting. The actual traffic is full of spoofed mac addresses and spoofed IPs, so there' s not much value to it. As you can see, all three customers in this pic are broadcasting, but the second and third are the worst offenders.
The gigafast routers don't seem to misbehave when on small networks, so we ocassionaly reuse them on a smaller routed networks. We are also gradually changing the network from a few seperately routed multi-site bridged networks to lots of small routed cells with small netblocks.
[/img]