Community discussions

MikroTik App
 
thermant
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 93
Joined: Sat Apr 21, 2007 6:17 am

Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 12:04 am

Hi all,

My PC ROS has 3 NICs: PUBLIC, LOCAL (192.168.1.0/24), and PRIVATE (192.168.2.0/24, for my personal use). Earlier today, my LOCAL network suddenly started acting up. VERY frequent connection drops & RTOs, high latency, and it took forever to reconnect. The PRIVATE network is fine, even now.

I then looked up the LOCAL interface traffic, and it shows 2.4Mbps Tx and 7.1Mbps Rx (holy smokes!! :shock:), like so:
Image

After I torched it, I see this:
Image

Here's what I put in my firewall rules:
Flags: X - disabled, I - invalid, D - dynamic 
 0   ;;; Allow established connections
     chain=input action=accept connection-state=established 

 1   ;;; Allow related connections
     chain=input action=accept connection-state=related 

 2   ;;; Drop invalid connections
     chain=input action=drop connection-state=invalid 

 3   ;;; Blocking outside relay
     chain=input action=drop protocol=tcp src-address=0.0.0.0/0 in-interface=PUBLIC dst-port=8080 

 4   ;;; Allow UDP
     chain=input action=accept protocol=udp 

 5   ;;; Allow limited ICMP
     chain=input action=accept protocol=icmp limit=50/5s,2 

 6   ;;; Drop excess pings
     chain=input action=drop protocol=icmp 

 7   ;;; Allow access from local network
     chain=input action=accept in-interface=!PUBLIC 

 8   ;;; Blocking telnet access method
     chain=input action=drop protocol=tcp dst-port=23 

 9   ;;; Log everything else
     chain=input action=log log-prefix="DROP INPUT" 

10   ;;; Drop everything else
     chain=input action=drop 

11   ;;; allow established connections
     chain=forward action=accept connection-state=established 

12   ;;; allow related connections
     chain=forward action=accept connection-state=related 

13   ;;; drop invalid connections
     chain=forward action=drop connection-state=invalid 

14   chain=forward action=drop src-address=0.0.0.0/8 

15   chain=forward action=drop dst-address=0.0.0.0/8 

16   chain=forward action=drop src-address=127.0.0.0/8 

17   chain=forward action=drop dst-address=127.0.0.0/8 

18   chain=forward action=drop src-address=224.0.0.0/3 

19   chain=forward action=drop dst-address=224.0.0.0/3 

20   chain=forward action=drop protocol=tcp dst-port=135 

21   chain=forward action=drop tcp-flags=syn protocol=tcp connection-limit=6,32 

22   ;;; jump to the tcp chain
     chain=forward action=jump jump-target=tcp protocol=tcp 

23   ;;; jump to the udp chain
     chain=forward action=jump jump-target=udp protocol=udp 

24   ;;; jump to the icmp chain
     chain=forward action=jump jump-target=icmp protocol=icmp 

25   ;;; Log dropped connections
     chain=forward action=log log-prefix="DROP FORWARD" 

26   ;;; Drop and log everything else
     chain=forward action=drop protocol=!tcp 

27   ;;; deny TFTP
     chain=tcp action=drop protocol=tcp dst-port=69 

28   ;;; deny RPC portmapper
     chain=tcp action=drop protocol=tcp dst-port=111 

29   ;;; deny RPC portmapper & NBT
     chain=tcp action=drop protocol=tcp dst-port=135-139 

30   ;;; deny cifs
     chain=tcp action=drop protocol=tcp dst-port=445 

31   ;;; deny NFS
     chain=tcp action=drop protocol=tcp dst-port=2049 

32   ;;; deny NetBus
     chain=tcp action=drop protocol=tcp dst-port=12345-12346 

33   ;;; deny NetBus
     chain=tcp action=drop protocol=tcp dst-port=20034 

34   ;;; deny BackOriffice
     chain=tcp action=drop protocol=tcp dst-port=3133 

35   ;;; deny DHCP
     chain=tcp action=drop protocol=tcp dst-port=67-68 

36   ;;; deny TFTP
     chain=udp action=drop protocol=udp dst-port=69 

37   ;;; deny PRC portmapper
     chain=udp action=drop protocol=udp dst-port=111 

38   ;;; deny PRC portmapper & NBT
     chain=udp action=drop protocol=udp dst-port=135-139 

39   ;;; deny NFS
     chain=udp action=drop protocol=udp dst-port=2049 

40   ;;; deny BackOriffice
     chain=udp action=drop protocol=udp dst-port=3133 

41   ;;; drop invalid connections
     chain=icmp action=accept protocol=icmp icmp-options=0:0 

42   ;;; allow established connections
     chain=icmp action=accept protocol=icmp icmp-options=3:0 

43   ;;; allow already established connections
     chain=icmp action=accept protocol=icmp icmp-options=3:1 

44   ;;; allow source quench
     chain=icmp action=accept protocol=icmp icmp-options=4:0 

45   ;;; allow echo request
     chain=icmp action=accept protocol=icmp icmp-options=8:0 

46   ;;; allow time exceed
     chain=icmp action=accept protocol=icmp icmp-options=11:0 

47   ;;; allow parameter bad
     chain=icmp action=accept protocol=icmp icmp-options=12:0 

48   ;;; deny all other types
     chain=icmp action=drop 
As you can see, I blocked TCP & UDP port 135-139 in rules 29 & 38, but I am still seeing traffic in port 137 & 138 from the clients to 192.168.1.255. Am I missing something here? This thing is clogging up my LOCAL traffic, and nothing can get through, locally and to the net. Well, in LOCAL it is clogged up. PRIVATE is running just fine. Everything was fine before today, and I didn't change anything in the rules.

Oh, by the way, that computer in the 2nd picture (192.168.1.165) is not powered on when I took the picture. I turned it off about an hour before the picture was taken, but it's still there! GAAAHHH!!!

I really don't think it's virus/trojan/worm/the like, because I keep the AV definitions updated with the latest version (well, it auto-updates, but meh...). And EVEN IF IT IS, the firewall should've blocked it, no?

Please, please, please help.

I've been working on this for 10 hours, been reading documentations, manuals, and tutorials till my eyes bleed, and I still can't figure out what's wrong. Plus, I only have several hours before my boss comes in and starts yelling at me.

Thanks a gazillion

-Ted-
 
fewi
Forum Guru
Forum Guru
Posts: 7717
Joined: Tue Aug 11, 2009 3:19 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 12:22 am

192.168.1.255 is the broadcast address for 192.168.1.0/24, so that traffic goes to all hosts on that network (including the firewall). You've done what you can on the firewall to ensure the traffic doesn't propagate outside of the LOCAL network, but the firewall can't do anything about traffic within the network, from the offending machines to the other hosts because that traffic doesn't flow through the firewall but rather flows between the hosts directly. Find the offending machines (at least 192.168.101, according to Torch) and clean it. It is most likely malicious traffic from an infection or a malicious user.

Your problem doesn't really have anything to do with the router, the router is configured correctly. Your issue is on the network behind it.

Torch will still show the traffic because the traffic still arrives on the firewall interface, the packets will be dropped by firewall rules but that doesn't keep them physically off the interface.
 
thermant
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 93
Joined: Sat Apr 21, 2007 6:17 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 12:31 am

Well, I can see why you would say what you said, but what about 192.168.1.165? The machine's been off for an hour. And just now, I turned on another PC (192.168.1.210) as a test, then turned it off again. After 5 minutes, I torched again, and the traffic appears!! So now there're three computers listed in Torch (101, 165, and 210), 2 of which are already turned off.

Is that possible?

I mean, I think that these broadcast packets would stop when the "offending" PC is turned off, am I wrong? So how could these traffic stays there?

I'll try cleaning the PC up, but this is just plain weird. And is it normal, a continuous broadcast of 1Mb+?

Edit: Well, I turned off 101 about 10 minutes ago, and guess what? It's still there. :(
Last edited by thermant on Mon Jan 11, 2010 12:41 am, edited 1 time in total.
 
fewi
Forum Guru
Forum Guru
Posts: 7717
Joined: Tue Aug 11, 2009 3:19 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 12:36 am

What's the entry time-out in Torch?

And no, that is not normal. It's probably malicious traffic.
 
thermant
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 93
Joined: Sat Apr 21, 2007 6:17 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 12:43 am

How do I look at the entry time out?

edit: Still downloading the newest AV definition, thought I would overwrite the ones that's already there. Or maybe get another AV altogether. Meh... Anyway, what I don't understand is how a turned-off PC can still broadcast?
 
fewi
Forum Guru
Forum Guru
Posts: 7717
Joined: Tue Aug 11, 2009 3:19 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 12:51 am

In the basic section of the Torch tool, right under the interface is a field for 'Entry Timeout'. Connections seen by Torch won't go away from the list of displayed connections until that timeout expires. If it's set to an hour, you would see connections from an hour ago still listed.
 
thermant
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 93
Joined: Sat Apr 21, 2007 6:17 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 12:55 am

Ooohh, that one. D'oh! Wonder why I didn't see it the first time.

Anyway, it says 00:00:03, which I assume would be, uh, 3 seconds?

How the hell does it becomes hours then?
Oh, and fewi, the numbers are constantly changing, so it's not like it stays at a certain number all the time. It really seems like something is still broadcasting (I made sure the PCs are really, truly, turned off).

Edit: Doh! I forgot to mention this before, but my clients can only go to certain websites that I allow, they can't download anything, and they can't plug in any USB flash drives, either. I installed all the programs meself too...
 
User avatar
butche
Trainer
Trainer
Posts: 430
Joined: Fri May 28, 2004 6:14 pm
Location: Missouri, USA
Contact:

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 1:02 am

You need to add this firewall rule and enable it for just a second or two:
/ip firewall filter
add chain=input place-before=0 action=log protocol=udp dst-port=137
This will put a filter rule at the top of your input chain that will capture the packet to the local log file. This will show you the mac address of the machine that is sending those packets. You can troubleshoot the pc from there.
 
thermant
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 93
Joined: Sat Apr 21, 2007 6:17 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 1:33 am

@fewi:
The bad news: I scanned 210, and it came out clean. No virus, malware, or whatever. Well, that is to be expected I suppose, since I limited the users' access that much. But it is still there in the torch window, even after I turned it off.

The good news: 101 and 165 is now gone. Just like that. Hmm... But now it's 210 that's eating up around 2M for port 138 and around 1.5M for port 137. Ouch... Other clients still can't connect.

@butche:
Did what you asked, and the IP & MAC for 192.168.1.210 came up again. Like I said to fewi, I already scanned it and it came out clean. Still nothing I can do about the stupid torch. Any other suggestions? :?
 
fewi
Forum Guru
Forum Guru
Posts: 7717
Joined: Tue Aug 11, 2009 3:19 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 1:50 am

Install Wireshark or some other protocol analyzer on a machine on the network and confirm it sees the same thing as Torch.

Just because a malware scan came up clean doesn't mean the machine is clean, though.
 
User avatar
jwcn
Forum Guru
Forum Guru
Posts: 1495
Joined: Sun Aug 27, 2006 6:49 am
Location: Maryland, USA
Contact:

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 2:04 am

Could be a bad NIC in that machine too...
 
thermant
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 93
Joined: Sat Apr 21, 2007 6:17 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 2:12 am

@jwcn:
Which machine? 192.168.1.210? You think so? The same problem happened on 3 machines before, remember? Or do you mean the router? I think that's not the case too, because other clients also can't connect to anywhere... Well, but that's just me.

@fewi:
Wireshark eh? Alright, I'll try that.
After that, what then? Say it sees the same thing, or not the same thing, as Torch. What does it mean?
 
fewi
Forum Guru
Forum Guru
Posts: 7717
Joined: Tue Aug 11, 2009 3:19 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 2:21 am

If it sees the same thing those machines are spewing out traffic. Maybe someone spoofed their IP and is spewing out traffic. At that point you're not dealing with RouterOS anymore so it's sort outside of the scope of the forum.
 
thermant
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 93
Joined: Sat Apr 21, 2007 6:17 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 2:35 am

Well...

Darn, maybe I should explain things a bit more clearly first.

OK, here's the situation at my workplace:
I work at a sort of training center that heavily utilizes computers. All my clients can only use the computer as is. The websites that can be visited are limited, no one can download anything, no one can plug in their own USB drives or CD/DVD roms. Control panel and task manager are blocked, too. Ergo, no one can install/uninstall anything, or even do anything other than use the programs already installed in the machines. I've taken all the precautions and necessary steps to ensure that the machines are, well, quite "secure".

And besides, my clients usually comes from those who only uses computers the way they are originally taught to. So IP/MAC spoofing is, honestly, quite beyond their knowledge. Heh... :)
 
rmichael
Forum Veteran
Forum Veteran
Posts: 718
Joined: Sun Mar 08, 2009 11:00 pm

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 3:32 am

Brodcast traffic could be a symptom of a network loop. If your switch is managed you'll be able utilization via GUI. If unmanaged look for fast blinking activity light, diconnect it and see if your problem disapears...

If all lights blink disconnect half of your network -if you cascade your switches you can start with unpliging uplinks first. What you're looking for is for fast blinking to stop or diminish substancially. If it happens - you'll know that what you disconnected is the source of brodcast storm.

At last I'd suggest that if you don't have a budget for all switches to be managed just get one - it helps a lot with troubleshooting.

EDIT: more details
Last edited by rmichael on Mon Jan 11, 2010 7:40 am, edited 1 time in total.
 
thermant
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 93
Joined: Sat Apr 21, 2007 6:17 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 3:40 am

@rmichael:
My switches are unmanaged. And YES, the lights are blinking like crazy. This one 8-port that I replaced earlier, all its lights were blinking like mad. The funny thing is, they all blink at the same time. I disconnected each cables, plugged them back, and even replaced the switch with a new one, but the problem didn't go away.

<sigh>

Any other ideas?
 
User avatar
butche
Trainer
Trainer
Posts: 430
Joined: Fri May 28, 2004 6:14 pm
Location: Missouri, USA
Contact:

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 5:12 am

First thing to verify is that the traffic you see in torch, coming from the 192.168.1.210 machine really is coming from that machine. That was the point of the rule I showed you earlier. If it really IS that machine (verify by comparing the mac address and IP from the logs to the actual machine). Second, turn off EVERY machine on that segment and then turn them on one at a time (or simply unplug their ethernet cable and plug them in one at a time). In spite of your insistence that there is no virus, there IS something causing at least one machine to misbehave. That's the only way to track it down is to eliminate all machines and reintroduce each one individually.
 
fewi
Forum Guru
Forum Guru
Posts: 7717
Joined: Tue Aug 11, 2009 3:19 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 5:24 am

And if it is a switching loop, unplug everything everywhere and plug things in one by one and wait for the traffic you're observing to re-appear - without a managed switching platform that's as good as you can do. That'd be after investigating machine by machine as butche described.
 
thermant
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 93
Joined: Sat Apr 21, 2007 6:17 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 8:22 am

So, the brute force approach, eh? Darn...

Oh well...

So what I'm looking for is the machine that, when I unplug its ethernet cable, would cause the storm to stop, yes?
Alright. I'll try it and let you all know how it goes.

Thanks, all.
 
thermant
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 93
Joined: Sat Apr 21, 2007 6:17 am

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 3:33 pm

....
...
..
.

IT'S DONE!!! YAAAAYYYYY!!!!

Turns out I left a LAN cable laying around one of the switches, and some idiot cleaning crew thought he somehow dropped it. Hence he plugged it back in. Both ends in the same switch. D'oh! Let the storm begins.

That'll teach me NOT to leave any cables laying around unplugged near a switch.

@fewi, butche, rmichael, jwcn (uhh, is that everyone? Did I miss anyone?):
THANK YOU SO VERY VERY MUCH!
If there's a "Thanks" button, I would clicked it, many times, for each one of you. Yeah, that makes me wonder, why isn't there a thank you point system here in the forum? Meh, whatever...

Again, thanks much.
Couldn't have done it without you all.

-Z-
 
User avatar
butche
Trainer
Trainer
Posts: 430
Joined: Fri May 28, 2004 6:14 pm
Location: Missouri, USA
Contact:

Re: Network suddenly flooded by port 137 & 138 traffic. Help...

Mon Jan 11, 2010 7:21 pm

Turns out I left a LAN cable laying around one of the switches, and some idiot cleaning crew thought he somehow dropped it. Hence he plugged it back in. Both ends in the same switch. D'oh! Let the storm begins.
Wow! I am glad you found the problem. FWIW, If your switches are capable, this is exactly why you'd want to turn on STP in the switches.

Who is online

Users browsing this forum: almdandi and 19 guests