Community discussions

MikroTik App
  4. RouterOS
  5. General

Logging web traffic to remote webserver

Post Reply
 
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 281
Joined: Sat Aug 24, 2013 8:26 pm

Logging web traffic to remote webserver

Sat May 10, 2014 8:26 pm

I want to send web traffic log to a remote server listening on UDP 514 from a CPE and then save it with a properly software.
CPE config:
ip firewall filter rules
Code: Select all
 5   ;;; default configuration
     chain=forward action=accept connection-state=established 

 6   ;;; default configuration
     chain=forward action=accept connection-state=related 

 7   chain=forward action=log tcp-flags=syn protocol=tcp log-prefix="clientlog>
system logging rules
Code: Select all
 4    topics=web-proxy prefix="clientlog" action=remote
system logging action
Code: Select all
 3 * name="remote" target=remote remote-port=514 src-address=0.0.0.0 
     bsd-syslog=no syslog-facility=daemon syslog-severity=auto 

 4   name="remotelog" target=remote remote=MY PUBLIC IP ADDRESS remote-port=514 
     src-address=0.0.0.0 bsd-syslog=no syslog-facility=daemon 
     syslog-severity=auto
port 514 on webserver firewall open
port 514 on modem firewall open

still no data received!
Something wrong with CPE config?
Top
 
jaykay2342
Member
Member
Posts: 336
Joined: Tue Dec 04, 2012 2:49 pm
Location: /Vigor/LocalGroup/Milky Way/Earth/Europe/Germany

Re: Logging web traffic to remote webserver

Sat May 10, 2014 9:06 pm

Are you using the buildin proxy-server? because topics=web-proxy selects the log from the buildin web-server not logs from the firewall
Top
 
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 281
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sat May 10, 2014 9:22 pm

now I set topic to firewall, but no data yet :(
Top
 
lordzar
Frequent Visitor
Frequent Visitor
Posts: 94
Joined: Sat May 29, 2004 7:47 pm

Re: Logging web traffic to remote webserver

Sat May 10, 2014 10:37 pm

First thing I see is that your logging rule says "remote" and not "remotelog"
Top
 
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 281
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sat May 10, 2014 10:50 pm

You're right . I changed to "remote" now but still no data.
Maybe it's my network since is natt'd.
Should I open port 514 for each NAT node?
Top
 
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 281
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 12:51 am

ok with topic=firewall now it works.
which is the different with web-proxy topic? Which is better to use?
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Sun May 11, 2014 5:11 am

the question is not what is better, but what is working with your actual configuration


remember:
for Italian law, as ISP, you must log:

all tcp and udp new connection with source and destionation ip and port and the exact time on GMT 00:00 and the username used.
and all of the other type of new connection (like ICMP, GRE, etc.)

and you MUST NOT LOG site and page visited for privacy LAW...
Top
 
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 281
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 1:41 pm

ok, sorry for my incompetence.
so what method do you think it's better for logging customer traffic respecting italian privacy law?
I thought to log to a remote server (e.g syslog watcher) listening on udp 514 and saving all firewall topics.
so data captured are like this:
Image
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Sun May 11, 2014 5:20 pm

I use (and I still to use) Syslog Watcer Pro v2!!!

Put those rules at the end of firewall filter drop on forward rules (if you have any):

lista_ip_clienti = automatic list made by pppoe-server profile on "address list" field
this log only the client traffic, and not all the network traffic.

The first rule log TCP, UDP, ICMP, IGMP etc. new connection,
the second the RELATED connection for the first,
the other rule log the other protocols.
Code: Select all
add action=log chain=forward comment="Invio Log connessioni in uscita al Server" connection-state=new src-address-list=lista_ip_clienti
add action=log chain=forward connection-state=related log-prefix=RELATED src-address-list=lista_ip_clienti
add action=log chain=forward comment="Tunnel IPv6 dentro IPv4" protocol=ipv6 src-address-list=lista_ip_clienti
add action=log chain=forward protocol=gre src-address-list=lista_ip_clienti
add action=log chain=forward protocol=ipsec-esp src-address-list=lista_ip_clienti
And configure accordingly system / logging

Code: Select all
/system logging action
add bsd-syslog=yes name=LogServerNotice remote=1.1.1.1 syslog-facility=local1 syslog-severity=notice target=remote
add bsd-syslog=yes name=LogServerInfo remote=1.1.1.1 syslog-facility=local1 syslog-severity=info target=remote

/system logging
add action=LogServerNotice topics=pppoe,info,!debug
add action=LogServerInfo topics=firewall,info,!debug
Those rules not only log the firewall, but also the pppoe autentication.

ALL THIS RULE ARE VALID ONLY FOR "PPP" CLIENTS, for HotSpot the method is different...

For HotSpot I suggest you to use "Cloud4Wi"...
Top
 
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 281
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 5:29 pm

if users do not log in via ppoe, should I manually configure each client's cpe for remote logging?
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Sun May 11, 2014 5:39 pm

if users do not log in via ppoe, should I manually configure each client's cpe for remote logging?
No, if you assign static public IP to client, you not need to log anything, each IP is exactly that customer :)
You must only track each time you change the "owner" of IP address :)

What method you use for login the CPE to Gateway?

any "ppp" method can be logged with that described on previous post, simply change pppoe in the log, with what you want.
Top
 
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 281
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:01 pm

by now I access cpe's via routing.
the problem is cpe addresses are assigned by Access Point dhcp server.
so manual config I think is necessary..
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:05 pm

by now I access cpe's via routing.
the problem is cpe addresses are assigned by Access Point dhcp server.
so manual config I think is necessary..
You decide that yourself or other "ISP starter" company as suggested that to you?

It's very hard log all easy with this method,
at this point the rule for logging must be put on access point, not on cpe.
Top
 
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 281
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:11 pm

why on the access point?
I configured a customer cpe to remote log on udp 514 to a pc into my lan and it seems to work as picture I posted in previous post.
Last edited by Arpanet on Sun May 11, 2014 6:20 pm, edited 2 times in total.
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:17 pm

why on the access point?
I configured a customer cpe to remote log on udp 514 to a pc into my lan and it seems to work as picture I posted in previous post.
1) Because you use user traffic for log the user, and you also log the logging of the user... ;)
2) If you want change any, you change only the AP, not all the CPE...
Top
 
Arpanet
Member Candidate
Member Candidate
Topic Author
Posts: 281
Joined: Sat Aug 24, 2013 8:26 pm

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:21 pm

multiple cpe's logging is allowed?
lf I set user cpe for logging this will be heavy on user connection speed?
Last edited by Arpanet on Sun May 11, 2014 6:27 pm, edited 1 time in total.
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Sun May 11, 2014 6:22 pm

Yes.

Syslog use less space than "packet capture"....
Top
 
simogere
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Wed Jul 09, 2014 6:28 pm


Those rules not only log the firewall, but also the pppoe autentication.

ALL THIS RULE ARE VALID ONLY FOR "PPP" CLIENTS, for HotSpot the method is different...

For HotSpot I suggest you to use "Cloud4Wi"...
Hi rextended, thanks for the conf.
What about hotspot loging configuration?

Thanks in advance, Simone.
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Wed Jul 09, 2014 9:47 pm

after this rule:
/ip firewall nat
add action=passthrough chain=unused-hs-chain comment="place hotspot rules here" disabled=yes

immediately after add this:
/ip firewall nat
add action=log chain=pre-hotspot

and this on system logging:
Code: Select all
/system logging action
add bsd-syslog=yes name=LogServerNotice2 remote=1.1.1.1 syslog-facility=local2 syslog-severity=notice target=remote
add bsd-syslog=yes name=LogServerInfo2 remote=1.1.1.1 syslog-facility=local2 syslog-severity=info target=remote

/system logging
set 0 prefix=INFO topics=info,!firewall
add action=LogServerInfo2 topics=firewall,info,!debug
add action=LogServerNotice2 topics=hotspot,info,!debug
Top
 
simogere
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Wed Jul 09, 2014 10:43 pm

Hi thanks for the post. I'll try it tomorrow.

What are the differences between this method?

[admin@Test-HotSpot] > ip firewall filter add chain=forward action=log connection-state=new
[admin@Test-HotSpot] > system logging add topics=firewall action=remote
[admin@Test-HotSpot] > system logging action set remote remote=[remote server ip]
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Thu Jul 10, 2014 10:39 am

Hi thanks for the post. I'll try it tomorrow.

What are the differences between this method?

[admin@Test-HotSpot] > ip firewall filter add chain=forward action=log connection-state=new
[admin@Test-HotSpot] > system logging add topics=firewall action=remote
[admin@Test-HotSpot] > system logging action set remote remote=[remote server ip]
"mine" is better and complete... ;)
Top
 
simogere
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 11:10 am

Hi thanks for the post. I'll try it tomorrow.

What are the differences between this method?

[admin@Test-HotSpot] > ip firewall filter add chain=forward action=log connection-state=new
[admin@Test-HotSpot] > system logging add topics=firewall action=remote
[admin@Test-HotSpot] > system logging action set remote remote=[remote server ip]
"mine" is better and complete... ;)
ok :D

Some questions:
- I can't see logs in log shell of rb. How i can do it? (without storing them)
- here a simple log (from my syslog)
Image
Any way to show not just calling ip but radius user too?
- How I can send radius / captive portal logs to syslog? (eg. user logins / failure, ip assigned, etc.)

Thanks, Simone.
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 3:59 pm

ci sono due regole apposta, la prima logga le connessioni, la seconda (add action=LogServerNotice2 topics=hotspot,info,!debug) logga gli utenti, così rimane la corrispondenza tra utente e mac address/IP


>>>- I can't see logs in log shell of rb. How i can do it? (without storing them)
Code: Select all
/system logging
add action=memory topics=firewall,info,!debug
add action=memory topics=hotspot,info,!debug
Remember to disable it after you have used...
Top
 
simogere
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 4:29 pm

ci sono due regole apposta, la prima logga le connessioni, la seconda (add action=LogServerNotice2 topics=hotspot,info,!debug) logga gli utenti, così rimane la corrispondenza tra utente e mac address/IP


>>>- I can't see logs in log shell of rb. How i can do it? (without storing them)
Code: Select all
/system logging
add action=memory topics=firewall,info,!debug
add action=memory topics=hotspot,info,!debug
Remember to disable it after you have used...

I'm sorry but with your conf I can't see hotspot infos.
Code: Select all
/ip firewall nat add action=log chain=pre-hotspot
/system logging action
add bsd-syslog=yes name=LogServerNotice2 remote=192.168.2.2 syslog-facility=local2 syslog-severity=notice target=remote
add bsd-syslog=yes name=LogServerInfo2 remote=192.168.2.2 syslog-facility=local2 syslog-severity=info target=remote

/system logging
set 0 prefix=INFO topics=info,!firewall
add action=LogServerInfo2 topics=firewall,info,!debug
add action=LogServerNotice2 topics=hotspot,info,!debug
About log shell: why I have to remember to disable it after use?

PS: skype?
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 4:40 pm

***NO SKYPE***

Because if the users make traffic, you can not reach to read other logs...
Top
 
simogere
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 4:47 pm

Ok, I don't uderstand :D

Any tips about hotspot logs that not working?
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 4:51 pm

really for me is working, is my production config.....

for see any log on hotspot side you must wait user login/logoff...
Top
 
simogere
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:04 pm

RouterOS version? Mine is 6.15
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:13 pm

RouterOS version? Mine is 6.15
RB1200 RouterOS 6.15
Top
 
simogere
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:18 pm

Mine is RB450. Damn, I can't find where is the problem...
Top
 
simogere
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:36 pm

I think I found the problem:
Code: Select all
/system logging
set 0 prefix=INFO topics=info,!firewall
Set 0 not working:
Code: Select all
[admin@MikroTik] /system logging> set action=
LogServerInfo2  LogServerNotice2  disk  echo  memory  remote
Which one I have to use?
Top
 
User avatar
rextended
Forum Guru
Forum Guru
Posts: 12558
Joined: Tue Feb 25, 2014 12:49 pm
Location: Italy
Contact:
Contact rextended

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:41 pm

I think I found the problem:
Code: Select all
/system logging
set 0 prefix=INFO topics=info,!firewall
Set 0 not working:
Code: Select all
[admin@MikroTik] /system logging> set action=
LogServerInfo2  LogServerNotice2  disk  echo  memory  remote
Which one I have to use?
memory, the default.

If you syslog is not bsd capable, remove that flag on logging action
Top
 
simogere
Frequent Visitor
Frequent Visitor
Posts: 56
Joined: Fri May 24, 2013 11:54 am

Re: Logging web traffic to remote webserver

Fri Jul 11, 2014 5:44 pm

With this one works fine:
Code: Select all
/system logging
set LogServerNotice2 prefix=INFO topics=info,!firewall
My syslog is a pc with The Dude 3.6
Top
 
User avatar
niamul
Frequent Visitor
Frequent Visitor
Posts: 50
Joined: Tue Dec 04, 2007 9:33 am
Location: Dhaka, Bangladesh
Contact:
Contact niamul

Re: Logging web traffic to remote webserver

Tue May 02, 2017 12:26 pm

Dear rextended,
thank you for the help. it has really saved me.
cheers
Niamul
Top
Post Reply

Who is online

Users browsing this forum: Mosmos and 32 guests
  4. RouterOS
  5. General