View clients' http requests and traffic

Arpanet · Sat Sep 21, 2013 7:31 pm

I'm looking for a RouterOs tool to see and possibly save as a log, all http requests, such as visited websites, of my clients CPE's.
I took a look into manual but I can't find anything yet!

thanks in advance

Sat Sep 21, 2013 7:50 pm

If you wanted to just look at the current traffic, and the IP of the web server the clients are visiting, you can just use "/tool torch".

If you want to see the contents and store (some of?) it, it's a little more complicated. I'd go for setting up a web server as a "transparent proxy":

Set up a web server with a server language such as PHP. Then make it make a request to whatever the HTTP request dictates with $_SERVER['HTTP_HOST'], $_SERVER['REQUEST_URI'] and so on, and output whatever the remote site outputs, with headers and everything. You can then store the request and/or response on the web server. At the router, in the "/ip firewall nat" menu, create a rule that does "dst-nat" to the web server whenever the client's dst-port is 80 (i.e. HTTP's default port). If you want to match HTTP content on an arbitrary port, you'd have to set up a layer7 match, but that will make the connection even more drastically slow than the transparent proxy already does.

This approach also allows you to potentially alter the site on-the-fly BTW (but if you do that, do it with care, because it's very performance costly).

P.S. In case it's not obvious, you can't capture HTTPS like that... maybe capture the encrypted traffic, but there's little use for that.

Arpanet · Sat Sep 21, 2013 8:17 pm

If you wanted to just look at the current traffic, and the IP of the web server the clients are visiting, you can just use "/tool torch".

If you want to see the contents and store (some of?) it, it's a little more complicated. I'd go for setting up a web server as a "transparent proxy":

Set up a web server with a server language such as PHP. Then make it make a request to whatever the HTTP request dictates with $_SERVER['HTTP_HOST'], $_SERVER['REQUEST_URI'] and so on, and output whatever the remote site outputs, with headers and everything. You can then store the request and/or response on the web server. At the router, in the "/ip firewall nat" menu, create a rule that does "dst-nat" to the web server whenever the client's dst-port is 80 (i.e. HTTP's default port). If you want to match HTTP content on an arbitrary port, you'd have to set up a layer7 match, but that will make the connection even more drastically slow than the transparent proxy already does.

This approach also allows you to potentially alter the site on-the-fly BTW (but if you do that, do it with care, because it's very performance costly).

P.S. In case it's not obvious, you can't capture HTTPS like that... maybe capture the encrypted traffic, but there's little use for that.

Uhm you explained everything very clearly.
As I see, it's very performance costly.
And if I set up an external webserver and then install on it a network protocol analyzer? Without compromising CPE performance?

thanks!

Sat Sep 21, 2013 8:56 pm

If the protocol analyzer has an option to store per-connection data, sure. Otherwise, you may need to create a custom one.

To make sure that you don't miss any traffic, you should connect the router and web server on the same switch, on the side of the local network. With this approach, you wouldn't be able to alter or deny any incoming/outgoing data, though from what I can see, you don't want to do that anyway.

Arpanet · Sat Sep 21, 2013 9:08 pm

yes, it can store per-ip address data.

you suggest to put webserver into the same LAN of CPEs?For avoid altering in/out data?

Sat Sep 21, 2013 9:14 pm

you suggest to put webserver into the same LAN of CPEs?

If you're going to use a protocol analyzer, it MUST be in the same LAN as the RouterOS router, yes.

If you place it in the same LAN as your public facing interface, you'll capture traffic that has already been processed by the router, and you wouldn't be able to tell MAC addresses of CPEs (and if your network is NAT-ted, you wouldn't be able to tell apart IPs either).

If you instead place it in the same LAN as your private facing interface (where the CPEs are), you'd be able to capture all traffic - allowed or merely attempted - as well as the router's response to it, which means a more complete data set. You'd also be able to store MACs too (and if the network is NAT-ted, keep private IPs too).

It doesn't "need" to be on the same switch, but depending on topology, and whether any of your switches are "smart", not connecting them on the same switch might result in some connections being missed.

For avoid altering in/out data?

With a protocol analyzer, you can't alter in/out data, even if you wanted to.

Arpanet · Sat Sep 21, 2013 9:53 pm

If you place it in the same LAN as your public facing interface, you'll capture traffic that has already been processed by the router, and you wouldn't be able to tell MAC addresses of CPEs (and if your network is NAT-ted, you wouldn't be able to tell apart IPs either).

If you instead place it in the same LAN as your private facing interface (where the CPEs are), you'd be able to capture all traffic - allowed or merely attempted - as well as the router's response to it, which means a more complete data set. You'd also be able to store MACs too (and if the network is NAT-ted, keep private IPs too).

It doesn't "need" to be on the same switch, but depending on topology, and whether any of your switches are "smart", not connecting them on the same switch might result in some connections being missed.

Tell me if if my understanding is correct, I need one webserver for each CPE/client?

thanks boen_robot

Sat Sep 21, 2013 10:09 pm

No.

Just one server that's "behind" your router, on EQUAL ground with all CPEs, like this:

SnifferTopology.png

Arpanet · Tue Sep 24, 2013 9:11 pm

this is my actual and simplified netword map:

CelticComms · Tue Sep 24, 2013 9:29 pm

To sniff traffic you would need a switch which can mirror traffic from the monitored port over to the port that the analyser is attached to.

Arpanet · Wed Sep 25, 2013 11:35 pm

What do you mean for "mirror"?

Sent from my iPhone using Tapatalk - now Free

Thu Sep 26, 2013 1:01 pm

What do you mean for "mirror"?

Basically, use a switch which is NOT a "smart" one. If you want to be 100% sure, use a "Hub" instead of "Switch", since a "Hub" is by design always "mirror"-ing traffic.

To "mirror" basically means to send the traffic to all of the switch's ports, as opposed to just the designated one. Many switches maintain a list of MAC addresses and a corresponding port the MAC is supposed to respond from. When a packet is intended for one already in the list, they send it only over its port, and not on other ones. That is, unless the device doesn't respond in a certain time interval, at which point they send it over all ports again. This is good for performance, bad for capturing traffic.

Some smart switches have the ability (via a web interface or whatever) to turn off this feature for a particular MAC address. If you can't find a hub, your best bet is one such switch, on which you can blacklist your router's private facing interface, so that messages intended for the router always reach the server also.

As far as your topology goes - attach it to the "switch" in your diagram.

Arpanet · Tue Apr 29, 2014 1:05 pm

so if I use a smart switch and place it into "switch" place on my network I could monitor and view all my clients CPEs traffic?
I need just this?Nothing else? Something like wireshark or similar?
what mikrotik switch can I buy?

arnoldmikro · Tue Apr 29, 2014 4:10 pm

Rb260gs has port mirror. You will need wireshark an filter port 80 on the capture.

Arpanet · Tue Apr 29, 2014 6:36 pm

is it better an ethernet hub?
how can I search for a mirror switch? What's specific name of mirroring in product specs?

Tue Apr 29, 2014 7:03 pm

Yes, as noted, a hub or a smart switch with mirroring is all you need... in terms of extra hardware. In terms of software, Wireshark or similar, to actually capture the incoming traffic for later analysis.

"Port Mirroring" seems to be the feature you need. It is supported by SwOS, and all MikroTik smart switches by extension.

If you look at SwOS' manual page, you'll notice in the "Forwarding" tab, the "Port Mirroring" section. You must simply set the mirror target on the port where your web server is connected, and enable mirroring from all other ports, so that their traffic will always reach the web server, even if the traffic is not intended for it.

Arpanet · Tue Apr 29, 2014 8:00 pm

yeah, now it's all clear.
Can I save data being captured?

CelticComms · Wed Apr 30, 2014 3:38 pm

Port mirroring is the only practical way to go since true hubs are difficult to find and generally are too slow for the Ethernet speeds that are currently in use. In many devices offering port mirroring you may be limited in the throughput that can be monitored. e.g. don't expect to monitor a saturated 1 Gbps connection in both directions since that would be 2 Gbps traffic and the mirror port can only receive 1 Gbps....

You should also check that what you are proposing is legal in your jurisdiction.

Arpanet · Wed Apr 30, 2014 4:08 pm

mirroring will work also into a NAT network?

Arpanet · Sun May 04, 2014 7:28 pm

mirroring is separate from ip configuration?

Sun May 04, 2014 7:53 pm

mirroring will work also into a NAT network?

mirroring is separate from ip configuration?

Yes. Port mirroring works on OSI layer 2 - the same level as switches work in general - while IP and NAT happen on layer 3 (i.e. in routers in general). As long as the smart switch is within your private network, it will appear to work transparently in it, the same way as a regular switch. Users wouldn't even know or be able to detect they're being monitored (you sneaky bastard

!).

Arpanet · Sat May 10, 2014 7:36 pm

does it work with multiple gateways/modem?

should I connect each modem to a separate switch lan port for mirroring?

Sat May 10, 2014 8:23 pm

You should connect a smart switch to every one of YOUR routers' LOCAL ends (those connected directly connected to the CPEs), not the upstream ends.

If you have a single router with multiple gateways/modems, and one interface from which all of your users are connected, you only need one smart switch, that you plug right before the last cable that goes into your router's local facing interface.

Arpanet · Sat May 10, 2014 8:34 pm

Omg! Does it mean I need one webserver (storing data) for each modem/smart switch?
what about cpe remote logging (topic=webproxy) to remote server? (I'm considering this option)

Sat May 10, 2014 8:46 pm

What?!? For each modem?!? No! The very opposite!

See the below topology:

TrafficCaptureTopology.png

And if you have multiple RouterOS routers (each connected to a separate modem), you can connect all smart switches to a single smart switch, from which instead of using port mirroring, you simply make a rule by which traffic from all other ports will unconditionally go to the web server's port.

e.g.:

TrafficCaptureTopologyExtended.png

Arpanet · Sat May 10, 2014 8:56 pm

I don't want load balancing between two isp's

Sat May 10, 2014 11:04 pm

I don't want load balancing between two isp's

Who's talking about any sort of load balancing?

Arpanet · Sat May 10, 2014 11:23 pm

I'm sorry. I meant I don't want to change actual network topology.
By now each Ap is connected directly to separate ISP modem

Sat May 10, 2014 11:29 pm

I'm sorry. I meant I don't want to change actual network topology.
By now each Ap is connected directly to separate ISP modem

Well, in that case, you can use the second topology above... though that does indeed mean replacing almost all switches with smart switches (so yeah, not exactly cheap, though cheaper and easier to pull off than the kind of rework the first topology would require). All of your CPE connected switches (in the above picture, labeled SwOS_1 and SwOS_2) should end up reaching, through their mirror port, a single smart switch from which all ports lead only to the web server.

Arpanet · Sat May 10, 2014 11:37 pm

In second topology why did you place a routeros after each ISP modem?
So for dual wan setups I need 3 smart switch right?

Sun May 11, 2014 12:04 am

In second topology why did you place a routeros after each ISP modem?
So for dual wan setups I need 3 smart switch right?

Wait, wait... How many non-CPE routers do you have that are under your control? And how many ISP modems are there connected to how many of them? How many of them do NAT?

When you said

By now each Ap is connected directly to separate ISP modem

I assumed you mean CPE connecting to AP, connecting to a switch to RouterOS to the ISP modem (similarly to your very first picture in this topic), with each AP going over a separate entire chain... i.e. I assumed you have X number of APs, X switches, X RouterOS, and X ISP modems... but apparently, that's not the case, so what is it?

Arpanet · Sun May 11, 2014 12:14 am

yes, @boenrobot
my network topology is as your second topology picture, I guess in first topology load-balancing is a must

anyway:
isp1=modem1->ap1->cpe's->client network
isp2=modem2->ap2->cpe's->client network
so only one difference with your pic, there's no mikrotik router between isp modem and ap.
with natted network I mean each node ap and cpe do NAT so they are in different lan's

Arpanet · Sun May 11, 2014 12:54 am

do you think this kind of logging is better or worse than syslog remote logging?

reinerotto · Sun May 11, 2014 2:26 am

Simplest solution for logging all stuff would be the use of squid as an upstream proxy. will also solve your question regarding "redirection" with "url_rewrite" or "session_control".

Arpanet · Sun May 11, 2014 4:32 am

You're talking about users redirection?
I always thought to use squid as caching server...so it could solve traffic logging & saving I guess

rextended · Sun May 11, 2014 5:23 am

Occhio che rischi il PENALE.
Informati da un avvocato, pensa se un tuo utente sta legendo questa discussione dove dici che raccogli i siti visitati da i tuoi utenti...

In Italia è assolutamente vietata la raccolta delle informazioni da parte degli ISP dei siti visitati dagli utenti.

Puoi (anzi devi) per legge fare solo i log delle connessioni tipo:
utente tizio, protocollo TCP, 12.2.3.4:6487 -> 65.12.45.22:80 11/05/2014 02:19:27 GMT +00:00

e i primi sei mesi devono essere custoditi da una persona, i successivi sei da un'altra,
e neanche il provider può controllarli ma solo la polizia su richiesta della questura...

**************************************************************

In Italy it is absolutely prohibited the collection of information from the sites visited by users.

You can (indeed must) to make law only logs the connection type:
user john, protocol TCP, 12.2.3.4:6487 -> 65.12.45.22:80 11/05/2014 02:19:27 GMT +00:00

reinerotto · Sun May 11, 2014 11:03 am

You're talking about users redirection?
I always thought to use squid as caching server...so it could solve traffic logging & saving I guess

First of all caching server, that is correct. But a lot of features besides, like configurable details of traffic logging, and various options to allow redirection. I.E. in case, user wants to visit a blocked site (blocked using a blocklist, filtering module for parental control etc.), to redirect to special URL displaying error msg. Or redirecting to splash page in case of a hotspot.
Ask me for details using email from my profile, as this is not really a MT issue.

Arpanet · Wed May 14, 2014 1:18 pm

What?!? For each modem?!? No! The very opposite!

See the below topology:
TrafficCaptureTopology.png
And if you have multiple RouterOS routers (each connected to a separate modem), you can connect all smart switches to a single smart switch, from which instead of using port mirroring, you simply make a rule by which traffic from all other ports will unconditionally go to the web server's port.

e.g.:
TrafficCaptureTopologyExtended.png

@boen_robot
which software did you use to create that map? thanks

Wed May 14, 2014 1:45 pm

@boen_robot
which software did you use to create that map? thanks

GNS3 - It's a great network simulator I recently learned about through this forum.

yes, @boenrobot
my network topology is as your second topology picture, I guess in first topology load-balancing is a must
anyway:
isp1=modem1->ap1->cpe's->client network
isp2=modem2->ap2->cpe's->client network
so only one difference with your pic, there's no mikrotik router between isp modem and ap.
with natted network I mean each node ap and cpe do NAT so they are in different lan's

So... the only MikroTik devices in your topology are the APs then, is that it?

Oh... since they're doing NAT, I don't think you'd be able to differentiate traffic from one CPE from traffic of another CPE. You'd only be able to differentiate traffic from one AP from traffic of another AP.

Arpanet · Wed May 14, 2014 2:07 pm

it requires cisco ios?

Wed May 14, 2014 2:15 pm

it requires cisco ios?

It can run MikroTik via Quemu or VirtualBox. For Cisco IOS, it supports specialized emulators that are more resource efficient. i.e. simulating a topology with hundreds of Cisco routers will consume less CPU and memory than the same number of RouterOS routers. It's important to note that with just ~3 routers or so, there isn't much of a difference, so for small scale "check it out, so that you get a feel for it" kind of scenarios, it all works just as well.

hakhunamathata · Sat Dec 10, 2016 7:44 am

Hello everybody, I hope you all doing well. I have a question to ask. I am using Mikro Tik Model - RB1100AHx2 , OS Level is 6 . How do I save or monitor my client's browsing history by a remote computer? I have already downloaded MT_Syslog software on my pc. Waiting for a suitable answer. Thanks in advance.

LEM1S · Wed Mar 28, 2018 4:48 am

If you wanted to just look at the current traffic, and the IP of the web server the clients are visiting, you can just use "/tool torch".

If you want to see the contents and store (some of?) it, it's a little more complicated. I'd go for setting up a web server as a "transparent proxy":

Set up a web server with a server language such as PHP. Then make it make a request to whatever the HTTP request dictates with $_SERVER['HTTP_HOST'], $_SERVER['REQUEST_URI'] and so on, and output whatever the remote site outputs, with headers and everything. You can then store the request and/or response on the web server. At the router, in the "/ip firewall nat" menu, create a rule that does "dst-nat" to the web server whenever the client's dst-port is 80 (i.e. HTTP's default port). If you want to match HTTP content on an arbitrary port, you'd have to set up a layer7 match, but that will make the connection even more drastically slow than the transparent proxy already does.

This approach also allows you to potentially alter the site on-the-fly BTW (but if you do that, do it with care, because it's very performance costly).

P.S. In case it's not obvious, you can't capture HTTPS like that... maybe capture the encrypted traffic, but there's little use for that.

Sorry,but can you explain how can i capture https??Thanks in advance

mkx · Wed Mar 28, 2018 10:48 am

Sorry,but can you explain how can i capture https??Thanks in advance

.
While you can capture any kind of traffic, including encrypted one (https or VPN or ...), it is extremely hard do decrypt it to see the contents of communication. And that's by design, encryption is there exactly for this reason.

LEM1S · Wed Mar 28, 2018 1:00 pm

Sorry,but can you explain how can i capture https??Thanks in advance
.
While you can capture any kind of traffic, including encrypted one (https or VPN or ...), it is extremely hard do decrypt it to see the contents of communication. And that's by design, encryption is there exactly for this reason.

if i have the website certificate,is possible to decrypt https?

mkx · Wed Mar 28, 2018 2:07 pm

With certificate you probably can decrypt server's data. For decrypting client's data you'd probably need server's private key. If the https server is yours, then you surely have access to private key. For other servers it should be impossible to get hold on private key, privacy of private key is the paramount of security of asymmetric encryption (used by number of protocols, including SSL/TLS).

Who is online