Bandwidth usage per IP

anserk · Fri May 13, 2022 1:14 am

Recently I started to get close to reaching my ISP data cap, so I wanted to track monthly bandwidth usage per each device behind my home router hAP ac2. I searched the forum and saw several ideas, but they weren't quite what I was looking for. So I wrote my own script that I would like to share with the community.

Key features

Everything is local on the router. You don't need any external servers or software to collect the data. You only need a computer to view the report.
The script uses kid-control functionality in RouterOS, which has a nice bonus - tracking per IP bandwidth.
I'm using it with a USB flash drive where I don't worry about write cycles. If your device doesn't support external storage, you could schedule the script to run less frequently (at the expense of more collected data loss due to unexpected reboots).

Limitations

Tested with RouterOS 7 only.
Kid-control counts all inter-VLAN traffic, not just WAN. Potentially you could re-write the script and use something like viewtopic.php?t=168427 instead of kid-control.
The HTML report (placed in the root of flash drive) has .txt extension - this is RouterOS limitation. You need to download the report from WinBox Files window and then remove .txt so that only .html is left. You could set up SFTP or FTP and use the fetch tool to rename it or upload/rename to another location on your network. There are some discussions about using the branding package for serving custom static web pages directly from the router. I haven't looked into that.
RouterOS supports only integers, so report accuracy is +/-1GB. If you need more precision, you can modify the script to report in megabytes. Note, all calculation and saving is done in bytes, so there are no repeated rounding losses. The rounding (or rather truncating) happens only at the last step when the report is generated.

How to use

Enable bandwidth tracking in kid-control. With configuration below kid-control doesn't limit anything but starts tracking traffic.
Code: Select all
```
/ip kid-control add name=Monitor mon=0s-1d tue=0s-1d wed=0s-1d thu=0s-1d fri=0s-1d sat=0s-1d sun=0s-1d
```
Install and format USB drive.
Create a directory called "bandwidth" on it (the easiest way is to create a folder on your PC and drag into Winbox Files window). You can change the paths as needed by changing the script variables.
Schedule the script to run every hour (or however often you like) at x:59:59. On the last day of the month at 23:59:59 the script will generate the final report for the month and won't touch it again. The next scheduled run will use the new month name for data files and the report.

Here is how the report looks like:

report.png

Edit: uploaded the revised script to address the new scripting date format and USB path.

Fri May 13, 2022 4:11 am

very interesting, thank you for sharing

Jotne · Mon May 16, 2022 8:22 am

5. How to read report?
can you do? http://your.router.ip/xxxx

rextended · Mon May 16, 2022 10:36 am

I have already seen that line on Splunk...

anav · Mon May 16, 2022 3:00 pm

Should be built into the router as a selectable function!!

anserk · Mon May 16, 2022 6:23 pm

You read the report by downloading it to your computer and removing .txt, so that the resulting file is report-2022may.html (for May). Then you can open it in any browser. It's a basic HTML with JavaScript leveraging Google Charts.

You are correct, the script doesn't do any magic and fully relies on kid-control to provide the data. The main point is storing the data persistently without anything external. And also presenting the data in a friendly format. It is especially useful for users who don't have any servers/Raspberry PI's/NAS/etc running somewhere to collect the data. It's very simple to set up, just copy and paste the script.

can you do? http://your.router.ip/xxxx

Ideally, if RouterOS had the ability to serve custom static web pages, then it would be possible. Like I mentioned above, the branding package might do this but I haven't looked into that.

Sat Jul 23, 2022 11:52 am

@anserk,
just a little shout out that I am using your script on an SXT device used for internet access in a vacation home setup.

MUCH prettier then simply mailing output from kid-control devices print stats

bpwl · Sat Jul 23, 2022 2:39 pm

VERY interesting feature of kid-control. Replacing "IP accounting" ?
TXS for the script!

HOW to get the authenticated user (RADIUS) into this Kid-Control device list ?

(Wifi AP authenticated via WPA2/Enterprise, Username value can be copied into Radius "MT Comment" attribute with User Manager 5, and remains visible in the Registration table, as long as the device is connected. DUDE collects in "RouterOS Info" all registration tables, including the comment field)

The username/password is stored in the client device, and is never entered again for the same SSID. So this should be acting as automated RSSO (Radius single signon).
But how to make Kid-control or Hotspot pick up this signon ?

anserk · Sat Jul 23, 2022 9:45 pm

If you downloaded the script before July 23 2022, please update line 35 from:

  :local ip [get $device ip-address]
# To
  :local ip [:tostr [get $device ip-address]]

I recently ran into issue with traffic counter overflow on the report, messing up all statistics. It turns out when a device has multiple IP addresses, the script was grabbing wrong fields for traffic counters. The issue is resolved by forcing the whole IP field to a string.

@bpwl I've never used RADIUS - wouldn't that authenticated user device get an IP from DHCP server? I don't know exactly how kid-control works under the hood, it seems to simply pick up devices by MAC address. Presumably anything that goes through the firewall, which is why it also counts traffic from one local VLAN to another. So as long as that user sends some traffic, it should be on the list. The difficult part is identifying who is who. If the device doesn't register a name with DHCP server, then only MAC and IP are shown. And with many phones (like Apple) periodically changing their MAC address, it becomes impossible to identify all devices accurately. So kid-control is quite a limited feature, but works well in simple configurations with mostly fixed number of devices.

bpwl · Sun Jul 24, 2022 12:29 am

The difficult part is identifying who is who.

That's why I use RADIUS with EAP/PEAP wifi authentication (= username/password).
The data is stored once in the device (until you 'forget" the wifi network in the known wifi network list.)

Limits are then per user. Hotspot login does the same but with many APP as primary use, the browser popup for logging in is sometimes missed. And Hotspot cookies are just a means for avoiding repeated login actions for the same MAC.

Whatever they do with the private MAC addresses, the user identification with EAP/PEAP is dependent on the stored login credentials only.
RADIUS accounting would collect the usage per user, but that is mostly all (wifi) traffic, and the interesting part is the WAN usage only.
RADIUS could give a specific range of IP addresses to one user. But when you have multiple network segments (eg VLAN) then it becomes more difficult.
RADIUS could also force a specific VLAN for a user.
RADIUS can be used for VPN, DHCP, Hotspot, wifi, and other services. But afaik only the wifi is persistent. (The device connects with that user without any interaction of the user, even when not around for more than a year, and when it comes with a different private MAC address.)

Sun Jul 24, 2022 12:32 am

I would think if the same sort of info can be retrieved from the Radius part (incl. user name), the concept of the script can still be used.

bpwl · Mon Jul 25, 2022 11:53 am

RADIUS itself is a bid special in this. Is some device authenticated to what user or IP address? Is not easy to tell.

RADIUS authentication is an event, that in the best case gets logged, username and MAC address of the AP. (not the client), in my case.
De-authentication in de most cases does not involve the RADIUS server. (There is CoA, but mostlly the wifi disconnects, "reason=3" by AP, "reason=8" by roaming client, extensive data loss (client is gone or out of range?)) So the RADIUS authenticator cannot tell if a user is still authenticated or not, and does not have the IP address of the client.
The AP maintains the registration (on MAC address) but does not register the user used for the authentication.

The way RADIUS knows about the disconnect is through the RADIUS accounting. (User name , IP address, MAC address, quota consumed). But RADIUS misses sometimes an accounting record.
Actually I'm using the old MT User Manager for collecting RADIUS accounting, even with another RADIUS server doing the authentication.
Reason is that MT limits the number of accounted sessions to 19, for a L4 license. Accounting updates is set to someting like every 10 minutes.
RADIUS authentication is quite central, for multiple AP, so the number of active sessions is likely above 19. Even L5 only allows for 50 sessions.
I have 10.000 RADIUS authentications per day, caused by 40 named users, 100 client devices on 34 MT AP. There is a lot of roaming in that large 1 km2 area, also between 2.4 and 5GHz on the same AP. (Playing kids, indoor/outdoor swap, ...)
Roaming can be every few seconds, and takes a split second. But steady clients their connections remain active for multiple days.
DHCP lease in this does not interfere (roaming does not renew the lease). Lease time is 24h, most client devices keep the same IP for the whole holiday stay period.

All data is there, but not correlated. DUDE collects the logs from (non-MT) RADIUS server, (non-MT) DHCP server, and MT AP's.
MT User manager only can follow 19 Active sessions with license level L4, via RADIUS accounting.

anserk · Fri Aug 12, 2022 11:00 pm

I encountered another bug with the script. One device that showed up in kid-control didn't have an IP address. That empty field was screwing up the report. I added the fix to the attached script above. It's just one new line.

  :if ([:len $name] = 0) do={ :set $name "unknown" }
  :local ip [:tostr [get $device ip-address]]
  :if ([:len $ip] = 0) do={ :set $ip "no-ip" }   <== new line
  :local down [get $device bytes-down]
  :local up [get $device bytes-up]

Dt17 · Tue Jan 03, 2023 8:54 am

Hi, your script works great and i really appreciate this tool that you provided

I seemed to have messed up some of the older reports by moving them before the last day of the month

Is there anyway to generate the report with all the older raw data? (in Txt format and also still viewable in Kid Control)

i am not familiar with Google Charts

anserk · Wed Jan 04, 2023 10:48 pm

Is there anyway to generate the report with all the older raw data? (in Txt format and also still viewable in Kid Control)

The script uses the current date to produce the txt filenames, which are then used for generating the report. You can make a copy of the script and edit the line:

:local sysdate [/system/clock/get date]

Change it to a static date for older month. The format is jan/04/2023.

As far as Kid Control goes, that data gets reset every time the script runs. So what you see there is only new data since the last run.

Dt17 · Thu Jan 05, 2023 9:11 am

Thank you, Appreciate your expertise and assistance!

Rox169 · Thu Jan 05, 2023 10:11 am

MIKROTIK PLEASE PLEASE INTEGTRATE THIS!!!!!!!! OR GIVE AS BACK IP ACCOUNTING!!!!!!! PLEASE PLEASE

Dt17 · Mon Feb 06, 2023 9:46 am

So this works for me out of 3 of 4 Mikrotik(2 x RB3011 and 2 x RB1100AHx4) devices that i currently run them

The 4th one(RB1100AHx4) keeps giving me this error for the last 3 months

The only result generated in the html is below

Does anyone has any idea what is the error? All the boxes were running ROS 7.6 and now ROS 7.7

<html><head>
<script type='text/javascript' src='https://www.gstatic.com/charts/loader.js'></script>
<script type='text/javascript'>
google.charts.load('current', {packages: ['corechart', 'bar']});
google.charts.setOnLoadCallback(drawMultSeries);

function drawMultSeries() {
var data = google.visualization.arrayToDataTable([
['Device', 'Download', 'Upload', 'Total'],
 
failure: new contents too long

anserk · Thu Feb 16, 2023 8:51 pm

That error

failure: new contents too long

seems to imply you are running out of space somewhere.

sebus46 · Thu Jun 22, 2023 8:27 pm

Pity RB4011 does not have USB...

Jotne · Thu Jun 22, 2023 9:33 pm

Without USB you can send information to an external server using Syslog

bingtonewali · Fri Jul 28, 2023 9:40 am

i tried to run on SXTsq Lite5 AC [6.48.6 (long-term) ]. i tried but i failed to setup this. I don't where i am getting stuck. No report generated. Can you please provide a video tutorial how to set it up?
1st i do turn on kid control through you give the code [ /ip kid-control add name=Monitor mon=0s-1d tue=0s-1d wed=0s-1d thu=0s-1d fri=0s-1d sat=0s-1d sun=0s-1d ]
and then
2nd i go to script and then paste the whole code and run interval of 00:59:59 .

nothing getting output. i change the directory according to me. i don't have external storage so i create a folder on Mikrotik and then i choose the path.

Troubleshoot by me:
I tried to run in scheduler not in script nothing happen
i tried to paste the whole code in script and then run it through schulder.
i tried to change the interval runtime to 10 sec to 1 min and 5min nothing results.

Please make a video tutorial and the installation. Please. I need this very badly to track my own internet ussage without any external thing.

Jotne · Fri Jul 28, 2023 12:25 pm

You can follow the steps to setup monitoring here:

viewtopic.php?t=179960

cmassey · Mon Aug 07, 2023 9:26 pm

Kid-control doesn't seem to give accurate bandwidth usage information. I'm using 7.10 on an hap ac2 as a test. When I run a speedtest on my computer, I get about 100Mbps down and about 45Mbps up. Kid control is showing about 30Mbps down and kbps up.

Is anyone else having the same issue? Does it seem to work with applications other than speedtest? It might just be showing one of the speedtest streams which would be a little unlucky.

I am currently using torch which gives accurate information but it would be nice to use kidcontrol instead so I don't need to add up all the connections for the same IPs.

Gerry123 · Wed Mar 27, 2024 3:36 pm

Thank you for your work, it works perfect.
When I try to save the report to a shared drive (runs on SMB raspberrypi) using
:local reportpath ("smb://user:password@192.168.3.19/home/pi/MyNASA/BWbyIP/report-" . $yearmonth . ".html")
either with or without the user/password

failure: cannot open output file

Is it my smb syntax or elsewhere ? Thanks.

anserk · Sat Mar 30, 2024 6:47 pm

i tried to run on SXTsq Lite5 AC [6.48.6 (long-term) ]. i tried but i failed to setup this. I don't where i am getting stuck. No report generated. Can you please provide a video tutorial how to set it up?

The script was developed and tested only for RouterOS 7.

anserk · Sat Mar 30, 2024 7:01 pm

Kid-control doesn't seem to give accurate bandwidth usage information. I'm using 7.10 on an hap ac2 as a test. When I run a speedtest on my computer, I get about 100Mbps down and about 45Mbps up. Kid control is showing about 30Mbps down and kbps up.

Is anyone else having the same issue? Does it seem to work with applications other than speedtest? It might just be showing one of the speedtest streams which would be a little unlucky.

I am currently using torch which gives accurate information but it would be nice to use kidcontrol instead so I don't need to add up all the connections for the same IPs.

I don't think kid control is a good tool for watching live traffic utilization, it's probably averaging out the numbers. However, the count is accurate to the best of my knowledge.
If you want to watch live traffic, I suggest using interface graphs or just interfaces window.

anserk · Sat Mar 30, 2024 7:18 pm

Thank you for your work, it works perfect.
When I try to save the report to a shared drive (runs on SMB raspberrypi) using
:local reportpath ("smb://user:password@192.168.3.19/home/pi/MyNASA/BWbyIP/report-" . $yearmonth . ".html")
either with or without the user/password

failure: cannot open output file

Is it my smb syntax or elsewhere ? Thanks.

RouterOS cannot use SMB shares as a client, unfortunately. There is the fetch tool that you can use to upload via SFTP, but to work with files you still need to keep them locally either in RAM or on USB drive.

jvanhambelgium · Sat Mar 30, 2024 7:47 pm

Thank you for your work, it works perfect.
When I try to save the report to a shared drive (runs on SMB raspberrypi) using
:local reportpath ("smb://user:password@192.168.3.19/home/pi/MyNASA/BWbyIP/report-" . $yearmonth . ".html")
either with or without the user/password

failure: cannot open output file

Is it my smb syntax or elsewhere ? Thanks.
RouterOS cannot use SMB shares as a client, unfortunately. There is the fetch tool that you can use to upload via SFTP, but to work with files you still need to keep them locally either in RAM or on USB drive.

ROSE package ??
You can mount a SMB-share from your NAS and write it on there.

SaeedYa · Sun Jun 30, 2024 1:07 am

Recently I started to get close to reaching my ISP data cap, so I wanted to track monthly bandwidth usage per each device behind my home router hAP ac2. I searched the forum and saw several ideas, but they weren't quite what I was looking for. So I wrote my own script that I would like to share with the community.

I just wanted to thank you for the script! I tried using Splunk and Grafana to get a simple total bandwidth usage per interface, but neither could calculate it out of the box. One suggestion: it would be great if the script could calculate the total bandwidth usage per interface listed as WAN. This would be helpful in scenarios where a WAN link is configured for failover.

Bandwidth usage per IP

Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Re: Bandwidth usage per IP

Who is online