Netflow Timestamps

turnip · Wed Apr 27, 2022 9:19 am

I'm collecting Netflow data using Fluentd and storing it as Parquet files on AWS S3. I'm then using Athena to query these files using SQL and Grafana to visualise the data.
I've tried using Netflow v5, v9 and IPFIX. When I use v5 I can get the timestamps easily:

SELECT flow_seq_num, from_iso8601_timestamp(first_switched) as startTime, from_iso8601_timestamp(last_switched) as endTime FROM "netflow"."netflow5_event";

Example output:
flow_seq_num: 783160
startTime: 2022-04-27 04:59:51.735 UTC
endTime: 2022-04-27 04:59:51.735 UTC

Netflow 9 and IPFIX don't store timestamps - I've read that it shows the time in milliseconds since system boot, but the numbers don't look right and systeminittimemilliseconds is 0 for most entries with IPFIX. I've also tried exporting all three at the same time, hoping I could get the timestamp from v5 and use the rest of the data from v9 or IPFIX, but the flows don't match (I presume RouterOS is processing them separately, not just exporting 3 different outputs for the same data set).

v5 is useful today, but it doesn't support IPv6, so I'd rather use v9 or IPFIX as we're planning on implementing IPv6 soon. Is there any way to get the timing of the flows from v9 or IPFIX?
All devices will be running RouterOS v7, as I need the REST API for other aspects of this project.

savage · Thu Sep 22, 2022 10:43 pm

+1 - really annoying. v5 does not support v6, and v9 does not support time stamps. I can't even remember for how long Netflow has been an issue in 'tik. Would be lovely to just get this fixed once and for all please.

Flow Record: 
  Flags        =              0x06 FLOW, Unsampled
  label        =            <none>
  export sysid =                25
  size         =                64
  first        =                 0 [1970-01-01 02:00:00]
  last         =                 0 [1970-01-01 02:00:00]
  msec_first   =                 0
  msec_last    =                 0
  src addr     =     89.248.165.57
  dst addr     =    102.x.x.231
  src port     =             42945
  dst port     =             30892
  fwd status   =                 0
  tcp flags    =              0x02 ......S.
  proto        =                 6 TCP  
  (src)tos     =                 0
  (in)packets  =                 1
  (in)bytes    =                40
  input        =                 1
  output       =                 2

Netflow Timestamps

Netflow Timestamps

Re: Netflow Timestamps

Who is online