Hi Paolo, Thank you for your explanation - very insightful in how the fields have slightly different meaning depending on where they are used.
However, that does not solve the underlying problem... We have determined that we have rebooted the PaloAlto device 13 days ago. We are seeing timestamp_start values in the file that are from 2015-02-11 which is more than 13 days ago. Before I am digging into this deeper, may I ask what the correct way is to produce the timestart_start value in the json output of a file? Currently I have configured this: aggregate[raw]: label, src_host, dst_host, src_port, dst_port, proto, tcpflags, flows, username, application, timestamp_start print_output_file[raw]: /usr/local/flow/data/traffic-%Y%m%d-%H%M.json print_output[raw]: json The label is configured via pretag, the username and application are configured via primitives to support username/application purpose from the palo alto - BTW - you have the only tool out there to actually let us get to this data - this is a big deal for us - so thank you for your efforts. I have tried to patch nfdump and almost ripped my hair out trying to add new fields... using your primitives and watching some of your published slides gave me the hints... So, is the above aggregate correct in terms of the timestamp_start ? The reason why I would like to have the timestamp is because when we ingest the data via logstash to elasticsearch, I would like it to use that timestamp and not make one up. Would a packet capture and json output file help? Steffen > -----Original Message----- > From: pmacct-discussion [mailto:[email protected]] On > Behalf Of Paolo Lucente > Sent: Thursday, April 02, 2015 4:56 AM > To: [email protected] > Subject: Re: [pmacct-discussion] timestamps in the past > > Hi Steffen, > > Please note that stamp_inserted/stamp_updated is different than > timestamp_start/timestamp_end. The former two, that you find in > the SQL table schema, are populated by enabling sql_history (and > companion settings) and affected by nfacctd_time_new (in other > words: assign flows to time-bins considering the time of arrival > to the collector rather than the flow start time). The equivalent > print_history setting for the print plugin does not populate the > JSON tuples with stamp_inserted/stamp_updated but stamp_inserted, > possibly the most important of the two fields (as the other like > Bill was saying is generated on the fly by MySQL with a NOW()), > can be optionally embedded in the filename, ie.: > > print_output_file: /path/to/spool/blabla-%Y%m%d-%H%M.txt > > timestamp_start/timestamp_end aggregation primitives are the same > in both MySQL and print (and any other) plugins and are not > influenced by the nfacctd_time_new setting. In NetFlow/IPFIX these > two primitives show the flow start/end times respectively (values > are literally taken from NetFlow/IPFIX flow and printed out). > > Hope this explains/helps. > > Cheers, > Paolo > > On Wed, Apr 01, 2015 at 08:30:31PM +0000, Steffen Plotner wrote: > > Hi, > > > > We have a Palo Alto firewall and are trying to use pmacct to collect > its netflow data. I have been able to get everything to work for netflow > v9 type data expect for the timestamps. Most timestamps are current and > some go back one month. The wireshark trace of the netflow data does > show such values in the Timestamp field. When writing out the data to > flat files the time goes back in the past. That same data written to the > mysql server is correct in terms of the timestamp - todays date/time. > > > > I have the following > > nfacctd_time_new: true > > > > which is apparently effective for the mysql side - but it appears not > on the file side... > > > > file output > > {"label": "netflow_fcnet_in_conv", "tcp_flags": "0", "application": > "bittorrent", "ip_src": "144.76.96.199", "port_src": 38914, "ip_dst": > "148.85.185.85", "port_dst": 41986, "timestamp_start": "2015-03-09 > 15:25:43.0", "ip_proto": "udp", "timestamp_end": "2015-03-09 > 15:45:43.0", "username": "", "packets": 4, "bytes": 409, "flows": 1} > > > > Mysql output > > ip_src ip_dst port_src port_dst tcp_flags > ip_proto packets bytes stamp_inserted stamp_updated > country_ip_src country_ip_dst username application > > 144.76.96.199 148.85.185.85 38914 41986 0 udp 4 > 409 4/1/2015 4:25:00 PM 4/1/2015 4:25:15 PM -- -- > bittorrent > > > > Steffen > > > > > > > ________________________________________________________________________ > _______________________ > > Steffen Plotner Amherst College > Tel (413) 542-2348 > > Systems/Network Administrator/Programmer PO BOX 5000 > Fax (413) 542-2626 > > Systems & Networking Amherst, MA 01002-5000 > [email protected] > > > > > > > > > _______________________________________________ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
