Hi Steinar, Thanks for getting back to this issue. The Github release is fixed now and behaves the same as 1.6.16 - so you should see again correct flow data. As of your timestamps and the pcaps you provided in https://sourceforge.net/p/nfdump/mailman/message/35481221/:
The v9 flows are decoded correctly - in 1.6.16 as well as with the current Github release. IPFIX is different and can not be decoded correctly with 1.6.16 as well as the current GitHub release, so you will still see 1970-1-1T00:00:00. So why? The v9 template as well as the ipfix templates announce the timestamp tags #21,#22. It means these are relative ms timestamps since the exporting device booted last. In v9 this boot time of the device can be found in the v9 header of each exported data record. So the absolute timestamps can be calculated. Here IPFIX differs. The timestamp in the header of the data record is the export time of the data flowset. The boot time in IPFIX should be announced by the tag #160. However, this element is missing in the pcap of your exporter. Therefore the absolute first/last timestamps can not be calculated. What you see in wireshark is a delta time. If you are still running VRP 8.120 V800R008C10SPC300, then this has not changed. If you upgraded to a newer release, this could have changed. If you could provide me a new pcap with IPFIX data records, I could check this. To make a long story short: You need to open a ticket at the vendor to fix this at the exporter side. Possible solutions: 1. If they want to stay with the tags #21, #22, they need to send also a tag #160. 2. If they want to base timestamps on the export header time, we would need tags #158,#159. 3. if they want absolute timestamps, they should announce #152, #153 https://www.iana.org/assignments/ipfix/ipfix.xhtml In IPFIX you have (too) many options to export a timestamp, but it needs to be consistent. I guess the vendor simply wanted to copy v9 to IPFIX. So far nfdump supports the options 2. and 3. Option 1 is ready as soon as I know which way the exporter sends #160. Hope this helps. - Peter On 30.12.17 22:15, sth...@nethelp.no wrote: >>> The current master on Github doesn't include a configure file or the >>> Makefile.in files. I'm having a bit of trouble running the autoconf >>> tools to produce this. Is there a version available (similar to version >>> 1.6.16) which includes these files - thus I should only have to run >>> ./configure to get started? >> >> I submitted the patch to remove ./configure. The idea being that for >> development >> one would always have the required autotools available (run ./bootstrap). >> Peter should add a source tarball (created with "make dist") to releases so >> it58/#159 >> should be available in a release. > > Thank you, this made it possible to get further. Unfortunately the > result was not a great success - it looks like there is some offset > mismatch between the data produced by nfcapd, and the result of > using nfdump. > > I switched from the 1.6.16 nfcapd to the github nfcapd at 21:49 this > evening. > > This is a typical example of 1.6.16 nfcapd and nfdump (-r > 2017/12/30/21/nfcapd.201712302140) > showing the same time stamp problem as before, but other fields are > reasonable: > > Date first seen Duration Proto Src IP Addr:Port Dst IP > Addr:Port Packets Bytes Flows > 1970-01-01 01:00:00.000 0.000 UDP 213.138.160.70:53 -> > 208.91.112.52:26218 1 166 1 > 1970-01-01 01:00:00.000 0.000 TCP 91.135.34.26:443 -> > 213.138.169.192:54447 12 18168 1 > 1970-01-01 01:00:00.000 0.000 TCP 213.138.174.158:58485 -> > 148.251.64.174:80 2 120 1 > 1970-01-01 01:00:00.000 0.000 TCP 193.90.147.79:443 -> > 213.138.168.174:50421 37 56018 1 > 1970-01-01 01:00:00.000 0.000 TCP 172.217.18.142:443 -> > 213.138.177.40:38406 1 66 1 > > This is what I get after building the github version - both nfcapd and > nfdump are the github version (-r 2017/12/30/21/nfcapd.201712302155): > > Date first seen Duration Proto Src IP Addr:Port Dst IP > Addr:Port Packets Bytes Flows > 1970-01-01 01:00:00.000 0.000 0 0.0.0.1:46931 -> > 0.0.0.0:54666 292 4647080.3 T 1 > 1970-01-01 01:00:00.000 0.000 0 0.0.0.2:40996 -> > 0.0.0.0:54666 126 2271150.0 T 1 > 1970-01-01 01:00:00.000 0.000 0 0.0.0.16:21024 -> > 0.0.0.0:54580 23424 13753711.8 T 1 > 1970-01-01 01:00:00.000 0.000 0 0.0.0.1:46247 -> > 0.0.0.0:54666 130 3039013.7 T 1 > 1970-01-01 01:00:00.000 0.000 0 0.0.0.54:33301 -> > 0.0.0.0:38010 81756 13753720.9 T 1 > > The problem seems to be nfcapd - if I use the new (github) version of > nfdump with an older nfcapd file (-r 2017/12/30/21/nfcapd.201712302140 > as above) it works as before (time stamps are still wrong but other > fields are okay): > > Date first seen Duration Proto Src IP Addr:Port Dst IP > Addr:Port Packets Bytes Flows > 1970-01-01 01:00:00.000 0.000 UDP 213.138.160.70:53 -> > 208.91.112.52:26218 1 166 1 > 1970-01-01 01:00:00.000 0.000 TCP 91.135.34.26:443 -> > 213.138.169.192:54447 12 18168 1 > 1970-01-01 01:00:00.000 0.000 TCP 213.138.174.158:58485 -> > 148.251.64.174:80 2 120 1 > 1970-01-01 01:00:00.000 0.000 TCP 193.90.147.79:443 -> > 213.138.168.174:50421 37 56018 1 > 1970-01-01 01:00:00.000 0.000 TCP 172.217.18.142:443 -> > 213.138.177.40:38406 1 66 1 > > Suggestions on how to debug this? > > Steinar Haug, AS2116 > > ------------------------------------------------------------------------------ > Check out the vibrant tech community on one of the world's most > engaging tech sites, Slashdot.org! http://sdm.link/slashdot > _______________________________________________ > Nfdump-discuss mailing list > Nfdump-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfdump-discuss > -- Be nice to your netflow data. Use NfSen and nfdump :) ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Nfdump-discuss mailing list Nfdump-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
