dear Paolo, thanks a lot for adding the feature of timestamp_arrival, it will be very useful in the cases when
- the probing router/switch has some bugs of sending out wrong timestamp for flows - the probing router/switch is not well synchronised with the NTP facilities. we will deploy it as soon as possible in our use cases. thanks and regards, maoke 2016-01-15 2:51 GMT+09:00 Paolo Lucente <[email protected]>: > VERSION. > 1.5.3 > > > DESCRIPTION. > pmacct is a small set of passive network monitoring tools to account, > classify, > aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable > architecture > allows to store collected data into memory tables, RDBMS (MySQL, > PostgreSQL, > SQLite), noSQL databases (MongoDB, BerkeleyDB) AMQP (RabbitMQ) and Kafka > message > exchanges and flat-files and export via NetFlow or sFlow protocols to > remote > collectors. pmacct offers customizable historical data breakdown, BGP and > IGP > correlation, BMP stats and events, GeoIP lookups, sampling and > renormalization, > filtering, tagging and triggers. Libpcap, Linux Netlink/ULOG, sFlow > v2/v4/v5, > NetFlow v5/v8/v9 and IPFIX are supported. Replication of incoming NetFlow > and > sFlow datagrams is also available. Statistics can be easily exported to > tools > like ElasticSearch, Cacti RRDtool MRTG, Net-SNMP, GNUPlot, etc. > > > HOMEPAGE. > http://www.pmacct.net/ > > > DOWNLOAD. > http://www.pmacct.net/pmacct-1.5.3.tar.gz > > > CHANGELOG. > + Introduced the Kafka plugin: Apache Kafka is publish-subscribe messaging > rethought as a distributed commit log. Its qualities being: fast, > scalable, > durable and distributed by design. pmacct Kafka plugin is designed to > send aggregated network traffic data, in JSON format, through a Kafka > broker to 3rd party applications. > + Introduced Kafka support to BGP and BMP daemons, in both their msglog > and dump flavors (ie. see [bgp|bmp]_daemon_msglog_kafka_broker_host and > [bgp_table|bmp]_dump_kafka_broker_host and companion config directives). > + Introduced support for a Kafka broker to be used for queueing and data > exchange between Core Process and plugins. plugin_pipe_kafka directive, > along with all other plugin_pipe_kafka_* directives, can be set globally > or apply on a per plugin basis - similarly to what was done for RabbitMQ > (ie. plugin_pipe_amqp). Support is currently restricted only to print > plugin. > + Added a new timestamp_arrival primitive to expose NetFlow/IPFIX records > observation time (ie. arrival at the collector), in addition to flows > start and end times (timestamp_start and timestamp_end respectively). > + plugin_pipe_amqp: feature extended to the plugins missing it: nfprobe, > sfprobe and tee. > + Introduced bgp_table_dump_latest_file: defines the full pathname to > pointer(s) to latest file(s). Update of the latest pointer is done > evaluating files modification time. Many thanks to Juan Camilo Cardona > ( @jccardonar ) for proposing the feature. > + Introduced pmacctd_nonroot config directive to allow to run pmacctd > from a user with non root privileges. This can be desirable on systems > supporting a tool like setcap, ie. 'setcap "cap_net_raw,cap_net_admin=ep" > /path/to/pmacctd', to assign specific system capabilities to unprivileged > users. Patch is courtesy by Laurent Oudot ( @loudot-tehtris ). > + Introduced plugin_pipe_check_core_pid: when enabled (default), validates > the sender of data at the plugin side. Useful when plugin_pipe_amqp or > plugin_pipe_kafka are enabled and hence a broker sits between the daemon > Core Process and the Plugins. > + A new debug_internal_msg config directive to specifically enable debug > of internal messaging between Core process and plugins. > ! bgp_table_dump_refresh_time, bmp_dump_refresh_time: max allowed value > raised to 86400 from 3600. > ! [n|s]facctd_as_new renamed [n|s]facctd_as; improved input checks to all > *_as (ie. nfacctd_as) and *_net (ie. nfacctd_net) config directives. > ! pkt_handlers.c: NF_sampling_rate_handler(), SF_sampling_rate_handler() > now perform a renormalization check at last (instead of at first) so to > report the case of unknown (0) sampling rate. > ! plugin_pipe_amqp_routing_key: default value changed to '$core_proc_name- > $plugin_name-$plugin_type'. Also, increased flexibility for customizing > the key with the use of variables (values computed at startup). > ! Improved amqp_receiver.py example with CL arguments and better exception > handling. Also removed file amqp_receiver_trace.py, example is now merged > in amqp_receiver.py. > ! fix, BGP daemon: several code optimizations and a few starving conditions > fixed. Thanks to Markus Weber ( @FvDxxx ) for his peer index round-robin > patch; thanks also to Job Snijders ( @job ) for his extensive support in > this area. > ! fix, BMP daemon: greatly improved message parsing and segment reassembly; > RabbitMQ broker support found broken; several code optimizations are also > included. > ! fix, bgp_table.c: bgp_table_top(), added input check to prevent crashes > in cases table contains no routes. > ! fix, networks_file: missing atoi() for networks_cache_entries. Patch is > courtesy by Markus Weber ( @FvDxxx ). > ! fix, plugin_pipe_amqp_routing_key: check introduced to prevent multiple > plugins to bind to the same RabbitMQ exchange, routing key combination. > Thanks to Jerred Horsman for reporting the issue. > ! fix, MongoDB plugin: added a custom oid fuzz generator to prevent > concurrent inserts to fail; switched from deprecated mongo_connect() to > mongo_client(); added MONGO_CONTINUE_ON_ERROR flag to mongo_insert_batch > along with more verbose error reporting. Patches are all courtesy by > Russell Heilling ( @xchewtoyx ). > ! fix, nl.c: increments made too early after introduction of MAX_GTP_TRIALS > Affected: pmacctd processing of GTP in releases 1.5.x. Patch is courtesy > by TANAKA Masayuki ( @tanakamasayuki ). > ! fix, pkt_handlers.c: improved case for no SAMPLER_ID, ALU & IPFIX in > NF_sampling_rate_handler() on par with NF_counters_renormalize_handler(). > ! fix, SQL scripts: always use "DROP TABLE IF EXISTS" for both PostgreSQL > and SQLite. Pathes are courtesy by Vincent Bernat ( @vincentbernat ). > ! fix, plugin_hooks.c: if p_amqp_publish_binary() calls were done while a > sleeper thread was launched, a memory corruption was observed. > ! fix, util.c: mkdir() calls in mkdir_multilevel() now default to mode 777 > instead of 700; this allows more play with files_umask (by default 077). > Thanks to Ruben Laban for reporting the issue. > ! fix, BMP daemon: solved a build issue under MacOS X. Path is courtesy by > Junpei YOSHINO ( @junpei-yoshino ). > ! fix, util.c: self-defined Malloc() can allocate more than 4GB of memory; > function is also now renamed pm_malloc(). > ! fix, PostgreSQL plugin: upon purge, call sql_query() only if status of > the entry is SQL_CACHE_COMMITTED. Thanks to Harry Foster ( @harryfoster ) > for his support resolving the issue. > ! fix, building system: link pfring before pcap to prevend failures when > linking. Patch is courtesy by @matthewsf . > ! fix, plugin_common.c: memory leak discovered when pending queries queue > was involved (ie. cases where print_refresh_time > print_history). Thanks > to Edward Henigin for reporting the issue. > > > NOTES. > See UPGRADE file. > > > Cheers, > Paolo > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists >
_______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
