To wrap-up on this. Bug was confirmed, reproduced and fixed. Fix has been also tested working by Ed. Log of the commit is here:
https://github.com/pmacct/pmacct/commit/6d518f4a2b0e808ae89e2b896fa3c0ba2c3fc64b Cheers, Paolo On Thu, Jan 07, 2016 at 11:00:43PM +0000, Paolo Lucente wrote: > Hi Ed, > > The config looks pretty standard so i'm puzzled - I'll try to reproduce > in lab and keep in touch; should that fail, i may ask you to debug myself > on your box. > > Cheers, > Paolo > > On Thu, Jan 07, 2016 at 04:32:43PM -0600, Edward Henigin wrote: > > Hi Paolo, > > > > You are wise my friend. > > > > 1) Yes just upgraded to 1.5.2. (Was having problems in 1.5.1 where the > > input ifindex was getting mangled.) > > 2) Correct - was not seeing memory leak before upgrade. > > 3) If I leave running, eventually it consumes all available memory on the > > server. From syslog prior to service restart, > > Jan 7 15:05:05 packet1 nfacctd[16988]: WARN ( default/print ): Unable to > > fork writer: Cannot allocate memory > > 4) config: > > > > aggregate: peer_src_ip,in_iface,dst_host,dst_mask > > plugins: print > > nfacctd_time_new: true > > plugin_buffer_size: 10240 > > imt_buckets: 157 > > imt_mem_pools_number: 256 > > imt_mem_pools_size: 32768 > > syslog: daemon > > daemonize: true > > ! > > print_refresh_time: 64 > > print_history: 1m > > print_output: csv > > print_output_file: /opt/pmacct/data/nfacct-%Y%m%d-%H%M.csv > > ! print_trigger_exec: /opt/gravitizer/bin/monitor.pl > > print_output_file_append: true > > ! > > nfacctd_port: 2055 > > > > Thanks, > > > > Ed > > > > On Thu, Jan 7, 2016 at 4:20 PM, Paolo Lucente <[email protected]> wrote: > > > > > Hi Ed, > > > > > > You mean you kind of just upgraded to 1.5.2 and you were not seeing > > > the same before the upgrade? If you leave it running, does it grow > > > undefinitely or at some stage it looks it reaches a stable memory > > > footprint? Especially if using malloc()ed structures, ie. derived > > > by BGP primitives, this could be a valid scenario. Finally, can you > > > share your config? > > > > > > Cheers, > > > Paolo > > > > > > On Thu, Jan 07, 2016 at 04:04:09PM -0600, Edward Henigin wrote: > > > > Hi Paolo, > > > > > > > > I'm seeing a significant memory leak in the print plugin in 1.5.2. > > > > > > > > # service nfacctd restart > > > > Shutting down sflow collection services (nfacctd): [ OK ] > > > > Starting sflow collection services (nfacctd): INFO ( default/core ): > > > > Daemonizing. Hmm, bye bye screen. > > > > [ OK ] > > > > # pgrep nfacctd > > > > 874 > > > > 876 > > > > # while [ 1 ]; do sleep 60; date; ps uwp 876; done > > > > Thu Jan 7 15:53:56 CST 2016 > > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > > > root 876 1.1 0.6 115276 104432 ? S 15:52 0:01 > > > > nfacctd: > > > > Print Plugin [default] > > > > Thu Jan 7 15:54:56 CST 2016 > > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > > > root 876 1.0 0.6 120884 109980 ? S 15:52 0:01 > > > > nfacctd: > > > > Print Plugin [default] > > > > Thu Jan 7 15:55:56 CST 2016 > > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > > > root 876 1.0 0.7 128408 117608 ? S 15:52 0:02 > > > > nfacctd: > > > > Print Plugin [default] > > > > Thu Jan 7 15:56:56 CST 2016 > > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > > > root 876 1.1 0.8 147680 136856 ? S 15:52 0:03 > > > > nfacctd: > > > > Print Plugin [default] > > > > Thu Jan 7 15:57:56 CST 2016 > > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > > > root 876 1.1 0.9 160748 149944 ? S 15:52 0:03 > > > > nfacctd: > > > > Print Plugin [default] > > > > Thu Jan 7 15:58:56 CST 2016 > > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > > > root 876 1.1 1.0 180548 169668 ? S 15:52 0:04 > > > > nfacctd: > > > > Print Plugin [default] > > > > Thu Jan 7 15:59:56 CST 2016 > > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > > > root 876 1.1 1.1 203384 192604 ? S 15:52 0:05 > > > > nfacctd: > > > > Print Plugin [default] > > > > Thu Jan 7 16:00:56 CST 2016 > > > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > > > > root 876 1.3 1.5 272024 261120 ? S 15:52 0:07 > > > > nfacctd: > > > > Print Plugin [default] > > > > ^C > > > > # strings - /usr/sbin/nfacctd | fgrep 1.5 > > > > NetFlow Accounting Daemon, nfacctd 1.5.2 > > > > /opt/pmacct/pmacct-1.5.2/src > > > > /opt/pmacct/pmacct-1.5.2/src/nfprobe_plugin > > > > /opt/pmacct/pmacct-1.5.2/src/sfprobe_plugin > > > > /opt/pmacct/pmacct-1.5.2/src/bgp > > > > /opt/pmacct/pmacct-1.5.2/src/tee_plugin > > > > /opt/pmacct/pmacct-1.5.2/src/isis > > > > /opt/pmacct/pmacct-1.5.2/src/bmp > > > > > > > > Any thoughts? Suggestions? > > > > > > > > Thanks, > > > > > > > > Ed > > > > > > > > On Mon, Sep 7, 2015 at 1:01 PM, Paolo Lucente <[email protected]> wrote: > > > > > > > > > VERSION. > > > > > 1.5.2 > > > > > > > > > > > > > > > DESCRIPTION. > > > > > pmacct is a small set of passive network monitoring tools to account, > > > > > classify, > > > > > aggregate, replicate and export IPv4 and IPv6 traffic; a pluggable > > > > > architecture > > > > > allows to store collected data into memory tables, RDBMS (MySQL, > > > > > PostgreSQL, > > > > > SQLite), noSQL databases (MongoDB, BerkeleyDB) AMQP message exchanges > > > > > (RabbitMQ) and flat-files and export through NetFlow or sFlow > > > protocols to > > > > > remote collectors. pmacct offers customizable historical data > > > breakdown, > > > > > BGP > > > > > and IGP correlation, BMP stats and events, GeoIP lookups, sampling and > > > > > renormalization, filtering, tagging and triggers. Libpcap, Linux > > > > > Netlink/ULOG, > > > > > sFlow v2/v4/v5, NetFlow v5/v8/v9 and IPFIX are supported. Replication > > > of > > > > > incoming NetFlow and sFlow datagrams is also available. Statistics can > > > be > > > > > easily > > > > > exported to tools like RRDtool, Net-SNMP, MRTG, GNUPlot and Cacti. > > > > > > > > > > > > > > > HOMEPAGE. > > > > > http://www.pmacct.net/ > > > > > > > > > > > > > > > DOWNLOAD. > > > > > http://www.pmacct.net/pmacct-1.5.2.tar.gz > > > > > > > > > > > > > > > CHANGELOG. > > > > > + Introduced support for a RabbitMQ broker to be used for queueing and > > > > > data exchange between Core Process and plugins. This is in > > > alternative to > > > > > the home-grown circular queue implementation. plugin_pipe_amqp > > > directive, > > > > > along with all other plugin_pipe_amqp_* directives, can be set > > > globally > > > > > or apply on a per plugin basis (ie. it is a valid scenario, if > > > multiple > > > > > plugins are instantiated, that some make use of home-grown queueing, > > > > > while others use RabbitMQ based queueing). > > > > > + Introducting support for Maximind GeoIP v2 (libmaxminddb) library: > > > > > if > > > > > pmacct is compiled with --enable-geoipv2, this defines full pathname > > > to > > > > > a Maxmind GeoIP database v2 (libmaxminddb) Only the binary database > > > > > format is supported (ie. it is not possible to load distinct CSVs > > > > > for > > > > > IPv4 and IPv6 addresses). > > > > > + Introduced infrastructure for sFlow counters and support > > > specifically for > > > > > generic, ethernet and vlan counters. Counters are exported in JSON > > > format > > > > > to files, specified via sfacctd_counter_file. The supplied filename > > > can > > > > > contain as variable the sFlow agent IP address. > > > > > + Introduced a new thread_stack config directive to allow to modify > > > > > the > > > > > thread stack size. Natanael Copa reported that some libc > > > implementations, > > > > > ie. musl libc, may set a stack size that is too small by default. > > > > > + Introduced networks_file_no_lpm feature: it applies when aggregation > > > > > method includes src_net and/or dst_net and nfacctd_net (or > > > equivalents) > > > > > and/or nfacctd_as_new (or equivalents) are set to longest (or > > > fallback): > > > > > an IP prefix defined as part of the supplied networks_file wins > > > always, > > > > > even if it's not longest. > > > > > + tee plugin: added support for (non-)transparent IPv6 replication > > > [further > > > > > QA required] > > > > > + plugin_common.c, sql_common.c: added log message to estimate base > > > cache > > > > > memory usage. > > > > > + print, AMQP, MongoDB plugins; sfacctd, BGP, BMP daemons: introducing > > > > > timestamps_since_epoch to write timestamps in 'since Epoch' format. > > > > > + nfacctd: flow bytes counter can now be sourced via element ID #352 > > > > > (layer2OctetDeltaCount) in addition to element ID's already > > > supported. > > > > > Thanks to Jonathan Thorpe for his support. > > > > > + Introducing proc_priority: redefines the process scheduling > > > > > priority, > > > > > equivalent to using the 'nice' tool. Each daemon process, ie. core, > > > > > plugins, etc., can define a different priority. > > > > > ! fix, BMP daemon: improved preliminar checks in bmp_log_msg() and > > > added > > > > > missing SIGHUP signal handling to reload bmp_daemon_msglog_file > > > files. > > > > > ! fix, bgp_logdump.c: under certain configuration conditions call to > > > both > > > > > write_and_free_json() and write_and_free_json_amqp() was leading to > > > SEGV. > > > > > Thanks to Yuriy Lachin for reporting the issue. > > > > > ! fix, BGP daemon: improved BGP dump output: more accurate > > > timestamping of > > > > > dump_init, dump_close events. dump_close now mentions amount of > > > entries > > > > > and tables dumped. Thanks to Yuriy Lachin for brainstorming around > > > this. > > > > > ! fix, cfg.c: raised amount of allowed config lines from 256 to 8K. > > > > > ! fix, print/AMQP/MongoDB plugins: SEGV observed when IPFIX vlen > > > variables > > > > > were stored in the pending_queries_queue structure (ie. as a result > > > of a > > > > > time mismatch among the IPFIX exporter and the collector box). > > > > > ! fix, vlen primitives: when 'raw' semantics was selected, print_hex() > > > was > > > > > returning wrong hex string length (one char short). As a consequence > > > > > occasionally some extra dirty chars were seen at the end of the > > > converted > > > > > string. > > > > > ! fix, vlen primitives: memory leak verified in print/AMQP/MongoDB > > > plugins. > > > > > ! fix, print, MongoDB & AMQP plugins: dirty values printed as part of > > > the > > > > > 'proto' under certain conditions. Thanks to Rene Stoutjesdijk for > > > > > his > > > > > support resolving the issue. > > > > > ! fix, amqp_common.c: amqp_exchange_declare() call changed so to > > > address > > > > > the change of rabbitmq-c API for support of auto_delete & internal > > > for > > > > > exchange.declare. Backward compatibility with rabbitmq-c <= 0.5.2 is > > > > > also taken care of. Thanks to Brent Van Dussen for reporting the > > > issue. > > > > > ! fix, compiling on recent FreeBSD: solved some errors catched by the > > > -Wall > > > > > compiler flag. Thanks to Stephen Fulton for reporting the issue. > > > Most of > > > > > the patch is courtesy by Mike Bowie. > > > > > ! fix, print/AMQP/MongoDB plugins: enforcing cleanup of malloc()ed > > > structs > > > > > part of entries added to the pending queue, ie. because seen as > > > future > > > > > entries due to a mismatch of the collector clock with the one of > > > NetFlow/ > > > > > IPFIX exporter(s). This may have lead to data inconsistencies. > > > > > ! fix, amqp_common.c: Content type was only specified for messages > > > > > published > > > > > when the amqp_persistent_msg configuration option is specified. This > > > info > > > > > should always be applied to describe the payload of the message. > > > Patch is > > > > > courtesy by Will Dowling. > > > > > ! fix, amqp_plugin.c: generate an error on compile if > > > --enable-rabbitmq is > > > > > specified without --enable-jansson. It's clear in the documentation > > > that > > > > > both are required for AMQP support, but if built without jansson it > > > will > > > > > silently not publish messages to AMQP. Patch is courtesy by Will > > > Dowling. > > > > > ! fix, amqp_common.c: modified the content type to "application/json" > > > in > > > > > line with RFC4627. Patch is courtesy by Will Dowling. > > > > > ! fix, setsockopt(): u_int64_t pipe_size vars changed to int, in line > > > with > > > > > typical OS buffer limits (Linux, Solaris). Introduced check supplied > > > pipe > > > > > size values are not bigger than INT_MAX. Many thanks to Markus Weber > > > for > > > > > reporting the issue. > > > > > ! fix, nl.c: removed pretag_free_label() from pcap_cb() and ensuring > > > init > > > > > of pptrs. Under certain conditions SEGVs could be noticed. > > > > > ! fix, flow stitching: when print/AMQP/MongoDB plugins were making use > > > of > > > > > the pending queries queue, ie. to compensate for time offsets/flows > > > in > > > > > the future, the stitching feature could potentially lead to SEGV due > > > to > > > > > unsettled pointers. > > > > > ! fix, pgsql plugin: SEGV were noticed when insert/update queries to > > > the > > > > > PostgreSQL database were returning different than PGRES_COMMAND_OK, > > > hence > > > > > triggering the reprocess mechanism. Thanks very much to Alan Turower > > > for > > > > > his support. > > > > > ! fix, improved logging of elements received/sent at buffering point > > > > > between > > > > > core process and plugins. Also added explicit start/end purge log > > > message > > > > > for cases in which there is no data to purge. > > > > > ! fix, signals.c: ignore_falling_child() now logs if a child process > > > exited > > > > > with abnormal conditions; this is useful to track writer processes > > > > > (created > > > > > by plugins) are terminated by a signal, ie. SEGV. This is already > > > > > the > > > > > case > > > > > for plugins themselves, with the Core Process reporting a simlar log > > > > > message in case of abnormal exit. Thanks very much to Rene > > > Stoutjesdijk > > > > > for his support. > > > > > ! fix, preprocess-data.h: added supported functions minf, minb, minbpp > > > and > > > > > minppf to non SQL plugins. Thanks to Jared Deyo for reporting the > > > issue. > > > > > ! fix, nfprobe_plugin.c: IP protocol was not set up correctly for IPv6 > > > > > traffic in NetFlow v9/IPFIX. Thanks to Gabriel Vermeulen his support > > > > > solving the issue. > > > > > > > > > > > > > > > NOTES. > > > > > See UPGRADE file. > > > > > > > > > > > > > > > Cheers, > > > > > Paolo > > > > > > > > > > _______________________________________________ > > > > > pmacct-discussion mailing list > > > > > http://www.pmacct.net/#mailinglists > > > > > > > > > > > > _______________________________________________ > > > > pmacct-discussion mailing list > > > > http://www.pmacct.net/#mailinglists > > > > > > > > > _______________________________________________ > > > pmacct-discussion mailing list > > > http://www.pmacct.net/#mailinglists > > > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
