Hi Tiernan, It is not clear to me whether you tried to set higher values for both plugin_buffer_size and plugin_pipe_size - say to 100MB and 1MB or 10MB respectively. A value of 16384 for plugin_buffer_size is definitely too small.
Cheers, Paolo On Wed, Jun 06, 2012 at 01:23:24PM +1000, Tiernan Messmer wrote: > Hi all, > > Using pmacct 0.14.0 on FreeBSD 9.0 amd64 with 64bit counters and mmap > enabled the core pmacct process maxes out a core when traffic levels are > above 10-20mbit and spams the following in syslog (as well as for the > in/mysql plugin too). > > pmacctd[82138]: ERROR ( out/mysql ): We are missing data. > pmacctd[82138]: If you see this message once in a while, discard it. > Otherwise some solutions follow: > pmacctd[82138]: - increase shared memory size, 'plugin_pipe_size'; now: > '134201344'. > pmacctd[82138]: - increase buffer size, 'plugin_buffer_size'; now: '16384'. > pmacctd[82138]: - increase system maximum socket size. > > >From what I have been able to gather, the above buffer and pipe sizes > should be more than sufficient for a 100mbit pipe leading me to believe > that simply increasing those values even further wouldn't solve the > issue. Additionally, whenever traffic goes over 10-20mbit and the above > spam appears in syslog the amount of data transferred in the sql table > gets arbitarily large (tens to hundreds of gigs) added to it which > doesn't exist. > > I have tried increasing kern.ipc.maxsockbuf from the default 2MB to 16MB > which did not seem to affect anything either. > > My pmacctd.conf is as follows: > > debug: false > daemonize: true > aggregate[in]: dst_host > aggregate[out]: src_host > aggregate[infree]: dst_host > aggregate[outfree]: src_host > aggregate[intotal]: dst_host > aggregate[outtotal]: src_host > aggregate_filter[in]: <big filter here> > aggregate_filter[out]: <big filter here> > aggregate_filter[infree]: <big filter here> > aggregate_filter[outfree]: <big filter here> > aggregate_filter[intotal]: <big filter here> > aggregate_filter[outtotal]: <big filter here> > plugins: mysql[in], mysql[out], mysql[infree], mysql[outfree], > memory[intotal], memory[outtotal] > plugin_buffer_size: 16384 > pmacctd_flow_buffer_buckets: 65536 > interface: tun0 > interface_wait: true > snaplen: 128 > promisc: false > syslog: daemon > pidfile: /var/run/pmacctd.pid > sql_db: pmacct > sql_table[in]: in > sql_table[out]: out > sql_table[infree]: infree > sql_table[outfree]: outfree > sql_table_version: 1 > sql_user: pmacct > sql_passwd: <masked> > sql_refresh_time: 60 > sql_table_schema[in]: /usr/local/etc/pmacctd/pmacct.in.db > sql_table_schema[out]: /usr/local/etc/pmacctd/pmacct.out.db > sql_table_schema[infree]: /usr/local/etc/pmacctd/pmacct.infree.db > sql_table_schema[outfree]: /usr/local/etc/pmacctd/pmacct.outfree.db > sql_history: 1h > sql_history_roundoff: m > sql_optimize_clauses: true > imt_path[intotal]: /tmp/pmacct_in.pipe > imt_path[outtotal]: /tmp/pmacct_out.pipe > > Is there anything ovbious wrong with my config causing this behaviour? > > ~Tiernan Messmer > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
