Hi Chris, I'm thinking to the possibility that given the aggregation method the SQL cache configured by default is not sufficient to keep all the aggregates over the time period - although the time period is very short. Can you as matter of test add the following line to your config and see if it makes any difference?
sql_cache_entries: 999991 About plugin_buffer_size and plugin_pipe_size, CONFIG-KEYS gives some guidelines. I suggest to start from those and take it from there (no error message is OK; still error message then move up by one order of magnitude; etc.). So try starting from: plugin_pipe_size: 10240000 plugin_buffer_size: 10240 Other than briefly experimenting around values (which i suggest as a start), you can check the INTERNALS document section 'V. Communications between core process and plugins'. Cheers, Paolo On Tue, Jun 12, 2012 at 10:45:34AM +0100, Chris Wilson wrote: > Hi all, > > We get many of these errors in our system logs: > > Jun 12 10:01:01 fen-fw2 pmacctd[2153]: ERROR ( short/mysql ): Duplicate > entry '72.232.223.58-82.68.244.70-80-46802-tcp-2012-06-12 09:56:00' for > key 1 > > They usually happen in batches. E.g. we had a few hundred at 07:27, then > another few hundred at 10:01, and a few dozen at 10:17. > > In our configuration these duplicate inserts should never happen. We > should get one INSERT per flow per minute, and the different minutes > should result in different values of the primary key. > > plugins: mysql[short], mysql[long] > aggregate[short]: src_host, src_port, dst_host, dst_port, proto > sql_db: pmacct > sql_table[short]: acct_v6 > sql_history[short]: 1m > sql_history_roundoff[short]: m > sql_refresh_time[short]: 60 > sql_dont_try_update: true > sql_optimize_clauses: true > > It's like pmacct is not correctly finding an existing flow when > aggregating, and creating a new one that duplicates the existing one. Is > there some way to test that? Would the memory plugin do it? > > Can anyone explain why this is happening or what I'm doing wrong? > > Also, we get a lot of these: > > Jun 10 05:13:01 fen-fw2 pmacctd[4070]: ERROR ( short/mysql ): We are > missing data. > Jun 10 05:13:01 fen-fw2 pmacctd[4070]: If you see this message once in a > while, discard it. Otherwise some solutions follow: > Jun 10 05:13:01 fen-fw2 pmacctd[4070]: - increase shared memory size, > 'plugin_pipe_size'; now: '3096576'. > Jun 10 05:13:01 fen-fw2 pmacctd[4070]: - increase buffer size, > 'plugin_buffer_size'; now: '192'. > Jun 10 05:13:01 fen-fw2 pmacctd[4070]: - increase system maximum socket > size. > > How would I know which parameter to increase? Could the writer tell us > exactly which limit it hit? > > Cheers, Chris. > -- > Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838 > Future Business, Cam City FC, Milton Rd, Cambridge, CB4 1UY, UK > > Aptivate is a not-for-profit company registered in England and Wales > with company number 04980791. > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
