Hi Ruben, Your email is very timely and i understand such fluctuations between low and high traffic periods can happen in a libpcap deployment. A new feature that has been introduced as part of 1.5.2 (which is currently in the CVS and about to be released) is passing buffers inside pmacct - so from the core process to the plugins - using a RabbitMQ broker, as an alternative to the homegrown queueing infrastructure. This should help you working with small buffers (so that data does not get "delayed" in case of low traffic times) while keeping the queue size dynamic.
Let me know if this looks of interest to you and, if yes, whether maybe you have room to give it a try before 1.5.2 release, say throughout next week. Cheers, Paolo On Fri, Aug 21, 2015 at 08:04:21AM +0200, Ruben Laban wrote: > Hi, > > Lately I've been experimenting a lot with plugin_buffer_size, > plugin_pipe_size and print_cache_size. My current conclusion is that > it is in no way possible to tune pmacct in such a way that it will > deal with low and (very) high traffic levels accurately: > > * Tune for low traffic: loss of data at high traffic > * Tune for high traffic: data gets "delayed" at low traffic > > Am I missing something here is this something I'll have to live with > (or otherwise work around). > > What I was hoping for was that by using large buffers (to > accommodate high traffic levels) I'd get to deal properly with low > and high traffic levels. I kinda expected buffers to flushed > whenever data is being flushed to disk (using print plugin right > now). This does not seem to be the case (completely). > The fact that tuning for high traffic levels requires (a lot) more > memory is easier to live with, memory's cheap after all. > > Some background info: > Ubuntu 14.01 > Pmacct 1.5.1 > libpcap, without PF_RING or anything > > Current config: > > ! > debug: false > ! > daemonize: true > ! > core_proc_name: bond0 > ! > aggregate[bond0-all]: src_mac, dst_mac, src_host, dst_host, > src_port, dst_port > aggregate[bond0-aggr]: none > ! > plugins: print[bond0-all], print[bond0-aggr] > ! > !plugin_buffer_size: 1024 > !plugin_buffer_size: 4096 > !plugin_pipe_size: 10240000 > plugin_pipe_size: 40960000 > ! > files_umask: 027 > !files_uid: 0 > files_gid: 112 > ! > interface: bond0.1234 > ! > promisc: false > ! > syslog: daemon > ! > pidfile: /var/run/pmacctd-bond0.pid > ! > print_output_file[bond0-all]: /tmp/pmacct-bond0-all-%Y%m%d-%H%M.csv > print_output_file[bond0-aggr]: /tmp/pmacct-bond0-aggr-%Y%m%d-%H%M.csv > ! > print_refresh_time: 60 > ! > print_time_roundoff: m > ! > print_history: 1m > ! > print_cache_entries: 323333 > !print_cache_entries: 100003 > ! > print_output_file_append: true > ! > print_markers: true > ! > !print_output: json > print_output: csv > > Regards, > Ruben Laban > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
