Hi Ruben, Yes, there is some config. You can read chapter "Internal buffering and queueing" of the QUICKSTART document:
https://github.com/paololucente/pmacct/blob/master/pmacct/QUICKSTART Essentially plugin_pipe_size becomes something not to care anymore and plugin_pipe_amqp goes to 'true'. Then any extra directive is to config aspects to connect to the RabbitMQ exchange (user, password, etc.). Cheers, Paolo On Sun, Aug 23, 2015 at 11:44:31PM +0200, Ruben Laban wrote: > Hi Paolo, > > This sure sounds interesting. I'll see if I can get around to > deploying latest CVS to my test setup asap. Does the change of the > queuing mechanism introduce a change in configuration as well? Like > new/changed options, different sizing recommendations, etc. > > Regards, > Ruben Laban > > On 22-8-2015 19:41, Paolo Lucente wrote: > >Hi Ruben, > > > >Your email is very timely and i understand such fluctuations between low > >and high traffic periods can happen in a libpcap deployment. A new feature > >that has been introduced as part of 1.5.2 (which is currently in the CVS > >and about to be released) is passing buffers inside pmacct - so from the > >core process to the plugins - using a RabbitMQ broker, as an alternative > >to the homegrown queueing infrastructure. This should help you working > >with small buffers (so that data does not get "delayed" in case of low > >traffic times) while keeping the queue size dynamic. > > > >Let me know if this looks of interest to you and, if yes, whether maybe > >you have room to give it a try before 1.5.2 release, say throughout next > >week. > > > >Cheers, > >Paolo > > > >On Fri, Aug 21, 2015 at 08:04:21AM +0200, Ruben Laban wrote: > >>Hi, > >> > >>Lately I've been experimenting a lot with plugin_buffer_size, > >>plugin_pipe_size and print_cache_size. My current conclusion is that > >>it is in no way possible to tune pmacct in such a way that it will > >>deal with low and (very) high traffic levels accurately: > >> > >>* Tune for low traffic: loss of data at high traffic > >>* Tune for high traffic: data gets "delayed" at low traffic > >> > >>Am I missing something here is this something I'll have to live with > >>(or otherwise work around). > >> > >>What I was hoping for was that by using large buffers (to > >>accommodate high traffic levels) I'd get to deal properly with low > >>and high traffic levels. I kinda expected buffers to flushed > >>whenever data is being flushed to disk (using print plugin right > >>now). This does not seem to be the case (completely). > >>The fact that tuning for high traffic levels requires (a lot) more > >>memory is easier to live with, memory's cheap after all. > >> > >>Some background info: > >>Ubuntu 14.01 > >>Pmacct 1.5.1 > >>libpcap, without PF_RING or anything > >> > >>Current config: > >> > >>! > >>debug: false > >>! > >>daemonize: true > >>! > >>core_proc_name: bond0 > >>! > >>aggregate[bond0-all]: src_mac, dst_mac, src_host, dst_host, > >>src_port, dst_port > >>aggregate[bond0-aggr]: none > >>! > >>plugins: print[bond0-all], print[bond0-aggr] > >>! > >>!plugin_buffer_size: 1024 > >>!plugin_buffer_size: 4096 > >>!plugin_pipe_size: 10240000 > >>plugin_pipe_size: 40960000 > >>! > >>files_umask: 027 > >>!files_uid: 0 > >>files_gid: 112 > >>! > >>interface: bond0.1234 > >>! > >>promisc: false > >>! > >>syslog: daemon > >>! > >>pidfile: /var/run/pmacctd-bond0.pid > >>! > >>print_output_file[bond0-all]: /tmp/pmacct-bond0-all-%Y%m%d-%H%M.csv > >>print_output_file[bond0-aggr]: /tmp/pmacct-bond0-aggr-%Y%m%d-%H%M.csv > >>! > >>print_refresh_time: 60 > >>! > >>print_time_roundoff: m > >>! > >>print_history: 1m > >>! > >>print_cache_entries: 323333 > >>!print_cache_entries: 100003 > >>! > >>print_output_file_append: true > >>! > >>print_markers: true > >>! > >>!print_output: json > >>print_output: csv > >> > >>Regards, > >>Ruben Laban > >> > >>_______________________________________________ > >>pmacct-discussion mailing list > >>http://www.pmacct.net/#mailinglists > > > >_______________________________________________ > >pmacct-discussion mailing list > >http://www.pmacct.net/#mailinglists > > > _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
