Hi Paolo,
This sure sounds interesting. I'll see if I can get around to deploying
latest CVS to my test setup asap. Does the change of the queuing
mechanism introduce a change in configuration as well? Like new/changed
options, different sizing recommendations, etc.
Regards,
Ruben Laban
On 22-8-2015 19:41, Paolo Lucente wrote:
Hi Ruben,
Your email is very timely and i understand such fluctuations between low
and high traffic periods can happen in a libpcap deployment. A new feature
that has been introduced as part of 1.5.2 (which is currently in the CVS
and about to be released) is passing buffers inside pmacct - so from the
core process to the plugins - using a RabbitMQ broker, as an alternative
to the homegrown queueing infrastructure. This should help you working
with small buffers (so that data does not get "delayed" in case of low
traffic times) while keeping the queue size dynamic.
Let me know if this looks of interest to you and, if yes, whether maybe
you have room to give it a try before 1.5.2 release, say throughout next
week.
Cheers,
Paolo
On Fri, Aug 21, 2015 at 08:04:21AM +0200, Ruben Laban wrote:
Hi,
Lately I've been experimenting a lot with plugin_buffer_size,
plugin_pipe_size and print_cache_size. My current conclusion is that
it is in no way possible to tune pmacct in such a way that it will
deal with low and (very) high traffic levels accurately:
* Tune for low traffic: loss of data at high traffic
* Tune for high traffic: data gets "delayed" at low traffic
Am I missing something here is this something I'll have to live with
(or otherwise work around).
What I was hoping for was that by using large buffers (to
accommodate high traffic levels) I'd get to deal properly with low
and high traffic levels. I kinda expected buffers to flushed
whenever data is being flushed to disk (using print plugin right
now). This does not seem to be the case (completely).
The fact that tuning for high traffic levels requires (a lot) more
memory is easier to live with, memory's cheap after all.
Some background info:
Ubuntu 14.01
Pmacct 1.5.1
libpcap, without PF_RING or anything
Current config:
!
debug: false
!
daemonize: true
!
core_proc_name: bond0
!
aggregate[bond0-all]: src_mac, dst_mac, src_host, dst_host,
src_port, dst_port
aggregate[bond0-aggr]: none
!
plugins: print[bond0-all], print[bond0-aggr]
!
!plugin_buffer_size: 1024
!plugin_buffer_size: 4096
!plugin_pipe_size: 10240000
plugin_pipe_size: 40960000
!
files_umask: 027
!files_uid: 0
files_gid: 112
!
interface: bond0.1234
!
promisc: false
!
syslog: daemon
!
pidfile: /var/run/pmacctd-bond0.pid
!
print_output_file[bond0-all]: /tmp/pmacct-bond0-all-%Y%m%d-%H%M.csv
print_output_file[bond0-aggr]: /tmp/pmacct-bond0-aggr-%Y%m%d-%H%M.csv
!
print_refresh_time: 60
!
print_time_roundoff: m
!
print_history: 1m
!
print_cache_entries: 323333
!print_cache_entries: 100003
!
print_output_file_append: true
!
print_markers: true
!
!print_output: json
print_output: csv
Regards,
Ruben Laban
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
