Hi Paolo,
Either I'm doing something wrong or there's a bug in this part of
pmacct. Current config:
!
debug: false
!
daemonize: true
!
core_proc_name: bond0
!
aggregate[bond0-all]: src_mac, dst_mac, src_host, dst_host, src_port,
dst_port
aggregate[bond0-aggr]: none
!
plugins: print[bond0-all], print[bond0-aggr]
!
!plugin_buffer_size: 1024
!plugin_buffer_size: 4096
!plugin_pipe_size: 10240000
!plugin_pipe_size: 40960000
plugin_pipe_amqp: true
!
!plugin_pipe_amqp_routing_key[bond0-all]: bond0-all
!plugin_pipe_amqp_routing_key[bond0-aggr]: bond0-aggr
!
files_umask: 027
!files_uid: 0
files_gid: 112
!
interface: bond0.1234
!
promisc: false
!
syslog: daemon
!
pidfile: /var/run/pmacctd-bond0.pid
!
print_output_file[bond0-all]: /tmp/pmacct-bond0-all-%Y%m%d-%H%M.csv
print_output_file[bond0-aggr]: /tmp/pmacct-bond0-aggr-%Y%m%d-%H%M.csv
!
print_refresh_time: 60
!
print_time_roundoff: m
!
print_history: 1m
!
print_cache_entries: 323333
!print_cache_entries: 100003
!
print_output_file_append: true
!
print_markers: true
!
!print_output: json
print_output: csv
I think there's something wrong with the plugin_pipe_amqp_routing_key
stuff. If I set it to some value, the data vanishes. If I leave it to
default, all data ends up in a single pile (data for both plugins get
written to /tmp/pmacct-bond0-all-*.csv).
These are my first steps in AMQP/RabbitMQ land, so it's quite possible
I'm doing something wrong. This is just a default installation of
rabbitmq-server on Ubuntu 14.04 (localhost, guest, etc).
Regards,
Ruben Laban
On 28-8-2015 10:42, Ruben Laban wrote:
Hi Paolo,
Strange, the bmp/ dir does show up in a fresh checkout (and nightly
tarball, and git). For some reason my previous checkout refused to pull
in that dir. I'll continue testing the rabbitmq stuff now.
Regards,
Ruben Laban
On 27-8-2015 12:03, Paolo Lucente wrote:
Hi Ruben,
Can you give it another try? I don't manage to reproduce this issue
with the CVS - all appears to work good for me. As a workaround, should
you still have problems, you can download the latest daily package from
the webapage: http://www.pmacct.net/pmacct-daily.tar.gz . As yet another
alternative you can check out latest code on GitHub. Please let me know.
Cheers,
Paolo
On Wed, Aug 26, 2015 at 09:41:29AM +0200, Ruben Laban wrote:
Hi Paolo,
I just did a fresh CVS checkout and tried to build it. Alas, it
failed. It complains about not finding ../bmp/bmp.h as referenced in
bgp/bgp_logdump.c. And my checkout doesn't even have that directory
(bmp).
Regards,
Ruben
On 26-8-2015 01:02, Paolo Lucente wrote:
Hi Ruben,
Yes, there is some config. You can read chapter "Internal buffering and
queueing" of the QUICKSTART document:
https://github.com/paololucente/pmacct/blob/master/pmacct/QUICKSTART
Essentially plugin_pipe_size becomes something not to care anymore and
plugin_pipe_amqp goes to 'true'. Then any extra directive is to config
aspects to connect to the RabbitMQ exchange (user, password, etc.).
Cheers,
Paolo
On Sun, Aug 23, 2015 at 11:44:31PM +0200, Ruben Laban wrote:
Hi Paolo,
This sure sounds interesting. I'll see if I can get around to
deploying latest CVS to my test setup asap. Does the change of the
queuing mechanism introduce a change in configuration as well? Like
new/changed options, different sizing recommendations, etc.
Regards,
Ruben Laban
On 22-8-2015 19:41, Paolo Lucente wrote:
Hi Ruben,
Your email is very timely and i understand such fluctuations
between low
and high traffic periods can happen in a libpcap deployment. A new
feature
that has been introduced as part of 1.5.2 (which is currently in
the CVS
and about to be released) is passing buffers inside pmacct - so
from the
core process to the plugins - using a RabbitMQ broker, as an
alternative
to the homegrown queueing infrastructure. This should help you
working
with small buffers (so that data does not get "delayed" in case of
low
traffic times) while keeping the queue size dynamic.
Let me know if this looks of interest to you and, if yes, whether
maybe
you have room to give it a try before 1.5.2 release, say
throughout next
week.
Cheers,
Paolo
On Fri, Aug 21, 2015 at 08:04:21AM +0200, Ruben Laban wrote:
Hi,
Lately I've been experimenting a lot with plugin_buffer_size,
plugin_pipe_size and print_cache_size. My current conclusion is that
it is in no way possible to tune pmacct in such a way that it will
deal with low and (very) high traffic levels accurately:
* Tune for low traffic: loss of data at high traffic
* Tune for high traffic: data gets "delayed" at low traffic
Am I missing something here is this something I'll have to live with
(or otherwise work around).
What I was hoping for was that by using large buffers (to
accommodate high traffic levels) I'd get to deal properly with low
and high traffic levels. I kinda expected buffers to flushed
whenever data is being flushed to disk (using print plugin right
now). This does not seem to be the case (completely).
The fact that tuning for high traffic levels requires (a lot) more
memory is easier to live with, memory's cheap after all.
Some background info:
Ubuntu 14.01
Pmacct 1.5.1
libpcap, without PF_RING or anything
Current config:
!
debug: false
!
daemonize: true
!
core_proc_name: bond0
!
aggregate[bond0-all]: src_mac, dst_mac, src_host, dst_host,
src_port, dst_port
aggregate[bond0-aggr]: none
!
plugins: print[bond0-all], print[bond0-aggr]
!
!plugin_buffer_size: 1024
!plugin_buffer_size: 4096
!plugin_pipe_size: 10240000
plugin_pipe_size: 40960000
!
files_umask: 027
!files_uid: 0
files_gid: 112
!
interface: bond0.1234
!
promisc: false
!
syslog: daemon
!
pidfile: /var/run/pmacctd-bond0.pid
!
print_output_file[bond0-all]: /tmp/pmacct-bond0-all-%Y%m%d-%H%M.csv
print_output_file[bond0-aggr]:
/tmp/pmacct-bond0-aggr-%Y%m%d-%H%M.csv
!
print_refresh_time: 60
!
print_time_roundoff: m
!
print_history: 1m
!
print_cache_entries: 323333
!print_cache_entries: 100003
!
print_output_file_append: true
!
print_markers: true
!
!print_output: json
print_output: csv
Regards,
Ruben Laban
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
_______________________________________________
pmacct-discussion mailing list
http://www.pmacct.net/#mailinglists
