Hello, How to check if the PF_RING is in action and active ? some forwarded packets counts, or etc ?
Thanks On Thu, Dec 27, 2018 at 3:00 PM Edvinas K <[email protected]> wrote: > thank you, > > seems all easy things didin't help. > > I tried to set up the buffer size in kernel: > > prod [root@netvpn001prpjay pmacct-1.7.2]# cat > /proc/sys/net/core/[rw]mem_max > 2000000000 > 2000000000 > > and then > > prod [root@netvpn001prpjay pmacct-1.7.2]# cat flowexport.cfg > ! > daemonize: no > aggregate: src_host, dst_host, src_port, dst_port, proto, tos > plugins: nfprobe > nfprobe_receiver: 10.3.14.101:2101 > nfprobe_version: 9 > > pmacctd_pipe_size: 2000000000 > plugin_pipe_size: 1000000 > plugin_buffer_size: 10000 > > ! nfprobe_engine: 1:1 > ! nfprobe_timeouts: tcp=120:maxlife=3600 > ! > ! networks_file: /path/to/networks.lst > !... > > maybe after putting plugin_pipe_size and plugin_buffer_size drops got > little bit lower, but still a lot. > also noticed strange log message: "INFO ( default/core ): short IPv4 > packet read (36/38/frags). Snaplen issue ?" > > I going to try that PF_RING stuff. > > On Thu, Dec 20, 2018 at 10:08 PM Paolo Lucente <[email protected]> wrote: > >> >> Hi Edvinas, >> >> I wanted to confirm that when you changed pmacctd_pipe_size to 2GB you >> ALSO changed /proc/sys/net/core/[rw]mem_max to 2GB and ALSO restarted >> pmacctd after having done so. >> >> Wrt PF_RING: i can't voice since i don't use it myself. While i never >> heard any horror story with it (thumbs up!), i think doing a proof of >> concept first is always a good idea; this is also to answer your second >> question: it will improve things for sure but how much you have to test. >> >> Another thing you may do is also to increase buffering internal to >> pmacct (it may help reduce CPU cycles by the core process and hence help >> it process more data), i see that in your config you have NO buffering >> enabled. For a quick test you could set: >> >> plugin_pipe_size: 1000000 >> plugin_buffer_size: 10000 >> >> And depending if you see any benefits/improvement and if you have memory >> you could ramp these values up. Or alternatively you could introduce >> ZeroMQ. Again, this is internal queueuing (whereas in my previous email >> i was tackling the queueing between kernel and pmacct): >> >> https://github.com/pmacct/pmacct/blob/master/CONFIG-KEYS#L234-#L292 >> >> Paolo >> >> On Wed, Dec 19, 2018 at 06:40:14PM +0200, Edvinas K wrote: >> > Hello, >> > >> > How would you recommend to test PF_RING: >> > >> > Some questions: >> > >> > Is't safe to install it on production server ? >> > Is't possible to hope, that this PF_RING will solve all the discards ? >> > >> > Thanks >> > >> > On Tue, Dec 18, 2018 at 5:59 PM Edvinas K <[email protected]> >> wrote: >> > >> > > thanks, >> > > >> > > I tried to change the pipe size. As i noticed my OS (centos) default >> and >> > > max size are the same: >> > > >> > > prod [root@netvpn001prpjay pmacct-1.7.2]# cat >> > > /proc/sys/net/core/[rw]mem_default >> > > 212992 >> > > 212992 >> > > >> > > prod [root@netvpn001prpjay pmacct-1.7.2]# cat >> > > /proc/sys/net/core/[rw]mem_max >> > > 212992 >> > > 212992 >> > > >> > > I tried to set the pmacctd_pipe_size: to 2000000000 and later to >> 212992. >> > > Seems the drops is still occuring. >> > > Tomorrow i will try to look at that PF_RING thing. >> > > >> > > Thanks >> > > >> > > >> > > >> > > >> > > >> > > On Tue, Dec 18, 2018 at 5:32 PM Paolo Lucente <[email protected]> >> wrote: >> > > >> > >> >> > >> Hi Edvinas, >> > >> >> > >> Easier thing first, i recommend to inject some test traffic and see >> that >> > >> one how it looks like. >> > >> >> > >> The dropped packets highlight a buffering issue. You could take an >> > >> intermediate step and see if enlarging buffers helps. Configure >> > >> pmacctd_pipe_size to 2000000000 and follow instructions here for the >> > >> /proc files to touch: >> > >> >> > >> https://github.com/pmacct/pmacct/blob/1.7.2/CONFIG-KEYS#L203-#L216 >> > >> >> > >> If it helps, good. If not: you should really look into one of the >> > >> frameworks i was pointing you to in my previous email. PF_RING, for >> > >> example, can do sampling and/or balancing. Sampling should not be >> done >> > >> inside pmacct because the dropped packets are between the kernel and >> the >> > >> application. >> > >> >> > >> Paolo >> > >> >> > >> On Mon, Dec 17, 2018 at 02:52:48PM +0200, Edvinas K wrote: >> > >> > Seems there're lots of dropped packets: >> > >> > >> > >> > prod [root@netvpn001prpjay pmacct-1.7.2]# pmacctd -i ens1f0.432 -f >> > >> > flowexport.cfg >> > >> > WARN: [flowexport.cfg:2] Invalid value. Ignored. >> > >> > INFO ( default/core ): Promiscuous Mode Accounting Daemon, pmacctd >> > >> > 1.7.2-git (20181018-00+c3) >> > >> > INFO ( default/core ): '--enable-l2' '--enable-ipv6' >> '--enable-64bit' >> > >> > '--enable-traffic-bins' '--enable-bgp-bins' '--enable-bmp-bins' >> > >> > '--enable-st-bins' >> > >> > INFO ( default/core ): Reading configuration file >> > >> > '/opt/pmacct-1.7.2/flowexport.cfg'. >> > >> > INFO ( default_nfprobe/nfprobe ): NetFlow probe plugin is >> originally >> > >> based >> > >> > on softflowd 0.9.7 software, Copyright 2002 Damien Miller < >> > >> [email protected]> >> > >> > All rights reserved. >> > >> > INFO ( default_nfprobe/nfprobe ): TCP timeout: 3600s >> > >> > INFO ( default_nfprobe/nfprobe ): TCP post-RST timeout: 120s >> > >> > INFO ( default_nfprobe/nfprobe ): TCP post-FIN timeout: 300s >> > >> > INFO ( default_nfprobe/nfprobe ): UDP timeout: 300s >> > >> > INFO ( default_nfprobe/nfprobe ): ICMP timeout: 300s >> > >> > INFO ( default_nfprobe/nfprobe ): General timeout: 3600s >> > >> > INFO ( default_nfprobe/nfprobe ): Maximum lifetime: 604800s >> > >> > INFO ( default_nfprobe/nfprobe ): Expiry interval: 60s >> > >> > INFO ( default_nfprobe/nfprobe ): Exporting flows to >> > >> > [10.3.14.101]:rtcm-sc104 >> > >> > INFO ( default/core ): [ens1f0.432,0] link type is: 1 >> > >> > ^C^C^C^C^C^C^C^C >> > >> > >> > >> > after 1 minute: >> > >> > >> > >> > WARN ( default_nfprobe/nfprobe ): Shutting down on user request. >> > >> > INFO ( default/core ): OK, Exiting ... >> > >> > NOTICE ( default/core ): +++ >> > >> > NOTICE ( default/core ): [ens1f0.432,0] received_packets=3441854 >> > >> > *dropped_packets=2365166* >> > >> > >> > >> > About 1GB of traffic is passing through the router where i'm >> capturing >> > >> the >> > >> > packets. Isn't it too much traffic for nfrpobe to process ? CPUs >> seems >> > >> not >> > >> > in 100% usage. We're using Intel Xeon E5-2620 0 @ 2.00GHz >> > >> > < >> > >> >> http://netmon.adform.com/device/device=531/tab=health/metric=processor/processor_id=1466/ >> > >> > >> > >> > x >> > >> > 24. >> > >> > >> > >> > prod [root@netvpn001prpjay ~]# ps -aux | grep pmacct >> > >> > root 41840 30.9 0.0 18964 7760 ? Rs Dec14 1309:50 >> > >> pmacctd: >> > >> > Core Process [default] >> > >> > root 41841 *68.4%* 0.0 22932 9756 ? R Dec14 >> 2898:29 >> > >> > pmacctd: Netflow Probe Plugin [default_nfprobe] >> > >> > root 41869 32.5 0.0 19360 8128 ? Ss Dec14 1378:29 >> > >> pmacctd: >> > >> > Core Process [default] >> > >> > root 41870 *67.6%* 0.0 22928 9760 ? R Dec14 2865:35 >> > >> > pmacctd: Netflow Probe Plugin [default_nfprobe] >> > >> > >> > >> > Before starting with your mentioned 'steroid' things, i would like >> to >> > >> ask, >> > >> > is't really worth to go to that kernel "things", or start with >> > >> techniques >> > >> > for example like sampling, or like Nikola recommended try to fidle >> with >> > >> > nfprobe_engine settings ? >> > >> > >> > >> > Thanks >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > >> > On Sun, Dec 16, 2018 at 6:25 PM Paolo Lucente <[email protected]> >> wrote: >> > >> > >> > >> > > >> > >> > > Hi Edvinas, >> > >> > > >> > >> > > You may want to check whether libpcap is dropping packets on >> input to >> > >> > > pmacctd. You can achieve that sending a SIGUSR1 and checking the >> > >> output >> > >> > > in the logfile/syslog/console. You will get something a-la: >> > >> > > >> > >> > > >> https://github.com/pmacct/pmacct/blob/master/docs/SIGNALS#L16-#L34 >> > >> > > >> > >> > > Should amount of dropped packets be non-zero and visibly >> increasing >> > >> then >> > >> > > you may want to put your libpcap on steroids: >> > >> > > >> > >> > > https://github.com/pmacct/pmacct/blob/master/FAQS#L71-#L101 >> > >> > > >> > >> > > Should, instead, that not be the case, i am unsure and would need >> > >> > > further investigation. You could try to produce a controlled >> stream of >> > >> > > data and sniff nfprobe output. Or collect with a different >> software >> > >> for >> > >> > > a quick counter-test (nfacctd itself or another of your choice). >> > >> > > >> > >> > > Paolo >> > >> > > >> > >> > > On Fri, Dec 14, 2018 at 03:02:35PM +0200, Edvinas K wrote: >> > >> > > > Thanks, i really appreciate your help. >> > >> > > > >> > >> > > > Everything seems working OK, on NFSEN (NFDUMP) graphs of flows >> > >> statistics >> > >> > > > looks good, but the traffic rate Mb/s (45 Mb/s) is somehow 10x >> > >> lower than >> > >> > > > really is. Maybe some tips to troubleshoot that ? >> > >> > > > >> > >> > > > [image: image.png] >> > >> > > > >> > >> > > > Is there any hidden things to check about ? >> > >> > > > >> > >> > > > My config: >> > >> > > > >> > >> > > > 1050 pmacctd -i ens1f0.432 -f flowexport.cfg >> > >> > > > 1051 pmacctd -i ens1f1.433 -f flowexport.cfg >> > >> > > > >> > >> > > > cat flowexport.cfg >> > >> > > > ! >> > >> > > > daemonize: true >> > >> > > > aggregate: src_host, dst_host, src_port, dst_port, proto, >> tos >> > >> > > > plugins: nfprobe >> > >> > > > nfprobe_receiver: 10.3.14.101:2101 >> > >> > > > nfprobe_version: 9 >> > >> > > > ! nfprobe_engine: 1:1 >> > >> > > > ! nfprobe_timeouts: tcp=120:maxlife=3600 >> > >> > > > ! >> > >> > > > ! networks_file: /path/to/networks.lst >> > >> > > > >> > >> > > > On Thu, Dec 13, 2018 at 4:32 AM Paolo Lucente < >> [email protected]> >> > >> wrote: >> > >> > > > >> > >> > > > > >> > >> > > > > Hi Nikola, >> > >> > > > > >> > >> > > > > I see, makes sense. Thanks very much for clarifying. >> > >> > > > > >> > >> > > > > Paolo >> > >> > > > > >> > >> > > > > On Wed, Dec 12, 2018 at 06:20:58PM -0800, Nikola Kolev wrote: >> > >> > > > > > Hi Paollo, >> > >> > > > > > >> > >> > > > > > Sorry for being cryptic - what I meant was that I wasn't >> able to >> > >> > > > > > launch pmacctd/uacctd in a way that it deals with dynamic >> > >> interfaces >> > >> > > as >> > >> > > > > > ppp. Basically I failed to find any reference in the docs >> on >> > >> how to >> > >> > > make >> > >> > > > > > it run in such a way, that it collects info from ppp* >> (a-la the >> > >> ppp+ >> > >> > > > > > syntax of iptables), without launching a separate pmacctd >> > >> instance >> > >> > > for >> > >> > > > > > each interface, hence the complicated setup with >> > >> > > > > > iptables-nflog-uacctd-nfdump. >> > >> > > > > > >> > >> > > > > > On Thu, 13 Dec 2018 01:35:00 +0000 >> > >> > > > > > Paolo Lucente <[email protected]> wrote: >> > >> > > > > > >> > >> > > > > > > >> > >> > > > > > > Hi Nikola, >> > >> > > > > > > >> > >> > > > > > > Can you please elaborate a bit more? The cryptic part >> for me >> > >> is "as >> > >> > > > > > > nfacctd is not supporting wildcard addresses to be bound >> to". >> > >> > > > > > > >> > >> > > > > > > Thanks, >> > >> > > > > > > Paolo >> > >> > > > > > > >> > >> > > > > > > On Wed, Dec 12, 2018 at 04:50:33PM -0800, Nikola Kolev >> wrote: >> > >> > > > > > > > Hey, >> > >> > > > > > > > >> > >> > > > > > > > If I may add to that: >> > >> > > > > > > > >> > >> > > > > > > > I'm doing something similar, but in a slightly >> different >> > >> manner: >> > >> > > > > > > > >> > >> > > > > > > > as nfacctd is not supporting wildcard addresses to be >> bound >> > >> to, >> > >> > > I'm >> > >> > > > > > > > using iptables' rules to export via nflog to uacctd, >> which >> > >> then >> > >> > > can >> > >> > > > > > > > send to nfdump. Just food for thought... >> > >> > > > > > > > >> > >> > > > > > > > On 2018-12-12 14:58, Paolo Lucente wrote: >> > >> > > > > > > > >Hi Edvinas, >> > >> > > > > > > > > >> > >> > > > > > > > >You are looking for the nfprobe plugin. You can >> follow the >> > >> > > relevant >> > >> > > > > > > > >section in the QUICKSTART to get going: >> > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > >> https://github.com/pmacct/pmacct/blob/1.7.2/QUICKSTART#L1167-#L1302 >> > >> > > > > > > > > >> > >> > > > > > > > >Paolo >> > >> > > > > > > > > >> > >> > > > > > > > >On Wed, Dec 12, 2018 at 03:12:39PM +0200, Edvinas K >> wrote: >> > >> > > > > > > > >>Hello, >> > >> > > > > > > > >> >> > >> > > > > > > > >>I managed to run basic pmacct to capture linux router >> > >> (FRR) >> > >> > > flows >> > >> > > > > > > > >>from libcap: >> > >> > > > > > > > >>"pmacctd -P print -O formatted -r 10 -i bond0.2170 -c >> > >> > > > > > > > >>src_host,dst_host,src_port,dst_port,proto" >> > >> > > > > > > > >> >> > >> > > > > > > > >>now I need to push all the flows as a netflow format >> to >> > >> the >> > >> > > > > > > > >>netflow collector (nfdump). Could you give me some >> advice >> > >> how >> > >> > > to >> > >> > > > > > > > >>configure that ? >> > >> > > > > > > > >>Thank you >> > >> > > > > > > > > >> > >> > > > > > > > >>_______________________________________________ >> > >> > > > > > > > >>pmacct-discussion mailing list >> > >> > > > > > > > >>http://www.pmacct.net/#mailinglists >> > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > >_______________________________________________ >> > >> > > > > > > > >pmacct-discussion mailing list >> > >> > > > > > > > >http://www.pmacct.net/#mailinglists >> > >> > > > > > > > >> > >> > > > > > > > -- >> > >> > > > > > > > Nikola >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > -- >> > >> > > > > > Nikola >> > >> > > > > >> > >> > > > > _______________________________________________ >> > >> > > > > pmacct-discussion mailing list >> > >> > > > > http://www.pmacct.net/#mailinglists >> > >> > > > > >> > >> > > >> > >> > > >> > >> > > >> > >> >> > > >> >
_______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
