On 2012-06-11 1026, Paolo Lucente wrote: > Hi Tiernan, > > On Sun, Jun 10, 2012 at 10:54:36PM +1000, Tiernan Messmer wrote: > >> I assume you meant buffer and pipe sizes the other way around? > > Correct. > >> plugin_pipe_size: 10485760 >> plugin_buffer_size: 1048576 >> >> This seems to have fixed the bogus data entries too, although during >> high load the CPU usage skyrockets. With 100MB pipe size and 10MB buffer >> size the CPU usage was about the same. Is there any way to get it down? > > Which process is maxing out the CPU? Also I would suggest to have a look > to Q7 of the FAQS document.
pmacctd: Core Process [default] (pmacctd) I've checked out Q7, polling is the only option I've not tried yet (I need to get around to rebooting) - but from what I understand that would result in high kernel time with a high % interrupts - which at present isn't that high (only a few % under full load). libpcap-mmap is in use (at least I assume that's what the MMAP option in FreeBSD's port tree for pmacct does?) and PF_RING is Linux only from what I can gather. >> A bit of an architectural question but why is such a large buffer needed >> between the core and plugin processes? Where is the aggregation being >> done, plugin side or core side? It seems like the core process is >> matching the filter then passing each packet header individually to the >> plugins incurring copying and requiring such large buffers - I could be >> wrong and I've not studied the internals of pmacct - but could resource >> usage be reduced by doing the aggregation and packet/byte counting in >> the core process then handing those figures off to the plugin every >> second or so help in high traffic instances? Possibly save moving around >> data in memory as much. > > Aggregation is done at the plugins. Idea is to have an as much as possible > lean and mean core process (network interface) and capture traffic once and > then have as many views of the same traffic as required: views are achieved > by binding an aggregation method to the defined plugins. With this in mind, and that the core process is what is using almost all of the CPU usage - the only intensive task I can think of the core process doing is moving around packet headers to the plugin processes - which certaintly seems at this stage to be a beefier task than incrementing counters - or is there another task that the core process does as well? > Cheers, > Paolo Cheers for the help, ~Tiernan _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
