Hi Simone, Thanks for your response.
Here is /etc/ntopng/ntopng.conf: --pid=/var/run/ntopng.pid --community --daemon --dns-mode=3 --user=root --interface=eth1 --local-networks="10.20.0.0/14, 10.40.0.0/14, 10.60.0.0/14, 10.80.0.0/13, 10.120.0.0/14" --dump-flows="es;flows;ntopng-%Y.%m.%d;http://localhost:9200/_bulk;"; I agree that this feels like a capacity issue somewhere, but I'm having a hard time figuring out where and what to do about it. The machine that this is running on is: - RAM: 16 GB - CPU: Intel Xeon L5520 Quad-Core 2.26GHz - Storage: 2x 7200 RPM, 1TB hard drives in RAID1 The same machine is running the elasticsearch node that ntopng is trying to write to. However, what seems puzzling, is that none of the metrics of the machine indicate the machine is over-taxed. Thanks again! > ------------------------------ > > Message: 2 > Date: Tue, 5 Jul 2016 10:25:03 +0200 > From: Simone Mainardi <[email protected]> > To: [email protected] > Cc: [email protected] > Subject: Re: [Ntop] ntopng -> elasticsearch - dropped flows > Message-ID: > < > cajcxkcbcdtrgneupxhvycehmn-hnrgauz0jqxa9qzdndbol...@mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hi, > > Could you please share ntopng configuration used? I think your setup > doesn't allow ntopng to be quick enough. Remember that there is one thread > per monitored interface and that thread has to > 1. capture packets / receive flows > 2. handle them > 3. export to ES > > > Simone > > On Wed, Jun 29, 2016 at 12:33 AM, Andris Bjornson <[email protected]> > wrote: > > > Hello, > > > > Hoping to find a little help here after unsuccessfully googling quite a > > bit. > > > > I've successfully setup the latest dev version of ntopng to dump flows > > using --dump-flows into elasticsearch. Elasticsearch is on the same > > machine as ntopng. My total traffic volume to process is about ~130Mbps > > peak. > > > > It works very well, but i think i am losing a lot of flows in the export > > process. > > > > My ntopng log file is rapidly filled (at the rate of ~600 per second) > like > > the following: > > > > 29/Jun/2016 01:26:02 [ElasticSearch.cpp:64] WARNING: [ES] Message > dropped. > > Total messages dropped: 2799026 > > > > However, I don't think this is an elasticsearch capacity problem, because > > I am not seeing the errors in elasticsearch.log that would normally > > accompany elasticsearch running out of capacity. I'm monitoring iostat, > > system load, and elasticsearch performance via marvel - and those all > look > > good. > > > > I'm not sure where to look next for more information about what might be > > causing the "message dropped" logs. > > > > Any help much appreciated! > > > > Andris > > > > > > > > --- > > Andris Bjornson | EveryLayer <http://www.everylayer.com/> > > skype: andris.bjornson > >
_______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
