The traffic charts enclosed show peaks at around .5 Gbps. I don't know how many flows you are generating but I think they are too much for a single ntopng and a single MySQL instance. MySQL tuning helps but not to the necessary extent. Also, we (as ntopng devs) may improve performances for example by batching insertions and thus avoiding a single INSERT INTO for each flow. This is something we will address in the future. However, I am not sure it be enough to handle the volume of flows you have using a single ntopng and a single database.
Also, I am not sure your data is inaccurate. You get only active hosts from the host pane so, if an host has transmitted TBs of data but is now inactive, it won't show up in the list. Use the report if you want to see historical host stats Simone On Wed, Mar 9, 2016 at 2:54 AM, <[email protected]> wrote: > ntopng 2.3.160204 & nprobe 7.3.160204 on Debian 8.3 > 8CPU / 16GB VM > > Using netflow from Cisco 6509 w/nprobe and ntop as collector on network > w/around 150K hosts. nprobe is started as such: > > nprobe -G --zmq "tcp://*:5556" -i none -n none --collector-port 2055 -V 9 > > > Ntop using ntopng.conf w/following parameters: > > -G=/var/tmp/ntopng.pid > -i=tcp://127.0.0.1:5556 > -S=all > -m=[NETWORKS] > -X=550000 > -x=550000 > -F="mysql;localhost;ntopng;flows;user;pass" > > When using MySQL the webui is extremely slow and I see constant writes to > disk, mostly inserts to flowsv4 table in ntopng db. I also am not seeing > accurate info in dashboard graphs & reports however I do see accurate host > information and historical data for ntop interface. I have taken steps to > expand various innodb configurations like buffer pool size, log buffer > size, innodb read/write io threads and there is no difference with > performance. Using various tools to view perf data for i/o I am seeing > contant 2+MB/sec - 10MB/sec disk writes and very high CPU wait percentages. > My VM infrastructure consists of 4 IBM M3 ESXi hosts and a gen 2 XIV SAN so > I'm pretty confident it's not the hardware. > > When I configure ntopng.conf to not use MySQL backend (everything written > to /var/tmp/ntopng) the UI is much more responsive and the dashboard is > accurate however, historical data for both hosts (traffic) and the ntop > interface is inaccurate. I can have ntop running for a week and see ~10TB > of data total for a given network but will not have info for hosts when I > sort on traffic totals. When I select hosts the default view may show a > host that has gigs and gigs more total traffic than any other host but when > I sort on traffic (descending) that host is not represented in the list. My > goal was to use SQL backend to retain that historical data which it does > but at a huge performance cost. > > I've attached a report screencap that shows on the left a typical day > without using MySQL highlighted in yellow. Today I started the day > configured the same (yellow highlight) but switched to MySQL backend (red) > and then back and forth after various tweaks. Historical info is there when > I view ntop interface and traffic for given time frame. I also see all > hosts traffic totals represented seemingly accurately. > > Not sure where to look next so any suggestions are appreciated. > > > _______________________________________________ > Ntop mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop >
_______________________________________________ Ntop mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop
