Just do not enable compression, collect data to ramdisk and move by script on rotation (-x switch, also you can apply compression here) to permanent storage. You will need some RAM (not so much by today's means), but I can't imagine load which will overload nfdump in this case.
22.02.2016 13:54, online264...@telkomsa.net пишет: > Hi > > Some years ago I wrote netflow collector s/w on RHEL3 for dealing with high > volumes of netflow (from multiple routers and interfaces including GigE and > ATM). Pretty simplistic design. Collector process reads UDP packet, adds some > IP header data to it, and writes it (round robin) to a SVR4 message queue. > Several message queues are configured. A reader process reads the message > from its queue and chucks it into a binary file (and rotates files as and > when needed/configured). This way the slow disk I/O is offloaded from the UDP > listener process and done via parallel processes. > > Using this approach I am processing around 30K to 40K flows/sec across 8 > (regionally distributed) collectors, dealing with over 1TB of IP traffic per > day. > > The servers we use as collectors are pretty old and we got new servers as > replacements. Only, my collection s/w on the newer RHEL5/6/7 kernels loose in > excess of 70% of messages send via the SVR4 IPC message queues. Unsure why > exactly. Did some hacking and debugging and it simply seems that these old > message queues are not well supported on newer Linux kernels. > > So I have been looking at Open Source replacement for my old s/w - tried a > number and like nfcapd the most. Well written, and works without an effort.. > > So my question is two fold: > > - any ideas as what the approx UDP/sec process rate of nfcapd on a new server > running RHEL6 or RHEL7 is? > > - has there been any attempts to separate the disk I/O factor from nfcapd via > some kind of threading/forking using IPC (any code laying around I can try > and perhaps add to?) > > > Comments/suggestions/beer (Belgium please?) will be much appreciated. ;-) > > > thanks, > Billy > > ------------------------------------------------------------------------------ > Site24x7 APM Insight: Get Deep Visibility into Application Performance > APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month > Monitor end-to-end web transactions and take corrective actions now > Troubleshoot faster and improve end-user experience. Signup Now! > http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 > _______________________________________________ > Nfdump-discuss mailing list > Nfdump-discuss@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfdump-discuss ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ Nfdump-discuss mailing list Nfdump-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
