Hi Some years ago I wrote netflow collector s/w on RHEL3 for dealing with high volumes of netflow (from multiple routers and interfaces including GigE and ATM). Pretty simplistic design. Collector process reads UDP packet, adds some IP header data to it, and writes it (round robin) to a SVR4 message queue. Several message queues are configured. A reader process reads the message from its queue and chucks it into a binary file (and rotates files as and when needed/configured). This way the slow disk I/O is offloaded from the UDP listener process and done via parallel processes.
Using this approach I am processing around 30K to 40K flows/sec across 8 (regionally distributed) collectors, dealing with over 1TB of IP traffic per day. The servers we use as collectors are pretty old and we got new servers as replacements. Only, my collection s/w on the newer RHEL5/6/7 kernels loose in excess of 70% of messages send via the SVR4 IPC message queues. Unsure why exactly. Did some hacking and debugging and it simply seems that these old message queues are not well supported on newer Linux kernels. So I have been looking at Open Source replacement for my old s/w - tried a number and like nfcapd the most. Well written, and works without an effort.. So my question is two fold: - any ideas as what the approx UDP/sec process rate of nfcapd on a new server running RHEL6 or RHEL7 is? - has there been any attempts to separate the disk I/O factor from nfcapd via some kind of threading/forking using IPC (any code laying around I can try and perhaps add to?) Comments/suggestions/beer (Belgium please?) will be much appreciated. ;-) thanks, Billy ------------------------------------------------------------------------------ Site24x7 APM Insight: Get Deep Visibility into Application Performance APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month Monitor end-to-end web transactions and take corrective actions now Troubleshoot faster and improve end-user experience. Signup Now! http://pubads.g.doubleclick.net/gampad/clk?id=272487151&iu=/4140 _______________________________________________ Nfdump-discuss mailing list Nfdump-discuss@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfdump-discuss
