We also have another report of the same in https://github.com/zeek/zeek/issues/1856 <https://github.com/zeek/zeek/issues/1856>. Is it possible for you to rebuild with jemalloc support and run the jemalloc profiling plugin on your logger node? That should give more information about what’s causing the bloat. We can use that issue to discuss more in depth what’s going on with it, if that’s easier than email.
Tim > On Dec 12, 2021, at 11:15 PM, Dheeraj Gupta <dheeraj.gup...@gmail.com> wrote: > > Hi, > > We have a Zeek node that sees high volumes on working days. Due to our > internal network configuration a lot of connections for our internal DNS > servers are generated by certain endpoints (because our DNS does not resolve > any external domains and certain applications keep repeating the DNS requests > at astronomical rates). The node is a 16 core, 128GB VM and we use ASCII > logger. > > We have observed that under high loads (~40k writes/s), the logger process > starts lagging behind and its memory usage goes up. Once the machine is using > >60% of its memory, Zeek starts dropping packets and a general drop in > performance is observed. Only solution is to restart the zeek process. > > My understanding is that logger is buffering the unwritten lines in memory > and so memory usage is going up. > > To work around this, I split the output files so that all connections to the > DNS server and all DNS requests to high velocity domains are logged to > separate files (conn-noise.log and dns-noise.log). These two files consume > nearly 80% of the disk usage under the current directory (E.g. in 30 minutes > the current directory use is 4.9G out of which these two files use 4.0G). > Doing this, I hoped that any lags would be limited to these two files and I > will lose less data on a restart. Also by using separate threads for heavily > written files, I may be able to get better performance. The idea has worked > partially as lags for other files are generally low now although we do need > to restart zeek if memory usage goes beyond 55%. > > The problem is that I have observed that logger memory usage does not > decrease on its own when the loads reduce (e.g. at night). E.g. If Zeek was > using 40G memory on Friday evening and dns-noise was showing a lag of 1800 > seconds, the memory usage on Monday morning is still 40G although the lag is > only around 1 second. Has anyone experienced anything similar? I am running > Zeek-4.1.1. > > Thanks, > Dheeraj > > -- > zeek mailing list -- z...@lists.zeek.org > To unsubscribe send an email to zeek-le...@lists.zeek.org
_______________________________________________ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
