Dheeraj, Whats the OS on the host running the logger node ?
I've seen same issues of bloating logger node with tcmalloc on FreeBSD. Mine crashes after 180+GB - takes a couple weeks to do so! Since last week I have been running with jemalloc and things seem better - but lets see I may risk speaking sooner here. (On a side note) I've been trying jemalloc and few hiccups (struggles) related to building zeek with jemalloc on FreeBSD: 1) fix for building zeek + jemalloc + FreeBSD: https://github.com/zeek/zeek/pull/1878 and, 2) Fix for building jemalloc itself on FreeBSD to --enable-profiling We've (Craig leres) got out a patch to be able to do so as well. (2) is mostly needed so that I can build zeek against jemalloc with --enable-profiling to run Justin's zeekctl jemalloc profiler. Aashish On Tue, Dec 14, 2021 at 10:33:23AM +0530, Dheeraj Gupta wrote: > Thanks for the pointer Tim. > > I will try to run jemalloc profiling and post back on the Github issue. > > - Dheeraj > > On Mon, Dec 13, 2021 at 11:18 PM Tim Wojtulewicz <t...@corelight.com> wrote: > > > We also have another report of the same in > > https://github.com/zeek/zeek/issues/1856. Is it possible for you to > > rebuild with jemalloc support and run the jemalloc profiling plugin on your > > logger node? That should give more information about what’s causing the > > bloat. We can use that issue to discuss more in depth what’s going on with > > it, if that’s easier than email. > > > > Tim > > > > On Dec 12, 2021, at 11:15 PM, Dheeraj Gupta <dheeraj.gup...@gmail.com> > > wrote: > > > > Hi, > > > > We have a Zeek node that sees high volumes on working days. Due to our > > internal network configuration a lot of connections for our internal DNS > > servers are generated by certain endpoints (because our DNS does not > > resolve any external domains and certain applications keep repeating the > > DNS requests at astronomical rates). The node is a 16 core, 128GB VM and we > > use ASCII logger. > > > > We have observed that under high loads (~40k writes/s), the logger process > > starts lagging behind and its memory usage goes up. Once the machine is > > using >60% of its memory, Zeek starts dropping packets and a general drop > > in performance is observed. Only solution is to restart the zeek process. > > > > My understanding is that logger is buffering the unwritten lines in memory > > and so memory usage is going up. > > > > To work around this, I split the output files so that all connections to > > the DNS server and all DNS requests to high velocity domains are logged to > > separate files (conn-noise.log and dns-noise.log). These two files consume > > nearly 80% of the disk usage under the current directory (E.g. in 30 > > minutes the current directory use is 4.9G out of which these two files use > > 4.0G). Doing this, I hoped that any lags would be limited to these two > > files and I will lose less data on a restart. Also by using separate > > threads for heavily written files, I may be able to get better performance. > > The idea has worked partially as lags for other files are generally low now > > although we do need to restart zeek if memory usage goes beyond 55%. > > > > The problem is that I have observed that logger memory usage does not > > decrease on its own when the loads reduce (e.g. at night). E.g. If Zeek was > > using 40G memory on Friday evening and dns-noise was showing a lag of 1800 > > seconds, the memory usage on Monday morning is still 40G although the lag > > is only around 1 second. Has anyone experienced anything similar? I am > > running Zeek-4.1.1. > > > > Thanks, > > Dheeraj > > > > -- > > zeek mailing list -- z...@lists.zeek.org > > To unsubscribe send an email to zeek-le...@lists.zeek.org > > > > > > > _______________________________________________ > zeek-dev mailing list -- zeek-dev@lists.zeek.org > To unsubscribe send an email to zeek-dev-le...@lists.zeek.org _______________________________________________ zeek-dev mailing list -- zeek-dev@lists.zeek.org To unsubscribe send an email to zeek-dev-le...@lists.zeek.org
