Good day Looking at the main Q statistics, the size remains mostly constant around 30. The max queue size currently sits at 400. There is also a queue (linked list + disk assisted) configured for the omkafka action, with the size not really going above single digits (and the DA queue stats remain at 0). Also note I completely disabled the omkafka action's queue previously as a test, but that didn't make a difference. There are no other queues.
Kind Regards --- ________________________________ From: Rainer Gerhards <rgerha...@hq.adiscon.com> Sent: 06 December 2023 17:05 To: rsyslog-users <rsyslog@lists.adiscon.com> Cc: Adriaan de Waal <adriaan.dewaal@nymbis.cloud> Subject: Re: [rsyslog] Memory Leak? Look at the queue sizes in impstats. Are they ever-increasing? Rainer El mié, 6 dic 2023 a las 14:30, Adriaan de Waal via rsyslog (<rsyslog@lists.adiscon.com>) escribió: > > Good day > > I am trying to diagnose and resolve an issue whereby the memory consumed by > the rsyslog daemon increases linearly over time. This continues until it > consumes most of the memory (including swap) of the system and the service > has to be restarted to free up memory. There are two servers with identical > configurations. What I noticed is that the server receiving a higher volume > of messages also consumes memory at a higher rate. In other word it appears > as if the message rate, or message volume, is directly proportional to the > rate at which memory is consumed. > > Below is the version information for the rsyslogd daemon: > rsyslogd 8.2310.0 (aka 2023.10) compiled with: > PLATFORM: x86_64-pc-linux-gnu > PLATFORM (lsb_release -d): > FEATURE_REGEXP: Yes > GSSAPI Kerberos 5 support: No > FEATURE_DEBUG (debug build, slow code): No > 32bit Atomic operations supported: Yes > 64bit Atomic operations supported: Yes > memory allocator: system default > Runtime Instrumentation (slow code): No > uuid support: Yes > systemd support: Yes > Config file: /etc/rsyslog.conf > PID file: /var/run/rsyslogd.pid > Number of Bits in RainerScript integers: 64 > > It is running on Debian 12 servers. > > To provide you with more background detail, initially I configured three > listeners: one UDP (port 514), one TCP (port 514) and one TLS (port 6514). A > single system was configured to push logs to the TLS port and that worked > well (no increase in memory usage over time). Recently I added another UDP > listener (port 10514) and started configured a number of systems to push > their logs to this port, but since then I've observed the described gradual > memory increase. > > This new listener is configured as follows: A ruleset was created and bound > to this listener (the ruleset doesn't have its own queue). The ruleset first > runs the mmutf8fix action then calls a different ruleset (named "normalise"), > which normalises the data (just sets specific variables that is later used in > a template to construct a JSON object). After the call to the "normalise" > ruleset returns, a mmnormalize action is performed and some additional > variables are set. Lastly the ruleset (the one bound to the listener) then > calls yet another ruleset (named "kafka_output"), which is used to construct > a JSON object from the various variables and uses the omkafka action to push > this to a Kafka cluster. > > The flow of the above can be visualised as: > Source -> Syslog Server [10514/UDP] -> [listener ruleset] -> [normalise > ruleset] -> [kafka_output ruleset] > > It should also be noted the original listeners are configured in much the > same way, apart from having calls to even more rulesets. I haven't tested if > the UDP listener on port 514 exhibits the same behaviour (it isn't currently > being used). > > This rsyslog daemon is also used to capture locally generated logs and the > statistics (impstats) module is also loaded. > > What can I do to troubleshoot what's causing this "memory leak"? > > Kind Regards > --- > > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
