The queue fills up because rsyslog is not able to deliver the logs fast enough.
You are sending the logs via TCP (encrypted) so the sending is throttled to the
speed that the receiving system can accept them.
Are you using Splunk as the syslog listener to accept the messages?
Splunk is a very poor performing syslog listener, you would be better of running
rsyslog on the splunk system and writing the files to disk there and then
reading those files into Splunk (frequent log rotation and sinkhole options to
have splunk delete the file after it reads it are your friends, you can also
have rsyslog write to different directories based on the sender
hostname/sourcetype/index and have splunk pick those up from the directory
rather than showing the splunk system as the sending host)
David Lang
On Tue, 5 Mar 2024, Andrew Heath via rsyslog wrote:
Greetings,
I have a rsyslog server we use to store and forward logs to a Splunk
instance, however we keep running into the issue where we hit max queue
size and it writes to disk witch is causing our log partition to fill up, I
have read the docs and added some more worker threads but to no avail. The
system has more resources available but for some reason rsyslog is not
using them to help process and forward logs. I have attached a copy of our
config file for reference.
[1] https://paste.centos.org/view/36386fa1
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
