Ok, one thing I see is that you have a dynafilecachesize that's too small
Tue Feb 4 22:10:36 2025: dynafile cache host_messages: origin=omfile requests=4316034 level0=3811573 missed=380153 evicted=380143 maxused=10 closetimeouts=0This has the system spending a lot of time closing a file, opening a new one, writing a log to it, and then repeating the process a LOT. That can drastically slow down your log processing. Since maxused is 10, that indicates defaults. Set the cach size to something larger than the number of files you expect to have open
another one is Tue Feb 4 22:30:36 2025: action-7-ommysql: origin=core.action processed=2270931 failed=0 suspended=0 suspended.duration=0 resumed=0It looks like many of your outputs are not seeing the processed number increase, so something is blocking, causing the queues to fill and then everything stops.
take a look at https://www.rsyslog.com/doc/whitepapers/queues_analogy.html and consider adding some queues to some of the actions that could block if the remote system pauses. you do have a queue on action 8 (looks like the elk forward) and that queue is not filling up
try setting the pstats to output more frequently than once every 10 min and have it running before the time that logs stop processing, it may give a little more detail. Look for a time when the queue size goes from near zero (normal operations) to something much larger, hopefully it happens slowly enough for us to see some stats.
I surprised we aren't seeing any failed or syspended (other than open failed for inbound tcp connections)
I don't think this section is doing what you expect: $ActionQueueType LinkedList # use asynchronous processing $ActionQueueFileName syslogdb1 # set file name, also enables disk mode $ActionResumeRetryCount -1 # infinite retries on insert failure $ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down$ActionQueueHighWatermark 51200000 # Switch to disk mode when queue => ~4.76GiB ( 5120000000 x 512 bytes avg msg size)
$ActionQueueLowWatermark 25600000 # Exit disk mode when queue < ~1.86GiB $ActionQueueMaxDiskSpace 30GThis is setting up a queue for the next action in the config that is not using the action() format (I think this is the writes to /var/log/rsyslog/%HOSTNAME%/messages but it's really hard to be sure). This is one of the biggest reasons why this format is depreiciated, it's hard even for experts to keep track of what's happening, especially when the config is spread across multiple files
David Lang On Wed, 5 Feb 2025, Levi Wilbert wrote:
Date: Wed, 5 Feb 2025 20:52:18 +0000 From: Levi Wilbert <lwilb...@uwyo.edu> To: David Lang <da...@lang.hm> Cc: Levi Wilbert via rsyslog <rsyslog@lists.adiscon.com> Subject: Re: [rsyslog] Rsyslog Losing Messages In fact, I've enabled rsyslog-stats already. It does look like the queue has been getting hammered, so much that the main queue is full, with millions of messages enqueued. I've attached an hours worth of rsyslog-stats during the impacted time. Levi Wilbert HPC & Linux Systems Administrator ARCC - Division of Research and Economic Development Information Technology Ctr 226 1000 E. University Avenue, Laramie, WY 82071-200 ________________________________ From: David Lang <da...@lang.hm> Sent: Wednesday, February 5, 2025 1:39 PM To: Levi Wilbert <lwilb...@uwyo.edu> Cc: David Lang <da...@lang.hm>; Levi Wilbert via rsyslog <rsyslog@lists.adiscon.com> Subject: Re: [rsyslog] Rsyslog Losing Messages I forgot to mention the other reason you may not see logs, if an output blocks, all log processing halts (unless you have action/ruleset queues in place) In your configs, you send logs to elasticsearch using omfwd over TCP, if eleasticsearch can't handle the message, all log processing will pause. The logs will get queued up (to the limits of your queues) and be delivered later. But if your queues get overrun, you may lose logs. I would suggest that you enable impstats, writing to a local file, and see what it shows during the time that log processing isn't happening. David Lang On Wed, 5 Feb 2025, Levi Wilbert wrote:Date: Wed, 5 Feb 2025 20:18:34 +0000 From: Levi Wilbert <lwilb...@uwyo.edu> To: David Lang <da...@lang.hm>, Levi Wilbert via rsyslog <rsyslog@lists.adiscon.com> Subject: Re: [rsyslog] Rsyslog Losing Messages 1. your VM server gets overloaded and stops scheduling your VM for a chunk of time * The VM itself appears fine. I can SSH to it, open 514/tcp w/ telnet from a remote host, cd, view files, etc. 2. you have a network problem (overload, interupption) that causes the packets to not get through. * It looks like packets are hitting the syslog server. I tested with a VM on which I generated logs during the impacted period, and could see traffic incoming into the syslog server using tcpdump. Granted, visually looking at it I can't guarantee every log is getting through, but I can see syslog traffic from that host, and no logs ever appear in the any of output destinations. 3. you have a flood of messages that are arriving faster than they can be processed and your network buffers on your OS are overflowing (from your description, this doesn't seem likely) for the network problem, this doesn't have to be on your VM server. For example, if you are sending the logs via UDP and have a router that gets overloaded with a nightly backup, it is allowed to drop UDP packets. * this is something I can look into, but as I mentioned in #2, I tested w/ a VM sending syslogs during the impacted times, could see syslog traffic arriving on the syslog server port, but those logs never appear to get passed along. So look at what else is going on on the network/systems around the time of your log outage? is that when a nightly backup runs somewhere? other big batch job? Additionally, I can share our config (included: /etc/rsyslog.conf, etc/rsyslog.d/*.conf). Levi Wilbert HPC & Linux Systems Administrator ARCC - Division of Research and Economic Development Information Technology Ctr 226 1000 E. University Avenue, Laramie, WY 82071-200 ________________________________ From: David Lang <da...@lang.hm> Sent: Wednesday, February 5, 2025 10:55 AM To: Levi Wilbert via rsyslog <rsyslog@lists.adiscon.com> Cc: Levi Wilbert <lwilb...@uwyo.edu> Subject: Re: [rsyslog] Rsyslog Losing Messages ◆ This message was sent from a non-UWYO address. Please exercise caution when clicking links or opening attachments from external sources. we would need to see your config to have any idea what's going on (are you sending via UDP, TCP, RELP, other??) But there are a few basic things that can go wrong 1. your VM server gets overloaded and stops scheduling your VM for a chunk of time 2. you have a network problem (overload, interupption) that causes the packets to not get through. 3. you have a flood of messages that are arriving faster than they can be processed and your network buffers on your OS are overflowing (from your description, this doesn't seem likely) for the network problem, this doesn't have to be on your VM server. For example, if you are sending the logs via UDP and have a router that gets overloaded with a nightly backup, it is allowed to drop UDP packets. So look at what else is going on on the network/systems around the time of your log outage? is that when a nightly backup runs somewhere? other big batch job? David Lang On Wed, 5 Feb 2025, Levi Wilbert via rsyslog wrote:Date: Wed, 5 Feb 2025 16:18:15 +0000 From: Levi Wilbert via rsyslog <rsyslog@lists.adiscon.com> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com> Cc: Levi Wilbert <lwilb...@uwyo.edu> Subject: [rsyslog] Rsyslog Losing Messages Greetings, We are using Rsyslog on RHEL9.3 to gather logs in an environment of around 600 or so servers. All of these servers forward directly to our single syslog server, which then forwards the logs along to a mysql db (runs on a separate server), ELK stack, and to file locally on the system. I've noticed at around the same time each night, rsyslog begins dropping most of the incoming logs, and there is a gap where almost all logs simply aren't recorded. Network connectivity seems ok, as I am able to connect to the syslog server, connect to 514/TCP port (we also use UDP), and I can see logs hitting the server using tcpdump on the NIC. The syslog server is a virtualized server w/ 4 CPU cores and 8G of RAM. Would anyone have any idea on how to tune rsyslog to avoid these periods of log loss? Levi Wilbert HPC & Linux Systems Administrator ARCC - Division of Research and Economic Development Information Technology Ctr 226 1000 E. University Avenue, Laramie, WY 82071-200 _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
rsyslog-stats
Description: rsyslog-stats
_______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
