we would need to see your config to have any idea what's going on (are you
sending via UDP, TCP, RELP, other??)
But there are a few basic things that can go wrong
1. your VM server gets overloaded and stops scheduling your VM for a chunk of
time
2. you have a network problem (overload, interupption) that causes the packets
to not get through.
3. you have a flood of messages that are arriving faster than they can be
processed and your network buffers on your OS are overflowing (from your
description, this doesn't seem likely)
for the network problem, this doesn't have to be on your VM server.
For example, if you are sending the logs via UDP and have a router that gets
overloaded with a nightly backup, it is allowed to drop UDP packets.
So look at what else is going on on the network/systems around the time of your
log outage? is that when a nightly backup runs somewhere? other big batch job?
David Lang
On Wed, 5 Feb 2025, Levi Wilbert via rsyslog wrote:
Date: Wed, 5 Feb 2025 16:18:15 +0000
From: Levi Wilbert via rsyslog <rsyslog@lists.adiscon.com>
To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
Cc: Levi Wilbert <lwilb...@uwyo.edu>
Subject: [rsyslog] Rsyslog Losing Messages
Greetings,
We are using Rsyslog on RHEL9.3 to gather logs in an environment of around 600
or so servers. All of these servers forward directly to our single syslog
server, which then forwards the logs along to a mysql db (runs on a separate
server), ELK stack, and to file locally on the system.
I've noticed at around the same time each night, rsyslog begins dropping most
of the incoming logs, and there is a gap where almost all logs simply aren't
recorded.
Network connectivity seems ok, as I am able to connect to the syslog server,
connect to 514/TCP port (we also use UDP), and I can see logs hitting the
server using tcpdump on the NIC.
The syslog server is a virtualized server w/ 4 CPU cores and 8G of RAM.
Would anyone have any idea on how to tune rsyslog to avoid these periods of log
loss?
Levi Wilbert
HPC & Linux Systems Administrator
ARCC - Division of Research and Economic Development
Information Technology Ctr 226
1000 E. University Avenue, Laramie, WY 82071-200
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
