1. your VM server gets overloaded and stops scheduling your VM for a chunk of
time
*
The VM itself appears fine. I can SSH to it, open 514/tcp w/ telnet from a
remote host, cd, view files, etc.
2. you have a network problem (overload, interupption) that causes the packets
to not get through.
*
It looks like packets are hitting the syslog server. I tested with a VM on
which I generated logs during the impacted period, and could see traffic
incoming into the syslog server using tcpdump. Granted, visually looking at it
I can't guarantee every log is getting through, but I can see syslog traffic
from that host, and no logs ever appear in the any of output destinations.
3. you have a flood of messages that are arriving faster than they can be
processed and your network buffers on your OS are overflowing (from your
description, this doesn't seem likely)
for the network problem, this doesn't have to be on your VM server.
For example, if you are sending the logs via UDP and have a router that gets
overloaded with a nightly backup, it is allowed to drop UDP packets.
*
this is something I can look into, but as I mentioned in #2, I tested w/ a VM
sending syslogs during the impacted times, could see syslog traffic arriving on
the syslog server port, but those logs never appear to get passed along.
So look at what else is going on on the network/systems around the time of your
log outage? is that when a nightly backup runs somewhere? other big batch job?
Additionally, I can share our config (included: /etc/rsyslog.conf,
etc/rsyslog.d/*.conf).
Levi Wilbert
HPC & Linux Systems Administrator
ARCC - Division of Research and Economic Development
Information Technology Ctr 226
1000 E. University Avenue, Laramie, WY 82071-200
________________________________
From: David Lang <da...@lang.hm>
Sent: Wednesday, February 5, 2025 10:55 AM
To: Levi Wilbert via rsyslog <rsyslog@lists.adiscon.com>
Cc: Levi Wilbert <lwilb...@uwyo.edu>
Subject: Re: [rsyslog] Rsyslog Losing Messages
◆ This message was sent from a non-UWYO address. Please exercise caution when
clicking links or opening attachments from external sources.
we would need to see your config to have any idea what's going on (are you
sending via UDP, TCP, RELP, other??)
But there are a few basic things that can go wrong
1. your VM server gets overloaded and stops scheduling your VM for a chunk of
time
2. you have a network problem (overload, interupption) that causes the packets
to not get through.
3. you have a flood of messages that are arriving faster than they can be
processed and your network buffers on your OS are overflowing (from your
description, this doesn't seem likely)
for the network problem, this doesn't have to be on your VM server.
For example, if you are sending the logs via UDP and have a router that gets
overloaded with a nightly backup, it is allowed to drop UDP packets.
So look at what else is going on on the network/systems around the time of your
log outage? is that when a nightly backup runs somewhere? other big batch job?
David Lang
On Wed, 5 Feb 2025, Levi Wilbert via rsyslog wrote:
> Date: Wed, 5 Feb 2025 16:18:15 +0000
> From: Levi Wilbert via rsyslog <rsyslog@lists.adiscon.com>
> To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
> Cc: Levi Wilbert <lwilb...@uwyo.edu>
> Subject: [rsyslog] Rsyslog Losing Messages
>
> Greetings,
>
> We are using Rsyslog on RHEL9.3 to gather logs in an environment of around
> 600 or so servers. All of these servers forward directly to our single syslog
> server, which then forwards the logs along to a mysql db (runs on a separate
> server), ELK stack, and to file locally on the system.
>
> I've noticed at around the same time each night, rsyslog begins dropping most
> of the incoming logs, and there is a gap where almost all logs simply aren't
> recorded.
>
> Network connectivity seems ok, as I am able to connect to the syslog server,
> connect to 514/TCP port (we also use UDP), and I can see logs hitting the
> server using tcpdump on the NIC.
>
> The syslog server is a virtualized server w/ 4 CPU cores and 8G of RAM.
>
> Would anyone have any idea on how to tune rsyslog to avoid these periods of
> log loss?
>
>
>
> Levi Wilbert
> HPC & Linux Systems Administrator
> ARCC - Division of Research and Economic Development
> Information Technology Ctr 226
> 1000 E. University Avenue, Laramie, WY 82071-200
>
>
>
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
>
# Name: 80-elk-syslog.conf
# Date: 2023-12-06
# Desc: Forwards all logs to ELK Logging server
# New syntax
action(
type="omfwd"
queue.filename="srvrFwd3"
queue.maxdiskspace="1g"
queue.saveonshutdown="on"
queue.type="LinkedList"
Target="elk-syslog.domain.edu"
Port="514"
Protocol="tcp"
template="json-template"
)
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# or latest version online at http://www.rsyslog.com/doc/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
#### GLOBAL DIRECTIVES ####
# Log statistics
module(load="impstats"
interval="600"
severity="7"
log.syslog="off"
log.file="/var/log/rsyslog-stats"
)
# to actually gather the data:
#syslog.=debug /var/log/rsyslog-stats
# Where to place auxiliary files
global(workDirectory="/var/lib/rsyslog")
# Use default timestamp format
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat")
#### MODULES ####
module(load="imuxsock" # provides support for local system logging (e.g. via
logger command)
SysSock.Use="off") # Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
module(load="imjournal" # provides access to the systemd journal
UsePid="system" # PID nummber is retrieved as the ID of the process the
journal entry originates from
StateFile="imjournal.state") # File to store the position in the journal
module(load="ompgsql") # postgres plugin
module(load="ommysql") # mysql/mariadb plugin
# Include all config files in /etc/rsyslog.d/
include(file="/etc/rsyslog.d/*.conf" mode="optional")
# Provides UDP syslog reception
# for parameters see http://www.rsyslog.com/doc/imudp.html
module(load="imudp" BatchSize="128" threads="4" ) # needs to be done just once
input(type="imudp" port="514" )
# Provides TCP syslog reception
# for parameters see http://www.rsyslog.com/doc/imtcp.html
module(load="imtcp") # needs to be done just once
input(type="imtcp" port="514")
#module(load="imptcp" threads="4" )
#input(type="imptcp" port="514")
#### RULES ####
# Only perform the following filters for the local host
if ($hostname == "syslog-server1") then {
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none
/var/log/messages
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.*
-/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
local7.*
/var/log/boot.log
}
# ### sample forwarding rule ###
#action(type="omfwd"
# # An on-disk queue is created for this action. If the remote host is
# # down, messages are spooled to disk and sent when it is up again.
#queue.filename="fwdRule1" # unique name prefix for spool files
#queue.maxdiskspace="1g" # 1gb space limit (use as much as possible)
#queue.saveonshutdown="on" # save messages to disk on shutdown
#queue.type="LinkedList" # run asynchronously
#action.resumeRetryCount="-1" # infinite retries if host is down
# # Remote Logging (we use TCP for reliable delivery)
# # remote_host is: name/ip, e.g. 192.168.0.1, port optional e.g. 10514
#Target="remote_host" Port="XXX" Protocol="tcp")
# Standard Setup
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName syslogdb1 # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
$ActionQueueHighWatermark 51200000 # Switch to disk mode when queue =>
~4.76GiB ( 5120000000 x 512 bytes avg msg size)
$ActionQueueLowWatermark 25600000 # Exit disk mode when queue < ~1.86GiB
$ActionQueueMaxDiskSpace 30G
# template
$template tpl,"insert into SystemEvents (Message, Facility, FromHost, Priority,
DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag) values ('%msg%',
%syslogfacility%, '%HOSTNAME%', %syslogpriority%,
'%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%,
'%syslogtag%')",SQL
# certificate files
$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/syslogdb1_2024-05-08.crt
# make gtls driver the default
$DefaultNetStreamDriver gtls
# TLS stuff
$ActionSendStreamDriverMode 1 # run driver in TLS-only mode
$ActionSendStreamDriverAuthMode x509/name
#$ActionSendStreamDriverPermittedPeer syslog-db1.domain.edu
$ActionSendStreamDriverPermittedPeer 10.10.11.22 # syslog-db1 as of
2024-03-04
# Send syslogs w/ severity WARN (4) thru EMERG (0) to MariaDB on syslog-db1
(10.10.11.22)
if ( $syslogseverity <= 4 ) then {
action(type="ommysql" server="10.10.11.22" serverport="3306"
db="Syslog" uid="rsyslog" pwd="<password>"
MySQLConfig.File="/etc/my.cnf.d/mariadb-client-tls.cnf"
MySQLConfig.Section="client-mariadb")
}
# Name: 20-remote-logging.conf
# Date: 2023-12-06
# Desc: This file processes remote rsyslog client logs and filters logs into
# their own discrete folders, without affecting the local host.
# Create templates to define log locations
template(name="host_messages" type="string"
string="/var/log/rsyslog/%HOSTNAME%/messages")
template(name="host_secure" type="string"
string="/var/log/rsyslog/%HOSTNAME%/secure")
template(name="host_maillog" type="string"
string="/var/log/rsyslog/%HOSTNAME%/maillog")
template(name="host_cron" type="string"
string="/var/log/rsyslog/%HOSTNAME%/cron")
template(name="host_spooler" type="string"
string="/var/log/rsyslog/%HOSTNAME%/spooler")
template(name="host_bootlog" type="string"
string="/var/log/rsyslog/%HOSTNAME%/boot.log")
#### RULES ####
# Match all logs from hosts that are not the local host
if ($hostname != "syslog-server1") then {
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none ?host_messages
# The authpriv file has restricted access.
authpriv.* ?host_secure
# Log all the mail messages in one place.
mail.* -?host_maillog
# Log cron stuff
cron.* ?host_cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit ?host_spooler
# Save boot messages also to boot.log
local7.* ?host_bootlog
}
template(name="json-template"
type="list") {
constant(value="{")
constant(value="\"@timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"@version\":\"1")
constant(value="\",\"message\":\"") property(name="msg" format="json")
constant(value="\",\"sysloghost\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"programname\":\"") property(name="programname")
constant(value="\",\"procid\":\"") property(name="procid")
constant(value="\"}\n")
}
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
