16 threads for UDP receive is very incorrect. Rsyslog should only need a single
thread, even when receiving messages at a rate of hundreds of thousands of
messages/sec. too many threads will slow rsyslog down and it will use
recvmmesg() to pull multiple udp messages from the OS buffers in a single
syscall.
I would set the impstats to something a bit longer than a dump every second, go
to every 10 to every 60 seconds.
you may want to adjust the OS buffers for UDP to be much larger (sysctl)
can you show us the pstats output for a time under high load where it's losing
some messages? It's important to check that you are not filling the queues and
dropping messages because you aren't processing them fast enough.
Also run top and look at the per-thread data (hit 'H' to see the threads) and
see if any threads are hitting 100% cpu. If they are, then it's an indication to
possibly use additional threads somewhere or refactor the config a bit.
David Lang
Hey folks,
Hoping someone has the expertise to help me out here. We have a Syslog server
running CentOS 7.9, kernel 5.18.5. It's acting as the centralized point for
Syslog (TCP + UDP) ingestion for 100+ Syslog devices. Something is causing a
what I think is the kernel (udp_queue_rcv_one_skb) to drop 10k+ packets in a
single go, intermittently. I expect given the nature of UDP in general to lose
some packets, but 5-10% of inbound packets is excessive, and I think,
avoidable. This is our only server experiencing issues - though it's also the
highest processing one by a significant margin. Dropwatch's output occurred in
the span of 1-2 seconds. The security-config-omsagent.conf file is the rsyslog
sub configuration for Forwarding to Microsoft Sentinel. That config has been
slightly modified to remove actual subnet ranges. I've removed all local disk
logging.
>From my research online, this appears to be a queue issue. I've tried fixing
>it using the rsyslog documentation, but evidently haven't made it very far.
>Any thoughts or obvious mistakes?
Cheers,
Mike
System Config
CPUs: 8
RAM: 16GB
Disk Space: 2x 128GB disks (OS + DATA)
net.core.rmem_default = 33554432
net.core.rmem_max = 268435456
net.ipv4.tcp_rmem = 4096 131072 6291456
net.ipv4.udp_rmem_min = 4096
Dropwatch -l kas
[cid:image001.png@01D8F80B.B4A87FE0]
/etc/rsyslog.conf
# rsyslog configuration file
# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html
global(net.enableDNS="off")
#### MODULES ####
module(
load="impstats"
interval="1"
severity="7"
resetCounters="on"
log.file="/var/syslog/impstats.log"
log.syslog="off"
)
# The imjournal module bellow is now used as a message source instead of
imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger
command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
module(load="imudp" threads="16")
input(type="imudp" port="514" rcvbufSize="256m")
# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514
#### Templates ####
$template RemoteIP,"/var/syslog/%FROMHOST-IP%.log"
#$template NetApp,"%timestamp% %fromhost-ip% %msg%\n"
#$template Ubiquiti,"%msg% %fromhost-ip% Mystic-Ubiquiti\n"
#### GLOBAL DIRECTIVES ####
# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog
# Use default timestamp format
#$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$ActionFileDefaultTemplate RSYSLOG_SyslogProtocol23Format
# File syncing capability is disabled by default. This feature is usually not
required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on
# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf
# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
$OmitLocalLogging on
# File to store the position in the journal
$IMJournalStateFile imjournal.state
# Performance Tuning #
$ActionQueueWorkerThreads 2000
$ActionQueueWorkerThreadMinimumMessages 1000
$ActionQueueSize 1000000
$ActionQueueDiscardMark 800000
$ActionQueueHighWaterMark 600000
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail authpriv, cron)
# Dont log private authentication messages!
#*.*;mail.none;authpriv.none;cron.none ?RemoteIP
# The authpriv file has restricted access.
authpriv.* /var/log/secure
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
# local7.* /var/syslog/boot.log
/etc/rsyslog.d/security-config-omsagent.conf
# [Firewall Log Filtering] #
:msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.2.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.3.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.5.[0-9]+.[0-9]+)" stop
:msg, ereregex, "(1.6.1[6-9].[0-9]+)" stop
:msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop
:msg, ereregex, "(1.8.68.[0-9]+)" stop
:msg, ereregex, "(1.9.69.[0-9]+)" stop
:msg, ereregex, "(1.10.82.[0-9]+)" stop
:msg, ereregex, "(IP multicast routing failed)" stop
:msg, ereregex, "(TCP_7680)" stop
if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226
& stop
if $rawmsg contains "infobloxgridmstr" then @127.0.0.1:25224
& stop
local0.info @127.0.0.1:25224
& stop
local1.info @127.0.0.1:25224
& stop
local2.info @127.0.0.1:25224
& stop
local3.info @127.0.0.1:25224
& stop
local4.info @127.0.0.1:25224
& stop
local5.info @127.0.0.1:25224
& stop
local6.info @127.0.0.1:25224
& stop
local7.info @127.0.0.1:25224
& stop
auth.* @127.0.0.1:25224
& stop
authpriv.* @127.0.0.1:25224
& stop
daemon.info @127.0.0.1:25224
& stop
syslog.* @127.0.0.1:25224
& stop
ftp.*<ftp://ftp.*> @127.0.0.1:25224
& stop
user.* @127.0.0.1:25224
& stop
[cid:image002.jpg@01D8F80B.B4A87FE0]
Michael Redbourne (BCS)
Pronouns: he/him
Senior Security Analyst
Main (C): 1-506-230-3071
VOIP : 1-506-606-0384
Service Desk: 1-877-274-2349
www.bulletproofsi.com<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Furldefense.com%2Fv3%2F__http%3A%2Fwww.bulletproofsi.com%2F__%3B!!NFWRZ6kECLqu!pxjMi1iqf_xCGX9L_kgynzgYHwxIacBtBbFWe63ZKWM7Mwo1M0-12T9CrPB_dAqlKGcyX6wHnrGaPpRliSwsShf6YtO2onamhERr%24&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7C9ebf1cab2a5041e45f2108da8ffc7cd0%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C637980611840534929%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=k5y21GBCF2DAfVgRxUUfVsTifx7G036ytfI0%2BPolIgw%3D&reserved=0>
________________________________________
This e-mail communication (including any or all attachments) is intended only
for the use of the person or entity to which it is addressed and may contain
confidential and/or privileged material. If you are not the intended recipient
of this e-mail, any use, review, retransmission, distribution, dissemination,
copying, printing, or other use of, or taking of any action in reliance upon
this e-mail, is strictly prohibited. If you have received this e-mail in error,
please contact the sender and delete the original and any copy of this e-mail
and any printout thereof, immediately. If you have any questions or concerns,
please contact our Customer Service Desk at 1-877-274-2349. Your co-operation
is appreciated.
Le pr?sent courriel (y compris toute pi?ce jointe) s'adresse uniquement ? son
destinataire, qu'il soit une personne ou un organisme, et pourrait comporter
des renseignements privil?gi?s ou confidentiels. Si vous n'?tes pas le
destinataire du courriel, il est interdit d'utiliser, de revoir, de
retransmettre, de distribuer, de diss?miner, de copier ou d'imprimer ce
courriel, d'agir en vous y fiant ou de vous en servir de toute autre fa?on. Si
vous avez re?u le pr?sent courriel par erreur, pri?re de communiquer avec
l'exp?diteur et d'?liminer l'original du courriel, ainsi que toute copie
?lectronique ou imprim?e de celui-ci, imm?diatement. Si vous avez des questions
ou des pr?occupations, veuillez contacter notre centre de service ? la
client?le au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
________________________________________
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
