Apologies for the lack of response. Your response got thrown into my spam
folder. You've been moved out of it.
- Impstats has been set to 30s intervals.
- Threads has been removed and changed back to default. That was from when I
was initially research and assumed (incorrectly) that it was responsible for
moving messages from the queue.
Section Modified in /etc/rsyslog.conf
#### MODULES ####
module(
load="impstats"
interval="30"
severity="7"
resetCounters="on"
log.file="/var/syslog/impstats.log"
log.syslog="off"
)
# The imjournal module bellow is now used as a message source instead of
imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger
command)
$ModLoad imjournal # provides access to the systemd journal
#$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark # provides --MARK-- message capability
# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514
module(load="imudp")
input(type="imudp" port="514" rcvbufSize="256m")
=============
When you talk about setting sysctl for UDP - which settings are you referring
too? These three?
net.ipv4.udp_mem = 381225 508301 762450
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
I had previously set - which I believe are the master ones for general buffers.
I can set the UDP ones too larger sizes as well. Would increasing the buffer
size in rsyslog from 256m to 512m have any impact?
net.core.rmem_max = 256m
net.core.rmem_default = 32m
The syslog server is outside of it's high load times right now. It'll be ~12h
until it's back under a full load with the various syslog clients reporting.
What do you mean by pstats?
Thread Information
in:udp and in:tcp @ 1-2%.
In_syslog.rb (omsagent) spikes between 5-25%
output.rb (omsagent) is usually low, but will also spike to 25% when dumping
logs back to Microsoft.
Michael Redbourne
-----Original Message-----
From: David Lang <da...@lang.hm>
Sent: Monday, November 14, 2022 9:52 AM
To: Redbourne,Michael via rsyslog <rsyslog@lists.adiscon.com>
Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com>
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
16 threads for UDP receive is very incorrect. Rsyslog should only need a single
thread, even when receiving messages at a rate of hundreds of thousands of
messages/sec. too many threads will slow rsyslog down and it will use
recvmmesg() to pull multiple udp messages from the OS buffers in a single
syscall.
I would set the impstats to something a bit longer than a dump every second, go
to every 10 to every 60 seconds.
you may want to adjust the OS buffers for UDP to be much larger (sysctl)
can you show us the pstats output for a time under high load where it's losing
some messages? It's important to check that you are not filling the queues and
dropping messages because you aren't processing them fast enough.
Also run top and look at the per-thread data (hit 'H' to see the threads) and
see if any threads are hitting 100% cpu. If they are, then it's an indication
to possibly use additional threads somewhere or refactor the config a bit.
David Lang
________________________________________
This e-mail communication (including any or all attachments) is intended only
for the use of the person or entity to which it is addressed and may contain
confidential and/or privileged material. If you are not the intended recipient
of this e-mail, any use, review, retransmission, distribution, dissemination,
copying, printing, or other use of, or taking of any action in reliance upon
this e-mail, is strictly prohibited. If you have received this e-mail in error,
please contact the sender and delete the original and any copy of this e-mail
and any printout thereof, immediately. If you have any questions or concerns,
please contact our Customer Service Desk at 1-877-274-2349. Your co-operation
is appreciated.
Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à son
destinataire, qu'il soit une personne ou un organisme, et pourrait comporter
des renseignements privilégiés ou confidentiels. Si vous n'êtes pas le
destinataire du courriel, il est interdit d'utiliser, de revoir, de
retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce
courriel, d'agir en vous y fiant ou de vous en servir de toute autre façon. Si
vous avez reçu le présent courriel par erreur, prière de communiquer avec
l'expéditeur et d'éliminer l'original du courriel, ainsi que toute copie
électronique ou imprimée de celui-ci, immédiatement. Si vous avez des questions
ou des préoccupations, veuillez contacter notre centre de service à la
clientèle au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
________________________________________
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
