On Tue, 15 Nov 2022, Redbourne,Michael wrote:
Apologies for the lack of response. Your response got thrown into my spam folder. You've been moved out of it.- Impstats has been set to 30s intervals. - Threads has been removed and changed back to default. That was from when I was initially research and assumed (incorrectly) that it was responsible for moving messages from the queue. Section Modified in /etc/rsyslog.conf #### MODULES #### module( load="impstats" interval="30" severity="7" resetCounters="on" log.file="/var/syslog/impstats.log" log.syslog="off" ) # The imjournal module bellow is now used as a message source instead of imuxsock. $ModLoad imuxsock # provides support for local system logging (e.g. via logger command) $ModLoad imjournal # provides access to the systemd journal #$ModLoad imklog # reads kernel messages (the same are read from journald) #$ModLoad immark # provides --MARK-- message capability # Provides UDP syslog reception #$ModLoad imudp #$UDPServerRun 514 module(load="imudp") input(type="imudp" port="514" rcvbufSize="256m") ============= When you talk about setting sysctl for UDP - which settings are you referring too? These three? net.ipv4.udp_mem = 381225 508301 762450 net.ipv4.udp_rmem_min = 4096 net.ipv4.udp_wmem_min = 4096
increase udp_rmem should improve things. There is a stat somewhere that records how many udp packets are dropped at the OS level (I know I've seen it in the output of monitoring tools, so it has to be under /proc somewhere, but I'm not sure where)
I had previously set - which I believe are the master ones for general buffers. I can set the UDP ones too larger sizes as well. Would increasing the buffer size in rsyslog from 256m to 512m have any impact? net.core.rmem_max = 256m net.core.rmem_default = 32m The syslog server is outside of it's high load times right now. It'll be ~12h until it's back under a full load with the various syslog clients reporting. What do you mean by pstats? Thread Information in:udp and in:tcp @ 1-2%.
I expect these to stay fairly low, which would indicate that the problem isn't in receiving the messages.
In_syslog.rb (omsagent) spikes between 5-25% output.rb (omsagent) is usually low, but will also spike to 25% when dumping logs back to Microsoft.
I expect that as load climbs, this is going to be where you run into grief, first this maxing out the cpu, then the queue size climbing as it can't process messages fast enough, and when the queue is full, then the OS buffers will fill up until they are full, at which point packets (logs) get dropped.
let's see if this is the case before doing more tinkering. Depending on how long the peaks last, increasing the queue/buffer sizes may let you ride out the peak, but if the peak lasts too long, you won't have enough ram to do so.
David Lang
Michael Redbourne -----Original Message----- From: David Lang <da...@lang.hm> Sent: Monday, November 14, 2022 9:52 AM To: Redbourne,Michael via rsyslog <rsyslog@lists.adiscon.com> Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events 16 threads for UDP receive is very incorrect. Rsyslog should only need a single thread, even when receiving messages at a rate of hundreds of thousands of messages/sec. too many threads will slow rsyslog down and it will use recvmmesg() to pull multiple udp messages from the OS buffers in a single syscall. I would set the impstats to something a bit longer than a dump every second, go to every 10 to every 60 seconds. you may want to adjust the OS buffers for UDP to be much larger (sysctl) can you show us the pstats output for a time under high load where it's losing some messages? It's important to check that you are not filling the queues and dropping messages because you aren't processing them fast enough. Also run top and look at the per-thread data (hit 'H' to see the threads) and see if any threads are hitting 100% cpu. If they are, then it's an indication to possibly use additional threads somewhere or refactor the config a bit. David Lang ________________________________________ This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated. Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privilégiés ou confidentiels. Si vous n'êtes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre façon. Si vous avez reçu le présent courriel par erreur, prière de communiquer avec l'expéditeur et d'éliminer l'original du courriel, ainsi que toute copie électronique ou imprimée de celui-ci, immédiatement. Si vous avez des questions ou des préoccupations, veuillez contacter notre centre de service à la clientèle au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration. ________________________________________
_______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
