ahh, I thought that was a rsyslog thread that could be maxing out a core.
my logging strategy is that everything should get sent to the central syslog server, and only there should it get thrown away (and before you throw it away, consider counting it, the number of times that an uninteresting thing happens can be an interesting thing to track)
per the impstats output you included, you are hitting up to ~240k messages/30 seconds, or around 8k messages/sec. That should not be challenging to rsyslog to receive (I've done over 300k messages/sec), we just need to watch the cost of processing them.
you don't show any times when the queue actually filled up, so messages should not be dropped because of that.
with the new configs, are you sure you are still dropping syslog messages? the proc stats you show below show drops on port hex 6288 (25224 decimal), not on hex 202 (514 decimal)
David Lang On Tue, 15 Nov 2022, Redbourne,Michael wrote:
Date: Tue, 15 Nov 2022 21:55:48 +0000 From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> To: David Lang <da...@lang.hm> Cc: rsyslog-users <rsyslog@lists.adiscon.com> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events I think my best course right now is to tie down and remove what I can from the Syslog Collector side of this. At minimum it'll reduce the amount of work rsyslog has to do. If I can completely remove the regex and contains searches in favour of syslog tags/properties, we'll all be better off and might help clear up what the actual issue is. If you're referencing this:1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1*It's owned by a completely different process (and package) from rsyslog. It's the Microsoft OMS Agent for Linux Package, available here: https://github.com/microsoft/OMS-Agent-for-Linux. Unfortunately, it's not something I can get rid of... yet. I'm waiting on Microsoft to support the newer Azure ARC agent for new builds of CentOS/RHEL 9. Cheers, Mike -----Original Message----- From: David Lang <da...@lang.hm> Sent: Tuesday, November 15, 2022 5:49 PM To: Redbourne,Michael <michael.redbou...@bulletproofsi.com> Cc: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events As I said before, log some of the messages with the template RSYSLOG_DebugFormat and see what you have and how you can filter more efficiently. Rsyslog is very efficient at processing messages, but regex and contains are about the most expensive tests that you can do If you really do need multiple threads to process the log messages, you need to set the queue parameters on the main queue with the global() stanza. I'm not convinced that we've caught the full problem during our sampling yet. As I said, it's probably a good idea to put a queue on the TCP forwarding that you have configured to isolate any issues there from the general processing. @rainer, what is the in_syslog.rb thread that he saw maxing out the cpu? David Lang On Tue, 15 Nov 2022, Redbourne,Michael wrote:Date: Tue, 15 Nov 2022 21:39:45 +0000 From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> To: David Lang <da...@lang.hm> Cc: rsyslog-users <rsyslog@lists.adiscon.com> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events I'm going to reach out to networking folks and see if I can get something better in place, especially around negating logs further up the chain then the syslog collector. (Moreso related to the Checkpoint firewalls - removing the forwarding/logging directly at the FW/MGMT server). I'm hoping if I can drop most of the filtering and replace $rawmsg contains with :fromhost-ip "1.1.1.1" that it'll solve most of my problems. It'll take a day or two before I get them into a call though. If I'm still havcing issues after that, I'll respond back. -----Original Message----- From: David Lang <da...@lang.hm> Sent: Tuesday, November 15, 2022 4:19 PM To: Redbourne,Michael <michael.redbou...@bulletproofsi.com> Cc: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events using the new action() syntax, you can name the actions so they aren't just numbered. starting rsyslog with -o /path/to/file will generate a config file that is the combination of all the included files (as rsyslog actually processes the config), which will make it easier to figure out which action is which. that could be the tcp forwarding of messages, which can stall if the device you are forwarding to can't keep up (that is a good place to have an action queue, but please use the action() syntax so that it's clear what the queue is doing) David Lang On Tue, 15 Nov 2022, Redbourne,Michael wrote:Date: Tue, 15 Nov 2022 20:01:52 +0000 From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> To: David Lang <da...@lang.hm> Cc: rsyslog-users <rsyslog@lists.adiscon.com> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events Tue Nov 15 00:22:18 2022: global: origin=dynstats Tue Nov 15 00:22:18 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0 Tue Nov 15 00:22:18 2022: action 0: origin=core.action processed=1628 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 1: origin=core.action processed=66 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 2: origin=core.action processed=37449 failed=37448 suspended=1 suspended.duration=30 resumed=0 Tue Nov 15 00:22:18 2022: action 3: origin=core.action processed=4582 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 4: origin=core.action processed=5039 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 8: origin=core.action processed=862 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 10: origin=core.action processed=397 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 11: origin=core.action processed=50 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 14: origin=core.action processed=766 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 15: origin=core.action processed=11058 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 17: origin=core.action processed=1099 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 18: origin=core.action processed=12 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: imudp(*:514): origin=imudp submitted=24227 Tue Nov 15 00:22:18 2022: imudp(*:514): origin=imudp submitted=0 Tue Nov 15 00:22:18 2022: imtcp(514): origin=imtcp submitted=68069 Tue Nov 15 00:22:18 2022: resource-usage: origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469 majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702 Tue Nov 15 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161 full=0 discarded.full=0 discarded.nf=0 maxqsize=7033 Tue Nov 15 00:22:18 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0 msgs.received=24227 Tue Nov 15 08:41:43 2022: global: origin=dynstats Tue Nov 15 08:41:43 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0 Tue Nov 15 08:41:43 2022: action 0: origin=core.action processed=1728 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 1: origin=core.action processed=47 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 2: origin=core.action processed=35778 failed=35777 suspended=1 suspended.duration=30 resumed=0 Tue Nov 15 08:41:43 2022: action 3: origin=core.action processed=9249 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 4: origin=core.action processed=2296 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 8: origin=core.action processed=1009 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 10: origin=core.action processed=558 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 11: origin=core.action processed=40 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 14: origin=core.action processed=720 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 15: origin=core.action processed=16593 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 17: origin=core.action processed=359 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 18: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: imudp(*:514): origin=imudp submitted=31257 Tue Nov 15 08:41:43 2022: imudp(*:514): origin=imudp submitted=0 Tue Nov 15 08:41:43 2022: imtcp(514): origin=imtcp submitted=60875 Tue Nov 15 08:41:43 2022: resource-usage: origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884 majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479 Tue Nov 15 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998 full=0 discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 08:41:43 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0 msgs.received=31257 Tue Nov 15 08:42:13 2022: global: origin=dynstats Tue Nov 15 08:42:13 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0 Tue Nov 15 08:42:13 2022: action 0: origin=core.action processed=2181 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 1: origin=core.action processed=57 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 2: origin=core.action processed=20793 failed=137 suspended=0 suspended.duration=30 resumed=0 Tue Nov 15 08:42:13 2022: action 3: origin=core.action processed=7599 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 4: origin=core.action processed=5540 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 8: origin=core.action processed=1120 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 10: origin=core.action processed=157807 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 11: origin=core.action processed=50 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 12: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 13: origin=core.action processed=3 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 14: origin=core.action processed=293 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 15: origin=core.action processed=20243 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 17: origin=core.action processed=866 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 18: origin=core.action processed=10 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: imudp(*:514): origin=imudp submitted=194836 Tue Nov 15 08:42:13 2022: imudp(*:514): origin=imudp submitted=0 Tue Nov 15 08:42:13 2022: imtcp(514): origin=imtcp submitted=34848 Tue Nov 15 08:42:13 2022: resource-usage: origin=impstats utime=5417168 stime=3346362 maxrss=16800 minflt=11196 majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993 Tue Nov 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698 full=0 discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595 called.recvmsg=0 msgs.received=194836 Tue Nov 15 10:15:04 2022: global: origin=dynstats Tue Nov 15 10:15:04 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 ratelimit.numratelimiters=0 Tue Nov 15 10:15:04 2022: action 0: origin=core.action processed=2009 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 1: origin=core.action processed=151 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 2: origin=core.action processed=52035 failed=52034 suspended=1 suspended.duration=60 resumed=0 Tue Nov 15 10:15:04 2022: action 3: origin=core.action processed=19442 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 4: origin=core.action processed=4480 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 5: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 6: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 7: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 8: origin=core.action processed=1125 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 9: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 10: origin=core.action processed=415 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 11: origin=core.action processed=144 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 12: origin=core.action processed=2 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 13: origin=core.action processed=13 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 14: origin=core.action processed=745 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 15: origin=core.action processed=23626 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 16: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 17: origin=core.action processed=682 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 18: origin=core.action processed=45 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 19: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 20: origin=core.action processed=0 failed=0 suspended=0 suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: imudp(*:514): origin=imudp submitted=51446 Tue Nov 15 10:15:04 2022: imudp(*:514): origin=imudp submitted=0 Tue Nov 15 10:15:04 2022: imtcp(514): origin=imtcp submitted=81918 Tue Nov 15 10:15:04 2022: resource-usage: origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979 majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668 Tue Nov 15 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235 full=0 discarded.full=0 discarded.nf=0 maxqsize=6830 Tue Nov 15 10:15:04 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0 msgs.received=51446 -----Original Message----- From: David Lang <da...@lang.hm> Sent: Tuesday, November 15, 2022 3:43 PM To: Redbourne,Michael <michael.redbou...@bulletproofsi.com> Cc: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events you have the impstats module loaded in your config and writing stats out, please post the output of this. David Lang On Tue, 15 Nov 2022, Redbourne,Michael wrote:Date: Tue, 15 Nov 2022 19:38:27 +0000 From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> To: David Lang <da...@lang.hm> Cc: rsyslog-users <rsyslog@lists.adiscon.com> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events I'm still not understanding what you mean by pstats - it's not a package or command available to me. It's apart of Unix from what I can tell. I've placed below the unparsed information form /proc/net/netstat and /proc/net/udp /proc/net/netstat TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected TCPMD5Failure TCPSackShifted TCPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge TCPChallengeACK TCPSYNChallenge TCPFastOpenActive TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash TcpDuplicateDataRehash TCPDSACKRec vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailureTcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0 IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0 /proc/net/udp sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode ref pointer drops 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23514 2 ffff9a4f8b46bf00 0 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 38 0 23513 2 ffff9a4f8b46c380 0 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14102 2 ffff9a4f81f11f80 0 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 14096 2 ffff9a4f81f11200 0 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38099 2 ffff9a4f8398c380 0 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41291 2 ffff9a4f81fc4800 0 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 00000000 982 0 41285 2 ffff9a4f81fc5580 0 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42071 2 ffff9a4f835a7500 0 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38107 2 ffff9a4f8398a880 0 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23974 2 ffff9a4f86392880 0 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38120 2 ffff9a4f8398c800 0 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 23973 2 ffff9a4f86392d00 0 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 39146 2 ffff9a4f863c7980 0 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 41233 2 ffff9a4f81fc2d00 0 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 26070 2 ffff9a4f894df500 0 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 42072 2 ffff9a4f835a6780 0 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 00000000 0 0 38108 2 ffff9a4f8398ad00 0 The ActionQueue for /var/log/secure has been commented out from the configuration. Dropwatch (Cycle 1): 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) 11 drops at skb_release_data+12b (0xffffffff9a7f8cab) 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e) 2 drops at skb_release_data+12b (0xffffffff9a7f8cab) 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f) 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5) 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5) 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d) 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f) 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f) 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d) 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) 6 drops at skb_release_data+12b (0xffffffff9a7f8cab) 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d) 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f) 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) 8 drops at skb_release_data+12b (0xffffffff9a7f8cab) 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d) 7 drops at skb_release_data+12b (0xffffffff9a7f8cab) 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d) 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) 11 drops at skb_release_data+12b (0xffffffff9a7f8cab) 7 drops at skb_release_data+12b (0xffffffff9a7f8cab) Dropwatch (Cycle 2): 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) 11 drops at skb_release_data+12b (0xffffffff937f8cab) 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f) 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5) 14 drops at skb_release_data+12b (0xffffffff937f8cab) 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f) 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d) 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) 6 drops at skb_release_data+12b (0xffffffff937f8cab) 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) 11 drops at skb_release_data+12b (0xffffffff937f8cab) 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f) 1 drops at nf_hook_slow+9d (0xffffffff938ae98d) 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) 10 drops at skb_release_data+12b (0xffffffff937f8cab) 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) 5 drops at skb_release_data+12b (0xffffffff937f8cab) 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) 1 drops at nf_hook_slow+9d (0xffffffff938ae98d) Top -H 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 in_syslog.rb:1* 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 output.rb:140 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main Q:Reg 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 in_syslog.rb:1* 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp /etc/sysctl.conf net.core.rmem_default = 33554432 net.core.rmem_max = 268435456 net.core.wmem_default = 33554432 net.core.wmem_max = 268435456 net.ipv4.tcp_mem = 190611 254150 381222 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304 net.ipv4.udp_mem = 762450 1524900 3049800 net.ipv4.udp_rmem_min = 33554432 net.ipv4.udp_wmem_min = 33554432 -----Original Message----- From: David Lang <da...@lang.hm> Sent: Tuesday, November 15, 2022 2:30 PM To: Redbourne,Michael <michael.redbou...@bulletproofsi.com> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang <da...@lang.hm> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events what does the pstats output look like when it's dropping messages? (give a couple cycles please) did you try to eliminate the action queue for /var/log/secure? David Lang On Tue, 15 Nov 2022, Redbourne,Michael wrote:Date: Tue, 15 Nov 2022 13:01:02 +0000 From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang <da...@lang.hm> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events Building on this - When the drop count spikes top is showing a spike in CPU usage among the previously listed threads: In:imdup spikes to ~10% in_syslog.rb spikes to 90-100% usage rs:main Q:Reg spikes to 25% usage. -----Original Message----- From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Redbourne,Michael via rsyslog Sent: Tuesday, November 15, 2022 8:42 AM To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang <da...@lang.hm> Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events Concerning the /proc and pstats. There is /proc/net/netstat, which looks something like this after a couple minutes of logs: Udp: 5820820 packets received 1504 packets to unknown port received. 798900 packet receive errors 3338814 packets sent 798900 receive buffer errors 0 send buffer errors I have doubled the values in net.ipv4.udp_mem. The intent behind the queue $ActionQueue* legacy directives was spawning additional worker threads when the queue became abnormally large. I've tried various settings assigned to it, high worker threads, low messages, and vice versa. Would it be beneficial (and possible) to move those legacy directives to /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load is going to be. (Though with less extreme settings). The ereregex filters are set to remove information from being forwarded to Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant for monitoring purpose. They mostly target /16s, /22s and /24s. I could change this to (pseudo): If fromhost-ip contains "<Sending Device>" and $rawmsg contains "<subnet>" stop Example Checkpoint Log: CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL Vendor-specific SQL Injection|Very-High| eventId=882492844392 msg=Application Intelligence mrt=1599552618944 in=-2147483648 out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 priority=8 deviceSeverity=Very-High rt=1599552617058 deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX cs2=asm_dynamic_prop_SQL_FINGERPRINT_A cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection flexString2=SQL Servers MSSQL Vendor-specific SQL Injection flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr> dvc=<dvc_ip_addr> That should help it cut down on the unnecessary checking of logs. Otherwise, it gets applied to every log inbound, not just the ones from the firewall assets. Checking for CEF: is not something I could easily remove. It controls event ingestion and separation from other log source types in Microsoft's system. I'll remove the ASA section though, it's not necessary for this collector. I can probably move the Infoblox setting to a syslog tag by source ip. -----Original Message----- From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Rainer Gerhards via rsyslog Sent: Tuesday, November 15, 2022 5:11 AM To: David Lang <da...@lang.hm> Cc: Rainer Gerhards <rgerha...@hq.adiscon.com>; rsyslog-users <rsyslog@lists.adiscon.com> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events Just wanted to make sure awareness of that option. Agree that it is not often needed. Rainer El mar, 15 nov 2022 a las 10:02, David Lang (<da...@lang.hm>) escribió:I haven't needed to do that to handle 300k messages/sec on UDP input (usually I run into bottlenecks in processing the messages long before I have problems accepting them) David Lang On Tue, 15 Nov 2022, Rainer Gerhards wrote:let me add: look into setting imudp to realtime priority. Doc: https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F w w w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.htm l & amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc61 6 2 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0 % 7 C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC J Q IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sda t a =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&reserved=0 Rainer El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog (<rsyslog@lists.adiscon.com>) escribió:Some additional comments on the config These action queue configs probably don't do what you intend them to do the first thing is that they only affect the next action, which is authpriv.* to /var/log/secure and you configure 2000 threads to write these logs out. That will create a HUGE amount of contention for the queue lock and under load you should see it maxing out quite quickly what is it that you are attempting to do here? # Performance Tuning # $ActionQueueWorkerThreads 2000 $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark 600000 #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail authpriv, cron) # Dont log private authentication messages! #*.*;mail.none;authpriv.none;cron.none ?RemoteIP # The authpriv file has restricted access. authpriv.* /var/log/secure since the queue only applied to the next action with this config, everything below this is operating from the main queue again as if there was no action queue configuration # Log all the mail messages in one place. mail.* -/var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log # local7.* /var/syslog/boot.log ereregex is a fairly expensive filter to apply, it's much better to figure out a non-regex approach to filtering these. Can you post some examples of what you are trying to filter? mmnormalize to parse the logs and then make decisions on the parsed results id probably much faster. /etc/rsyslog.d/security-config-omsagent.conf # [Firewall Log Filtering] # :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)" stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex, "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop check the messages to see where CEF: and ASA- are in the message, can you filter on something smaller than rawmsg? (say syslogtag), and can you use 'startswith' instead of 'contains'?, again mmnormalize may be much faster if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr" then @127.0.0.1:25224 & stop combining multiple filters into one action, or having the filters call a ruleset can be far more efficient than all of them writing things out independently. the if..then filter structure lets you easily combine filters local0.info @127.0.0.1:25224 & stop local1.info @127.0.0.1:25224 & stop local2.info @127.0.0.1:25224 & stop local3.info @127.0.0.1:25224 & stop local4.info @127.0.0.1:25224 & stop local5.info @127.0.0.1:25224 & stop local6.info @127.0.0.1:25224 & stop local7.info @127.0.0.1:25224 & stop auth.* @127.0.0.1:25224 & stop authpriv.* @127.0.0.1:25224 & stop daemon.info @127.0.0.1:25224 & stop syslog.* @127.0.0.1:25224 & stop ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop user.* @127.0.0.1:25224 & stop _______________________________________________ rsyslog mailing list https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2 F l ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01% 7 C michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac 6 e 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C63804100329703 1 5 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC J B TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BlY86%2FvQ y n hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&reserved=0 https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2F w w w.rsyslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmicha e l .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e % 7 C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7C U n known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I k 1 haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W96%2BKd2Th68p6gYB6 I o nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&reserved=0 What's up with rsyslog? Follow https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2 F t witter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bul l e tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411 b b e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZs b 3 d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 % 3 D%7C3000%7C%7C%7C&sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3 O Y kHGY%3D&reserved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT._______________________________________________ rsyslog mailing list https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli s t s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01%7Cmich a e l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7 C 9 a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno w n %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL C J XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ % 2 BRRguRnpir9RsV8%3D&reserved=0 https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww. r%2F&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44 e 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7 C 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL C JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sda t a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&reserved=0 syslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmichael.red b o urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1 3 8 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW F p bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 M n 0%3D%7C3000%7C%7C%7C&sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3 w N KcbIFqhg%3D&reserved=0 What's up with rsyslog? Follow https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw i t ter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bulletpro o f si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7 e 2 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi M C 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7 C % 7C&sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&re s e rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. ________________________________________ This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated. Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privilégiés ou confidentiels. Si vous n'êtes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre façon. Si vous avez reçu le présent courriel par erreur, prière de communiquer avec l'expéditeur et d'éliminer l'original du courriel, ainsi que toute copie électronique ou imprimée de celui-ci, immédiatement. Si vous avez des questions ou des préoccupations, veuillez contacter notre centre de service à la clientèle au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration. ________________________________________ _______________________________________________ rsyslog mailing list https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli s t s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01%7Cmich a e l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7 C 9 a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno w n %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL C J XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ % 2 BRRguRnpir9RsV8%3D&reserved=0 https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww. r%2F&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44 e 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7 C 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL C JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sda t a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&reserved=0 syslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmichael.red b o urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1 3 8 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW F p bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 M n 0%3D%7C3000%7C%7C%7C&sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3 w N KcbIFqhg%3D&reserved=0 What's up with rsyslog? Follow https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw i t ter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bulletpro o f si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7 e 2 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi M C 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7 C % 7C&sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&re s e rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
