A little late to the party here, but I had a similar issue a few months ago with firewall logs...
Ended up being an issue/bug on the firewall (Palo Alto) that would cause TCP syslog to be unreliable. Switching to UDP corrected the issue, but I had to prove it wasn't the syslog receiver side first to the firewall team. I used loggen (https://www.systutorials.com/docs/linux/man/1-loggen/) to "replay" some firewall logs into the syslog receivers at a rate 2~3x what the firewalls delivered on average to prove it. Might be helpful from a testing perspective ~ $.02 ________________________________ From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of Rainer Gerhards via rsyslog <rsyslog@lists.adiscon.com> Sent: Wednesday, November 16, 2022 3:14 AM To: rsyslog-users <rsyslog@lists.adiscon.com> Cc: Rainer Gerhards <rgerha...@hq.adiscon.com> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events > @rainer, what is the in_syslog.rb thread that he saw maxing out the cpu? Not rsyslog, looks like fluentd: https://github.com/fluent/fluentd/blob/master/lib/fluent/plugin/in_syslog.rb Rainer > > David Lang > > > On Tue, 15 Nov 2022, Redbourne,Michael wrote: > > > Date: Tue, 15 Nov 2022 21:39:45 +0000 > > From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> > > To: David Lang <da...@lang.hm> > > Cc: rsyslog-users <rsyslog@lists.adiscon.com> > > Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events > > > > I'm going to reach out to networking folks and see if I can get something > > better in place, especially around negating logs further up the chain then > > the syslog collector. (Moreso related to the Checkpoint firewalls - > > removing the forwarding/logging directly at the FW/MGMT server). I'm hoping > > if I can drop most of the filtering and replace $rawmsg contains with > > :fromhost-ip "1.1.1.1" that it'll solve most of my problems. It'll take a > > day or two before I get them into a call though. > > > > If I'm still havcing issues after that, I'll respond back. > > > > > > -----Original Message----- > > From: David Lang <da...@lang.hm> > > Sent: Tuesday, November 15, 2022 4:19 PM > > To: Redbourne,Michael <michael.redbou...@bulletproofsi.com> > > Cc: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com> > > Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events > > > > using the new action() syntax, you can name the actions so they aren't just > > numbered. > > > > starting rsyslog with -o /path/to/file will generate a config file that is > > the combination of all the included files (as rsyslog actually processes > > the config), which will make it easier to figure out which action is which. > > > > that could be the tcp forwarding of messages, which can stall if the device > > you are forwarding to can't keep up (that is a good place to have an action > > queue, but please use the action() syntax so that it's clear what the queue > > is doing) > > > > David Lang > > > > > > On Tue, 15 Nov 2022, Redbourne,Michael wrote: > > > >> Date: Tue, 15 Nov 2022 20:01:52 +0000 > >> From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> > >> To: David Lang <da...@lang.hm> > >> Cc: rsyslog-users <rsyslog@lists.adiscon.com> > >> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events > >> > >> Tue Nov 15 00:22:18 2022: global: origin=dynstats Tue Nov 15 00:22:18 > >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 > >> ratelimit.numratelimiters=0 Tue Nov 15 00:22:18 2022: action 0: > >> origin=core.action processed=1628 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 1: > >> origin=core.action processed=66 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 2: > >> origin=core.action processed=37449 failed=37448 suspended=1 > >> suspended.duration=30 resumed=0 Tue Nov 15 00:22:18 2022: action 3: > >> origin=core.action processed=4582 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 4: > >> origin=core.action processed=5039 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 5: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 6: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 7: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 8: > >> origin=core.action processed=862 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 9: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 10: > >> origin=core.action processed=397 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 11: > >> origin=core.action processed=50 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 12: > >> origin=core.action processed=2 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 13: > >> origin=core.action processed=13 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 14: > >> origin=core.action processed=766 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 15: > >> origin=core.action processed=11058 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 16: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 17: > >> origin=core.action processed=1099 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 18: > >> origin=core.action processed=12 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 19: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: action 20: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 00:22:18 2022: imudp(*:514): > >> origin=imudp submitted=24227 Tue Nov 15 00:22:18 2022: imudp(*:514): > >> origin=imudp submitted=0 Tue Nov 15 00:22:18 2022: imtcp(514): > >> origin=imtcp submitted=68069 Tue Nov 15 00:22:18 2022: resource-usage: > >> origin=impstats utime=1984677 stime=640671 maxrss=19788 minflt=8469 > >> majflt=22 inblock=2496 oublock=1512 nvcsw=31964 nivcsw=1702 Tue Nov 15 > >> 00:22:18 2022: main Q: origin=core.queue size=0 enqueued=94161 full=0 > >> discarded.full=0 discarded.nf=0 maxqsize=7033 Tue Nov 15 00:22:18 > >> 2022: imudp(w0): origin=imudp called.recvmmsg=26626 called.recvmsg=0 > >> msgs.received=24227 > >> > >> > >> Tue Nov 15 08:41:43 2022: global: origin=dynstats Tue Nov 15 08:41:43 > >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 > >> ratelimit.numratelimiters=0 Tue Nov 15 08:41:43 2022: action 0: > >> origin=core.action processed=1728 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 1: > >> origin=core.action processed=47 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 2: > >> origin=core.action processed=35778 failed=35777 suspended=1 > >> suspended.duration=30 resumed=0 Tue Nov 15 08:41:43 2022: action 3: > >> origin=core.action processed=9249 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 4: > >> origin=core.action processed=2296 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 5: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 6: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 7: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 8: > >> origin=core.action processed=1009 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 9: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 10: > >> origin=core.action processed=558 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 11: > >> origin=core.action processed=40 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 12: > >> origin=core.action processed=2 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 13: > >> origin=core.action processed=13 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 14: > >> origin=core.action processed=720 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 15: > >> origin=core.action processed=16593 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 16: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 17: > >> origin=core.action processed=359 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 18: > >> origin=core.action processed=2 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 19: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: action 20: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:41:43 2022: imudp(*:514): > >> origin=imudp submitted=31257 Tue Nov 15 08:41:43 2022: imudp(*:514): > >> origin=imudp submitted=0 Tue Nov 15 08:41:43 2022: imtcp(514): > >> origin=imtcp submitted=60875 Tue Nov 15 08:41:43 2022: resource-usage: > >> origin=impstats utime=2106346 stime=935022 maxrss=16800 minflt=7884 > >> majflt=22 inblock=2464 oublock=1496 nvcsw=43531 nivcsw=2479 Tue Nov 15 > >> 08:41:43 2022: main Q: origin=core.queue size=0 enqueued=93998 full=0 > >> discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 08:41:43 > >> 2022: imudp(w0): origin=imudp called.recvmmsg=37966 called.recvmsg=0 > >> msgs.received=31257 > >> > >> > >> Tue Nov 15 08:42:13 2022: global: origin=dynstats Tue Nov 15 08:42:13 > >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 > >> ratelimit.numratelimiters=0 Tue Nov 15 08:42:13 2022: action 0: > >> origin=core.action processed=2181 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 1: > >> origin=core.action processed=57 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 2: > >> origin=core.action processed=20793 failed=137 suspended=0 > >> suspended.duration=30 resumed=0 Tue Nov 15 08:42:13 2022: action 3: > >> origin=core.action processed=7599 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 4: > >> origin=core.action processed=5540 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 5: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 6: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 7: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 8: > >> origin=core.action processed=1120 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 9: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 10: > >> origin=core.action processed=157807 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 11: > >> origin=core.action processed=50 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 12: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 13: > >> origin=core.action processed=3 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 14: > >> origin=core.action processed=293 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 15: > >> origin=core.action processed=20243 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 16: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 17: > >> origin=core.action processed=866 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 18: > >> origin=core.action processed=10 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 19: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: action 20: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 08:42:13 2022: imudp(*:514): > >> origin=imudp submitted=194836 Tue Nov 15 08:42:13 2022: imudp(*:514): > >> origin=imudp submitted=0 Tue Nov 15 08:42:13 2022: imtcp(514): > >> origin=imtcp submitted=34848 Tue Nov 15 08:42:13 2022: resource-usage: > >> origin=impstats utime=5417168 stime=3346362 maxrss=16800 minflt=11196 > >> majflt=22 inblock=2496 oublock=1544 nvcsw=161168 nivcsw=4993 Tue Nov > >> 15 08:42:13 2022: main Q: origin=core.queue size=0 enqueued=229698 > >> full=0 discarded.full=0 discarded.nf=0 maxqsize=4893 Tue Nov 15 > >> 08:42:13 2022: imudp(w0): origin=imudp called.recvmmsg=125595 > >> called.recvmsg=0 msgs.received=194836 > >> > >> > >> Tue Nov 15 10:15:04 2022: global: origin=dynstats Tue Nov 15 10:15:04 > >> 2022: imuxsock: origin=imuxsock submitted=0 ratelimit.discarded=0 > >> ratelimit.numratelimiters=0 Tue Nov 15 10:15:04 2022: action 0: > >> origin=core.action processed=2009 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 1: > >> origin=core.action processed=151 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 2: > >> origin=core.action processed=52035 failed=52034 suspended=1 > >> suspended.duration=60 resumed=0 Tue Nov 15 10:15:04 2022: action 3: > >> origin=core.action processed=19442 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 4: > >> origin=core.action processed=4480 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 5: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 6: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 7: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 8: > >> origin=core.action processed=1125 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 9: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 10: > >> origin=core.action processed=415 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 11: > >> origin=core.action processed=144 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 12: > >> origin=core.action processed=2 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 13: > >> origin=core.action processed=13 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 14: > >> origin=core.action processed=745 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 15: > >> origin=core.action processed=23626 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 16: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 17: > >> origin=core.action processed=682 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 18: > >> origin=core.action processed=45 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 19: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: action 20: > >> origin=core.action processed=0 failed=0 suspended=0 > >> suspended.duration=0 resumed=0 Tue Nov 15 10:15:04 2022: imudp(*:514): > >> origin=imudp submitted=51446 Tue Nov 15 10:15:04 2022: imudp(*:514): > >> origin=imudp submitted=0 Tue Nov 15 10:15:04 2022: imtcp(514): > >> origin=imtcp submitted=81918 Tue Nov 15 10:15:04 2022: resource-usage: > >> origin=impstats utime=2709285 stime=1419920 maxrss=21636 minflt=9979 > >> majflt=22 inblock=2368 oublock=1528 nvcsw=69973 nivcsw=3668 Tue Nov 15 > >> 10:15:04 2022: main Q: origin=core.queue size=0 enqueued=135235 full=0 > >> discarded.full=0 discarded.nf=0 maxqsize=6830 Tue Nov 15 10:15:04 > >> 2022: imudp(w0): origin=imudp called.recvmmsg=64324 called.recvmsg=0 > >> msgs.received=51446 -----Original Message----- > >> From: David Lang <da...@lang.hm> > >> Sent: Tuesday, November 15, 2022 3:43 PM > >> To: Redbourne,Michael <michael.redbou...@bulletproofsi.com> > >> Cc: David Lang <da...@lang.hm>; rsyslog-users > >> <rsyslog@lists.adiscon.com> > >> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events > >> > >> you have the impstats module loaded in your config and writing stats out, > >> please post the output of this. > >> > >> David Lang > >> > >> On Tue, 15 Nov 2022, Redbourne,Michael wrote: > >> > >>> Date: Tue, 15 Nov 2022 19:38:27 +0000 > >>> From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> > >>> To: David Lang <da...@lang.hm> > >>> Cc: rsyslog-users <rsyslog@lists.adiscon.com> > >>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP > >>> Events > >>> > >>> I'm still not understanding what you mean by pstats - it's not a > >>> package or command available to me. It's apart of Unix from what I > >>> can tell. I've placed below the unparsed information form > >>> /proc/net/netstat and /proc/net/udp > >>> > >>> /proc/net/netstat > >>> TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts > >>> PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps > >>> ArpFilter TW TWRecycled TWKilled PAWSActive PAWSEstab DelayedACKs > >>> DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPHPHits > >>> TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging > >>> TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo > >>> TCPDSACKUndo TCPLossUndo TCPLostRetransmit TCPRenoFailures > >>> TCPSackFailures TCPLossFailures TCPFastRetrans TCPSlowStartRetrans > >>> TCPTimeouts TCPLossProbes TCPLossProbeRecovery TCPRenoRecoveryFail > >>> TCPSackRecoveryFail TCPRcvCollapsed TCPBacklogCoalesce > >>> TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv > >>> TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout > >>> TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures > >>> TCPMemoryPressuresChrono TCPSACKDiscard TCPDSACKIgnoredOld > >>> TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected > >>> TCPMD5Failure TCPSackShifted T > >> CPSackMerged TCPSackShiftFallback TCPBacklogDrop PFMemallocDrop > >> TCPMinTTLDrop TCPDeferAcceptDrop IPReversePathFilter > >> TCPTimeWaitOverflow TCPReqQFullDoCookies TCPReqQFullDrop > >> TCPRetransFail TCPRcvCoalesce TCPOFOQueue TCPOFODrop TCPOFOMerge > >> TCPChallengeACK TCPSYNChallenge TCPFastOpenActive > >> TCPFastOpenActiveFail TCPFastOpenPassive TCPFastOpenPassiveFail > >> TCPFastOpenListenOverflow TCPFastOpenCookieReqd TCPFastOpenBlackhole > >> TCPSpuriousRtxHostQueues BusyPollRxPackets TCPAutoCorking > >> TCPFromZeroWindowAdv TCPToZeroWindowAdv TCPWantZeroWindowAdv > >> TCPSynRetrans TCPOrigDataSent TCPHystartTrainDetect > >> TCPHystartTrainCwnd TCPHystartDelayDetect TCPHystartDelayCwnd > >> TCPACKSkippedSynRecv TCPACKSkippedPAWS TCPACKSkippedSeq > >> TCPACKSkippedFinWait2 TCPACKSkippedTimeWait TCPACKSkippedChallenge > >> TCPWinProbe TCPKeepAlive TCPMTUPFail TCPMTUPSuccess TCPDelivered > >> TCPDeliveredCE TCPAckCompressed TCPZeroWindowDrop TCPRcvQDrop > >> TCPWqueueTooBig TCPFastOpenPassiveAltKey TcpTimeoutRehash > >> TcpDuplicateDataRehash TCPDSACKRe > > c vSegs TCPDSACKIgnoredDubious TCPMigrateReqSuccess TCPMigrateReqFailure > >>> TcpExt: 0 0 0 0 0 0 0 0 0 0 60 0 0 0 0 147 0 45 0 0 127557 13402 5231 > >>> 0 2 0 0 0 0 0 0 1 0 0 0 0 0 254 0 5 49 0 0 0 0 2245 53 0 47 0 0 0 0 0 > >>> 0 0 0 0 0 0 42 0 0 0 0 295 52 12 0 0 0 0 0 0 0 0 0 53916 24 0 0 0 0 0 > >>> 0 0 0 0 0 0 0 0 123 2259 2259 413 4 33447 1 1242 0 0 0 0 6 0 0 0 24 0 > >>> 0 0 33561 0 0 0 0 0 0 1 0 47 0 0 0 > >>> IpExt: InNoRoutes InTruncatedPkts InMcastPkts OutMcastPkts > >>> InBcastPkts OutBcastPkts InOctets OutOctets InMcastOctets > >>> OutMcastOctets InBcastOctets OutBcastOctets InCsumErrors InNoECTPkts > >>> InECT1Pkts InECT0Pkts InCEPkts ReasmOverlaps > >>> IpExt: 0 0 2 0 30 0 869144236 408176181 72 0 8723 0 0 1142299 0 1 0 0 > >>> > >>> /proc/net/udp > >>> sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt > >>> uid timeout inode ref pointer drops > >>> 31: 3050810A:007B 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 38 0 23514 2 ffff9a4f8b46bf00 0 > >>> 31: 3224200A:007B 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 38 0 23513 2 ffff9a4f8b46c380 0 > >>> 31: 0100007F:007B 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 14102 2 ffff9a4f81f11f80 0 > >>> 31: 00000000:007B 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 14096 2 ffff9a4f81f11200 0 > >>> 422: 00000000:0202 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 38099 2 ffff9a4f8398c380 0 > >>> 556: 0100007F:6288 00000000:0000 07 00000000:000A5F00 00:00000000 > >>> 00000000 982 0 41299 2 ffff9a4f81fc5a00 62728 > >>> 559: 00000000:628B 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 982 0 41291 2 ffff9a4f81fc4800 0 > >>> 560: 00000000:628C 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 982 0 41285 2 ffff9a4f81fc5580 0 > >>> 3008: 00000000:8C1C 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 42071 2 ffff9a4f835a7500 0 > >>> 3263: 00000000:8D1B 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 38107 2 ffff9a4f8398a880 0 > >>> 3520: 00000000:8E1C 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 23974 2 ffff9a4f86392880 0 > >>> 4172: 00000000:B0A8 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 38120 2 ffff9a4f8398c800 0 > >>> 4203: 00000000:B0C7 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 23973 2 ffff9a4f86392d00 0 > >>> 5106: 00000000:D44E 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 39146 2 ffff9a4f863c7980 0 > >>> 5961: 00000000:B7A5 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 41233 2 ffff9a4f81fc2d00 0 > >>> 6077: 00000000:B819 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 26070 2 ffff9a4f894df500 0 > >>> 7203: 00000000:9C7F 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 42072 2 ffff9a4f835a6780 0 > >>> 7781: 00000000:BEC1 00000000:0000 07 00000000:00000000 00:00000000 > >>> 00000000 0 0 38108 2 ffff9a4f8398ad00 0 > >>> > >>> The ActionQueue for /var/log/secure has been commented out from the > >>> configuration. > >>> > >>> Dropwatch (Cycle 1): > >>> 6562 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) > >>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab) > >>> 9521 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) > >>> 4 drops at unix_dgram_sendmsg+3fe (0xffffffff9a95cb9e) > >>> 2 drops at skb_release_data+12b (0xffffffff9a7f8cab) > >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f) > >>> 4625 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) > >>> 2 drops at unix_stream_connect+295 (0xffffffff9a95d2f5) > >>> 1 drops at unix_stream_connect+295 (0xffffffff9a95d2f5) > >>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d) > >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f) > >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f) > >>> 1 drops at tcp_v4_rcv+7d (0xffffffff9a8eef4d) > >>> 16564 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) > >>> 6 drops at skb_release_data+12b (0xffffffff9a7f8cab) > >>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d) > >>> 1 drops at tcp_drop_reason+3f (0xffffffff9a8d5b2f) > >>> 9411 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) > >>> 8 drops at skb_release_data+12b (0xffffffff9a7f8cab) > >>> 3790 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) > >>> 2 drops at nf_hook_slow+9d (0xffffffff9a8ae98d) > >>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab) > >>> 1 drops at nf_hook_slow+9d (0xffffffff9a8ae98d) > >>> 6612 drops at udp_queue_rcv_one_skb+438 (0xffffffff9a8fd528) > >>> 11 drops at skb_release_data+12b (0xffffffff9a7f8cab) > >>> 7 drops at skb_release_data+12b (0xffffffff9a7f8cab) > >>> > >>> Dropwatch (Cycle 2): > >>> 12308 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) > >>> 11 drops at skb_release_data+12b (0xffffffff937f8cab) > >>> 1262 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) > >>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f) > >>> 2 drops at unix_stream_connect+295 (0xffffffff9395d2f5) > >>> 14 drops at skb_release_data+12b (0xffffffff937f8cab) > >>> 7654 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) > >>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f) > >>> 1 drops at tcp_v4_rcv+7d (0xffffffff938eef4d) > >>> 6326 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) > >>> 6 drops at skb_release_data+12b (0xffffffff937f8cab) > >>> 19601 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) > >>> 11 drops at skb_release_data+12b (0xffffffff937f8cab) > >>> 1 drops at tcp_drop_reason+3f (0xffffffff938d5b2f) > >>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d) > >>> 8994 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) > >>> 10 drops at skb_release_data+12b (0xffffffff937f8cab) > >>> 6422 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) > >>> 5 drops at skb_release_data+12b (0xffffffff937f8cab) > >>> 161 drops at udp_queue_rcv_one_skb+438 (0xffffffff938fd528) > >>> 1 drops at nf_hook_slow+9d (0xffffffff938ae98d) > >>> > >>> Top -H > >>> 1842 omsagent 20 0 1304664 251636 9828 R 52.2 1.5 0:18.48 > >>> in_syslog.rb:1* > >>> 1779 omsagent 20 0 1304664 251636 9828 S 41.9 1.5 0:13.58 > >>> output.rb:140 > >>> 1453 root 20 0 589760 10560 5464 R 26.6 0.1 0:09.74 rs:main > >>> Q:Reg > >>> 1838 omsagent 20 0 1304664 251636 9828 S 16.9 1.5 0:17.69 > >>> in_syslog.rb:1* > >>> 1447 root 20 0 589760 10560 5464 S 9.6 0.1 0:02.71 in:imudp > >>> 1448 root 20 0 589760 10560 5464 S 1.7 0.1 0:01.66 in:imtcp > >>> > >>> /etc/sysctl.conf > >>> net.core.rmem_default = 33554432 > >>> net.core.rmem_max = 268435456 > >>> net.core.wmem_default = 33554432 > >>> net.core.wmem_max = 268435456 > >>> net.ipv4.tcp_mem = 190611 254150 381222 > >>> net.ipv4.tcp_rmem = 4096 131072 6291456 > >>> net.ipv4.tcp_wmem = 4096 16384 4194304 > >>> net.ipv4.udp_mem = 762450 1524900 3049800 > >>> net.ipv4.udp_rmem_min = 33554432 > >>> net.ipv4.udp_wmem_min = 33554432 > >>> > >>> > >>> -----Original Message----- > >>> From: David Lang <da...@lang.hm> > >>> Sent: Tuesday, November 15, 2022 2:30 PM > >>> To: Redbourne,Michael <michael.redbou...@bulletproofsi.com> > >>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang > >>> <da...@lang.hm> > >>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP > >>> Events > >>> > >>> what does the pstats output look like when it's dropping messages? > >>> (give a couple cycles please) > >>> > >>> did you try to eliminate the action queue for /var/log/secure? > >>> > >>> David Lang > >>> > >>> On Tue, 15 Nov 2022, Redbourne,Michael wrote: > >>> > >>>> Date: Tue, 15 Nov 2022 13:01:02 +0000 > >>>> From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> > >>>> To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang > >>>> <da...@lang.hm> > >>>> Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP > >>>> Events > >>>> > >>>> Building on this - > >>>> > >>>> When the drop count spikes top is showing a spike in CPU usage among the > >>>> previously listed threads: > >>>> In:imdup spikes to ~10% > >>>> in_syslog.rb spikes to 90-100% usage rs:main Q:Reg spikes to 25% > >>>> usage. > >>>> > >>>> -----Original Message----- > >>>> From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of > >>>> Redbourne,Michael via rsyslog > >>>> Sent: Tuesday, November 15, 2022 8:42 AM > >>>> To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang > >>>> <da...@lang.hm> > >>>> Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com> > >>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP > >>>> Events > >>>> > >>>> Concerning the /proc and pstats. There is /proc/net/netstat, which looks > >>>> something like this after a couple minutes of logs: > >>>> Udp: > >>>> 5820820 packets received > >>>> 1504 packets to unknown port received. > >>>> 798900 packet receive errors > >>>> 3338814 packets sent > >>>> 798900 receive buffer errors > >>>> 0 send buffer errors > >>>> > >>>> I have doubled the values in net.ipv4.udp_mem. > >>>> > >>>> The intent behind the queue $ActionQueue* legacy directives was spawning > >>>> additional worker threads when the queue became abnormally large. I've > >>>> tried various settings assigned to it, high worker threads, low > >>>> messages, and vice versa. Would it be beneficial (and possible) to move > >>>> those legacy directives to > >>>> /etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the > >>>> load is going to be. (Though with less extreme settings). > >>>> > >>>> The ereregex filters are set to remove information from being forwarded > >>>> to Sentinel, in most cases, large swaths of IP subnet ranges that are > >>>> irrelevant for monitoring purpose. They mostly target /16s, /22s and > >>>> /24s. I could change this to (pseudo): > >>>> If fromhost-ip contains "<Sending Device>" and $rawmsg contains > >>>> "<subnet>" stop > >>>> > >>>> Example Checkpoint Log: > >>>> CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL > >>>> Vendor-specific SQL Injection|Very-High| eventId=882492844392 > >>>> msg=Application Intelligence mrt=1599552618944 in=-2147483648 > >>>> out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 > >>>> priority=8 deviceSeverity=Very-High rt=1599552617058 > >>>> deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX > >>>> sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX > >>>> cs2=asm_dynamic_prop_SQL_FINGERPRINT_A > >>>> cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection > >>>> flexString2=SQL Servers MSSQL Vendor-specific SQL Injection > >>>> flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr> > >>>> dvc=<dvc_ip_addr> > >>>> > >>>> That should help it cut down on the unnecessary checking of logs. > >>>> Otherwise, it gets applied to every log inbound, not just the ones from > >>>> the firewall assets. > >>>> > >>>> Checking for CEF: is not something I could easily remove. It controls > >>>> event ingestion and separation from other log source types in > >>>> Microsoft's system. I'll remove the ASA section though, it's not > >>>> necessary for this collector. I can probably move the Infoblox setting > >>>> to a syslog tag by source ip. > >>>> > >>>> -----Original Message----- > >>>> From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of > >>>> Rainer Gerhards via rsyslog > >>>> Sent: Tuesday, November 15, 2022 5:11 AM > >>>> To: David Lang <da...@lang.hm> > >>>> Cc: Rainer Gerhards <rgerha...@hq.adiscon.com>; rsyslog-users > >>>> <rsyslog@lists.adiscon.com> > >>>> Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP > >>>> Events > >>>> > >>>> Just wanted to make sure awareness of that option. Agree that it is not > >>>> often needed. > >>>> > >>>> Rainer > >>>> > >>>> El mar, 15 nov 2022 a las 10:02, David Lang (<da...@lang.hm>) escribió: > >>>>> > >>>>> I haven't needed to do that to handle 300k messages/sec on UDP > >>>>> input (usually I run into bottlenecks in processing the messages > >>>>> long before I have problems accepting them) > >>>>> > >>>>> David Lang > >>>>> > >>>>> On Tue, 15 Nov 2022, Rainer Gerhards wrote: > >>>>> > >>>>>> let me add: look into setting imudp to realtime priority. Doc: > >>>>>> > >>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2F > >>>>>> w > >>>>>> w > >>>>>> w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.htm > >>>>>> l > >>>>>> & > >>>>>> amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc61 > >>>>>> 6 > >>>>>> 2 > >>>>>> 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0 > >>>>>> % > >>>>>> 7 > >>>>>> C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLC > >>>>>> J > >>>>>> Q > >>>>>> IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sda > >>>>>> t > >>>>>> a > >>>>>> =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&reserved=0 > >>>>>> > >>>>>> Rainer > >>>>>> > >>>>>> El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog > >>>>>> (<rsyslog@lists.adiscon.com>) escribió: > >>>>>>> > >>>>>>> Some additional comments on the config > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> These action queue configs probably don't do what you intend them > >>>>>>> to do > >>>>>>> > >>>>>>> the first thing is that they only affect the next action, which > >>>>>>> is > >>>>>>> authpriv.* to /var/log/secure and you configure 2000 threads to > >>>>>>> write these logs out. That will create a HUGE amount of > >>>>>>> contention for the queue lock and under load you should see it > >>>>>>> maxing out quite quickly > >>>>>>> > >>>>>>> what is it that you are attempting to do here? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> # Performance Tuning # > >>>>>>> $ActionQueueWorkerThreads 2000 > >>>>>>> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize > >>>>>>> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark > >>>>>>> 600000 > >>>>>>> > >>>>>>> #### RULES #### > >>>>>>> # Log all kernel messages to the console. > >>>>>>> # Logging much else clutters up the screen. > >>>>>>> #kern.* /dev/console > >>>>>>> > >>>>>>> # Log anything (except mail authpriv, cron) # Dont log private > >>>>>>> authentication messages! > >>>>>>> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP > >>>>>>> > >>>>>>> # The authpriv file has restricted access. > >>>>>>> authpriv.* > >>>>>>> /var/log/secure > >>>>>>> > >>>>>>> > >>>>>>> since the queue only applied to the next action with this config, > >>>>>>> everything below this is operating from the main queue again as > >>>>>>> if there was no action queue configuration > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> # Log all the mail messages in one place. > >>>>>>> mail.* > >>>>>>> -/var/log/maillog > >>>>>>> > >>>>>>> # Log cron stuff > >>>>>>> cron.* /var/log/cron > >>>>>>> > >>>>>>> # Everybody gets emergency messages > >>>>>>> *.emerg :omusrmsg:* > >>>>>>> > >>>>>>> # Save news errors of level crit and higher in a special file. > >>>>>>> uucp,news.crit > >>>>>>> /var/log/spooler > >>>>>>> > >>>>>>> # Save boot messages also to boot.log > >>>>>>> # local7.* > >>>>>>> /var/syslog/boot.log > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> ereregex is a fairly expensive filter to apply, it's much better > >>>>>>> to figure out a non-regex approach to filtering these. Can you > >>>>>>> post some examples of what you are trying to filter? mmnormalize > >>>>>>> to parse the logs and then make decisions on the parsed results id > >>>>>>> probably much faster. > >>>>>>> > >>>>>>> > >>>>>>> /etc/rsyslog.d/security-config-omsagent.conf > >>>>>>> # [Firewall Log Filtering] # > >>>>>>> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex, > >>>>>>> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)" > >>>>>>> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex, > >>>>>>> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)" > >>>>>>> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex, > >>>>>>> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop > >>>>>>> :msg, ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP > >>>>>>> multicast routing failed)" stop :msg, ereregex, "(TCP_7680)" stop > >>>>>>> > >>>>>>> > >>>>>>> check the messages to see where CEF: and ASA- are in the message, > >>>>>>> can you filter on something smaller than rawmsg? (say syslogtag), and > >>>>>>> can you use 'startswith' > >>>>>>> instead of 'contains'?, again mmnormalize may be much faster > >>>>>>> > >>>>>>> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then > >>>>>>> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr" > >>>>>>> then @127.0.0.1:25224 & stop > >>>>>>> > >>>>>>> > >>>>>>> combining multiple filters into one action, or having the filters > >>>>>>> call a ruleset can be far more efficient than all of them writing > >>>>>>> things out independently. > >>>>>>> > >>>>>>> the if..then filter structure lets you easily combine filters > >>>>>>> > >>>>>>> local0.info @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> local1.info @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> local2.info @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> local3.info @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> local4.info @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> local5.info @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> local6.info @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> local7.info @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> auth.* @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> authpriv.* @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> daemon.info @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> syslog.* @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop > >>>>>>> user.* @127.0.0.1:25224 > >>>>>>> & stop > >>>>>>> _______________________________________________ > >>>>>>> rsyslog mailing list > >>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2 > >>>>>>> F > >>>>>>> l > >>>>>>> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01% > >>>>>>> 7 > >>>>>>> C > >>>>>>> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac > >>>>>>> 6 > >>>>>>> e > >>>>>>> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C63804100329703 > >>>>>>> 1 > >>>>>>> 5 > >>>>>>> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLC > >>>>>>> J > >>>>>>> B > >>>>>>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BlY86%2FvQ > >>>>>>> y > >>>>>>> n > >>>>>>> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&reserved=0 > >>>>>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2F > >>>>>>> w > >>>>>>> w > >>>>>>> w.rsyslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmicha > >>>>>>> e > >>>>>>> l > >>>>>>> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e > >>>>>>> % > >>>>>>> 7 > >>>>>>> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7C > >>>>>>> U > >>>>>>> n > >>>>>>> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I > >>>>>>> k > >>>>>>> 1 > >>>>>>> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W96%2BKd2Th68p6gYB6 > >>>>>>> I > >>>>>>> o > >>>>>>> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&reserved=0 > >>>>>>> What's up with rsyslog? Follow > >>>>>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2 > >>>>>>> F > >>>>>>> t > >>>>>>> witter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bul > >>>>>>> l > >>>>>>> e > >>>>>>> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411 > >>>>>>> b > >>>>>>> b > >>>>>>> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZs > >>>>>>> b > >>>>>>> 3 > >>>>>>> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0 > >>>>>>> % > >>>>>>> 3 > >>>>>>> D%7C3000%7C%7C%7C&sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3 > >>>>>>> O > >>>>>>> Y > >>>>>>> kHGY%3D&reserved=0 NOTE WELL: This is a PUBLIC mailing list, > >>>>>>> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE > >>>>>>> UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. > >>>>>> > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli > >>>> s > >>>> t > >>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01%7Cmich > >>>> a > >>>> e > >>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7 > >>>> C > >>>> 9 > >>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno > >>>> w > >>>> n > >>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL > >>>> C > >>>> J > >>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ > >>>> % > >>>> 2 > >>>> BRRguRnpir9RsV8%3D&reserved=0 > >>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww. > >>>> r%2F&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44 > >>>> e > >>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7 > >>>> C > >>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL > >>>> C > >>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sda > >>>> t > >>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&reserved=0 > >>>> syslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmichael.red > >>>> b > >>>> o > >>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1 > >>>> 3 > >>>> 8 > >>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW > >>>> F > >>>> p > >>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 > >>>> M > >>>> n > >>>> 0%3D%7C3000%7C%7C%7C&sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3 > >>>> w > >>>> N > >>>> KcbIFqhg%3D&reserved=0 What's up with rsyslog? Follow > >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw > >>>> i > >>>> t > >>>> ter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bulletpro > >>>> o > >>>> f > >>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7 > >>>> e > >>>> 2 > >>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi > >>>> M > >>>> C > >>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7 > >>>> C > >>>> % > >>>> 7C&sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&re > >>>> s > >>>> e > >>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > >>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > >>>> POST if you DON'T LIKE THAT. > >>>> ________________________________________ > >>>> This e-mail communication (including any or all attachments) is intended > >>>> only for the use of the person or entity to which it is addressed and > >>>> may contain confidential and/or privileged material. If you are not the > >>>> intended recipient of this e-mail, any use, review, retransmission, > >>>> distribution, dissemination, copying, printing, or other use of, or > >>>> taking of any action in reliance upon this e-mail, is strictly > >>>> prohibited. If you have received this e-mail in error, please contact > >>>> the sender and delete the original and any copy of this e-mail and any > >>>> printout thereof, immediately. If you have any questions or concerns, > >>>> please contact our Customer Service Desk at 1-877-274-2349. Your > >>>> co-operation is appreciated. > >>>> > >>>> Le présent courriel (y compris toute pièce jointe) s'adresse uniquement > >>>> à son destinataire, qu'il soit une personne ou un organisme, et pourrait > >>>> comporter des renseignements privilégiés ou confidentiels. Si vous > >>>> n'êtes pas le destinataire du courriel, il est interdit d'utiliser, de > >>>> revoir, de retransmettre, de distribuer, de disséminer, de copier ou > >>>> d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de > >>>> toute autre façon. Si vous avez reçu le présent courriel par erreur, > >>>> prière de communiquer avec l'expéditeur et d'éliminer l'original du > >>>> courriel, ainsi que toute copie électronique ou imprimée de celui-ci, > >>>> immédiatement. Si vous avez des questions ou des préoccupations, > >>>> veuillez contacter notre centre de service à la clientèle au > >>>> 1-877-274-2349. Nous sommes reconnaissants de votre collaboration. > >>>> ________________________________________ > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fli > >>>> s > >>>> t > >>>> s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01%7Cmich > >>>> a > >>>> e > >>>> l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7 > >>>> C > >>>> 9 > >>>> a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnkno > >>>> w > >>>> n > >>>> %7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiL > >>>> C > >>>> J > >>>> XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ > >>>> % > >>>> 2 > >>>> BRRguRnpir9RsV8%3D&reserved=0 > >>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww. > >>>> r%2F&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ccf44 > >>>> e > >>>> 6a59c9a49036c8b08dac7418ef5%7C9a63d13853ea411bbe8458b7e2570747%7C1%7 > >>>> C > >>>> 0%7C638041381634074303%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL > >>>> C > >>>> JQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sda > >>>> t > >>>> a=DWG7CbFvTRvkJx9pNSncNIZXJhf%2B4VtnbyAps2Gbk%2B8%3D&reserved=0 > >>>> syslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmichael.red > >>>> b > >>>> o > >>>> urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d1 > >>>> 3 > >>>> 8 > >>>> 53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTW > >>>> F > >>>> p > >>>> bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6 > >>>> M > >>>> n > >>>> 0%3D%7C3000%7C%7C%7C&sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3 > >>>> w > >>>> N > >>>> KcbIFqhg%3D&reserved=0 What's up with rsyslog? Follow > >>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftw > >>>> i > >>>> t > >>>> ter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bulletpro > >>>> o > >>>> f > >>>> si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7 > >>>> e > >>>> 2 > >>>> 570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoi > >>>> M > >>>> C > >>>> 4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7 > >>>> C > >>>> % > >>>> 7C&sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&re > >>>> s > >>>> e > >>>> rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > >>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > >>>> POST if you DON'T LIKE THAT. > >>>> > >>> > >> > > > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
