Building on this -
When the drop count spikes top is showing a spike in CPU usage among the
previously listed threads:
In:imdup spikes to ~10%
in_syslog.rb spikes to 90-100% usage
rs:main Q:Reg spikes to 25% usage.
-----Original Message-----
From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of
Redbourne,Michael via rsyslog
Sent: Tuesday, November 15, 2022 8:42 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang <da...@lang.hm>
Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com>
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
Concerning the /proc and pstats. There is /proc/net/netstat, which looks
something like this after a couple minutes of logs:
Udp:
5820820 packets received
1504 packets to unknown port received.
798900 packet receive errors
3338814 packets sent
798900 receive buffer errors
0 send buffer errors
I have doubled the values in net.ipv4.udp_mem.
The intent behind the queue $ActionQueue* legacy directives was spawning
additional worker threads when the queue became abnormally large. I've tried
various settings assigned to it, high worker threads, low messages, and vice
versa. Would it be beneficial (and possible) to move those legacy directives to
/etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load
is going to be. (Though with less extreme settings).
The ereregex filters are set to remove information from being forwarded to
Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant
for monitoring purpose. They mostly target /16s, /22s and /24s. I could change
this to (pseudo):
If fromhost-ip contains "<Sending Device>" and $rawmsg contains "<subnet>" stop
Example Checkpoint Log:
CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
Vendor-specific SQL Injection|Very-High| eventId=882492844392 msg=Application
Intelligence mrt=1599552618944 in=-2147483648 out=-2147483648 customerURI=XXXX
catdt=Firewall severity=0 priority=8 deviceSeverity=Very-High rt=1599552617058
deviceDirection=0 shost=XXXX src=<src_ip_addr> sourceZoneURI=XXXX
sourceGeoCountryCode=XXXX sourceGeoRegionCode=XXXX
cs2=asm_dynamic_prop_SQL_FINGERPRINT_A cs3=IPS cs4=SQL Servers MSSQL
Vendor-specific SQL Injection flexString2=SQL Servers MSSQL Vendor-specific SQL
Injection flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
dvc=<dvc_ip_addr>
That should help it cut down on the unnecessary checking of logs. Otherwise, it
gets applied to every log inbound, not just the ones from the firewall assets.
Checking for CEF: is not something I could easily remove. It controls event
ingestion and separation from other log source types in Microsoft's system.
I'll remove the ASA section though, it's not necessary for this collector. I
can probably move the Infoblox setting to a syslog tag by source ip.
-----Original Message-----
From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Rainer Gerhards
via rsyslog
Sent: Tuesday, November 15, 2022 5:11 AM
To: David Lang <da...@lang.hm>
Cc: Rainer Gerhards <rgerha...@hq.adiscon.com>; rsyslog-users
<rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
Just wanted to make sure awareness of that option. Agree that it is not often
needed.
Rainer
El mar, 15 nov 2022 a las 10:02, David Lang (<da...@lang.hm>) escribió:
>
> I haven't needed to do that to handle 300k messages/sec on UDP input
> (usually I run into bottlenecks in processing the messages long before
> I have problems accepting them)
>
> David Lang
>
> On Tue, 15 Nov 2022, Rainer Gerhards wrote:
>
> > let me add: look into setting imudp to realtime priority. Doc:
> >
> > https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
> > w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.html&
> > amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc6162
> > 80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7
> > C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
> > IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata
> > =OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&reserved=0
> >
> > Rainer
> >
> > El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribió:
> >>
> >> Some additional comments on the config
> >>
> >>
> >>
> >> These action queue configs probably don't do what you intend them
> >> to do
> >>
> >> the first thing is that they only affect the next action, which is
> >> authpriv.* to /var/log/secure and you configure 2000 threads to
> >> write these logs out. That will create a HUGE amount of contention
> >> for the queue lock and under load you should see it maxing out
> >> quite quickly
> >>
> >> what is it that you are attempting to do here?
> >>
> >>
> >>
> >> # Performance Tuning #
> >> $ActionQueueWorkerThreads 2000
> >> $ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
> >> 1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
> >> 600000
> >>
> >> #### RULES ####
> >> # Log all kernel messages to the console.
> >> # Logging much else clutters up the screen.
> >> #kern.* /dev/console
> >>
> >> # Log anything (except mail authpriv, cron) # Dont log private
> >> authentication messages!
> >> #*.*;mail.none;authpriv.none;cron.none ?RemoteIP
> >>
> >> # The authpriv file has restricted access.
> >> authpriv.* /var/log/secure
> >>
> >>
> >> since the queue only applied to the next action with this config,
> >> everything below this is operating from the main queue again as if
> >> there was no action queue configuration
> >>
> >>
> >>
> >>
> >> # Log all the mail messages in one place.
> >> mail.* -/var/log/maillog
> >>
> >> # Log cron stuff
> >> cron.* /var/log/cron
> >>
> >> # Everybody gets emergency messages
> >> *.emerg :omusrmsg:*
> >>
> >> # Save news errors of level crit and higher in a special file.
> >> uucp,news.crit /var/log/spooler
> >>
> >> # Save boot messages also to boot.log
> >> # local7.*
> >> /var/syslog/boot.log
> >>
> >>
> >>
> >> ereregex is a fairly expensive filter to apply, it's much better to
> >> figure out a non-regex approach to filtering these. Can you post
> >> some examples of what you are trying to filter? mmnormalize to
> >> parse the logs and then make decisions on the parsed results id probably
> >> much faster.
> >>
> >>
> >> /etc/rsyslog.d/security-config-omsagent.conf
> >> # [Firewall Log Filtering] #
> >> :msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
> >> "(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
> >> stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
> >> "(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
> >> stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
> >> "(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop :msg,
> >> ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP multicast
> >> routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
> >>
> >>
> >> check the messages to see where CEF: and ASA- are in the message,
> >> can you filter on something smaller than rawmsg? (say syslogtag), and can
> >> you use 'startswith'
> >> instead of 'contains'?, again mmnormalize may be much faster
> >>
> >> if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
> >> @@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
> >> then @127.0.0.1:25224 & stop
> >>
> >>
> >> combining multiple filters into one action, or having the filters
> >> call a ruleset can be far more efficient than all of them writing things
> >> out independently.
> >>
> >> the if..then filter structure lets you easily combine filters
> >>
> >> local0.info @127.0.0.1:25224
> >> & stop
> >> local1.info @127.0.0.1:25224
> >> & stop
> >> local2.info @127.0.0.1:25224
> >> & stop
> >> local3.info @127.0.0.1:25224
> >> & stop
> >> local4.info @127.0.0.1:25224
> >> & stop
> >> local5.info @127.0.0.1:25224
> >> & stop
> >> local6.info @127.0.0.1:25224
> >> & stop
> >> local7.info @127.0.0.1:25224
> >> & stop
> >> auth.* @127.0.0.1:25224
> >> & stop
> >> authpriv.* @127.0.0.1:25224
> >> & stop
> >> daemon.info @127.0.0.1:25224
> >> & stop
> >> syslog.* @127.0.0.1:25224
> >> & stop
> >> ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
> >> user.* @127.0.0.1:25224
> >> & stop
> >> _______________________________________________
> >> rsyslog mailing list
> >> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
> >> ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01%7C
> >> michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e
> >> 9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C6380410032970315
> >> 74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
> >> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BlY86%2FvQyn
> >> hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&reserved=0
> >> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fww
> >> w.rsyslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmichael
> >> .redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7
> >> C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUn
> >> known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
> >> haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W96%2BKd2Th68p6gYB6Io
> >> nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&reserved=0
> >> What's up with rsyslog? Follow
> >> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft
> >> witter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bulle
> >> tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bb
> >> e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3
> >> d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> >> D%7C3000%7C%7C%7C&sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3OY
> >> kHGY%3D&reserved=0 NOTE WELL: This is a PUBLIC mailing list,
> >> posts are ARCHIVED by a myriad of sites beyond our control. PLEASE
> >> UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >
_______________________________________________
rsyslog mailing list
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2Bt7rCJdsO4aJG5xTyEOpCPEln7yisFg36N%2F2nR0sHHE%3D&reserved=0
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YXII1LE3Cc7kV9GZytjwiZ1dpeNeqaL2HqyTPT90UpI%3D&reserved=0
What's up with rsyslog? Follow
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=y%2FSsPs3j4Rrmfb7dtuxq6aia%2Fgssf5LPaAG1wPHwTfM%3D&reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
________________________________________
This e-mail communication (including any or all attachments) is intended only
for the use of the person or entity to which it is addressed and may contain
confidential and/or privileged material. If you are not the intended recipient
of this e-mail, any use, review, retransmission, distribution, dissemination,
copying, printing, or other use of, or taking of any action in reliance upon
this e-mail, is strictly prohibited. If you have received this e-mail in error,
please contact the sender and delete the original and any copy of this e-mail
and any printout thereof, immediately. If you have any questions or concerns,
please contact our Customer Service Desk at 1-877-274-2349. Your co-operation
is appreciated.
Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à son
destinataire, qu'il soit une personne ou un organisme, et pourrait comporter
des renseignements privilégiés ou confidentiels. Si vous n'êtes pas le
destinataire du courriel, il est interdit d'utiliser, de revoir, de
retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce
courriel, d'agir en vous y fiant ou de vous en servir de toute autre façon. Si
vous avez reçu le présent courriel par erreur, prière de communiquer avec
l'expéditeur et d'éliminer l'original du courriel, ainsi que toute copie
électronique ou imprimée de celui-ci, immédiatement. Si vous avez des questions
ou des préoccupations, veuillez contacter notre centre de service à la
clientèle au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
________________________________________
_______________________________________________
rsyslog mailing list
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2Bt7rCJdsO4aJG5xTyEOpCPEln7yisFg36N%2F2nR0sHHE%3D&reserved=0
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=YXII1LE3Cc7kV9GZytjwiZ1dpeNeqaL2HqyTPT90UpI%3D&reserved=0
What's up with rsyslog? Follow
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Cb593a6a093354465fb5c08dac706d17c%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C1%7C638041129350221959%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=y%2FSsPs3j4Rrmfb7dtuxq6aia%2Fgssf5LPaAG1wPHwTfM%3D&reserved=0
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
