Date: Tue, 15 Nov 2022 13:01:02 +0000
From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com>
To: rsyslog-users <rsyslog@lists.adiscon.com>, David Lang
<da...@lang.hm>
Subject: RE: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
Building on this -
When the drop count spikes top is showing a spike in CPU usage among the
previously listed threads:
In:imdup spikes to ~10%
in_syslog.rb spikes to 90-100% usage
rs:main Q:Reg spikes to 25% usage.
-----Original Message-----
From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of
Redbourne,Michael via rsyslog
Sent: Tuesday, November 15, 2022 8:42 AM
To: rsyslog-users <rsyslog@lists.adiscon.com>; David Lang
<da...@lang.hm>
Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com>
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
Concerning the /proc and pstats. There is /proc/net/netstat, which looks
something like this after a couple minutes of logs:
Udp:
5820820 packets received
1504 packets to unknown port received.
798900 packet receive errors
3338814 packets sent
798900 receive buffer errors
0 send buffer errors
I have doubled the values in net.ipv4.udp_mem.
The intent behind the queue $ActionQueue* legacy directives was spawning
additional worker threads when the queue became abnormally large. I've tried
various settings assigned to it, high worker threads, low messages, and vice
versa. Would it be beneficial (and possible) to move those legacy directives to
/etc/rsyslog.d/security-confiig-omsagent.conf? That is where most of the load
is going to be. (Though with less extreme settings).
The ereregex filters are set to remove information from being forwarded to
Sentinel, in most cases, large swaths of IP subnet ranges that are irrelevant
for monitoring purpose. They mostly target /16s, /22s and /24s. I could change
this to (pseudo):
If fromhost-ip contains "<Sending Device>" and $rawmsg contains
"<subnet>" stop
Example Checkpoint Log:
CEF:0|Check Point|SmartDefense|Check Point|IPS|SQL Servers MSSQL
Vendor-specific SQL Injection|Very-High| eventId=882492844392
msg=Application Intelligence mrt=1599552618944 in=-2147483648
out=-2147483648 customerURI=XXXX catdt=Firewall severity=0 priority=8
deviceSeverity=Very-High rt=1599552617058 deviceDirection=0 shost=XXXX
src=<src_ip_addr> sourceZoneURI=XXXX sourceGeoCountryCode=XXXX
sourceGeoRegionCode=XXXX cs2=asm_dynamic_prop_SQL_FINGERPRINT_A
cs3=IPS cs4=SQL Servers MSSQL Vendor-specific SQL Injection
flexString2=SQL Servers MSSQL Vendor-specific SQL Injection
flexNumber1=5 flexNumber2=3 locality=1 amac=<mac_addr>
dvc=<dvc_ip_addr>
That should help it cut down on the unnecessary checking of logs. Otherwise, it
gets applied to every log inbound, not just the ones from the firewall assets.
Checking for CEF: is not something I could easily remove. It controls event
ingestion and separation from other log source types in Microsoft's system.
I'll remove the ASA section though, it's not necessary for this collector. I
can probably move the Infoblox setting to a syslog tag by source ip.
-----Original Message-----
From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Rainer
Gerhards via rsyslog
Sent: Tuesday, November 15, 2022 5:11 AM
To: David Lang <da...@lang.hm>
Cc: Rainer Gerhards <rgerha...@hq.adiscon.com>; rsyslog-users
<rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] rsyslog Performance Tuning - Dropped UDP Events
Just wanted to make sure awareness of that option. Agree that it is not often
needed.
Rainer
El mar, 15 nov 2022 a las 10:02, David Lang (<da...@lang.hm>) escribió:
I haven't needed to do that to handle 300k messages/sec on UDP input
(usually I run into bottlenecks in processing the messages long
before I have problems accepting them)
David Lang
On Tue, 15 Nov 2022, Rainer Gerhards wrote:
let me add: look into setting imudp to realtime priority. Doc:
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fww
w.rsyslog.com%2Fdoc%2Fmaster%2Fconfiguration%2Fmodules%2Fimudp.html&
amp;data=05%7C01%7Cmichael.redbourne%40bulletproofsi.com%7Ca6adc6162
80047e6f3dd08dac6e9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7
C638041003297031574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQ
IjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata
=OYRW6vzy9wKL556zxhIVEQ5TdTYYo23ij1dvEermN2c%3D&reserved=0
Rainer
El mar, 15 nov 2022 a las 5:04, David Lang via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
Some additional comments on the config
These action queue configs probably don't do what you intend them
to do
the first thing is that they only affect the next action, which is
authpriv.* to /var/log/secure and you configure 2000 threads to
write these logs out. That will create a HUGE amount of contention
for the queue lock and under load you should see it maxing out
quite quickly
what is it that you are attempting to do here?
# Performance Tuning #
$ActionQueueWorkerThreads 2000
$ActionQueueWorkerThreadMinimumMessages 1000 $ActionQueueSize
1000000 $ActionQueueDiscardMark 800000 $ActionQueueHighWaterMark
600000
#### RULES ####
# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.* /dev/console
# Log anything (except mail authpriv, cron) # Dont log private
authentication messages!
#*.*;mail.none;authpriv.none;cron.none ?RemoteIP
# The authpriv file has restricted access.
authpriv.* /var/log/secure
since the queue only applied to the next action with this config,
everything below this is operating from the main queue again as if
there was no action queue configuration
# Log all the mail messages in one place.
mail.* -/var/log/maillog
# Log cron stuff
cron.* /var/log/cron
# Everybody gets emergency messages
*.emerg :omusrmsg:*
# Save news errors of level crit and higher in a special file.
uucp,news.crit /var/log/spooler
# Save boot messages also to boot.log
# local7.* /var/syslog/boot.log
ereregex is a fairly expensive filter to apply, it's much better to
figure out a non-regex approach to filtering these. Can you post
some examples of what you are trying to filter? mmnormalize to
parse the logs and then make decisions on the parsed results id probably much
faster.
/etc/rsyslog.d/security-config-omsagent.conf
# [Firewall Log Filtering] #
:msg, ereregex, "(1.1.[0-9]+.[0-9]+)" stop :msg, ereregex,
"(1.2.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.3.[0-9]+.[0-9]+)"
stop :msg, ereregex, "(1.4.[0-9]+.[0-9]+)" stop :msg, ereregex,
"(1.5.[0-9]+.[0-9]+)" stop :msg, ereregex, "(1.6.1[6-9].[0-9]+)"
stop :msg, ereregex, "(1.7.2[0-3].[0-9]+)" stop :msg, ereregex,
"(1.8.68.[0-9]+)" stop :msg, ereregex, "(1.9.69.[0-9]+)" stop :msg,
ereregex, "(1.10.82.[0-9]+)" stop :msg, ereregex, "(IP multicast
routing failed)" stop :msg, ereregex, "(TCP_7680)" stop
check the messages to see where CEF: and ASA- are in the message,
can you filter on something smaller than rawmsg? (say syslogtag), and can you
use 'startswith'
instead of 'contains'?, again mmnormalize may be much faster
if $rawmsg contains "CEF:" or $rawmsg contains "ASA-" then
@@127.0.0.1:25226 & stop if $rawmsg contains "infobloxgridmstr"
then @127.0.0.1:25224 & stop
combining multiple filters into one action, or having the filters
call a ruleset can be far more efficient than all of them writing things out
independently.
the if..then filter structure lets you easily combine filters
local0.info @127.0.0.1:25224
& stop
local1.info @127.0.0.1:25224
& stop
local2.info @127.0.0.1:25224
& stop
local3.info @127.0.0.1:25224
& stop
local4.info @127.0.0.1:25224
& stop
local5.info @127.0.0.1:25224
& stop
local6.info @127.0.0.1:25224
& stop
local7.info @127.0.0.1:25224
& stop
auth.* @127.0.0.1:25224
& stop
authpriv.* @127.0.0.1:25224
& stop
daemon.info @127.0.0.1:25224
& stop
syslog.* @127.0.0.1:25224
& stop
ftp.*<ftp://ftp.*> @127.0.0.1:25224 & stop
user.* @127.0.0.1:25224
& stop
_______________________________________________
rsyslog mailing list
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fl
ists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01%7C
michael.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e
9784e%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C6380410032970315
74%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB
TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BlY86%2FvQyn
hVyFKzkpfWQHP%2BDhyNqfx3yTEpO9CEdQg%3D&reserved=0
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fww
w.rsyslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmichael
.redbourne%40bulletproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7
C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041003297031574%7CUn
known%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1
haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=W96%2BKd2Th68p6gYB6Io
nLtwuK26mJ4KFhWe6k%2BLYKvg%3D&reserved=0
What's up with rsyslog? Follow
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ft
witter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bulle
tproofsi.com%7Ca6adc616280047e6f3dd08dac6e9784e%7C9a63d13853ea411bb
e8458b7e2570747%7C1%7C0%7C638041003297031574%7CUnknown%7CTWFpbGZsb3
d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
D%7C3000%7C%7C%7C&sdata=qmPgnCgvUSjmACoXE6qWPKmb7SpWOFvpzVZV3OY
kHGY%3D&reserved=0 NOTE WELL: This is a PUBLIC mailing list,
posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01%7Cmichae
l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9
a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown
%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ
XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%2
BRRguRnpir9RsV8%3D&reserved=0
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.r
syslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmichael.redbo
urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d138
53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFp
bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
0%3D%7C3000%7C%7C%7C&sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3wN
KcbIFqhg%3D&reserved=0 What's up with rsyslog? Follow
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwit
ter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bulletproof
si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e2
570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%
7C&sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&rese
rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
________________________________________
This e-mail communication (including any or all attachments) is intended only
for the use of the person or entity to which it is addressed and may contain
confidential and/or privileged material. If you are not the intended recipient
of this e-mail, any use, review, retransmission, distribution, dissemination,
copying, printing, or other use of, or taking of any action in reliance upon
this e-mail, is strictly prohibited. If you have received this e-mail in error,
please contact the sender and delete the original and any copy of this e-mail
and any printout thereof, immediately. If you have any questions or concerns,
please contact our Customer Service Desk at 1-877-274-2349. Your co-operation
is appreciated.
Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à son
destinataire, qu'il soit une personne ou un organisme, et pourrait comporter
des renseignements privilégiés ou confidentiels. Si vous n'êtes pas le
destinataire du courriel, il est interdit d'utiliser, de revoir, de
retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce
courriel, d'agir en vous y fiant ou de vous en servir de toute autre façon. Si
vous avez reçu le présent courriel par erreur, prière de communiquer avec
l'expéditeur et d'éliminer l'original du courriel, ainsi que toute copie
électronique ou imprimée de celui-ci, immédiatement. Si vous avez des questions
ou des préoccupations, veuillez contacter notre centre de service à la
clientèle au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
________________________________________
_______________________________________________
rsyslog mailing list
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flist
s.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C01%7Cmichae
l.redbourne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9
a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown
%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ
XVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BfubHLcKhnssSFxmSNcnqGQjlhfZ%2
BRRguRnpir9RsV8%3D&reserved=0
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.r
syslog.com%2Fprofessional-services%2F&data=05%7C01%7Cmichael.redbo
urne%40bulletproofsi.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d138
53ea411bbe8458b7e2570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFp
bGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn
0%3D%7C3000%7C%7C%7C&sdata=lcpnEcpHkgHX%2BbeYzPuKTEzKQcsstXB%2B3wN
KcbIFqhg%3D&reserved=0 What's up with rsyslog? Follow
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwit
ter.com%2Frgerhards&data=05%7C01%7Cmichael.redbourne%40bulletproof
si.com%7Ce9f9bc5a7e4b4a01b59708dac7375b35%7C9a63d13853ea411bbe8458b7e2
570747%7C1%7C0%7C638041337811269412%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC
4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%
7C&sdata=Zjf%2Bpcx71yJyPb7JWkIlN70THvNnyqzd6yXHJ7lUmU4%3D&rese
rved=0 NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by
a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
