rsyslog documentation suffers from being written by people too close to it's
development (too many assumptions baked in), given the troubleshooting excercise
that you just went through, would it be reasonable for you to try and writeup a
guide based on what we talked through with you to help the next person?
David Lang
On Wed, 11 May 2022, Steven D wrote:
Date: Wed, 11 May 2022 22:56:38 +0000
From: Steven D <pheerl...@hotmail.com>
To: John Chivian <jchiv...@chivian.com>,
rsyslog-users <rsyslog@lists.adiscon.com>
Cc: David Lang <da...@lang.hm>
Subject: RE: [rsyslog] Basic Rsyslog Troubleshooting
Wanted to circle back around to this topic for a closure update.
We were able to diagnose that one particular data source (a specific firewall
vendor) appears to have trouble consistently sending TCP ... So we switched it
to UDP and problem is gone. Ultimately no issues with rsyslog or the OS were at
fault.
The rest of the diagnostic help from here helped us locate a couple other
issues we wouldnt have noticed otherwise.
Thanks a lot all!
Regards,
Steven.
-------- Original message --------
From: John Chivian <jchiv...@chivian.com>
Date: 4/26/22 12:58 PM (GMT-05:00)
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: David Lang <da...@lang.hm>, Steven D <pheerl...@hotmail.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
I second David's thought to switch to imptcp. It is designed for plain TCP
performance.
I would also like to say that I previously thought having rsyslog write to a
file for a SIEM’s agent ingestion was a bad idea. Generally I make the SIEM do
the TCP listening itself and relay through rsyslog to the SIEM in this manner.
However, David makes some excellent points, and sometimes it’s easier to do
some of the in-transit work with a SIEM agent than with rsyslog. Throw in the
fact that storage virtualization often hides the rust or bits beneath, and I
could see it being viable and equally as performant. Things that make you go,
“Hmm…”
Thanks Gents!
On Apr 26, 2022, at 11:21, Steven D via rsyslog <rsyslog@lists.adiscon.com>
wrote:
Interesting... thanks for the tips. As long as there's no overt issues, I won't
press my luck by clobbering the config 🙂
I think what we're seeing at this point is some system side inability to handle
TCP at the same volume as UDP.... Could be OS side, still plinko-ing down that
rabbithole.
________________________________
From: David Lang <da...@lang.hm>
Sent: Tuesday, April 26, 2022 10:23 AM
To: Steven D <pheerl...@hotmail.com>
Cc: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
that looks pretty good to me, if it's still running slow, I would look at the
threads in top to see what their utilization is, and try the imptcp vs imtcp and
see if there's a difference
the way you are specifying a ruleset in an input means that you don't have lots
of rule processing happening.
one thing you could have is contention for the lock on the main queue. By
default, all messages go to the main queue. If you are tieing an input to a
ruleset, you can give that ruleset a queue and the messages will never touch the
main queue (in effect, it's like you were running a separate rsyslog instance
for that input and ruleset)
David Lang
On Tue, 26 Apr 2022, Steven D wrote:
Date: Tue, 26 Apr 2022 14:12:36 +0000
From: Steven D <pheerl...@hotmail.com>
To: David Lang <da...@lang.hm>
Cc: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
Good link thanks for that.... Lack of file probably on me, i'm sure I forgot to
attach. Coffee hadn't kicked in.
________________________________
From: David Lang <da...@lang.hm>
Sent: Tuesday, April 26, 2022 9:57 AM
To: Steven D <pheerl...@hotmail.com>
Cc: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
hmm, looks like it didn't make it through the mail server
https://cromwell-intl.com/open-source/performance-tuning/tcp.html
(also note there are similar UDP buffer tuning parameters, if those buffers fill
up udp logs will be dropped silently)
David Lang
On Tue, 26 Apr 2022, Steven D wrote:
Date: Tue, 26 Apr 2022 11:00:16 +0000
From: Steven D <pheerl...@hotmail.com>
To: David Lang <da...@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
Here's most recent few rotations of pstats data, any additional input would be
appreciated.
With the keepalives set, TCP connections don't seem to be growing...however it
still seems that using TCP (at least for the high throughput firewalls) causes
ingest to drop significantly. Are there any OS side (RHEL 8) optimizations that
youz could suggestion that make help?
Regards,
Steven
________________________________
From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of Steven D via rsyslog
<rsyslog@lists.adiscon.com>
Sent: Monday, April 25, 2022 9:52 AM
To: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Steven D <pheerl...@hotmail.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
Kind of depends on SIEM agent... With our set up, having the files written to
disk allows us to target/optimize the front-end parsing applied by the SIEM
agent a little better. Make is so there's less work to be done elsewhere in the
solution, but it could be done other ways.
Glad there are other SIEM jockeys on here... haha.
________________________________
From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of Mariusz Kruk via
rsyslog <rsyslog@lists.adiscon.com>
Sent: Monday, April 25, 2022 9:08 AM
To: David Lang <da...@lang.hm>
Cc: Mariusz Kruk <k...@epsilon.eu.org>; Mariusz Kruk via rsyslog
<rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
On 25.04.2022 15:04, David Lang wrote:
Sure. I work with them, I know ;-)
It's just that for some, you can do the same but using rsyslog to
process the message (even filter some events out or trim them or do
many other fancy stuff) an send them directly to SIEM (by means of
native SIEM API, not by syslog)
instead of killing the server with IOPS. That's all.
fair point, but I'll point out that since rsyslog is buffering the
writes to disk, the IOPS load is not as high as you would think.
It's sequential writes and reads, and in practice should be just
sequential writes as the reads should be able to be served from ram
(the files should be read shortly after they are written, so should
still be cached)
It would be interesting to look into the performance (CPU and I/O)
of writing to disk and having the files read in batches vs posting
each message to the SIEM. I think there are enough variables
involved that it's not an obvious win either way.
Sure, there is also another layer of OS-level caching/buffering. But
in the end it all has to get written to the disk. ;-) I just think
that passing the events straight through rulesets from input to
output (and I mean the real destination, not to the files) is simply
more straightforward and easier to manage (after all you don't have
to - for example - deal with file rotation and so on). On the other
hand, you may run into some other problems when your rulesets get too
complicated and "run away".
buffered, sequential writes to disk are pretty cheap and perform well
even on slow spinning rust, and file rotation (should) happen during
off-peak periods, and as previously mentioned gives you a handy backup
for those cases where something goes wrong with your SIEM. Yes, you
could have disk backed queues in rsyslog, but the performance of such
queues is horrid and significantly more I/O load than writing to disk.
Not saying that sending directly is wrong, just listing things I've
run into that make me default the other way.
Yep. I wasn't talking about file rotation in terms of performance (since
it should be relatively cheap anyway, especially if you don't want to
compress old files). I was referring more to the manageability of the
whole solution.
Adding another ruleset seems easier than making sure that you have a
proper directory prepared for the files, possibly reconfiguring your
SIEM agent, making sure the files are included in the rotation scheme...
Seems like a headache.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
