Good link thanks for that.... Lack of file probably on me, i'm sure I forgot to attach. Coffee hadn't kicked in. ________________________________ From: David Lang <da...@lang.hm> Sent: Tuesday, April 26, 2022 9:57 AM To: Steven D <pheerl...@hotmail.com> Cc: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com> Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting
hmm, looks like it didn't make it through the mail server https://cromwell-intl.com/open-source/performance-tuning/tcp.html (also note there are similar UDP buffer tuning parameters, if those buffers fill up udp logs will be dropped silently) David Lang On Tue, 26 Apr 2022, Steven D wrote: > Date: Tue, 26 Apr 2022 11:00:16 +0000 > From: Steven D <pheerl...@hotmail.com> > To: David Lang <da...@lang.hm>, rsyslog-users <rsyslog@lists.adiscon.com> > Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting > > Here's most recent few rotations of pstats data, any additional input would > be appreciated. > > With the keepalives set, TCP connections don't seem to be growing...however > it still seems that using TCP (at least for the high throughput firewalls) > causes ingest to drop significantly. Are there any OS side (RHEL 8) > optimizations that youz could suggestion that make help? > > Regards, > Steven > ________________________________ > From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of Steven D via > rsyslog <rsyslog@lists.adiscon.com> > Sent: Monday, April 25, 2022 9:52 AM > To: David Lang <da...@lang.hm>; rsyslog-users <rsyslog@lists.adiscon.com> > Cc: Steven D <pheerl...@hotmail.com> > Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting > > Kind of depends on SIEM agent... With our set up, having the files written to > disk allows us to target/optimize the front-end parsing applied by the SIEM > agent a little better. Make is so there's less work to be done elsewhere in > the solution, but it could be done other ways. > > Glad there are other SIEM jockeys on here... haha. > ________________________________ > From: rsyslog <rsyslog-boun...@lists.adiscon.com> on behalf of Mariusz Kruk > via rsyslog <rsyslog@lists.adiscon.com> > Sent: Monday, April 25, 2022 9:08 AM > To: David Lang <da...@lang.hm> > Cc: Mariusz Kruk <k...@epsilon.eu.org>; Mariusz Kruk via rsyslog > <rsyslog@lists.adiscon.com> > Subject: Re: [rsyslog] Basic Rsyslog Troubleshooting > > > On 25.04.2022 15:04, David Lang wrote: >>>>> Sure. I work with them, I know ;-) >>>>> >>>>> It's just that for some, you can do the same but using rsyslog to >>>>> process the message (even filter some events out or trim them or do >>>>> many other fancy stuff) an send them directly to SIEM (by means of >>>>> native SIEM API, not by syslog) >>>>> >>>>> instead of killing the server with IOPS. That's all. >>>> >>>> fair point, but I'll point out that since rsyslog is buffering the >>>> writes to disk, the IOPS load is not as high as you would think. >>>> It's sequential writes and reads, and in practice should be just >>>> sequential writes as the reads should be able to be served from ram >>>> (the files should be read shortly after they are written, so should >>>> still be cached) >>>> >>>> It would be interesting to look into the performance (CPU and I/O) >>>> of writing to disk and having the files read in batches vs posting >>>> each message to the SIEM. I think there are enough variables >>>> involved that it's not an obvious win either way. >>>> >>> Sure, there is also another layer of OS-level caching/buffering. But >>> in the end it all has to get written to the disk. ;-) I just think >>> that passing the events straight through rulesets from input to >>> output (and I mean the real destination, not to the files) is simply >>> more straightforward and easier to manage (after all you don't have >>> to - for example - deal with file rotation and so on). On the other >>> hand, you may run into some other problems when your rulesets get too >>> complicated and "run away". >> >> buffered, sequential writes to disk are pretty cheap and perform well >> even on slow spinning rust, and file rotation (should) happen during >> off-peak periods, and as previously mentioned gives you a handy backup >> for those cases where something goes wrong with your SIEM. Yes, you >> could have disk backed queues in rsyslog, but the performance of such >> queues is horrid and significantly more I/O load than writing to disk. >> >> Not saying that sending directly is wrong, just listing things I've >> run into that make me default the other way. >> > Yep. I wasn't talking about file rotation in terms of performance (since > it should be relatively cheap anyway, especially if you don't want to > compress old files). I was referring more to the manageability of the > whole solution. > > Adding another ruleset seems easier than making sure that you have a > proper directory prepared for the files, possibly reconfiguring your > SIEM agent, making sure the files are included in the rotation scheme... > Seems like a headache. > > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. >
pstats_04262022.log
Description: pstats_04262022.log
_______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
