Wouldn't be that sure. Firstly, you spare iops. But that can be offset by caching. But secondly, if you post to event endpoint, you skip the whole linebreaking and timestamp detection and parsing part (and you have the timestamp already since you parsed it in rsyslog). Not to mention the flexibility. Been there, done that, got the t-shirt ;) I've seen working rsyslog->HEC setups running with several thousand EPS with no sweat.
On 2 November 2021 22:37:52 CET, David Lang <da...@lang.hm> wrote: >local file reads are much more efficient than HEC > >David Lang > >On Tue, 2 Nov 2021, Mariusz Kruk via rsyslog wrote: > >> Date: Tue, 02 Nov 2021 22:25:09 +0100 >> From: Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com> >> To: rsyslog-users <rsyslog@lists.adiscon.com> >> Cc: Mariusz Kruk <k...@epsilon.eu.org> >> Subject: Re: [rsyslog] (no subject) >> >> Regarding rsyslog and Splunk - why don't you use omhttp and post directly to >> HEC? >> >> >> On 2 November 2021 20:17:40 CET, Milan Koudelka via rsyslog >> <rsyslog@lists.adiscon.com> wrote: >>> Hi David, >>> thank you for your answer as well. This is a good hint. I'll remove it. >>> >>> I'm trying to solve a problem with high-traffic log files which are >>> read by the log management system Splunk. Sometimes rsyslog writes >>> only part of the line, Splunk reads it and then rsyslog finishes the >>> line. That causes corrupted events in Splunk. I didn't find any >>> solution for that on Splunk. From the Splunk side, there are >>> recommendations to wait longer before the file is considered as closed >>> (Splunk parameters time_before_close). But that didn't help. I hoped >>> that some fine-tuning of how rsyslog writes the file could help, but I >>> see that it would be probably even worse with queues. >>> >>> Milan Koudelka >>> >>> Principal SW engineer >>> >>> milan.koude...@gooddata.com >>> >>> +420 776 313 414 >>> >>> >>> Danube House >>> >>> Karolinská 650/1 >>> >>> 186 00 Prague 8, Czech Republic >>> >>> Twitter | Facebook | LinkedIn | Blog >>> >>> >>> >>> On Tue, Nov 2, 2021 at 7:24 PM David Lang <da...@lang.hm> wrote: >>>> >>>> It's almost always a bad idea to use a queue with omfile, it's slower to >>>> put the >>>> messages into the queue than to write them to disk >>>> >>>> David Lang >>>> >>>> On Tue, 2 Nov 2021, Milan Koudelka via rsyslog wrote: >>>> >>>> > Date: Tue, 2 Nov 2021 11:15:29 +0100 >>>> > From: Milan Koudelka via rsyslog <rsyslog@lists.adiscon.com> >>>> > To: rsyslog@lists.adiscon.com >>>> > Cc: Milan Koudelka <milan.koude...@gooddata.com> >>>> > Subject: [rsyslog] (no subject) >>>> > >>>> > Hi, >>>> > I tried to switch some rsyslog configurations to advanced format to >>>> > fine-tune actions. >>>> > >>>> > Instead of >>>> > local1.* /mnt/log/gdc;RawMsg >>>> > >>>> > I wrote >>>> > local1.* action(type="omfile" file="/mnt/log/gdc" template="RawMsg" >>>> > ioBufferSize="128k" queue.size="50000" queue.type="linkedlist" >>>> > queue.filename="gdc") >>>> > >>>> > As recommended, I kept other configurations as they were, if I don't >>>> > need any advanced configuration. >>>> > >>>> > Eg. >>>> > *.info;mail.none;authpriv.none;cron.none /mnt/log/messages >>>> > >>>> > But, I also changed legacy setting of file group ownership >>>> > $FileGroup splunk >>>> > $FileCreateMode 0640 >>>> > $DirGroup splunk >>>> > $DirCreateMode 0650 >>>> > >>>> > And I've put that directly to module load >>>> > module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat" >>>> > fileGroup="splunk" FileCreateMode="0640" dirGroup="splunk" >>>> > DirCreateMode="0650") >>>> > >>>> > The problem is, only the files configured with advanced format are >>>> > created with the correct group owner. /mnt/log/messages is created >>>> > under root user. Do I need to add back the legacy setting and keep >>>> > both in-module and legacy setting? >>>> > >>>> > Milan Koudelka >>>> > Principal SW engineer >>>> > milan.koude...@gooddata.com >>>> > +420 776 313 414 >>>> > >>>> > Danube House >>>> > Karolinská 650/1 >>>> > 186 00 Prague 8, Czech Republic >>>> > Twitter | Facebook | LinkedIn | Blog >>>> > _______________________________________________ >>>> > rsyslog mailing list >>>> > https://lists.adiscon.net/mailman/listinfo/rsyslog >>>> > http://www.rsyslog.com/professional-services/ >>>> > What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> > DON'T LIKE THAT. >>> _______________________________________________ >>> rsyslog mailing list >>> https://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >> -- >> Sent from my Android device with K-9 Mail. Please excuse my brevity. >> _______________________________________________ >> rsyslog mailing list >> https://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
