Regarding rsyslog and Splunk - why don't you use omhttp and post directly to HEC?
On 2 November 2021 20:17:40 CET, Milan Koudelka via rsyslog <rsyslog@lists.adiscon.com> wrote: >Hi David, >thank you for your answer as well. This is a good hint. I'll remove it. > >I'm trying to solve a problem with high-traffic log files which are >read by the log management system Splunk. Sometimes rsyslog writes >only part of the line, Splunk reads it and then rsyslog finishes the >line. That causes corrupted events in Splunk. I didn't find any >solution for that on Splunk. From the Splunk side, there are >recommendations to wait longer before the file is considered as closed >(Splunk parameters time_before_close). But that didn't help. I hoped >that some fine-tuning of how rsyslog writes the file could help, but I >see that it would be probably even worse with queues. > >Milan Koudelka > >Principal SW engineer > >milan.koude...@gooddata.com > >+420 776 313 414 > > >Danube House > >Karolinská 650/1 > >186 00 Prague 8, Czech Republic > >Twitter | Facebook | LinkedIn | Blog > > > >On Tue, Nov 2, 2021 at 7:24 PM David Lang <da...@lang.hm> wrote: >> >> It's almost always a bad idea to use a queue with omfile, it's slower to put >> the >> messages into the queue than to write them to disk >> >> David Lang >> >> On Tue, 2 Nov 2021, Milan Koudelka via rsyslog wrote: >> >> > Date: Tue, 2 Nov 2021 11:15:29 +0100 >> > From: Milan Koudelka via rsyslog <rsyslog@lists.adiscon.com> >> > To: rsyslog@lists.adiscon.com >> > Cc: Milan Koudelka <milan.koude...@gooddata.com> >> > Subject: [rsyslog] (no subject) >> > >> > Hi, >> > I tried to switch some rsyslog configurations to advanced format to >> > fine-tune actions. >> > >> > Instead of >> > local1.* /mnt/log/gdc;RawMsg >> > >> > I wrote >> > local1.* action(type="omfile" file="/mnt/log/gdc" template="RawMsg" >> > ioBufferSize="128k" queue.size="50000" queue.type="linkedlist" >> > queue.filename="gdc") >> > >> > As recommended, I kept other configurations as they were, if I don't >> > need any advanced configuration. >> > >> > Eg. >> > *.info;mail.none;authpriv.none;cron.none /mnt/log/messages >> > >> > But, I also changed legacy setting of file group ownership >> > $FileGroup splunk >> > $FileCreateMode 0640 >> > $DirGroup splunk >> > $DirCreateMode 0650 >> > >> > And I've put that directly to module load >> > module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat" >> > fileGroup="splunk" FileCreateMode="0640" dirGroup="splunk" >> > DirCreateMode="0650") >> > >> > The problem is, only the files configured with advanced format are >> > created with the correct group owner. /mnt/log/messages is created >> > under root user. Do I need to add back the legacy setting and keep >> > both in-module and legacy setting? >> > >> > Milan Koudelka >> > Principal SW engineer >> > milan.koude...@gooddata.com >> > +420 776 313 414 >> > >> > Danube House >> > Karolinská 650/1 >> > 186 00 Prague 8, Czech Republic >> > Twitter | Facebook | LinkedIn | Blog >> > _______________________________________________ >> > rsyslog mailing list >> > https://lists.adiscon.net/mailman/listinfo/rsyslog >> > http://www.rsyslog.com/professional-services/ >> > What's up with rsyslog? Follow https://twitter.com/rgerhards >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> > DON'T LIKE THAT. >_______________________________________________ >rsyslog mailing list >https://lists.adiscon.net/mailman/listinfo/rsyslog >http://www.rsyslog.com/professional-services/ >What's up with rsyslog? Follow https://twitter.com/rgerhards >NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE >THAT. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
