I actually don't need it real-time. 1 to 5 minutes delay would be acceptable.
That means, the log file wouldn't be available locally longer than 1 minute. It could cause some problems to engineers if they would like to tail such a file locally. Anyway, it is a good idea. I'll read more about it. Thank you. Milan Koudelka Principal SW engineer milan.koude...@gooddata.com +420 776 313 414 Danube House Karolinská 650/1 186 00 Prague 8, Czech Republic Twitter | Facebook | LinkedIn | Blog On Tue, Nov 2, 2021 at 8:51 PM David Lang <da...@lang.hm> wrote: > > do you really need the file contents in real-time? what I like to do is to > rotate the file every min and mv the file to a directory that splunk reads > (and > let splunk delete the file after it's read see the sinkhole feature) > > that way the file is known to be complete when splunk gets it and if splunk > stops reading, the files accumulate until splunk gets around to it wihtout > leaving stray files around. > > David Lang > > On Tue, 2 Nov 2021, Milan Koudelka > wrote: > > > Date: Tue, 2 Nov 2021 20:17:40 +0100 > > From: Milan Koudelka <milan.koude...@gooddata.com> > > To: David Lang <da...@lang.hm> > > Cc: Milan Koudelka via rsyslog <rsyslog@lists.adiscon.com> > > Subject: Re: [rsyslog] (no subject) > > > > Hi David, > > thank you for your answer as well. This is a good hint. I'll remove it. > > > > I'm trying to solve a problem with high-traffic log files which are > > read by the log management system Splunk. Sometimes rsyslog writes > > only part of the line, Splunk reads it and then rsyslog finishes the > > line. That causes corrupted events in Splunk. I didn't find any > > solution for that on Splunk. From the Splunk side, there are > > recommendations to wait longer before the file is considered as closed > > (Splunk parameters time_before_close). But that didn't help. I hoped > > that some fine-tuning of how rsyslog writes the file could help, but I > > see that it would be probably even worse with queues. > > > > Milan Koudelka > > > > Principal SW engineer > > > > milan.koude...@gooddata.com > > > > +420 776 313 414 > > > > > > Danube House > > > > Karolinská 650/1 > > > > 186 00 Prague 8, Czech Republic > > > > Twitter | Facebook | LinkedIn | Blog > > > > > > > > On Tue, Nov 2, 2021 at 7:24 PM David Lang <da...@lang.hm> wrote: > >> > >> It's almost always a bad idea to use a queue with omfile, it's slower to > >> put the > >> messages into the queue than to write them to disk > >> > >> David Lang > >> > >> On Tue, 2 Nov 2021, Milan Koudelka via rsyslog wrote: > >> > >>> Date: Tue, 2 Nov 2021 11:15:29 +0100 > >>> From: Milan Koudelka via rsyslog <rsyslog@lists.adiscon.com> > >>> To: rsyslog@lists.adiscon.com > >>> Cc: Milan Koudelka <milan.koude...@gooddata.com> > >>> Subject: [rsyslog] (no subject) > >>> > >>> Hi, > >>> I tried to switch some rsyslog configurations to advanced format to > >>> fine-tune actions. > >>> > >>> Instead of > >>> local1.* /mnt/log/gdc;RawMsg > >>> > >>> I wrote > >>> local1.* action(type="omfile" file="/mnt/log/gdc" template="RawMsg" > >>> ioBufferSize="128k" queue.size="50000" queue.type="linkedlist" > >>> queue.filename="gdc") > >>> > >>> As recommended, I kept other configurations as they were, if I don't > >>> need any advanced configuration. > >>> > >>> Eg. > >>> *.info;mail.none;authpriv.none;cron.none /mnt/log/messages > >>> > >>> But, I also changed legacy setting of file group ownership > >>> $FileGroup splunk > >>> $FileCreateMode 0640 > >>> $DirGroup splunk > >>> $DirCreateMode 0650 > >>> > >>> And I've put that directly to module load > >>> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat" > >>> fileGroup="splunk" FileCreateMode="0640" dirGroup="splunk" > >>> DirCreateMode="0650") > >>> > >>> The problem is, only the files configured with advanced format are > >>> created with the correct group owner. /mnt/log/messages is created > >>> under root user. Do I need to add back the legacy setting and keep > >>> both in-module and legacy setting? > >>> > >>> Milan Koudelka > >>> Principal SW engineer > >>> milan.koude...@gooddata.com > >>> +420 776 313 414 > >>> > >>> Danube House > >>> Karolinská 650/1 > >>> 186 00 Prague 8, Czech Republic > >>> Twitter | Facebook | LinkedIn | Blog > >>> _______________________________________________ > >>> rsyslog mailing list > >>> https://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > > _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
