Thank you for another idea. I always preferred log files, because in case of any temporary outage, I can read them later. We use HEC with Kubernetes anyway. It is another idea what we can try. It looks like to me you are experienced with such a solution. Can you point me to some docs or presentations how to make this bullet proof?
Milan Koudelka Principal SW engineer milan.koude...@gooddata.com +420 776 313 414 Danube House Karolinská 650/1 186 00 Prague 8, Czech Republic Twitter | Facebook | LinkedIn | Blog On Tue, Nov 2, 2021 at 10:25 PM Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com> wrote: > > Regarding rsyslog and Splunk - why don't you use omhttp and post directly to > HEC? > > > On 2 November 2021 20:17:40 CET, Milan Koudelka via rsyslog > <rsyslog@lists.adiscon.com> wrote: > >Hi David, > >thank you for your answer as well. This is a good hint. I'll remove it. > > > >I'm trying to solve a problem with high-traffic log files which are > >read by the log management system Splunk. Sometimes rsyslog writes > >only part of the line, Splunk reads it and then rsyslog finishes the > >line. That causes corrupted events in Splunk. I didn't find any > >solution for that on Splunk. From the Splunk side, there are > >recommendations to wait longer before the file is considered as closed > >(Splunk parameters time_before_close). But that didn't help. I hoped > >that some fine-tuning of how rsyslog writes the file could help, but I > >see that it would be probably even worse with queues. > > > >Milan Koudelka > > > >Principal SW engineer > > > >milan.koude...@gooddata.com > > > >+420 776 313 414 > > > > > >Danube House > > > >Karolinská 650/1 > > > >186 00 Prague 8, Czech Republic > > > >Twitter | Facebook | LinkedIn | Blog > > > > > > > >On Tue, Nov 2, 2021 at 7:24 PM David Lang <da...@lang.hm> wrote: > >> > >> It's almost always a bad idea to use a queue with omfile, it's slower to > >> put the > >> messages into the queue than to write them to disk > >> > >> David Lang > >> > >> On Tue, 2 Nov 2021, Milan Koudelka via rsyslog wrote: > >> > >> > Date: Tue, 2 Nov 2021 11:15:29 +0100 > >> > From: Milan Koudelka via rsyslog <rsyslog@lists.adiscon.com> > >> > To: rsyslog@lists.adiscon.com > >> > Cc: Milan Koudelka <milan.koude...@gooddata.com> > >> > Subject: [rsyslog] (no subject) > >> > > >> > Hi, > >> > I tried to switch some rsyslog configurations to advanced format to > >> > fine-tune actions. > >> > > >> > Instead of > >> > local1.* /mnt/log/gdc;RawMsg > >> > > >> > I wrote > >> > local1.* action(type="omfile" file="/mnt/log/gdc" template="RawMsg" > >> > ioBufferSize="128k" queue.size="50000" queue.type="linkedlist" > >> > queue.filename="gdc") > >> > > >> > As recommended, I kept other configurations as they were, if I don't > >> > need any advanced configuration. > >> > > >> > Eg. > >> > *.info;mail.none;authpriv.none;cron.none /mnt/log/messages > >> > > >> > But, I also changed legacy setting of file group ownership > >> > $FileGroup splunk > >> > $FileCreateMode 0640 > >> > $DirGroup splunk > >> > $DirCreateMode 0650 > >> > > >> > And I've put that directly to module load > >> > module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat" > >> > fileGroup="splunk" FileCreateMode="0640" dirGroup="splunk" > >> > DirCreateMode="0650") > >> > > >> > The problem is, only the files configured with advanced format are > >> > created with the correct group owner. /mnt/log/messages is created > >> > under root user. Do I need to add back the legacy setting and keep > >> > both in-module and legacy setting? > >> > > >> > Milan Koudelka > >> > Principal SW engineer > >> > milan.koude...@gooddata.com > >> > +420 776 313 414 > >> > > >> > Danube House > >> > Karolinská 650/1 > >> > 186 00 Prague 8, Czech Republic > >> > Twitter | Facebook | LinkedIn | Blog > >> > _______________________________________________ > >> > rsyslog mailing list > >> > https://lists.adiscon.net/mailman/listinfo/rsyslog > >> > http://www.rsyslog.com/professional-services/ > >> > What's up with rsyslog? Follow https://twitter.com/rgerhards > >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> > DON'T LIKE THAT. > >_______________________________________________ > >rsyslog mailing list > >https://lists.adiscon.net/mailman/listinfo/rsyslog > >http://www.rsyslog.com/professional-services/ > >What's up with rsyslog? Follow https://twitter.com/rgerhards > >NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > >sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > >LIKE THAT. > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
