Thank Mariusz for all the info! I have great ground where to start some tests with Rsyslog - HEC configuration.
Milan Koudelka Principal SW engineer milan.koude...@gooddata.com +420 776 313 414 Danube House Karolinská 650/1 186 00 Prague 8, Czech Republic Twitter | Facebook | LinkedIn | Blog On Wed, Nov 3, 2021 at 9:37 AM Mariusz Kruk via rsyslog <rsyslog@lists.adiscon.com> wrote: > > I don't think there's a ready-made howto for the rsyslog->HEC connectivity. > > There's a general presentation about syslog to HEC - > https://conf.splunk.com/files/2017/slides/to-hec-with-syslog-scalable-aggregated-data-collection-in-splunk.pdf > > But it's quite dated and doesn't use omhttp but custom script. > > In general - posting to HEC is relatively easy. It requires omhttp with > properly rendered message using a template that produces proper json > containing at least the "event" field. I think it's easiest to create a > json object within rsyslog, set appropriate fields and just render it as > json > > The good thing about generating events straight from rsyslog into HEC is > that you can manipulate metadata on the fly - > https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/FormateventsforHTTPEventCollector > > You can also add "fields" json object with custom indexed fields! > > And if you have some weird logs format which is not your typical > syslog-conformant and for which rsyslog cannot detect and parse > timestamp, you can send it to the event endpoint with > auto_extract_timestamp=1 so that Splunk does the datetime parsing. > https://docs.splunk.com/Documentation/Splunk/8.2.3/Data/HECRESTendpoints > > So it's very, very flexible. > > One thing to remember is that if you supply the time field, it needs to > be rendered to a unix timestamp _with milliseconds part_. > > For example. > > template(name="timems" type="string" > string="%timereported:::date-unixtimestamp%.%timereported:::date-subseconds%") > template(name="hec" type="list") > { > property(name="!hec") > } > set $.time = exec_template("timems"); > /* That's the bare minimum needed for the event */ > set $!hec!event = $msg; > /* These are optional */ > set $!hec!time = $.time > set $!hec!source = $fromhost-ip; > /* Do your own logic here - for example set sourcetype and index */ > action(name="splunk-hec" template="hec" type="omhttp" > server="your.splunk.server" serverport="8088" > httpheaderkey="Authorization" httpheadervalue="Splunk > <your-splunk-hec-token>" > restpath="services/collector/event" > checkpath="services/collector/health" batch="on" batch.format="newline" ) > > That's pretty much it. Of course you can add TLS-related options, you > can do some heavy logic before calling the omhttp so you can easily > filter/redirect/manipulate the events before sending them to splunk. > > > On 03.11.2021 08:34, Rainer Gerhards via rsyslog wrote: > > Just a side-note: we once had a bug where omfile wrote only on buffer > > boundary and thus incomplete lines. I think this was fixed in 8.1905, > > but I may be wrong. If you want to pursue the file path, I would > > suggest upgrading to the current version and see if the problem you > > experience persists. It could, because the two processes (rsyslog > > writing and another one reading) are inherently racy. > > > > Rainer > > > > El mar, 2 nov 2021 a las 21:16, Milan Koudelka > > (<milan.koude...@gooddata.com>) escribió: > >> I'm using rsyslog-8.1911.0-7.el8_4.2.x86_64 > >> > >> Milan Koudelka > >> > >> Principal SW engineer > >> > >> milan.koude...@gooddata.com > >> > >> +420 776 313 414 > >> > >> > >> Danube House > >> > >> Karolinská 650/1 > >> > >> 186 00 Prague 8, Czech Republic > >> > >> Twitter | Facebook | LinkedIn | Blog > >> > >> > >> > >> > >> On Tue, Nov 2, 2021 at 9:14 PM Rainer Gerhards <rgerha...@hq.adiscon.com> > >> wrote: > >>> Which rsyslog version do you use? > >>> > >>> Rainer > >>> > >>> Milan Koudelka via rsyslog <rsyslog@lists.adiscon.com> schrieb am Di., 2. > >>> Nov. 2021, 20:18: > >>>> Hi David, > >>>> thank you for your answer as well. This is a good hint. I'll remove it. > >>>> > >>>> I'm trying to solve a problem with high-traffic log files which are > >>>> read by the log management system Splunk. Sometimes rsyslog writes > >>>> only part of the line, Splunk reads it and then rsyslog finishes the > >>>> line. That causes corrupted events in Splunk. I didn't find any > >>>> solution for that on Splunk. From the Splunk side, there are > >>>> recommendations to wait longer before the file is considered as closed > >>>> (Splunk parameters time_before_close). But that didn't help. I hoped > >>>> that some fine-tuning of how rsyslog writes the file could help, but I > >>>> see that it would be probably even worse with queues. > >>>> > >>>> Milan Koudelka > >>>> > >>>> Principal SW engineer > >>>> > >>>> milan.koude...@gooddata.com > >>>> > >>>> +420 776 313 414 > >>>> > >>>> > >>>> Danube House > >>>> > >>>> Karolinská 650/1 > >>>> > >>>> 186 00 Prague 8, Czech Republic > >>>> > >>>> Twitter | Facebook | LinkedIn | Blog > >>>> > >>>> > >>>> > >>>> On Tue, Nov 2, 2021 at 7:24 PM David Lang <da...@lang.hm> wrote: > >>>>> It's almost always a bad idea to use a queue with omfile, it's slower > >>>>> to put the > >>>>> messages into the queue than to write them to disk > >>>>> > >>>>> David Lang > >>>>> > >>>>> On Tue, 2 Nov 2021, Milan Koudelka via rsyslog wrote: > >>>>> > >>>>>> Date: Tue, 2 Nov 2021 11:15:29 +0100 > >>>>>> From: Milan Koudelka via rsyslog <rsyslog@lists.adiscon.com> > >>>>>> To: rsyslog@lists.adiscon.com > >>>>>> Cc: Milan Koudelka <milan.koude...@gooddata.com> > >>>>>> Subject: [rsyslog] (no subject) > >>>>>> > >>>>>> Hi, > >>>>>> I tried to switch some rsyslog configurations to advanced format to > >>>>>> fine-tune actions. > >>>>>> > >>>>>> Instead of > >>>>>> local1.* /mnt/log/gdc;RawMsg > >>>>>> > >>>>>> I wrote > >>>>>> local1.* action(type="omfile" file="/mnt/log/gdc" template="RawMsg" > >>>>>> ioBufferSize="128k" queue.size="50000" queue.type="linkedlist" > >>>>>> queue.filename="gdc") > >>>>>> > >>>>>> As recommended, I kept other configurations as they were, if I don't > >>>>>> need any advanced configuration. > >>>>>> > >>>>>> Eg. > >>>>>> *.info;mail.none;authpriv.none;cron.none /mnt/log/messages > >>>>>> > >>>>>> But, I also changed legacy setting of file group ownership > >>>>>> $FileGroup splunk > >>>>>> $FileCreateMode 0640 > >>>>>> $DirGroup splunk > >>>>>> $DirCreateMode 0650 > >>>>>> > >>>>>> And I've put that directly to module load > >>>>>> module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat" > >>>>>> fileGroup="splunk" FileCreateMode="0640" dirGroup="splunk" > >>>>>> DirCreateMode="0650") > >>>>>> > >>>>>> The problem is, only the files configured with advanced format are > >>>>>> created with the correct group owner. /mnt/log/messages is created > >>>>>> under root user. Do I need to add back the legacy setting and keep > >>>>>> both in-module and legacy setting? > >>>>>> > >>>>>> Milan Koudelka > >>>>>> Principal SW engineer > >>>>>> milan.koude...@gooddata.com > >>>>>> +420 776 313 414 > >>>>>> > >>>>>> Danube House > >>>>>> Karolinská 650/1 > >>>>>> 186 00 Prague 8, Czech Republic > >>>>>> Twitter | Facebook | LinkedIn | Blog > >>>>>> _______________________________________________ > >>>>>> rsyslog mailing list > >>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com/professional-services/ > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > >>>>>> if you DON'T LIKE THAT. > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> https://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > https://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > > LIKE THAT. > _______________________________________________ > rsyslog mailing list > https://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
