I don't think there's a ready-made howto for the rsyslog->HEC connectivity.
There's a general presentation about syslog to HEC -
https://conf.splunk.com/files/2017/slides/to-hec-with-syslog-scalable-aggregated-data-collection-in-splunk.pdf
But it's quite dated and doesn't use omhttp but custom script.
In general - posting to HEC is relatively easy. It requires omhttp with
properly rendered message using a template that produces proper json
containing at least the "event" field. I think it's easiest to create a
json object within rsyslog, set appropriate fields and just render it as
json
The good thing about generating events straight from rsyslog into HEC is
that you can manipulate metadata on the fly -
https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/FormateventsforHTTPEventCollector
You can also add "fields" json object with custom indexed fields!
And if you have some weird logs format which is not your typical
syslog-conformant and for which rsyslog cannot detect and parse
timestamp, you can send it to the event endpoint with
auto_extract_timestamp=1 so that Splunk does the datetime parsing.
https://docs.splunk.com/Documentation/Splunk/8.2.3/Data/HECRESTendpoints
So it's very, very flexible.
One thing to remember is that if you supply the time field, it needs to
be rendered to a unix timestamp _with milliseconds part_.
For example.
template(name="timems" type="string"
string="%timereported:::date-unixtimestamp%.%timereported:::date-subseconds%")
template(name="hec" type="list")
{
property(name="!hec")
}
set $.time = exec_template("timems");
/* That's the bare minimum needed for the event */
set $!hec!event = $msg;
/* These are optional */
set $!hec!time = $.time
set $!hec!source = $fromhost-ip;
/* Do your own logic here - for example set sourcetype and index */
action(name="splunk-hec" template="hec" type="omhttp"
server="your.splunk.server" serverport="8088"
httpheaderkey="Authorization" httpheadervalue="Splunk
<your-splunk-hec-token>"
restpath="services/collector/event"
checkpath="services/collector/health" batch="on" batch.format="newline" )
That's pretty much it. Of course you can add TLS-related options, you
can do some heavy logic before calling the omhttp so you can easily
filter/redirect/manipulate the events before sending them to splunk.
On 03.11.2021 08:34, Rainer Gerhards via rsyslog wrote:
Just a side-note: we once had a bug where omfile wrote only on buffer
boundary and thus incomplete lines. I think this was fixed in 8.1905,
but I may be wrong. If you want to pursue the file path, I would
suggest upgrading to the current version and see if the problem you
experience persists. It could, because the two processes (rsyslog
writing and another one reading) are inherently racy.
Rainer
El mar, 2 nov 2021 a las 21:16, Milan Koudelka
(<milan.koude...@gooddata.com>) escribió:
I'm using rsyslog-8.1911.0-7.el8_4.2.x86_64
Milan Koudelka
Principal SW engineer
milan.koude...@gooddata.com
+420 776 313 414
Danube House
Karolinská 650/1
186 00 Prague 8, Czech Republic
Twitter | Facebook | LinkedIn | Blog
On Tue, Nov 2, 2021 at 9:14 PM Rainer Gerhards <rgerha...@hq.adiscon.com> wrote:
Which rsyslog version do you use?
Rainer
Milan Koudelka via rsyslog <rsyslog@lists.adiscon.com> schrieb am Di., 2. Nov.
2021, 20:18:
Hi David,
thank you for your answer as well. This is a good hint. I'll remove it.
I'm trying to solve a problem with high-traffic log files which are
read by the log management system Splunk. Sometimes rsyslog writes
only part of the line, Splunk reads it and then rsyslog finishes the
line. That causes corrupted events in Splunk. I didn't find any
solution for that on Splunk. From the Splunk side, there are
recommendations to wait longer before the file is considered as closed
(Splunk parameters time_before_close). But that didn't help. I hoped
that some fine-tuning of how rsyslog writes the file could help, but I
see that it would be probably even worse with queues.
Milan Koudelka
Principal SW engineer
milan.koude...@gooddata.com
+420 776 313 414
Danube House
Karolinská 650/1
186 00 Prague 8, Czech Republic
Twitter | Facebook | LinkedIn | Blog
On Tue, Nov 2, 2021 at 7:24 PM David Lang <da...@lang.hm> wrote:
It's almost always a bad idea to use a queue with omfile, it's slower to put the
messages into the queue than to write them to disk
David Lang
On Tue, 2 Nov 2021, Milan Koudelka via rsyslog wrote:
Date: Tue, 2 Nov 2021 11:15:29 +0100
From: Milan Koudelka via rsyslog <rsyslog@lists.adiscon.com>
To: rsyslog@lists.adiscon.com
Cc: Milan Koudelka <milan.koude...@gooddata.com>
Subject: [rsyslog] (no subject)
Hi,
I tried to switch some rsyslog configurations to advanced format to
fine-tune actions.
Instead of
local1.* /mnt/log/gdc;RawMsg
I wrote
local1.* action(type="omfile" file="/mnt/log/gdc" template="RawMsg"
ioBufferSize="128k" queue.size="50000" queue.type="linkedlist"
queue.filename="gdc")
As recommended, I kept other configurations as they were, if I don't
need any advanced configuration.
Eg.
*.info;mail.none;authpriv.none;cron.none /mnt/log/messages
But, I also changed legacy setting of file group ownership
$FileGroup splunk
$FileCreateMode 0640
$DirGroup splunk
$DirCreateMode 0650
And I've put that directly to module load
module(load="builtin:omfile" Template="RSYSLOG_TraditionalFileFormat"
fileGroup="splunk" FileCreateMode="0640" dirGroup="splunk"
DirCreateMode="0650")
The problem is, only the files configured with advanced format are
created with the correct group owner. /mnt/log/messages is created
under root user. Do I need to add back the legacy setting and keep
both in-module and legacy setting?
Milan Koudelka
Principal SW engineer
milan.koude...@gooddata.com
+420 776 313 414
Danube House
Karolinská 650/1
186 00 Prague 8, Czech Republic
Twitter | Facebook | LinkedIn | Blog
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
