There is also some inconsistency within the documentation page (it lists
OMSR_TPL_AS_MSG as the default template whereas few lines later it shows
a definition of a template named BSON which is supposed to be default).
So I suspect that something wasn't quite finished here ;-)
On 5.10.2022 12:03, Mariusz Kruk wrote:
Out of sheer curiosity I looked into the ommongodb.c and found this:
557 if(pData->tplName == NULL) {
558 doc = getDefaultBSON(*(smsg_t**)pMsgData);
559 } else {
560 doc = BSONFromJSONObject(*(struct json_object **)pMsgData);
561 }
I'm not that good in rsyslog internals but for me it seems as if the
module does not use the specified template at all. If the template is
not specified, it creates a BSON with pre-defined files. If the
template is specified, it just dumps a bson object from the message data.
On 5.10.2022 11:28, Marcin Mirosław via rsyslog wrote:
Hi!
If I use omfile then logs contains messages in my desired format but
not with ommongodb. tcpdump shows:
..msg.....{"foo":"bar"}..
Marcin
W dniu 2022-10-05 08:05, Mariusz Kruk via rsyslog napisał(a):
Looks relatively normal.
You can of course try writing to a file with your "i-json" template to
make sure that's what you want to be sent to mongodb.
But then, if your ommongodb action does contain the template="i-json"
parameter, it simply should work.
The only other thing you can do to make sure what's going on over the
wire is of course the tcpdump (unless your traffic is encrypted).
Ot simply looks that it should work - it should not need any more
"processing" as it is.
On 4.10.2022 15:45, Marcin Mirosław wrote:
:)
I had to censore log...
Debug line with all properties:
FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: 'linux',
PRI: 174,
syslogtag 'a1', programname: 'x1', APP-NAME: 'x1', PROCID: '-',
MSGID: '-',
TIMESTAMP: 'Oct 4 15:38:53', STRUCTURED-DATA: '[timeQuality
tzKnown="1" isSynced="1" syncAccuracy="415383"]',
msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
escaped msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
inputname: imtcp rawmsg: '<174>1 2022-10-04T15:38:53.219052+02:00
linux a1 - - [timeQuality tzKnown="1" isSynced="1"
syncAccuracy="415383"] {"foo":"bar
","mode":750,"date":"2022-10-04T15:38:53"}'
$!:
$.:
$/:
W dniu 2022-10-04 09:18, Mariusz Kruk via rsyslog napisał(a):
:-)
OK, I understand that you did
template(name="i-json" type="string" string="%msg%")
action(type="ommongodb" server="..." [...] template="i-json")
right?
This way you should indeed be pushing the %msg% part of the incoming
syslog message.
Try adding
action(type="omfile" file="/tmp/debug.log"
template="RSYSLOG_DebugFormat")
immediately before your ommongodb action to see what exactly your
properties look like. That usually helps finding what's happening
inside your processing pipeline.
On 4.10.2022 09:03, Marcin Mirosław wrote:
I did in the part about connection to mongodb ;)
"It will be used automatically if no other template is specified
to be used"
But I specified template: template(name="ui-json" type="string"
string="%msg%")
so if I understand qutoed docs correctly, all about default,
canned template doesn't apply to my case, am I right?
W dniu 2022-10-04 08:27, Mariusz Kruk napisał(a):
You didn't read the docs, did you? ;-)
https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
<https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
"Note rsyslog contains a canned default template to write to the
MongoDB. It will be used automatically if no other template is
specified to be used. This template is:
template(name="BSON" type="string" string="\\"sys\\" :
\\"%hostname%\\",
\\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
\\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
\\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\" :
\\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
\\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
\\"level\\" : \\"%syslogpriority-text%\\"")
This creates the BSON document needed for MongoDB if no template
is specified. The default schema is aligned to CEE and project
lumberjack. As such, the field names are standard lumberjack
field names, and *not* rsyslog property names
<https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
On 3.10.2022 22:02, Marcin Mirosław wrote:
W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
Don't know about this particular output module but in general
what you want is for rsyslog to parse the message and insert
it as json object.
Meseems that now rsyslog put %msg% as json object. (
ex: msg: '{"foo":"bar"}' }
)
So you need to use parse_json() function on the input string
and then use proper template which will render the json to
appropirate string. I use similar approach (without the
parsing part) to create output json for Splunk's HEC input -
the idea is roughly the same.
On 3.10.2022 18:35, Marcin Mirosław via rsyslog wrote:
Maybe when I show examples from mongo it will be more clear.
> db.log2.find()
[
{ _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
'{"foo":"bar"}' },
{ _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
]
First record shows how msg is inserted to mongo by rsyslog.
Second record is what I would like to get.
rsyslog do:
db.log2.insert({msg:'{"foo":"bar"}'})
but I'd like to have:
db.log2.insert({"foo":"bar"})
Marcin
W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
I do not fully understand the question (maybe language issue
on my
side), but there is a syntax error:
In a string template, properties must be enclosed in percent
sign. so:
... string="%msg%
HTH
Rainer
El lun, 3 oct 2022 a las 13:18, Marcin Mirosław via rsyslog
(<rsyslog@lists.adiscon.com>
<mailto:rsyslog@lists.adiscon.com>) escribió:
Hello!
Field msg contains complete json with data. I would like to
instert it
to mongodb as is. But now rsyslog inserts it as a value of
key "msg".
So now is:
msg: '{"foo":"bar"}
a i'd like to insert: only:
'{"foo","bar"}'
I tried with template:
template(name="ui-json" type="string" string="%msg")
but it doesn't do what I need. Is it possible to configure
it using
template or this is imposible due to ommnongodb limitation?
Marcin Mirosław
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
<https://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/professional-services/
<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow
https://twitter.com/rgerhards <https://twitter.com/rgerhards>
NOTE WELL: This is a PUBLIC mailing list, posts are
ARCHIVED by a myriad of sites beyond our control. PLEASE
UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
<https://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/professional-services/
<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
<https://twitter.com/rgerhards>
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
<https://lists.adiscon.net/mailman/listinfo/rsyslog>
http://www.rsyslog.com/professional-services/
<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://twitter.com/rgerhards
<https://twitter.com/rgerhards>
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
