It's contributed code.
Albeit I have done some improvements, I need to familiarize myself
again with the module. IAW: it will take some time ;-)
Rainer
El mié, 5 oct 2022 a las 16:04, Marcin Mirosław via rsyslog
(<rsyslog@lists.adiscon.com>) escribió:
>
> It would be easier to fix if it would be my misconfiguration :(
> Rainer, would you look into ommongodb module, please?
> Marcin
>
> W dniu 2022-10-05 12:20, Mariusz Kruk via rsyslog napisał(a):
> > There is also some inconsistency within the documentation page (it
> > lists OMSR_TPL_AS_MSG as the default template whereas few lines later
> > it shows a definition of a template named BSON which is supposed to be
> > default).
> >
> > So I suspect that something wasn't quite finished here ;-)
> >
> > On 5.10.2022 12:03, Mariusz Kruk wrote:
> >> Out of sheer curiosity I looked into the ommongodb.c and found this:
> >>
> >> 557 if(pData->tplName == NULL) {
> >> 558 doc = getDefaultBSON(*(smsg_t**)pMsgData);
> >> 559 } else {
> >> 560 doc = BSONFromJSONObject(*(struct json_object
> >> **)pMsgData);
> >> 561 }
> >>
> >> I'm not that good in rsyslog internals but for me it seems as if the
> >> module does not use the specified template at all. If the template is
> >> not specified, it creates a BSON with pre-defined files. If the
> >> template is specified, it just dumps a bson object from the message
> >> data.
> >>
> >>
> >> On 5.10.2022 11:28, Marcin Mirosław via rsyslog wrote:
> >>> Hi!
> >>> If I use omfile then logs contains messages in my desired format but
> >>> not with ommongodb. tcpdump shows:
> >>>
> >>> ..msg.....{"foo":"bar"}..
> >>>
> >>> Marcin
> >>>
> >>>
> >>> W dniu 2022-10-05 08:05, Mariusz Kruk via rsyslog napisał(a):
> >>>> Looks relatively normal.
> >>>>
> >>>> You can of course try writing to a file with your "i-json" template
> >>>> to
> >>>> make sure that's what you want to be sent to mongodb.
> >>>>
> >>>> But then, if your ommongodb action does contain the
> >>>> template="i-json"
> >>>> parameter, it simply should work.
> >>>>
> >>>> The only other thing you can do to make sure what's going on over
> >>>> the
> >>>> wire is of course the tcpdump (unless your traffic is encrypted).
> >>>>
> >>>> Ot simply looks that it should work - it should not need any more
> >>>> "processing" as it is.
> >>>>
> >>>> On 4.10.2022 15:45, Marcin Mirosław wrote:
> >>>>> :)
> >>>>> I had to censore log...
> >>>>>
> >>>>>
> >>>>> Debug line with all properties:
> >>>>> FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: 'linux',
> >>>>> PRI: 174,
> >>>>> syslogtag 'a1', programname: 'x1', APP-NAME: 'x1', PROCID: '-',
> >>>>> MSGID: '-',
> >>>>> TIMESTAMP: 'Oct 4 15:38:53', STRUCTURED-DATA: '[timeQuality
> >>>>> tzKnown="1" isSynced="1" syncAccuracy="415383"]',
> >>>>> msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
> >>>>> escaped msg:
> >>>>> '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
> >>>>> inputname: imtcp rawmsg: '<174>1 2022-10-04T15:38:53.219052+02:00
> >>>>> linux a1 - - [timeQuality tzKnown="1" isSynced="1"
> >>>>> syncAccuracy="415383"] {"foo":"bar
> >>>>> ","mode":750,"date":"2022-10-04T15:38:53"}'
> >>>>> $!:
> >>>>> $.:
> >>>>> $/:
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> W dniu 2022-10-04 09:18, Mariusz Kruk via rsyslog napisał(a):
> >>>>>> :-)
> >>>>>>
> >>>>>> OK, I understand that you did
> >>>>>>
> >>>>>> template(name="i-json" type="string" string="%msg%")
> >>>>>>
> >>>>>> action(type="ommongodb" server="..." [...] template="i-json")
> >>>>>>
> >>>>>> right?
> >>>>>>
> >>>>>> This way you should indeed be pushing the %msg% part of the
> >>>>>> incoming
> >>>>>> syslog message.
> >>>>>>
> >>>>>> Try adding
> >>>>>>
> >>>>>> action(type="omfile" file="/tmp/debug.log"
> >>>>>> template="RSYSLOG_DebugFormat")
> >>>>>>
> >>>>>> immediately before your ommongodb action to see what exactly your
> >>>>>> properties look like. That usually helps finding what's happening
> >>>>>> inside your processing pipeline.
> >>>>>>
> >>>>>> On 4.10.2022 09:03, Marcin Mirosław wrote:
> >>>>>>>
> >>>>>>> I did in the part about connection to mongodb ;)
> >>>>>>>
> >>>>>>> "It will be used automatically if no other template is specified
> >>>>>>> to be used"
> >>>>>>>
> >>>>>>> But I specified template: template(name="ui-json" type="string"
> >>>>>>> string="%msg%")
> >>>>>>>
> >>>>>>> so if I understand qutoed docs correctly, all about default,
> >>>>>>> canned template doesn't apply to my case, am I right?
> >>>>>>>
> >>>>>>>
> >>>>>>> W dniu 2022-10-04 08:27, Mariusz Kruk napisał(a):
> >>>>>>>
> >>>>>>>> You didn't read the docs, did you? ;-)
> >>>>>>>>
> >>>>>>>> https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
> >>>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
> >>>>>>>> "Note rsyslog contains a canned default template to write to the
> >>>>>>>> MongoDB. It will be used automatically if no other template is
> >>>>>>>> specified to be used. This template is:
> >>>>>>>>
> >>>>>>>> template(name="BSON" type="string" string="\\"sys\\" :
> >>>>>>>> \\"%hostname%\\",
> >>>>>>>> \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
> >>>>>>>> \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
> >>>>>>>> \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\"
> >>>>>>>> :
> >>>>>>>> \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
> >>>>>>>> \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
> >>>>>>>> \\"level\\" : \\"%syslogpriority-text%\\"")
> >>>>>>>>
> >>>>>>>> This creates the BSON document needed for MongoDB if no template
> >>>>>>>> is specified. The default schema is aligned to CEE and project
> >>>>>>>> lumberjack. As such, the field names are standard lumberjack
> >>>>>>>> field names, and *not* rsyslog property names
> >>>>>>>> <https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 3.10.2022 22:02, Marcin Mirosław wrote:
> >>>>>>>>> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
> >>>>>>>>>> Don't know about this particular output module but in general
> >>>>>>>>>> what you want is for rsyslog to parse the message and insert
> >>>>>>>>>> it as json object.
> >>>>>>>>>
> >>>>>>>>> Meseems that now rsyslog put %msg% as json object. (
> >>>>>>>>> ex: msg: '{"foo":"bar"}' }
> >>>>>>>>> )
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>> So you need to use parse_json() function on the input string
> >>>>>>>>>> and then use proper template which will render the json to
> >>>>>>>>>> appropirate string. I use similar approach (without the
> >>>>>>>>>> parsing part) to create output json for Splunk's HEC input -
> >>>>>>>>>> the idea is roughly the same.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 3.10.2022 18:35, Marcin Mirosław via rsyslog wrote:
> >>>>>>>>>>> Maybe when I show examples from mongo it will be more clear.
> >>>>>>>>>>>
> >>>>>>>>>>> > db.log2.find()
> >>>>>>>>>>> [
> >>>>>>>>>>> { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
> >>>>>>>>>>> '{"foo":"bar"}' },
> >>>>>>>>>>> { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
> >>>>>>>>>>> ]
> >>>>>>>>>>>
> >>>>>>>>>>> First record shows how msg is inserted to mongo by rsyslog.
> >>>>>>>>>>> Second record is what I would like to get.
> >>>>>>>>>>>
> >>>>>>>>>>> rsyslog do:
> >>>>>>>>>>> db.log2.insert({msg:'{"foo":"bar"}'})
> >>>>>>>>>>>
> >>>>>>>>>>> but I'd like to have:
> >>>>>>>>>>> db.log2.insert({"foo":"bar"})
> >>>>>>>>>>>
> >>>>>>>>>>> Marcin
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
> >>>>>>>>>>>> I do not fully understand the question (maybe language issue
> >>>>>>>>>>>> on my
> >>>>>>>>>>>> side), but there is a syntax error:
> >>>>>>>>>>>>
> >>>>>>>>>>>> In a string template, properties must be enclosed in percent
> >>>>>>>>>>>> sign. so:
> >>>>>>>>>>>>
> >>>>>>>>>>>> ... string="%msg%
> >>>>>>>>>>>>
> >>>>>>>>>>>> HTH
> >>>>>>>>>>>> Rainer
> >>>>>>>>>>>>
> >>>>>>>>>>>> El lun, 3 oct 2022 a las 13:18, Marcin Mirosław via rsyslog
> >>>>>>>>>>>> (<rsyslog@lists.adiscon.com>
> >>>>>>>>>>>> <mailto:rsyslog@lists.adiscon.com>) escribió:
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Hello!
> >>>>>>>>>>>>> Field msg contains complete json with data. I would like to
> >>>>>>>>>>>>> instert it
> >>>>>>>>>>>>> to mongodb as is. But now rsyslog inserts it as a value of
> >>>>>>>>>>>>> key "msg".
> >>>>>>>>>>>>> So now is:
> >>>>>>>>>>>>> msg: '{"foo":"bar"}
> >>>>>>>>>>>>> a i'd like to insert: only:
> >>>>>>>>>>>>> '{"foo","bar"}'
> >>>>>>>>>>>>> I tried with template:
> >>>>>>>>>>>>> template(name="ui-json" type="string" string="%msg")
> >>>>>>>>>>>>> but it doesn't do what I need. Is it possible to configure
> >>>>>>>>>>>>> it using
> >>>>>>>>>>>>> template or this is imposible due to ommnongodb limitation?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Marcin Mirosław
> >>>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
> >>>>>>>>>>>>> http://www.rsyslog.com/professional-services/
> >>>>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
> >>>>>>>>>>>>> What's up with rsyslog? Follow
> >>>>>>>>>>>>> https://twitter.com/rgerhards
> >>>>>>>>>>>>> <https://twitter.com/rgerhards>
> >>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
> >>>>>>>>>>>>> ARCHIVED by a myriad of sites beyond our control. PLEASE
> >>>>>>>>>>>>> UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
> >>>>>>>>>>>
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>> rsyslog mailing list
> >>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
> >>>>>>>>>>> http://www.rsyslog.com/professional-services/
> >>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
> >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>>>>>>>>> <https://twitter.com/rgerhards>
> >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> >>>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
> >>>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> rsyslog mailing list
> >>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
> >>>>>>>>>> http://www.rsyslog.com/professional-services/
> >>>>>>>>>> <http://www.rsyslog.com/professional-services/>
> >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>>>>>>>> <https://twitter.com/rgerhards>
> >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> >>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
> >>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
> >>>>>>>
> >>>>>>>
> >>>>>> _______________________________________________
> >>>>>> rsyslog mailing list
> >>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>>>> http://www.rsyslog.com/professional-services/
> >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >>>>>> POST
> >>>>>> if you DON'T LIKE THAT.
> >>>> _______________________________________________
> >>>> rsyslog mailing list
> >>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com/professional-services/
> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >>>> POST
> >>>> if you DON'T LIKE THAT.
> >>> _______________________________________________
> >>> rsyslog mailing list
> >>> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
> >>> POST if you DON'T LIKE THAT.
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> > if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
