It would be easier to fix if it would be my misconfiguration :(
Rainer, would you look into ommongodb module, please?
Marcin
W dniu 2022-10-05 12:20, Mariusz Kruk via rsyslog napisał(a):
> There is also some inconsistency within the documentation page (it
> lists OMSR_TPL_AS_MSG as the default template whereas few lines later
> it shows a definition of a template named BSON which is supposed to be
> default).
>
> So I suspect that something wasn't quite finished here ;-)
>
> On 5.10.2022 12:03, Mariusz Kruk wrote:
>> Out of sheer curiosity I looked into the ommongodb.c and found this:
>>
>> 557 if(pData->tplName == NULL) {
>> 558 doc = getDefaultBSON(*(smsg_t**)pMsgData);
>> 559 } else {
>> 560 doc = BSONFromJSONObject(*(struct json_object
>> **)pMsgData);
>> 561 }
>>
>> I'm not that good in rsyslog internals but for me it seems as if the
>> module does not use the specified template at all. If the template is
>> not specified, it creates a BSON with pre-defined files. If the
>> template is specified, it just dumps a bson object from the message
>> data.
>>
>>
>> On 5.10.2022 11:28, Marcin Mirosław via rsyslog wrote:
>>> Hi!
>>> If I use omfile then logs contains messages in my desired format but
>>> not with ommongodb. tcpdump shows:
>>>
>>> ..msg.....{"foo":"bar"}..
>>>
>>> Marcin
>>>
>>>
>>> W dniu 2022-10-05 08:05, Mariusz Kruk via rsyslog napisał(a):
>>>> Looks relatively normal.
>>>>
>>>> You can of course try writing to a file with your "i-json" template
>>>> to
>>>> make sure that's what you want to be sent to mongodb.
>>>>
>>>> But then, if your ommongodb action does contain the
>>>> template="i-json"
>>>> parameter, it simply should work.
>>>>
>>>> The only other thing you can do to make sure what's going on over
>>>> the
>>>> wire is of course the tcpdump (unless your traffic is encrypted).
>>>>
>>>> Ot simply looks that it should work - it should not need any more
>>>> "processing" as it is.
>>>>
>>>> On 4.10.2022 15:45, Marcin Mirosław wrote:
>>>>> :)
>>>>> I had to censore log...
>>>>>
>>>>>
>>>>> Debug line with all properties:
>>>>> FROMHOST: 'localhost', fromhost-ip: '127.0.0.1', HOSTNAME: 'linux',
>>>>> PRI: 174,
>>>>> syslogtag 'a1', programname: 'x1', APP-NAME: 'x1', PROCID: '-',
>>>>> MSGID: '-',
>>>>> TIMESTAMP: 'Oct 4 15:38:53', STRUCTURED-DATA: '[timeQuality
>>>>> tzKnown="1" isSynced="1" syncAccuracy="415383"]',
>>>>> msg: '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>>>>> escaped msg:
>>>>> '{"foo":"bar","mode":750,"date":"2022-10-04T15:38:53"}'
>>>>> inputname: imtcp rawmsg: '<174>1 2022-10-04T15:38:53.219052+02:00
>>>>> linux a1 - - [timeQuality tzKnown="1" isSynced="1"
>>>>> syncAccuracy="415383"] {"foo":"bar
>>>>> ","mode":750,"date":"2022-10-04T15:38:53"}'
>>>>> $!:
>>>>> $.:
>>>>> $/:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> W dniu 2022-10-04 09:18, Mariusz Kruk via rsyslog napisał(a):
>>>>>> :-)
>>>>>>
>>>>>> OK, I understand that you did
>>>>>>
>>>>>> template(name="i-json" type="string" string="%msg%")
>>>>>>
>>>>>> action(type="ommongodb" server="..." [...] template="i-json")
>>>>>>
>>>>>> right?
>>>>>>
>>>>>> This way you should indeed be pushing the %msg% part of the
>>>>>> incoming
>>>>>> syslog message.
>>>>>>
>>>>>> Try adding
>>>>>>
>>>>>> action(type="omfile" file="/tmp/debug.log"
>>>>>> template="RSYSLOG_DebugFormat")
>>>>>>
>>>>>> immediately before your ommongodb action to see what exactly your
>>>>>> properties look like. That usually helps finding what's happening
>>>>>> inside your processing pipeline.
>>>>>>
>>>>>> On 4.10.2022 09:03, Marcin Mirosław wrote:
>>>>>>>
>>>>>>> I did in the part about connection to mongodb ;)
>>>>>>>
>>>>>>> "It will be used automatically if no other template is specified
>>>>>>> to be used"
>>>>>>>
>>>>>>> But I specified template: template(name="ui-json" type="string"
>>>>>>> string="%msg%")
>>>>>>>
>>>>>>> so if I understand qutoed docs correctly, all about default,
>>>>>>> canned template doesn't apply to my case, am I right?
>>>>>>>
>>>>>>>
>>>>>>> W dniu 2022-10-04 08:27, Mariusz Kruk napisał(a):
>>>>>>>
>>>>>>>> You didn't read the docs, did you? ;-)
>>>>>>>>
>>>>>>>>
https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html
>>>>>>>>
<https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommongodb.html>
>>>>>>>> "Note rsyslog contains a canned default template to write to the
>>>>>>>> MongoDB. It will be used automatically if no other template is
>>>>>>>> specified to be used. This template is:
>>>>>>>>
>>>>>>>> template(name="BSON" type="string" string="\\"sys\\" :
>>>>>>>> \\"%hostname%\\",
>>>>>>>> \\"time\\" : \\"%timereported:::rfc3339%\\", \\"time\_rcvd\\" :
>>>>>>>> \\"%timegenerated:::rfc3339%\\", \\"msg\\" : \\"%msg%\\",
>>>>>>>> \\"syslog\_fac\\" : \\"%syslogfacility%\\", \\"syslog\_server\\"
>>>>>>>> :
>>>>>>>> \\"%syslogseverity%\\", \\"syslog\_tag\\" : \\"%syslogtag%\\",
>>>>>>>> \\"procid\\" : \\"%programname%\\", \\"pid\\" : \\"%procid%\\",
>>>>>>>> \\"level\\" : \\"%syslogpriority-text%\\"")
>>>>>>>>
>>>>>>>> This creates the BSON document needed for MongoDB if no template
>>>>>>>> is specified. The default schema is aligned to CEE and project
>>>>>>>> lumberjack. As such, the field names are standard lumberjack
>>>>>>>> field names, and *not* rsyslog property names
>>>>>>>>
<https://www.rsyslog.com/doc/v8-stable/configuration/modules/property_replacer.html>."
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3.10.2022 22:02, Marcin Mirosław wrote:
>>>>>>>>> W dniu 03.10.2022 o 18:55, Mariusz Kruk via rsyslog pisze:
>>>>>>>>>> Don't know about this particular output module but in general
>>>>>>>>>> what you want is for rsyslog to parse the message and insert
>>>>>>>>>> it as json object.
>>>>>>>>>
>>>>>>>>> Meseems that now rsyslog put %msg% as json object. (
>>>>>>>>> ex: msg: '{"foo":"bar"}' }
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> So you need to use parse_json() function on the input string
>>>>>>>>>> and then use proper template which will render the json to
>>>>>>>>>> appropirate string. I use similar approach (without the
>>>>>>>>>> parsing part) to create output json for Splunk's HEC input -
>>>>>>>>>> the idea is roughly the same.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 3.10.2022 18:35, Marcin Mirosław via rsyslog wrote:
>>>>>>>>>>> Maybe when I show examples from mongo it will be more clear.
>>>>>>>>>>>
>>>>>>>>>>> > db.log2.find()
>>>>>>>>>>> [
>>>>>>>>>>> { _id: ObjectId("633b0ea6b8f2a532cfa6c64c"), msg:
>>>>>>>>>>> '{"foo":"bar"}' },
>>>>>>>>>>> { _id: ObjectId("633b0eb6b8f2a532cfa6c64d"), foo: 'bar' }
>>>>>>>>>>> ]
>>>>>>>>>>>
>>>>>>>>>>> First record shows how msg is inserted to mongo by rsyslog.
>>>>>>>>>>> Second record is what I would like to get.
>>>>>>>>>>>
>>>>>>>>>>> rsyslog do:
>>>>>>>>>>> db.log2.insert({msg:'{"foo":"bar"}'})
>>>>>>>>>>>
>>>>>>>>>>> but I'd like to have:
>>>>>>>>>>> db.log2.insert({"foo":"bar"})
>>>>>>>>>>>
>>>>>>>>>>> Marcin
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> W dniu 03.10.2022 o 17:36, Rainer Gerhards pisze:
>>>>>>>>>>>> I do not fully understand the question (maybe language issue
>>>>>>>>>>>> on my
>>>>>>>>>>>> side), but there is a syntax error:
>>>>>>>>>>>>
>>>>>>>>>>>> In a string template, properties must be enclosed in percent
>>>>>>>>>>>> sign. so:
>>>>>>>>>>>>
>>>>>>>>>>>> ... string="%msg%
>>>>>>>>>>>>
>>>>>>>>>>>> HTH
>>>>>>>>>>>> Rainer
>>>>>>>>>>>>
>>>>>>>>>>>> El lun, 3 oct 2022 a las 13:18, Marcin Mirosław via rsyslog
>>>>>>>>>>>> (<rsyslog@lists.adiscon.com>
>>>>>>>>>>>> <mailto:rsyslog@lists.adiscon.com>) escribió:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hello!
>>>>>>>>>>>>> Field msg contains complete json with data. I would like to
>>>>>>>>>>>>> instert it
>>>>>>>>>>>>> to mongodb as is. But now rsyslog inserts it as a value of
>>>>>>>>>>>>> key "msg".
>>>>>>>>>>>>> So now is:
>>>>>>>>>>>>> msg: '{"foo":"bar"}
>>>>>>>>>>>>> a i'd like to insert: only:
>>>>>>>>>>>>> '{"foo","bar"}'
>>>>>>>>>>>>> I tried with template:
>>>>>>>>>>>>> template(name="ui-json" type="string" string="%msg")
>>>>>>>>>>>>> but it doesn't do what I need. Is it possible to configure
>>>>>>>>>>>>> it using
>>>>>>>>>>>>> template or this is imposible due to ommnongodb limitation?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Marcin Mirosław
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>>>>>> What's up with rsyslog? Follow
>>>>>>>>>>>>> https://twitter.com/rgerhards
>>>>>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are
>>>>>>>>>>>>> ARCHIVED by a myriad of sites beyond our control. PLEASE
>>>>>>>>>>>>> UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> rsyslog mailing list
>>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>>>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>>>> _______________________________________________
>>>>>>>>>> rsyslog mailing list
>>>>>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>>>>>> <https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>>>>>>>>> http://www.rsyslog.com/professional-services/
>>>>>>>>>> <http://www.rsyslog.com/professional-services/>
>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>>>>>> <https://twitter.com/rgerhards>
>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
>>>>>>>>>> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE
>>>>>>>>>> and DO NOT POST if you DON'T LIKE THAT.
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>>>> POST
>>>>>> if you DON'T LIKE THAT.
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>>> POST
>>>> if you DON'T LIKE THAT.
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
>>> POST if you DON'T LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
