Hello,
Thanks for your answer but David figured out: my CentOS 7 rsyslog package does
not accept inline rule definition but wants a rulebase file.
It works fine like this, however, recreating timestamp from the msg payload
itself has been a nightmare but I got something (quite ugly) that works.
I'll update this thread with my solution, hopefully it'll help someone else and
maybe I'll get some comments.
Adam.
On May 22, 2024 7:58:16 AM GMT+02:00, Mariusz Kruk via rsyslog
<rsyslog@lists.adiscon.com> wrote:
>Check your spaces in msg and rawmsg and compare it with your pattern because
>they are inconsistent.
>
>$msg contains [something]-two spaces-[something]-two spaces-something
>
>$rawmsg contains [something]-two spaces-[something]-one space-something
>
>Your pattern contains [something]-one space-[something]-one space-something
>
>On 22.05.2024 00:24, Adam Cecile via rsyslog wrote:
>> No ;-)
>>
>> Debug line with all properties:
>> FROMHOST: '', fromhost-ip: '', HOSTNAME: 'gz-tuma', PRI: 133,
>> syslogtag 'trace.log', programname: 'trace.log', APP-NAME: 'trace.log',
>> PROCID: '-', MSGID: '-',
>> TIMESTAMP: 'May 21 23:56:46', STRUCTURED-DATA: '-',
>> msg: '[21/05/2024 23:56:37] [pcc->cct] 00100t7cc'
>> escaped msg: '[21/05/2024 23:56:37] [pcc->cct] 00100t7cc'
>> inputname: imfile rawmsg: '[21/05/2024 23:56:37] [pcc->cct] 00100t7cc'
>> $!:{ "metadata": { "filename": "<redacted>", "fileoffset": "3396674" } }
>> $.:{ "pcc": "t7" }
>> $/:
>>
>> On 5/22/24 00:20, David Lang wrote:
>>> if you look at the msg field in the RSYSLOG_DebugFormat output, you will
>>> see that it does have a leading space. your pattern doesn't
>>>
>>> David Lang
>>>
>>> On Tue, 21 May 2024, Adam Cecile via rsyslog wrote:
>>>
>>>> Date: Tue, 21 May 2024 23:58:23 +0200
>>>> From: Adam Cecile via rsyslog <rsyslog@lists.adiscon.com>
>>>> To: Adam Cecile via rsyslog <rsyslog@lists.adiscon.com>
>>>> Cc: Adam Cecile <acec...@letz-it.lu>
>>>> Subject: Re: [rsyslog] Unable to re-use variable generated by mmnormalize
>>>>
>>>> Just tried, no leading space, message is correct however the log does not
>>>> contains any trace indicating mmnormalize has tried to do something.
>>>> I guess it is just NOT active for some reason...
>>>>
>>>> On 5/21/24 23:50, David Lang wrote:
>>>>> log the message with RSYSLOG_DebugFormat so that you can see the
>>>>> variables that exist.
>>>>>
>>>>> my guess is that your rule needs a leading space, because the msg field
>>>>> you are parsing starts with a space (a very common problem when you are
>>>>> starting to use mmnormalize)
>>>>>
>>>>> David Lang
>>>>>
>>>>> On Tue, 21 May 2024, Adam Cecile via rsyslog wrote:
>>>>>
>>>>>> Date: Tue, 21 May 2024 23:37:14 +0200
>>>>>> From: Adam Cecile via rsyslog <rsyslog@lists.adiscon.com>
>>>>>> To: rsyslog@lists.adiscon.com
>>>>>> Cc: Adam Cecile <acec...@letz-it.lu>
>>>>>> Subject: [rsyslog] Unable to re-use variable generated by mmnormalize
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>>
>>>>>> I'm struggling to understand how to use mmnormalize module. My
>>>>>> configuration snippet is the following:
>>>>>>
>>>>>> template(name="recordRfc3164FileOutput" type="list") {
>>>>>> property(name="timestamp" dateFormat="rfc3164")
>>>>>> constant(value=" ")
>>>>>> property(name="hostname")
>>>>>> constant(value=" ")
>>>>>> property(name="$.pcc")
>>>>>> constant(value="@")
>>>>>> property(name="syslogtag")
>>>>>> constant(value=":")
>>>>>> constant(value=" ")
>>>>>> property(name="msg")
>>>>>> constant(value="\n")
>>>>>> }
>>>>>>
>>>>>> ruleset(
>>>>>> name="extractPccNameFromRecordLog"
>>>>>> )
>>>>>> {
>>>>>> set
>>>> $.pcc=re_extract($!metadata!filename,"/record_([a-z0-9]{2})/",0,1,"UNMATCHED_PCC");
>>>>
>>>>>> action(
>>>>>> type="mmnormalize"
>>>>>> rule=["rule=:[%day:number%/%month:number%/%year:number%
>>>>>> %hour:number%:%minutes:number%:%seconds:number%]
>>>>>> [%dsd-src:string-to:->%->%dsd-dst:char-to:]%] %dsd-message:rest%"]
>>>>>> useRawMsg="off"
>>>>>> )
>>>>>> action(
>>>>>> type="omfile"
>>>>>> dirCreateMode="0755"
>>>>>> FileCreateMode="0644"
>>>>>> File="/var/log/record.log"
>>>>>> template="recordRfc3164FileOutput"
>>>>>> )
>>>>>> }
>>>>>>
>>>>>> input(
>>>>>> type="imfile"
>>>>>> file="/path/to/record_*/*.log"
>>>>>> tag="trace.log"
>>>>>> addmetadata="on"
>>>>>> ruleset="extractPccNameFromRecordLog"
>>>>>> )
>>>>>>
>>>>>> Extraction of pattern from imfile filepath is working fine, mmnormalize
>>>>>> rule is working fine too, according to my test using "lognormalizer"
>>>>>> command line tool to test it.
>>>>>>
>>>>>> So I guessed, I could be able to replace property(name="msg") with
>>>>>> property(name="$.dsd-message") or property(name="$!dsd-message") but all
>>>>>> I get is an empty string.
>>>>>>
>>>>>> Is there something I missed ?
>>>>>>
>>>>>>
>>>>>> Thanks in advance,
>>>>>>
>>>>>> Regards, Adam.
>>>>>>
>>>>>> _______________________________________________
>>>>>> rsyslog mailing list
>>>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>>>> http://www.rsyslog.com/professional-services/
>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>>>> DON'T LIKE THAT.
>>>>
>>>>
>>>> _______________________________________________
>>>> rsyslog mailing list
>>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>>> http://www.rsyslog.com/professional-services/
>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>>> DON'T LIKE THAT.
>>
>>
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards
>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
>> LIKE THAT.
>_______________________________________________
>rsyslog mailing list
>https://lists.adiscon.net/mailman/listinfo/rsyslog
>http://www.rsyslog.com/professional-services/
>What's up with rsyslog? Follow https://twitter.com/rgerhards
>NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
>sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
>THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
