Date: Wed, 22 May 2024 09:31:49 +0200
From: Adam Cecile via rsyslog <rsyslog@lists.adiscon.com>
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Adam Cecile <acec...@letz-it.lu>
Subject: Re: [rsyslog] Unable to re-use variable generated by
mmnormalize
So for the record, here is my awful hack.
I'm parsing messages like "[21/05/2024 23:56:37] [pcc->cct]
00100t7cc" from existing files using imfile and want to re-use the
timestamp from msg payload in Rsyslog. Timestamp formatting is
dd/mm/YYYY HH:MM:ss in local server timezone
So first, I used mmnormalize with the following rule to extract
individual timestamp fields:
rule=:[%day:number%/%month:number%/%year:number%
%hour:number%:%minute:number%:%second:number%]
[%dsd-src:string-to:->%->%dsd-dst:char-to:]%] %dsd-message:rest%
Then I used a custom static mapping table to convert month as digits
into RFC 3164 "3 letters" month name:
{
"version": 1,
"nomatch": "",
"type": "array",
"table": [
{ "index": 1, "value": "Jan" },
{ "index": 2, "value": "Feb" },
{ "index": 3, "value": "Mar" },
{ "index": 4, "value": "Apr" },
{ "index": 5, "value": "May" },
{ "index": 6, "value": "Jun" },
{ "index": 7, "value": "Jul" },
{ "index": 8, "value": "Aug" },
{ "index": 9, "value": "Sep" },
{ "index": 10, "value": "Oct" },
{ "index": 11, "value": "Nov" },
{ "index": 12, "value": "Dec" },
]
}
Finally, I used a ruleset to apply this liblognorm pattern and use so
internal Rsyslog scripting to build a RFC 3164 timestamp:
lookup_table(
name="month-num-to-three-letters-lookup-table"
file="/etc/rsyslog.d/month-num-to-three-letters-lookup-table.json"
reloadonhup="off"
)
ruleset(
name="extractData"
)
{
set
$.pcc=re_extract($!metadata!filename,"/REP_RECORD_([a-z0-9]{2})/",0,1,"UNMATCHED_PCC");
action(
type="mmnormalize"
path="$!extracted"
rulebase="/etc/rsyslog.d/dsd-trace-parser.rules"
useRawMsg="off"
)
set $!extracted!monthInt = cnum($!extracted!month);
set $!extracted!dayInt = cnum($!extracted!day);
if ($!extracted!dayInt < 10) then {
set $!extracted!dayLeadingSpace = " " & $!extracted!dayInt;
} else {
set $!extracted!dayLeadingSpace = "" & $!extracted!dayInt;
}
set $!extracted!monthThreeLetters =
lookup("month-num-to-three-letters-lookup-table", $!extracted!monthInt);
set $!extracted!timestamp!rfc3164 =
$!extracted!!monthThreeLetters & " " & $!extracted!dayLeadingSpace &
" " & $!extracted!hour & ":" & $!extracted!minute & ":" &
$!extracted!second;
action(
type="omfile"
dirCreateMode="0755"
FileCreateMode="0644"
File="/var/log/rep-record.log"
template="repRecordRfc3164FileOutput"
# template="RSYSLOG_DebugFormat"
)
}
So... Yeah it works, but this is ugly and I really thought I'd come
up with a cleaner solution...
I have a couple of questions, maybe someone can points me to
improvement or confirm I did the best that was possible:
1. I did not find any way to parse my timestamp into date-time object
form liblognorm rule directly, did I missed something ?
2. Is there any better way to convert 1 / "01" -> Jan or 12 / "12" ->
Dec than using the static lookup table hack ?
3. Is there any way to localize a my timestamp with server local
timezone ? If so I could use RFC 5424 date-time instead of RFC 3164
4. I tried to use parse_time with my RFC 3164 date-time string but it
returned 0 or and empty string, I think such function does not exist
on CentOS 7 / Rsyslog 8.24, confirmed ?
5. I there any other way to convert this string timestamp into a
date-time object ?
6. I considered re-injecting my generated full RFC 3164 syslog
message into rsyslog itself, so it can then be handled by normal
rules, is that stupid ? Is there any output module to re-feed the
message into internal rsyslog processing queue ?
7. At last, but not least, is there any "bounty" system or something
to contribute to date-time parsing improvments for Rsyslog ?
Best regards, Adam.
On 5/22/24 08:03, Adam Cecile via rsyslog wrote:
Hello,
Thanks for your answer but David figured out: my CentOS 7 rsyslog
package
does not accept inline rule definition but wants a rulebase file.
It works fine like this, however, recreating timestamp from the msg
payload
itself has been a nightmare but I got something (quite ugly) that works.
I'll update this thread with my solution, hopefully it'll help
someone else
and maybe I'll get some comments.
Adam.
On May 22, 2024 7:58:16 AM GMT+02:00, Mariusz Kruk via
rsyslog<rsyslog@lists.adiscon.com> wrote:
Check your spaces in msg and rawmsg and compare it with your pattern
because they are inconsistent.
$msg contains [something]-two spaces-[something]-two spaces-something
$rawmsg contains [something]-two spaces-[something]-one
space-something
Your pattern contains [something]-one space-[something]-one
space-something
On 22.05.2024 00:24, Adam Cecile via rsyslog wrote:
No ;-)
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'gz-tuma', PRI: 133,
syslogtag 'trace.log', programname: 'trace.log', APP-NAME:
'trace.log',
PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 21 23:56:46', STRUCTURED-DATA: '-',
msg: '[21/05/2024 23:56:37] [pcc->cct] 00100t7cc'
escaped msg: '[21/05/2024 23:56:37] [pcc->cct] 00100t7cc'
inputname: imfile rawmsg: '[21/05/2024 23:56:37] [pcc->cct]
00100t7cc'
$!:{ "metadata": { "filename": "<redacted>", "fileoffset":
"3396674" } }
$.:{ "pcc": "t7" }
$/:
On 5/22/24 00:20, David Lang wrote:
if you look at the msg field in the RSYSLOG_DebugFormat output,
you will
see that it does have a leading space. your pattern doesn't
David Lang
On Tue, 21 May 2024, Adam Cecile via rsyslog wrote:
Date: Tue, 21 May 2024 23:58:23 +0200
From: Adam Cecile via rsyslog<rsyslog@lists.adiscon.com>
To: Adam Cecile via rsyslog<rsyslog@lists.adiscon.com>
Cc: Adam Cecile<acec...@letz-it.lu>
Subject: Re: [rsyslog] Unable to re-use variable generated by
mmnormalize
Just tried, no leading space, message is correct however the log
does
not contains any trace indicating mmnormalize has tried to do something.
I guess it is just NOT active for some reason...
On 5/21/24 23:50, David Lang wrote:
log the message with RSYSLOG_DebugFormat so that you can see the
variables that exist.
my guess is that your rule needs a leading space, because the msg
field you are parsing starts with a space (a very common problem when
you are starting to use mmnormalize)
David Lang
On Tue, 21 May 2024, Adam Cecile via rsyslog wrote:
Date: Tue, 21 May 2024 23:37:14 +0200
From: Adam Cecile via rsyslog<rsyslog@lists.adiscon.com>
To:rsyslog@lists.adiscon.com
Cc: Adam Cecile<acec...@letz-it.lu>
Subject: [rsyslog] Unable to re-use variable generated by
mmnormalize
Hello,
I'm struggling to understand how to use mmnormalize module. My
configuration snippet is the following:
template(name="recordRfc3164FileOutput" type="list") {
property(name="timestamp" dateFormat="rfc3164")
constant(value=" ")
property(name="hostname")
constant(value=" ")
property(name="$.pcc")
constant(value="@")
property(name="syslogtag")
constant(value=":")
constant(value=" ")
property(name="msg")
constant(value="\n")
}
ruleset(
name="extractPccNameFromRecordLog"
)
{
set
$.pcc=re_extract($!metadata!filename,"/record_([a-z0-9]{2})/",0,1,"UNMATCHED_PCC");
action(
type="mmnormalize"
rule=["rule=:[%day:number%/%month:number%/%year:number%
%hour:number%:%minutes:number%:%seconds:number%]
[%dsd-src:string-to:->%->%dsd-dst:char-to:]%] %dsd-message:rest%"]
useRawMsg="off"
)
action(
type="omfile"
dirCreateMode="0755"
FileCreateMode="0644"
File="/var/log/record.log"
template="recordRfc3164FileOutput"
)
}
input(
type="imfile"
file="/path/to/record_*/*.log"
tag="trace.log"
addmetadata="on"
ruleset="extractPccNameFromRecordLog"
)
Extraction of pattern from imfile filepath is working fine,
mmnormalize rule is working fine too, according to my test using
"lognormalizer" command line tool to test it.
So I guessed, I could be able to replace property(name="msg")
with
property(name="$.dsd-message") or property(name="$!dsd-message") but
all I get is an empty string.
Is there something I missed ?
Thanks in advance,
Regards, Adam.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
