On Wed, 22 May 2024, Adam Cecile via rsyslog wrote:
Yes I'll consider that if needed, those are old servers, most of other
are a lot newer and are running recent rsyslogd, so maybe I'll leave it
like this.
Can you explain me a bit more what kind of input should I use to
re-inject my imfile parsed entries back into local rsyslog ? Should I
just send it to local UDP port ?
that works well, especially over localhost
Regarding sponsoring, I guess this is going to be too much for me, but
do you have an estimation of the cost ? I can at least talk about it and
see if we can help.
I'm not part of adiscon (the company formed by the Rainer, the initial author to
maintain rsyslog), you would have to ask him.
David Lang
On 5/22/24 09:49, David Lang wrote:
8.24 was released back in 2017. RedHat has backported some fixes and
features from newer versions of rsyslog (which were released every 6
weeks for years, now every 8 weeks), but only they track what is and
isn't in there.
If you are going to really start using the power of rsyslog, I would
suggest upgrading to the community version (it will be much easier
than figuring out what features are and aren't in such an old version)
the time parsing is MUCH newer than 8.24, and there are a ton of new
mmnormalize features that aren't in that old a version.
timezone handling is a headache still, glibc has internal functions to
handle timezones, but they don't expose it to userspace in a way that
lets userspace use arbitrary timezones (the functions are there, but
they insist on using the system environment rather than something
passed as a parameter)
overall, your approach works.
when you are considering sending the message back into rsyslog, you
need to be VERY sure that you don't setup a message loop. The best way
to do this is to have your imfile input go into a separate ruleset
(with it's own queue) so that there is no way for your output message
to be re-processed.
thinking out loud, you may also want to look at the pmnormalize parser
that lets the liblognorm parser populate the standard properties. I
don't know that the time parsing you are dealing with here will work
in that, but there has been some talk about implementing some time
parsing code in mmnormalize that would use date-style fields, allowing
for you to do what you are wanting in one step, but there hasn't been
enough demand for it to rise to the top of the list for Adiscon to
allocate manpower for it, and nobody has contributed the code or
offered to sponsor development of it. (adiscon is a very small
company, a half dozen or fewer people AFAIK)
David Lang
On Wed, 22 May 2024, Adam Cecile via rsyslog wrote:
Date: Wed, 22 May 2024 09:31:49 +0200
From: Adam Cecile via rsyslog <rsyslog@lists.adiscon.com>
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Adam Cecile <acec...@letz-it.lu>
Subject: Re: [rsyslog] Unable to re-use variable generated by
mmnormalize
So for the record, here is my awful hack.
I'm parsing messages like "[21/05/2024 23:56:37] [pcc->cct]
00100t7cc" from existing files using imfile and want to re-use the
timestamp from msg payload in Rsyslog. Timestamp formatting is
dd/mm/YYYY HH:MM:ss in local server timezone
So first, I used mmnormalize with the following rule to extract
individual timestamp fields:
rule=:[%day:number%/%month:number%/%year:number%
%hour:number%:%minute:number%:%second:number%]
[%dsd-src:string-to:->%->%dsd-dst:char-to:]%] %dsd-message:rest%
Then I used a custom static mapping table to convert month as digits
into RFC 3164 "3 letters" month name:
{
"version": 1,
"nomatch": "",
"type": "array",
"table": [
{ "index": 1, "value": "Jan" },
{ "index": 2, "value": "Feb" },
{ "index": 3, "value": "Mar" },
{ "index": 4, "value": "Apr" },
{ "index": 5, "value": "May" },
{ "index": 6, "value": "Jun" },
{ "index": 7, "value": "Jul" },
{ "index": 8, "value": "Aug" },
{ "index": 9, "value": "Sep" },
{ "index": 10, "value": "Oct" },
{ "index": 11, "value": "Nov" },
{ "index": 12, "value": "Dec" },
]
}
Finally, I used a ruleset to apply this liblognorm pattern and use so
internal Rsyslog scripting to build a RFC 3164 timestamp:
lookup_table(
name="month-num-to-three-letters-lookup-table"
file="/etc/rsyslog.d/month-num-to-three-letters-lookup-table.json"
reloadonhup="off"
)
ruleset(
name="extractData"
)
{
set
$.pcc=re_extract($!metadata!filename,"/REP_RECORD_([a-z0-9]{2})/",0,1,"UNMATCHED_PCC");
action(
type="mmnormalize"
path="$!extracted"
rulebase="/etc/rsyslog.d/dsd-trace-parser.rules"
useRawMsg="off"
)
set $!extracted!monthInt = cnum($!extracted!month);
set $!extracted!dayInt = cnum($!extracted!day);
if ($!extracted!dayInt < 10) then {
set $!extracted!dayLeadingSpace = " " & $!extracted!dayInt;
} else {
set $!extracted!dayLeadingSpace = "" & $!extracted!dayInt;
}
set $!extracted!monthThreeLetters =
lookup("month-num-to-three-letters-lookup-table", $!extracted!monthInt);
set $!extracted!timestamp!rfc3164 =
$!extracted!!monthThreeLetters & " " & $!extracted!dayLeadingSpace &
" " & $!extracted!hour & ":" & $!extracted!minute & ":" &
$!extracted!second;
action(
type="omfile"
dirCreateMode="0755"
FileCreateMode="0644"
File="/var/log/rep-record.log"
template="repRecordRfc3164FileOutput"
# template="RSYSLOG_DebugFormat"
)
}
So... Yeah it works, but this is ugly and I really thought I'd come
up with a cleaner solution...
I have a couple of questions, maybe someone can points me to
improvement or confirm I did the best that was possible:
1. I did not find any way to parse my timestamp into date-time object
form liblognorm rule directly, did I missed something ?
2. Is there any better way to convert 1 / "01" -> Jan or 12 / "12" ->
Dec than using the static lookup table hack ?
3. Is there any way to localize a my timestamp with server local
timezone ? If so I could use RFC 5424 date-time instead of RFC 3164
4. I tried to use parse_time with my RFC 3164 date-time string but it
returned 0 or and empty string, I think such function does not exist
on CentOS 7 / Rsyslog 8.24, confirmed ?
5. I there any other way to convert this string timestamp into a
date-time object ?
6. I considered re-injecting my generated full RFC 3164 syslog
message into rsyslog itself, so it can then be handled by normal
rules, is that stupid ? Is there any output module to re-feed the
message into internal rsyslog processing queue ?
7. At last, but not least, is there any "bounty" system or something
to contribute to date-time parsing improvments for Rsyslog ?
Best regards, Adam.
On 5/22/24 08:03, Adam Cecile via rsyslog wrote:
Hello,
Thanks for your answer but David figured out: my CentOS 7 rsyslog
package
does not accept inline rule definition but wants a rulebase file.
It works fine like this, however, recreating timestamp from the msg
payload
itself has been a nightmare but I got something (quite ugly) that works.
I'll update this thread with my solution, hopefully it'll help
someone else
and maybe I'll get some comments.
Adam.
On May 22, 2024 7:58:16 AM GMT+02:00, Mariusz Kruk via
rsyslog<rsyslog@lists.adiscon.com> wrote:
Check your spaces in msg and rawmsg and compare it with your pattern
because they are inconsistent.
$msg contains [something]-two spaces-[something]-two spaces-something
$rawmsg contains [something]-two spaces-[something]-one
space-something
Your pattern contains [something]-one space-[something]-one
space-something
On 22.05.2024 00:24, Adam Cecile via rsyslog wrote:
No ;-)
Debug line with all properties:
FROMHOST: '', fromhost-ip: '', HOSTNAME: 'gz-tuma', PRI: 133,
syslogtag 'trace.log', programname: 'trace.log', APP-NAME:
'trace.log',
PROCID: '-', MSGID: '-',
TIMESTAMP: 'May 21 23:56:46', STRUCTURED-DATA: '-',
msg: '[21/05/2024 23:56:37] [pcc->cct] 00100t7cc'
escaped msg: '[21/05/2024 23:56:37] [pcc->cct] 00100t7cc'
inputname: imfile rawmsg: '[21/05/2024 23:56:37] [pcc->cct]
00100t7cc'
$!:{ "metadata": { "filename": "<redacted>", "fileoffset":
"3396674" } }
$.:{ "pcc": "t7" }
$/:
On 5/22/24 00:20, David Lang wrote:
if you look at the msg field in the RSYSLOG_DebugFormat output,
you will
see that it does have a leading space. your pattern doesn't
David Lang
On Tue, 21 May 2024, Adam Cecile via rsyslog wrote:
Date: Tue, 21 May 2024 23:58:23 +0200
From: Adam Cecile via rsyslog<rsyslog@lists.adiscon.com>
To: Adam Cecile via rsyslog<rsyslog@lists.adiscon.com>
Cc: Adam Cecile<acec...@letz-it.lu>
Subject: Re: [rsyslog] Unable to re-use variable generated by
mmnormalize
Just tried, no leading space, message is correct however the log
does
not contains any trace indicating mmnormalize has tried to do something.
I guess it is just NOT active for some reason...
On 5/21/24 23:50, David Lang wrote:
log the message with RSYSLOG_DebugFormat so that you can see the
variables that exist.
my guess is that your rule needs a leading space, because the msg
field you are parsing starts with a space (a very common problem when
you are starting to use mmnormalize)
David Lang
On Tue, 21 May 2024, Adam Cecile via rsyslog wrote:
Date: Tue, 21 May 2024 23:37:14 +0200
From: Adam Cecile via rsyslog<rsyslog@lists.adiscon.com>
To:rsyslog@lists.adiscon.com
Cc: Adam Cecile<acec...@letz-it.lu>
Subject: [rsyslog] Unable to re-use variable generated by
mmnormalize
Hello,
I'm struggling to understand how to use mmnormalize module. My
configuration snippet is the following:
template(name="recordRfc3164FileOutput" type="list") {
property(name="timestamp" dateFormat="rfc3164")
constant(value=" ")
property(name="hostname")
constant(value=" ")
property(name="$.pcc")
constant(value="@")
property(name="syslogtag")
constant(value=":")
constant(value=" ")
property(name="msg")
constant(value="\n")
}
ruleset(
name="extractPccNameFromRecordLog"
)
{
set
$.pcc=re_extract($!metadata!filename,"/record_([a-z0-9]{2})/",0,1,"UNMATCHED_PCC");
action(
type="mmnormalize"
rule=["rule=:[%day:number%/%month:number%/%year:number%
%hour:number%:%minutes:number%:%seconds:number%]
[%dsd-src:string-to:->%->%dsd-dst:char-to:]%] %dsd-message:rest%"]
useRawMsg="off"
)
action(
type="omfile"
dirCreateMode="0755"
FileCreateMode="0644"
File="/var/log/record.log"
template="recordRfc3164FileOutput"
)
}
input(
type="imfile"
file="/path/to/record_*/*.log"
tag="trace.log"
addmetadata="on"
ruleset="extractPccNameFromRecordLog"
)
Extraction of pattern from imfile filepath is working fine,
mmnormalize rule is working fine too, according to my test using
"lognormalizer" command line tool to test it.
So I guessed, I could be able to replace property(name="msg")
with
property(name="$.dsd-message") or property(name="$!dsd-message") but
all I get is an empty string.
Is there something I missed ?
Thanks in advance,
Regards, Adam.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Followhttps://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
