Date: Mon, 20 Jan 2025 01:25:08 +0000
From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com>
To: David Lang <da...@lang.hm>,
"Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Building Custom pstats Template
Hey David,
I read that documentation. As far as I can tell, it only discusses using
rsyslog/adiscon-built templates. I don't think I can use those templates -
regardless of "bracketing". The SAM we use doesn't appear to support
multi-line matching through PCRE(2) via the /s flag.
Hence why I'm looking to build and use my own (normalized) JSON template.
The JSON template that adiscon uses is both multi line and isn't actual
JSON. It contains elements of JSON.
Cheers,
Mike
[cid:e628dcc4-f053-42b6-90a6-79771b1ae670]
Michael Redbourne (he/him)
Senior Security Analyst
Office: +1 (506) 606-0384
Cell: +61 04 2647 3071
SOC: 1-833-415-2424
www.bulletproofsi.com<http://www.bulletproofsi.com/>
Book a
Meeting<https://outlook.office365.com/owa/calendar/michaelredbou...@bulletproofsolutions.onmicrosoft.com/bookings/>
[cid:ea725086-9fec-4d13-8b97-056dba7feac4]
Notes: Please be advised that I live in Sydney, Australia. My normal hours
are 8AM-5PM (Australia/Sydney). I allow 24-hour calendar bookings, but
bookings made outside of official working hours should be discussed with me
prior to booking.
________________________________
From: David Lang <da...@lang.hm>
Sent: Monday, January 20, 2025 12:04 PM
To: Redbourne,Michael via rsyslog <rsyslog@lists.adiscon.com>
Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com>
Subject: Re: [rsyslog] Building Custom pstats Template
CAUTION: The Sender is located Outside The Organization. Do not click links
or open attachments unless you recognize the sender and know the content is
safe.
see
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rsyslog.com%2Fdoc%2Fconfiguration%2Fmodules%2Fimpstats.html&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9781aa28dc084468627308dd38ee6516%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729318758914939%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=7XiAjfDEAt8QZyju2Od1NnGFNhxjlpZUz8CT1Xd0i1Q%3D&reserved=0<https://www.rsyslog.com/doc/configuration/modules/impstats.html>
Format
type
default
mandatory
obsolete legacy directive
word
legacy
no
none
New in version 8.16.0.
Specifies the format of emitted stats messages. The default of “legacy” is
compatible with pre v6-rsyslog. The other options provide support for
structured
formats (note the “cee” is actually “project lumberjack” logging).
The json-elasticsearch format supports the broken ElasticSearch JSON
implementation. ES 2.0 no longer supports valid JSON and disallows dots
inside
names. The “json-elasticsearch” format option replaces those dots by the
bang
(“!”) character. So “discarded.full” becomes “discarded!full”. Options:
json/json-elasticsearch/cee/legacy
David Lang
On Mon, 20 Jan 2025, Redbourne,Michael via
rsyslog wrote:
Date: Mon, 20 Jan 2025 01:01:47 +0000
From: "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com>
To: "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com>
Cc: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com>
Subject: [rsyslog] Building Custom pstats Template
Hey Folks,
I'm wondering if anyone has implemented (or knows how to) implement their
own pstats template? The current formatting works well for human-readable
monitoring but isn't so good for machine-based monitoring. What I want to
do is essentially transform the current format json to a single line of
properly formatted JSON that I can do LLD (low level discovery) on in our
SAM. This is necessary so I can tell SAM to dynamically add or remove
entries anytime it sees or doesn't see [a given item], as compared to the
last entries. I'm aware of the 'bracketing' option, however, I can't get
my SAM to play nicely with multi-line log matching, not to mention the
post-processing (pre-processing) needed would be a bit of a nightmare to
handle.
The current pstats formatting looks like this:
Sun Jan 19 23:57:54 2025: { "name": "action-15-builtin:omfile", "origin":
"core.action", "processed": 32079, "failed": 0, "suspended": 0,
"suspended.duration": 0, "resumed": 0 }
[...]
Sun Jan 19 23:57:54 2025: { "name": "action-16-builtin:omfile", "origin":
"core.action", "processed": 0, "failed": 0, "suspended": 0,
"suspended.duration": 0, "resumed": 0 }
I want to transform it to something like this:
{
"time": "Sun Jan 19 23:40:26 2025",
"host": "$HOSTNAME",
"stats": [
{ [...] },
{ "name": "action-15-builtin:omfile", "origin": "core.action",
"processed": 32079, "failed": 0, "suspended": 0, "suspended.duration": 0,
"resumed": 0 },
{ "name": "action-16-builtin:omfile", "origin": "core.action",
"processed": 0, "failed": 0, "suspended": 0, "suspended.duration": 0,
"resumed": 0 },
{ [...] }
]
}
I put it in multi-line form (for us humans), but the actual log one would
a single line for processing purposes
________________________________________
This e-mail communication (including any or all attachments) is intended
only for the use of the person or entity to which it is addressed and may
contain confidential and/or privileged material. If you are not the
intended recipient of this e-mail, any use, review, retransmission,
distribution, dissemination, copying, printing, or other use of, or taking
of any action in reliance upon this e-mail, is strictly prohibited. If you
have received this e-mail in error, please contact the sender and delete
the original and any copy of this e-mail and any printout thereof,
immediately. If you have any questions or concerns, please contact our
Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à
son destinataire, qu'il soit une personne ou un organisme, et pourrait
comporter des renseignements privilégiés ou confidentiels. Si vous n'êtes
pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de
retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce
courriel, d'agir en vous y fiant ou de vous en servir de toute autre
façon. Si vous avez reçu le présent courriel par erreur, prière de
communiquer avec l'expéditeur et d'éliminer l'original du courriel, ainsi
que toute copie électronique ou imprimée de celui-ci, immédiatement. Si
vous avez des questions ou des préoccupations, veuillez contacter notre
centre de service à la clientèle au 1-877-274-2349. Nous sommes
reconnaissants de votre collaboration.
________________________________________
_______________________________________________
rsyslog mailing list
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9781aa28dc084468627308dd38ee6516%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729318758942409%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=gNipllHXQ1gqvgzw1qTsQIVZNCLnSG2MKdnvYLi0FOQ%3D&reserved=0<https://lists.adiscon.net/mailman/listinfo/rsyslog>
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9781aa28dc084468627308dd38ee6516%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729318758956309%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=e0voCn8CQ8utb1XenUEOnDVJ8iFem48E8WUnfq45YwA%3D&reserved=0<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9781aa28dc084468627308dd38ee6516%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729318758968361%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2BSXJKV2srCDSajX%2B8mKhj3NbDqwD3r%2BVIHzKvhN%2BTRc%3D&reserved=0<https://twitter.com/rgerhards>
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
