first off, I would suggest that you check the output of those templates, I don't think they are multi-line output (remember, per the syslog spec, newlines indicate new messages, so multiline messages are problematic to send), I've used some of them in the past and I know they include valid json.

start with the format that includes the best json for your use, and then you can parse the output and manipulate it with your own logic and output template

however, adding in all the newlines that you want may be harder than you think.

David Lang

On Mon, 20 Jan 2025, Redbourne,Michael wrote:

Date: Mon, 20 Jan 2025 01:25:08 +0000
From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com>
To: David Lang <da...@lang.hm>,
    "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Building Custom pstats Template

Hey David,

I read that documentation. As far as I can tell, it only discusses using 
rsyslog/adiscon-built templates. I don't think I can use those templates - regardless of 
"bracketing". The SAM we use doesn't appear to support multi-line matching 
through PCRE(2) via the /s flag.

Hence why I'm looking to build and use my own (normalized) JSON template. The 
JSON template that adiscon uses is both multi line and isn't actual JSON. It 
contains elements of JSON.


Cheers,
Mike




[cid:e628dcc4-f053-42b6-90a6-79771b1ae670]



Michael Redbourne (he/him)
Senior Security Analyst

Office: +1 (506) 606-0384
Cell: +61 04 2647 3071

SOC: 1-833-415-2424

www.bulletproofsi.com<http://www.bulletproofsi.com/>
Book a 
Meeting<https://outlook.office365.com/owa/calendar/michaelredbou...@bulletproofsolutions.onmicrosoft.com/bookings/>

[cid:ea725086-9fec-4d13-8b97-056dba7feac4]

Notes: Please be advised that I live in Sydney, Australia. My normal hours are 
8AM-5PM (Australia/Sydney). I allow 24-hour calendar bookings, but bookings 
made outside of official working hours should be discussed with me prior to 
booking.



________________________________
From: David Lang <da...@lang.hm>
Sent: Monday, January 20, 2025 12:04 PM
To: Redbourne,Michael via rsyslog <rsyslog@lists.adiscon.com>
Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com>
Subject: Re: [rsyslog] Building Custom pstats Template

CAUTION: The Sender is located Outside The Organization. Do not click links or 
open attachments unless you recognize the sender and know the content is safe.


see 
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rsyslog.com%2Fdoc%2Fconfiguration%2Fmodules%2Fimpstats.html&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9781aa28dc084468627308dd38ee6516%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729318758914939%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=7XiAjfDEAt8QZyju2Od1NnGFNhxjlpZUz8CT1Xd0i1Q%3D&reserved=0<https://www.rsyslog.com/doc/configuration/modules/impstats.html>

Format
type

default

mandatory

obsolete legacy directive

word

legacy

no

none

New in version 8.16.0.

Specifies the format of emitted stats messages. The default of “legacy” is
compatible with pre v6-rsyslog. The other options provide support for structured
formats (note the “cee” is actually “project lumberjack” logging).

The json-elasticsearch format supports the broken ElasticSearch JSON
implementation. ES 2.0 no longer supports valid JSON and disallows dots inside
names. The “json-elasticsearch” format option replaces those dots by the bang
(“!”) character. So “discarded.full” becomes “discarded!full”. Options:
json/json-elasticsearch/cee/legacy

David Lang

 On Mon, 20 Jan 2025, Redbourne,Michael via
rsyslog wrote:

Date: Mon, 20 Jan 2025 01:01:47 +0000
From: "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com>
To: "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com>
Cc: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com>
Subject: [rsyslog] Building Custom pstats Template

Hey Folks,

I'm wondering if anyone has implemented (or knows how to) implement their own 
pstats template? The current formatting works well for human-readable 
monitoring but isn't so good for machine-based monitoring. What I want to do is 
essentially transform the current format json to a single line of properly 
formatted JSON that I can do LLD (low level discovery) on in our SAM. This is 
necessary so I can tell SAM to dynamically add or remove entries anytime it 
sees or doesn't see [a given item], as compared to the last entries. I'm aware 
of the 'bracketing' option, however, I can't get my SAM to play nicely with 
multi-line log matching, not to mention the post-processing (pre-processing) 
needed would be a bit of a nightmare to handle.

The current pstats formatting looks like this:
Sun Jan 19 23:57:54 2025: { "name": "action-15-builtin:omfile", "origin": "core.action", "processed": 32079, 
"failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 }
[...]
Sun Jan 19 23:57:54 2025: { "name": "action-16-builtin:omfile", "origin": "core.action", "processed": 0, 
"failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 }


I want to transform it to something like this:
{
 "time": "Sun Jan 19 23:40:26 2025",
 "host": "$HOSTNAME",
 "stats": [
   { [...] },
   { "name": "action-15-builtin:omfile", "origin": "core.action", "processed": 32079, "failed": 0, 
"suspended": 0, "suspended.duration": 0, "resumed": 0 },
   { "name": "action-16-builtin:omfile", "origin": "core.action", "processed": 0, "failed": 0, 
"suspended": 0, "suspended.duration": 0, "resumed": 0 },
   { [...] }
 ]
}

I put it in multi-line form (for us humans), but the actual log one would a 
single line for processing purposes
________________________________________
This e-mail communication (including any or all attachments) is intended only 
for the use of the person or entity to which it is addressed and may contain 
confidential and/or privileged material. If you are not the intended recipient 
of this e-mail, any use, review, retransmission, distribution, dissemination, 
copying, printing, or other use of, or taking of any action in reliance upon 
this e-mail, is strictly prohibited. If you have received this e-mail in error, 
please contact the sender and delete the original and any copy of this e-mail 
and any printout thereof, immediately. If you have any questions or concerns, 
please contact our Customer Service Desk at 1-877-274-2349. Your co-operation 
is appreciated.

Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à son 
destinataire, qu'il soit une personne ou un organisme, et pourrait comporter 
des renseignements privilégiés ou confidentiels. Si vous n'êtes pas le 
destinataire du courriel, il est interdit d'utiliser, de revoir, de 
retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce 
courriel, d'agir en vous y fiant ou de vous en servir de toute autre façon. Si 
vous avez reçu le présent courriel par erreur, prière de communiquer avec 
l'expéditeur et d'éliminer l'original du courriel, ainsi que toute copie 
électronique ou imprimée de celui-ci, immédiatement. Si vous avez des questions 
ou des préoccupations, veuillez contacter notre centre de service à la 
clientèle au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration.
________________________________________
_______________________________________________
rsyslog mailing list
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9781aa28dc084468627308dd38ee6516%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729318758942409%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=gNipllHXQ1gqvgzw1qTsQIVZNCLnSG2MKdnvYLi0FOQ%3D&reserved=0<https://lists.adiscon.net/mailman/listinfo/rsyslog>
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9781aa28dc084468627308dd38ee6516%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729318758956309%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=e0voCn8CQ8utb1XenUEOnDVJ8iFem48E8WUnfq45YwA%3D&reserved=0<http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow 
https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9781aa28dc084468627308dd38ee6516%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729318758968361%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2BSXJKV2srCDSajX%2B8mKhj3NbDqwD3r%2BVIHzKvhN%2BTRc%3D&reserved=0<https://twitter.com/rgerhards>
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to