Hey David,
I think we're getting mixed up here. To answer your question directly, we're
using Zabbix as our SAM which is not capable of multi-line log support when
reading log data. With bracketing-based JSON (and JSON directly), the logs
actually look like this:
Mon Jan 20 03:18:03 2025: BEGIN
Mon Jan 20 03:18:03 2025: { "name": "global", "origin": "dynstats", "values": {
} }
Mon Jan 20 03:18:03 2025: { "name": "global", "origin": "percentile", "values":
{ } }
[...]
Mon Jan 20 03:18:03 2025: { "name": "main Q", "origin": "core.queue", "size":
0, "enqueued": 45839, "full": 0, "discarded.full": 0, "discarded.nf": 0,
"maxqsize": 1836 }
Mon Jan 20 03:18:03 2025: { "name": "imudp(w0)", "origin": "imudp",
"called.recvmmsg": 5081, "called.recvmsg": 0, "msgs.received": 45839 }
Mon Jan 20 03:18:03 2025: END
I want it to look like:
{"time": "Sun Jan 19 23:40:26 2025","host": "$HOSTNAME","stats": [{ [...] },{
"name": "action-15-builtin:omfile", "origin": "core.action", "processed":
32079, "failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 },{
"name": "action-16-builtin:omfile", "origin": "core.action", "processed": 0,
"failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 },{ [...] }]}
OR
Sun Jan 19 23:40:26 2025 {"stats": [{ [...] },{ "name":
"action-15-builtin:omfile", "origin": "core.action", "processed": 32079,
"failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 },{ "name":
"action-16-builtin:omfile", "origin": "core.action", "processed": 0, "failed":
0, "suspended": 0, "suspended.duration": 0, "resumed": 0 },{ [...] }]}
Here's /etc/rsyslog.conf's pstats config:
module(
load="impstats"
interval="60"
severity="9"
resetCounters="on"
log.file="/var/log/impstats.log"
log.syslog="off"
format="json"
bracketing="on"
)
You're correct of course that the output of pstats has valid JSON. It is not
valid JSON in its entire form though, nor is the entire pstats output handled
as a single JSON entity. The ultimate goal here is to have Zabbix read a
single-line entry of JSON (either has JSON or is JSON) for a block of pstats
output (for example, all stats received between Jan 20 2025 12:00:00 to Jan 20
2025 12:01:00) and have it dynamically add and delete items as the
configurations of rsyslog are updated. What I'm understanding is, this probably
isn't possibly with rsyslog. Which isn't the end of the world. The bit that I
mostly care about is dynamic creation of objects in Zabbix for the Kafka
modules. We have other ways of monitoring Kafka, they just wouldn't integrate
with other systems like Ansible to handle recovering failed nodes.
________________________________
From: David Lang <da...@lang.hm>
Sent: Monday, January 20, 2025 12:45 PM
To: David Lang <da...@lang.hm>
Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com>; Redbourne,Michael
via rsyslog <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] Building Custom pstats Template
CAUTION: The Sender is located Outside The Organization. Do not click links or
open attachments unless you recognize the sender and know the content is safe.
what is it that you are sending the message to that needs all those newlines in
the json message?
David Lang
On Sun, 19 Jan 2025, David Lang wrote:
> Date: Sun, 19 Jan 2025 17:44:56 -0800 (PST)
> From: David Lang <da...@lang.hm>
> To: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com>
> Cc: David Lang <da...@lang.hm>,
> "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] Building Custom pstats Template
>
> first off, I would suggest that you check the output of those templates, I
> don't think they are multi-line output (remember, per the syslog spec,
> newlines indicate new messages, so multiline messages are problematic to
> send), I've used some of them in the past and I know they include valid json.
>
> start with the format that includes the best json for your use, and then you
> can parse the output and manipulate it with your own logic and output
> template
>
> however, adding in all the newlines that you want may be harder than you
> think.
>
> David Lang
>
> On Mon, 20 Jan 2025, Redbourne,Michael wrote:
>
>> Date: Mon, 20 Jan 2025 01:25:08 +0000
>> From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com>
>> To: David Lang <da...@lang.hm>,
>> "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] Building Custom pstats Template
>>
>> Hey David,
>>
>> I read that documentation. As far as I can tell, it only discusses using
>> rsyslog/adiscon-built templates. I don't think I can use those templates -
>> regardless of "bracketing". The SAM we use doesn't appear to support
>> multi-line matching through PCRE(2) via the /s flag.
>>
>> Hence why I'm looking to build and use my own (normalized) JSON template.
>> The JSON template that adiscon uses is both multi line and isn't actual
>> JSON. It contains elements of JSON.
>>
>>
>> Cheers,
>> Mike
>>
>>
>>
>>
>> [cid:e628dcc4-f053-42b6-90a6-79771b1ae670]
>>
>>
>>
>> Michael Redbourne (he/him)
>> Senior Security Analyst
>>
>> Office: +1 (506) 606-0384
>> Cell: +61 04 2647 3071
>>
>> SOC: 1-833-415-2424
>>
>> www.bulletproofsi.com<http://www.bulletproofsi.com/><http://www.bulletproofsi.com<http://www.bulletproofsi.com/>>
>> Book a
>> Meeting<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Foutlook.office365.com%2Fowa%2Fcalendar%2FMichaelRedbourne%40bulletproofsolutions.onmicrosoft.com%2Fbookings%2F&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9b99a1b4d34746e1d02a08dd38f4267a%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729343463491376%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=WRJDyAmSSR3s4j3lm5kHyC%2Fqw8sfVEzpCsoLpOUK3eM%3D&reserved=0<https://outlook.office365.com/owa/calendar/michaelredbou...@bulletproofsolutions.onmicrosoft.com/bookings/>>
>>
>> [cid:ea725086-9fec-4d13-8b97-056dba7feac4]
>>
>> Notes: Please be advised that I live in Sydney, Australia. My normal hours
>> are 8AM-5PM (Australia/Sydney). I allow 24-hour calendar bookings, but
>> bookings made outside of official working hours should be discussed with me
>> prior to booking.
>>
>>
>>
>> ________________________________
>> From: David Lang <da...@lang.hm>
>> Sent: Monday, January 20, 2025 12:04 PM
>> To: Redbourne,Michael via rsyslog <rsyslog@lists.adiscon.com>
>> Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com>
>> Subject: Re: [rsyslog] Building Custom pstats Template
>>
>> CAUTION: The Sender is located Outside The Organization. Do not click links
>> or open attachments unless you recognize the sender and know the content is
>> safe.
>>
>>
>> see
>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rsyslog.com%2Fdoc%2Fconfiguration%2Fmodules%2Fimpstats.html&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9b99a1b4d34746e1d02a08dd38f4267a%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729343463509526%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=uZ1uoEIbWGGtLvGMoonLYjQdAYs5dG98MAsyK1ayAp8%3D&reserved=0<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rsyslog.com%2Fdoc%2Fconfiguration%2Fmodules%2Fimpstats.html&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9b99a1b4d34746e1d02a08dd38f4267a%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729343463521760%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=R3bT1cTlcYxTzWF7ueDZYRrzJQtV8TR%2BdEuMtUXwN%2FE%3D&reserved=0><https://www.rsyslog.com/doc/configuration/modules/impstats.html>
>>
>> Format
>> type
>>
>> default
>>
>> mandatory
>>
>> obsolete legacy directive
>>
>> word
>>
>> legacy
>>
>> no
>>
>> none
>>
>> New in version 8.16.0.
>>
>> Specifies the format of emitted stats messages. The default of “legacy” is
>> compatible with pre v6-rsyslog. The other options provide support for
>> structured
>> formats (note the “cee” is actually “project lumberjack” logging).
>>
>> The json-elasticsearch format supports the broken ElasticSearch JSON
>> implementation. ES 2.0 no longer supports valid JSON and disallows dots
>> inside
>> names. The “json-elasticsearch” format option replaces those dots by the
>> bang
>> (“!”) character. So “discarded.full” becomes “discarded!full”. Options:
>> json/json-elasticsearch/cee/legacy
>>
>> David Lang
>>
>> On Mon, 20 Jan 2025, Redbourne,Michael via
>> rsyslog wrote:
>>
>>> Date: Mon, 20 Jan 2025 01:01:47 +0000
>>> From: "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com>
>>> To: "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com>
>>> Cc: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com>
>>> Subject: [rsyslog] Building Custom pstats Template
>>>
>>> Hey Folks,
>>>
>>> I'm wondering if anyone has implemented (or knows how to) implement their
>>> own pstats template? The current formatting works well for human-readable
>>> monitoring but isn't so good for machine-based monitoring. What I want to
>>> do is essentially transform the current format json to a single line of
>>> properly formatted JSON that I can do LLD (low level discovery) on in our
>>> SAM. This is necessary so I can tell SAM to dynamically add or remove
>>> entries anytime it sees or doesn't see [a given item], as compared to the
>>> last entries. I'm aware of the 'bracketing' option, however, I can't get
>>> my SAM to play nicely with multi-line log matching, not to mention the
>>> post-processing (pre-processing) needed would be a bit of a nightmare to
>>> handle.
>>>
>>> The current pstats formatting looks like this:
>>> Sun Jan 19 23:57:54 2025: { "name": "action-15-builtin:omfile", "origin":
>>> "core.action", "processed": 32079, "failed": 0, "suspended": 0,
>>> "suspended.duration": 0, "resumed": 0 }
>>> [...]
>>> Sun Jan 19 23:57:54 2025: { "name": "action-16-builtin:omfile", "origin":
>>> "core.action", "processed": 0, "failed": 0, "suspended": 0,
>>> "suspended.duration": 0, "resumed": 0 }
>>>
>>>
>>> I want to transform it to something like this:
>>> {
>>> "time": "Sun Jan 19 23:40:26 2025",
>>> "host": "$HOSTNAME",
>>> "stats": [
>>> { [...] },
>>> { "name": "action-15-builtin:omfile", "origin": "core.action",
>>> "processed": 32079, "failed": 0, "suspended": 0, "suspended.duration": 0,
>>> "resumed": 0 },
>>> { "name": "action-16-builtin:omfile", "origin": "core.action",
>>> "processed": 0, "failed": 0, "suspended": 0, "suspended.duration": 0,
>>> "resumed": 0 },
>>> { [...] }
>>> ]
>>> }
>>>
>>> I put it in multi-line form (for us humans), but the actual log one would
>>> a single line for processing purposes
>>> ________________________________________
>>> This e-mail communication (including any or all attachments) is intended
>>> only for the use of the person or entity to which it is addressed and may
>>> contain confidential and/or privileged material. If you are not the
>>> intended recipient of this e-mail, any use, review, retransmission,
>>> distribution, dissemination, copying, printing, or other use of, or taking
>>> of any action in reliance upon this e-mail, is strictly prohibited. If you
>>> have received this e-mail in error, please contact the sender and delete
>>> the original and any copy of this e-mail and any printout thereof,
>>> immediately. If you have any questions or concerns, please contact our
>>> Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated.
>>>
>>> Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à
>>> son destinataire, qu'il soit une personne ou un organisme, et pourrait
>>> comporter des renseignements privilégiés ou confidentiels. Si vous n'êtes
>>> pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de
>>> retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce
>>> courriel, d'agir en vous y fiant ou de vous en servir de toute autre
>>> façon. Si vous avez reçu le présent courriel par erreur, prière de
>>> communiquer avec l'expéditeur et d'éliminer l'original du courriel, ainsi
>>> que toute copie électronique ou imprimée de celui-ci, immédiatement. Si
>>> vous avez des questions ou des préoccupations, veuillez contacter notre
>>> centre de service à la clientèle au 1-877-274-2349. Nous sommes
>>> reconnaissants de votre collaboration.
>>> ________________________________________
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9b99a1b4d34746e1d02a08dd38f4267a%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729343463533617%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=YLoyW4P3tG0QhmSnT1NYZR5sWAZjiBkrvDHBOrWwbAw%3D&reserved=0<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9b99a1b4d34746e1d02a08dd38f4267a%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729343463544976%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=qGn0mOqSM434YD3eFmr6Kh9r9QPNiOzki6DtT4wUvRc%3D&reserved=0><https://lists.adiscon.net/mailman/listinfo/rsyslog>
>>> https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9b99a1b4d34746e1d02a08dd38f4267a%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729343463556376%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=8ZkZcyZbP4E1fEXC%2BJXKvGQpEEBCPftV9V6ef%2BKvZh0%3D&reserved=0<https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9b99a1b4d34746e1d02a08dd38f4267a%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729343463568344%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=zZhPuG64rFqnFX%2BgewYiGm%2FCFiTAfW7oSCug8owLKfM%3D&reserved=0><http://www.rsyslog.com/professional-services/>
>>> What's up with rsyslog? Follow
>>> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9b99a1b4d34746e1d02a08dd38f4267a%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729343463580065%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=pKwEPgCaj3V99%2BNjL9TXK87L6pEq0e1Nxnt3kfz0POA%3D&reserved=0<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7C9b99a1b4d34746e1d02a08dd38f4267a%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729343463592050%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=1dwu0vvbjsIAYOe3mhmyXhEzCV5ciOSilRUeoydlzpY%3D&reserved=0><https://twitter.com/rgerhards>
>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
>>> DON'T LIKE THAT.
>>>
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
