Redbourne,Michael wrote:
Will using a ruleset even help here? As far as I can tell (and I'm not a developer), rsyslog is grabbing internal stats and printing or sending individual lines. Which if that's the case, I don't think creating a different ruleset and/or template will help me, because it'd still be formatted as one JSON message per stat, and we have dozens of stats to pull in.
if you want to combine all of the stats into a single line, then you would need to send them through an external program.
using a ruleset would mean that the stats logs would be processed independently of all other log messages.
you could then send them to an external program (like Simple Event Correlator) that could reformat them into a single message and then log that message to syslog or send it to zabbix (in either case, watch out for maximum message length limits)
David Lang
________________________________ From: David Lang <da...@lang.hm> Sent: Monday, January 20, 2025 7:30 PM To: Redbourne,Michael <michael.redbou...@bulletproofsi.com> Cc: David Lang <da...@lang.hm>; Redbourne,Michael via rsyslog <rsyslog@lists.adiscon.com> Subject: Re: [rsyslog] Building Custom pstats Template CAUTION: The Sender is located Outside The Organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. I would create a ruleset (with it's own queue so that blockages in the main queue won't block stats) and set log.syslog to true. then in the ruleset you can reformat it with a template before you setn. first I would parse the json into $! variable, then create the additionsl fields ($!time, $!host, etc) and then create a templste that sends the message with $! in the body. David Lang On Mon, 20 Jan 2025, Redbourne,Michael wrote:Date: Mon, 20 Jan 2025 03:35:56 +0000 From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> To: David Lang <da...@lang.hm> Cc: "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com> Subject: Re: [rsyslog] Building Custom pstats Template Hey David, I think we're getting mixed up here. To answer your question directly, we're using Zabbix as our SAM which is not capable of multi-line log support when reading log data. With bracketing-based JSON (and JSON directly), the logs actually look like this: Mon Jan 20 03:18:03 2025: BEGIN Mon Jan 20 03:18:03 2025: { "name": "global", "origin": "dynstats", "values": { } } Mon Jan 20 03:18:03 2025: { "name": "global", "origin": "percentile", "values": { } } [...] Mon Jan 20 03:18:03 2025: { "name": "main Q", "origin": "core.queue", "size": 0, "enqueued": 45839, "full": 0, "discarded.full": 0, "discarded.nf": 0, "maxqsize": 1836 } Mon Jan 20 03:18:03 2025: { "name": "imudp(w0)", "origin": "imudp", "called.recvmmsg": 5081, "called.recvmsg": 0, "msgs.received": 45839 } Mon Jan 20 03:18:03 2025: END I want it to look like: {"time": "Sun Jan 19 23:40:26 2025","host": "$HOSTNAME","stats": [{ [...] },{ "name": "action-15-builtin:omfile", "origin": "core.action", "processed": 32079, "failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 },{ "name": "action-16-builtin:omfile", "origin": "core.action", "processed": 0, "failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 },{ [...] }]} OR Sun Jan 19 23:40:26 2025 {"stats": [{ [...] },{ "name": "action-15-builtin:omfile", "origin": "core.action", "processed": 32079, "failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 },{ "name": "action-16-builtin:omfile", "origin": "core.action", "processed": 0, "failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 },{ [...] }]} Here's /etc/rsyslog.conf's pstats config: module( load="impstats" interval="60" severity="9" resetCounters="on" log.file="/var/log/impstats.log" log.syslog="off" format="json" bracketing="on" ) You're correct of course that the output of pstats has valid JSON. It is not valid JSON in its entire form though, nor is the entire pstats output handled as a single JSON entity. The ultimate goal here is to have Zabbix read a single-line entry of JSON (either has JSON or is JSON) for a block of pstats output (for example, all stats received between Jan 20 2025 12:00:00 to Jan 20 2025 12:01:00) and have it dynamically add and delete items as the configurations of rsyslog are updated. What I'm understanding is, this probably isn't possibly with rsyslog. Which isn't the end of the world. The bit that I mostly care about is dynamic creation of objects in Zabbix for the Kafka modules. We have other ways of monitoring Kafka, they just wouldn't integrate with other systems like Ansible to handle recovering failed nodes. ________________________________ From: David Lang <da...@lang.hm> Sent: Monday, January 20, 2025 12:45 PM To: David Lang <da...@lang.hm> Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com>; Redbourne,Michael via rsyslog <rsyslog@lists.adiscon.com> Subject: Re: [rsyslog] Building Custom pstats Template CAUTION: The Sender is located Outside The Organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. what is it that you are sending the message to that needs all those newlines in the json message? David Lang On Sun, 19 Jan 2025, David Lang wrote:Date: Sun, 19 Jan 2025 17:44:56 -0800 (PST) From: David Lang <da...@lang.hm> To: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> Cc: David Lang <da...@lang.hm>, "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com> Subject: Re: [rsyslog] Building Custom pstats Template first off, I would suggest that you check the output of those templates, I don't think they are multi-line output (remember, per the syslog spec, newlines indicate new messages, so multiline messages are problematic to send), I've used some of them in the past and I know they include valid json. start with the format that includes the best json for your use, and then you can parse the output and manipulate it with your own logic and output template however, adding in all the newlines that you want may be harder than you think. David Lang On Mon, 20 Jan 2025, Redbourne,Michael wrote:Date: Mon, 20 Jan 2025 01:25:08 +0000 From: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> To: David Lang <da...@lang.hm>, "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com> Subject: Re: [rsyslog] Building Custom pstats Template Hey David, I read that documentation. As far as I can tell, it only discusses using rsyslog/adiscon-built templates. I don't think I can use those templates - regardless of "bracketing". The SAM we use doesn't appear to support multi-line matching through PCRE(2) via the /s flag. Hence why I'm looking to build and use my own (normalized) JSON template. The JSON template that adiscon uses is both multi line and isn't actual JSON. It contains elements of JSON. Cheers, Mike [cid:e628dcc4-f053-42b6-90a6-79771b1ae670] Michael Redbourne (he/him) Senior Security Analyst Office: +1 (506) 606-0384 Cell: +61 04 2647 3071 SOC: 1-833-415-2424 www.bulletproofsi.com<http://www.bulletproofsi.com/><http://www.bulletproofsi.com<http://www.bulletproofsi.com/>><http://www.bulletproofsi.com<http://www.bulletproofsi.com/><http://www.bulletproofsi.com<http://www.bulletproofsi.com/>>> Book a Meeting<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Foutlook.office365.com%2Fowa%2Fcalendar%2FMichaelRedbourne%40bulletproofsolutions.onmicrosoft.com%2Fbookings%2F&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607376467%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=M1m7E8sAxJUG4XXjfk14KST68OShMxPqBQMBV%2BhciMk%3D&reserved=0<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Foutlook.office365.com%2Fowa%2Fcalendar%2FMichaelRedbourne%40bulletproofsolutions.onmicrosoft.com%2Fbookings%2F&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607393857%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&s
data=6gsKrPj8T8mMMhpHFx%2FNTAyxm%2B4lfBKRqlDzqi4WI1k%3D&reserved=0<https://outlook.office365.com/owa/calendar/michaelredbou...@bulletproofsolutions.onmicrosoft.com/bookings/>>>
[cid:ea725086-9fec-4d13-8b97-056dba7feac4] Notes: Please be advised that I live in Sydney, Australia. My normal hours are 8AM-5PM (Australia/Sydney). I allow 24-hour calendar bookings, but bookings made outside of official working hours should be discussed with me prior to booking. ________________________________ From: David Lang <da...@lang.hm> Sent: Monday, January 20, 2025 12:04 PM To: Redbourne,Michael via rsyslog <rsyslog@lists.adiscon.com> Cc: Redbourne,Michael <michael.redbou...@bulletproofsi.com> Subject: Re: [rsyslog] Building Custom pstats Template CAUTION: The Sender is located Outside The Organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. see https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rsyslog.com%2Fdoc%2Fconfiguration%2Fmodules%2Fimpstats.html&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607405715%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=fvvB2GabbWM0vwu2ArHw2Bay3JYo6rNeCcFuFeUpZIg%3D&reserved=0<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.rsyslog.com%2Fdoc%2Fconfiguration%2Fmodules%2Fimpstats.html&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607417312%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=862tprYIefyOQG7I7WdDRgq%2B7RJvPHoR%2B9ua%2FUhLLPE%3D&reserved=0><https://can01.safelinks.protection
.outlook.com/?url=https%3A%2F%2Fwww.rsyslog.com%2Fdoc%2Fconfigura&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607427949%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2F7nOTFV9sSPvuXOM0BjLZ8rHwFg5rkE3lalRNbI%2FIoU%3D&reserved=0<https://www.rsyslog.com/doc/configuration/modules/impstats.html>
tion/modules/impstats.html>Format type default mandatory obsolete legacy directive word legacy no none New in version 8.16.0. Specifies the format of emitted stats messages. The default of “legacy” is compatible with pre v6-rsyslog. The other options provide support for structured formats (note the “cee” is actually “project lumberjack” logging). The json-elasticsearch format supports the broken ElasticSearch JSON implementation. ES 2.0 no longer supports valid JSON and disallows dots inside names. The “json-elasticsearch” format option replaces those dots by the bang (“!”) character. So “discarded.full” becomes “discarded!full”. Options: json/json-elasticsearch/cee/legacy David Lang On Mon, 20 Jan 2025, Redbourne,Michael via rsyslog wrote:Date: Mon, 20 Jan 2025 01:01:47 +0000 From: "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com> To: "Redbourne,Michael via rsyslog" <rsyslog@lists.adiscon.com> Cc: "Redbourne,Michael" <michael.redbou...@bulletproofsi.com> Subject: [rsyslog] Building Custom pstats Template Hey Folks, I'm wondering if anyone has implemented (or knows how to) implement their own pstats template? The current formatting works well for human-readable monitoring but isn't so good for machine-based monitoring. What I want to do is essentially transform the current format json to a single line of properly formatted JSON that I can do LLD (low level discovery) on in our SAM. This is necessary so I can tell SAM to dynamically add or remove entries anytime it sees or doesn't see [a given item], as compared to the last entries. I'm aware of the 'bracketing' option, however, I can't get my SAM to play nicely with multi-line log matching, not to mention the post-processing (pre-processing) needed would be a bit of a nightmare to handle. The current pstats formatting looks like this: Sun Jan 19 23:57:54 2025: { "name": "action-15-builtin:omfile", "origin": "core.action", "processed": 32079, "failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 } [...] Sun Jan 19 23:57:54 2025: { "name": "action-16-builtin:omfile", "origin": "core.action", "processed": 0, "failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 } I want to transform it to something like this: { "time": "Sun Jan 19 23:40:26 2025", "host": "$HOSTNAME", "stats": [ { [...] }, { "name": "action-15-builtin:omfile", "origin": "core.action", "processed": 32079, "failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 }, { "name": "action-16-builtin:omfile", "origin": "core.action", "processed": 0, "failed": 0, "suspended": 0, "suspended.duration": 0, "resumed": 0 }, { [...] } ] } I put it in multi-line form (for us humans), but the actual log one would a single line for processing purposes ________________________________________ This e-mail communication (including any or all attachments) is intended only for the use of the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient of this e-mail, any use, review, retransmission, distribution, dissemination, copying, printing, or other use of, or taking of any action in reliance upon this e-mail, is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete the original and any copy of this e-mail and any printout thereof, immediately. If you have any questions or concerns, please contact our Customer Service Desk at 1-877-274-2349. Your co-operation is appreciated. Le présent courriel (y compris toute pièce jointe) s'adresse uniquement à son destinataire, qu'il soit une personne ou un organisme, et pourrait comporter des renseignements privilégiés ou confidentiels. Si vous n'êtes pas le destinataire du courriel, il est interdit d'utiliser, de revoir, de retransmettre, de distribuer, de disséminer, de copier ou d'imprimer ce courriel, d'agir en vous y fiant ou de vous en servir de toute autre façon. Si vous avez reçu le présent courriel par erreur, prière de communiquer avec l'expéditeur et d'éliminer l'original du courriel, ainsi que toute copie électronique ou imprimée de celui-ci, immédiatement. Si vous avez des questions ou des préoccupations, veuillez contacter notre centre de service à la clientèle au 1-877-274-2349. Nous sommes reconnaissants de votre collaboration. ________________________________________ _______________________________________________ rsyslog mailing list https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607439570%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2BFBBPktOpZQ3be9GBpViMEzUCWnOa%2BluPfHbLN4cmYU%3D&reserved=0<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607455791%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=X6Ncwc942m9yguBBuxPd48rag7Uh7Zns2OZ1PKGUQhs%3D&reserved=0><https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2
Flists.adiscon.net%2Fmailman%2Flistinfo%2Frsyslog&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607472084%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=2sSBti5hrhtonCKQFb13vgYUI%2BjPfRoW%2F4E4XiG8IWU%3D&reserved=0><https://lists.adiscon.net/mailman/listinfo/rsyslog>
https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607489656%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=gNdibkEONg85OEUk3dCR6j51SOEWDrqq0xH%2BX7yGGH8%3D&reserved=0<https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rsyslog.com%2Fprofessional-services%2F&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607506125%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Q%2B1GLq%2BZCOIapo%2BoTJoSLi%2B8UfUSE6M1%2FhqPnRNsMno%3D&reserved=0><https://can01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.rs
yslog.com%2Fprofessional-services%2F&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607522333%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ITguJre82V09sQiL2OLqAlEy5%2FoEpYGBQnRrfvKe6HQ%3D&reserved=0><http://www.rsyslog.com/professional-services/>
What's up with rsyslog? Follow https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607538057%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=hiQ4qhtqTkoCTAf6K9RTHo6DlrbUBD4%2Blx5ppmxN4lM%3D&reserved=0<https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=05%7C02%7Cmichael.redbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607549690%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=xhpxRdhFt2g0Y9DhrQGR%2BOeiDRqKaScI6S0QcsM4hkQ%3D&reserved=0><https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Frgerhards&data=05%7C02%7Cmichael.re
dbourne%40bulletproofsi.com%7Cf11ca76e8845447ad8c708dd392cc228%7C9a63d13853ea411bbe8458b7e2570747%7C1%7C0%7C638729586607560744%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=J2Sjn1FeakqcuHpf90ELyB89f9mfMKK1xF8KNz89Qf4%3D&reserved=0><https://twitter.com/rgerhards>
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________ rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
