Morning Sean,
Are you using the stdout output plugin to view the logs from logstash?
IIRC it tries to escape the data.
Try outputing the logs to a file.
Using tcpdump to look at the syslog data after rsyslog sends it and/or
before rsyslog receives it might also help.
BR,
- Simon
On Mon, 2023-09-18 at 17:04:25 +0200, Lennon, Sean (UK) via rsyslog wrote:
> This email may contain proprietary information of BAE Systems and/or third
> parties.
>
> Sorry, but for ‘reasons’ I can only give you a severely edited version, I
> have used debug output from mmexternal first and the received message from
> logstash second:
>
>
> 1. mexternal debug output – I am satisfied with this.
> { “msg” : {“messageGroup”:[{“field1”:1,”field2”:2},{“field1”:3,”field2”:4}]}}
>
> 2. what logstash receives
> “message” => “{ \“msg\” :
> {\“messageGroup\”:[{\“field1\”:1,\”field2\”:2},{\“field1\”:3,\”field2\”:4}]}}”
>
>
> From: Rainer Gerhards <rgerha...@hq.adiscon.com>
> Sent: 18 September 2023 15:47
> To: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with escaped
> quotations and additional quotations
>
>
> PHISHING ALERT
> This email has been sent from an account outside of the BAE Systems network.
>
> Please treat the email with caution, especially if you are requested to click
> on a link or open an attachment.
> For further information on how to spot and report a phishing email please
> access the Global Intranet then select <Functions> / <IT>.
> If you think this is a phishing email, please report it by using the "Report
> Phishing" button in Outlook.
>
>
> Output the message with RSYSLOG_DebugFormat template. I need to see which
> data msg actually has.
>
> Rainer
> Sent from phone, thus brief.
>
> Lennon, Sean (UK)
> <sean.lenn...@baesystems.com<mailto:sean.lenn...@baesystems.com>> schrieb am
> Mo., 18. Sept. 2023, 16:41:
>
>
>
>
> This email may contain proprietary information of BAE Systems and/or third
> parties.
>
> Thanks for your response Rainer. I don't think it answers my question, I
> have property fields from the Rsyslog message that are fine, they get
> formatted correctly, for example 'timereported' or 'syslogseverity-text'.
> So, the output json for these and others are correct, it's the msg field that
> is returned from my custom code (using mmexternal) that is the problem.
>
> I have created a newer template that is more upto date and looks something
> similar to this:
>
> template(name="json-template" type="list" option.jsonf="on") {
> property(outname="@timestamp" name="timereported"
> dataformat="rfc3339" format="jsonf")
> property(outname="message" name="msg" format="jsonf")
> }
>
> -----Original Message-----
> From: Rainer Gerhards
> <rgerha...@hq.adiscon.com<mailto:rgerha...@hq.adiscon.com>>
> Sent: 18 September 2023 15:26
> To: rsyslog-users
> <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>
> Cc: Lennon, Sean (UK)
> <sean.lenn...@baesystems.com<mailto:sean.lenn...@baesystems.com>>
> Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with escaped
> quotations and additional quotations
>
> ----------------------------- PHISHING ALERT -----------------------------
> This email has been sent from an account outside of the BAE Systems network.
>
> Please treat the email with caution, especially if you are requested to click
> on a link or open an attachment.
> For further information on how to spot and report a phishing email please
> access the Global Intranet, then select <Functions> / <IT>.
>
> ------------------------------------------------------------------------------------
>
> Does this example from the rsyslog testbench help?
>
> https://github.com/rsyslog/rsyslog/blob/761cb2bc51e3046b242b45994cff11ff8be3990e/tests/json-nonstring.sh#L4
>
> Rainer
>
> El lun, 18 sept 2023 a las 15:10, Lennon, Sean (UK) via rsyslog
> (<rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>) escribió:
> >
> >
> >
> >
> >
> > This email may contain proprietary information of BAE Systems and/or third
> > parties.
> >
> > This is the one I meant.
> >
> > -----Original Message-----
> > From: rsyslog
> > <rsyslog-boun...@lists.adiscon.com<mailto:rsyslog-boun...@lists.adiscon.com>>
> > On Behalf Of Lennon,
> > Sean (UK) via rsyslog
> > Sent: 29 August 2023 17:39
> > To: rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>
> > Cc: Lennon, Sean (UK)
> > <sean.lenn...@baesystems.com<mailto:sean.lenn...@baesystems.com>>
> > Subject: [rsyslog] rsyslog mmextenal logstash json output with escaped
> > quotations and additional quotations
> >
> > ----------------------------- PHISHING ALERT
> > ----------------------------- This email has been sent from an account
> > outside of the BAE Systems network.
> >
> > Please treat the email with caution, especially if you are requested to
> > click on a link or open an attachment.
> > For further information on how to spot and report a phishing email please
> > access the Global Intranet, then select <Functions> / <IT>.
> >
> > ----------------------------------------------------------------------
> > --------------
> >
> > This email may contain proprietary information of BAE Systems and/or third
> > parties.
> >
> > Hi all,
> >
> > I've encountered an issue with formatting json output to logstash. I'm
> > using mmexternal to reformat data received from a remote system, the data
> > is project specific and needs to be massaged into json for use with
> > logstash. The intention is to create a json message for logstash with the
> > mmexternal output being part of that message. I'm able to receive this
> > json output at logstash but the message field (which contains the
> > mmexternal output) is encapsulated within double quotes and all json fields
> > within have escaped double quotes. This means that logstash is not able to
> > interpret part of the message. If I take the raw output of the mmextenal
> > code and send it to a omfile then it looks perfectly fine.
> >
> > I have asked a more detailed question, on Stackoverflow:
> > https://stackoverflow.com/questions/77001549/rsyslog-mmextenal-logstas
> > h-json-output-with-escaped-quotations-and-additional-qu
> >
> > What am I missing?
> >
> > I appreciate your help.
> >
> > Sean
> >
> > ********************************************************************
> > This email and any attachments are confidential to the intended recipient
> > and may also be privileged. If you are not the intended recipient please
> > delete it from your system and notify the sender.
> > You should not copy it or use it for any purpose nor disclose or distribute
> > its contents to any other person.
> > ********************************************************************
> >
> > BAE Systems may process information about you that may be subject to
> > data protection laws. For more information about how we use your
> > personal information, how we protect your information, our legal basis
> > for using your information, your rights and who you can contact,
> > please refer to our Privacy Notice at
> > www.baesystems.com/en/privacy<http://www.baesystems.com/en/privacy>
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
> > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
> > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
> > THAT.
> >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > LIKE THAT.
> _______________________________________________
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
