This email may contain proprietary information of BAE Systems and/or third
parties.
Fixed it, I think there was some confusion over the template and quotes and
formatting the message field, so the end of the template should be:
template(name="json-template"
type="list") {
constant(value="{")
constant(value="\"timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"message\":") property(name="msg")
constant(value="}\n")
}
Not:
template(name="json-template"
type="list") {
constant(value="{")
constant(value="\"timestamp\":\"") property(name="timereported"
dateFormat="rfc3339")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"")
property(name="syslogseverity-text")
constant(value="\",\"facility\":\"")
property(name="syslogfacility-text")
constant(value="\",\"message\":\"") property(name="msg" format="jsonf")
#<--remove /" & format
constant(value="\"}\n") #<--remove /"
}
'Tis now working perfectly, thank you all.
Sean.
-----Original Message-----
From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Lennon, Sean
(UK) via rsyslog
Sent: 20 September 2023 13:21
To: rsyslog-users <rsyslog@lists.adiscon.com>
Cc: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with escaped
quotations and additional quotations
----------------------------- PHISHING ALERT -----------------------------
This email has been sent from an account outside of the BAE Systems network.
Please treat the email with caution, especially if you are requested to click
on a link or open an attachment.
For further information on how to spot and report a phishing email please
access the Global Intranet, then select <Functions> / <IT>.
------------------------------------------------------------------------------------
This email may contain proprietary information of BAE Systems and/or third
parties.
Hi Simon,
Yes, I did during early development of the mmexternal code, I tried all the
combinations. However, all of this Linux dev is new to me and there is so much
to take in, so given that there were a lot of unknowns a few weeks ago I may
have missed something, so I will go through it again this afternoon and check
out those other options.
Cheers,
Sean.
-----Original Message-----
From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Simon Lundström
via rsyslog
Sent: 20 September 2023 12:44
To: rsyslog@lists.adiscon.com
Cc: Simon Lundström <si...@su.se>
Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with escaped
quotations and additional quotations
----------------------------- PHISHING ALERT -----------------------------
This email has been sent from an account outside of the BAE Systems network.
Please treat the email with caution, especially if you are requested to click
on a link or open an attachment.
For further information on how to spot and report a phishing email please
access the Global Intranet, then select <Functions> / <IT>.
------------------------------------------------------------------------------------
That's excellent Sean!
Have you tried using any of the other json formats on your template:
https://rsyslog.readthedocs.io/en/latest/configuration/templates.html#property-statement
template(name="json-template" type="list" option.jsonf="on") {
property(outname="@timestamp" name="timereported" dataformat="rfc3339"
format="jsonf")
property(outname="message" name="msg" format="jsonf") }
So format="json" and so on?
BR,
- Simon
On Wed, 2023-09-20 at 13:22:09 +0200, Lennon, Sean (UK) wrote:
>
>
>
>
> This email may contain proprietary information of BAE Systems and/or third
> parties.
>
> Rainer/David/Simon,
>
> Thanks for your help, I have done what you have requested/suggested:
>
> The following are redacted versions of the shortest message I have. I've
> used David's suggestion of using sed to process the messages to remove
> sensitive data, I've opted to change numbers to 0. I have left some
> inconsequential fields un-redacted for clarity. These messages were captured
> using tcpdump as suggested by Simon, therefore they are what rsyslog puts out
> to logstash.
>
> The first message is what I've settled on for the time being, i.e. logstash
> will accept it and we can get logstash to process it to remove the quotes
> around the value for the message field and also remove the escape characters.
> It's a bit of a fudge and not ideal, but it 'works'.
> {
> "@timestamp":"2023-09-20T11:06:03.571548+01:00",
> "@version":"1",
> "sysloghost":"XXXXXXXXXXXXXXX",
> "severity":"info",
> "facility":"local6",
> "message":"{
> \"XXXX\": [
> {
> \"XXXXXXXXXXX\":0,
> \"XXXXXXX\":0,
> \"XXXXXXXXXXXXX\": {
> \"XXXXXX\": \"XXXXXXXXXXXXXXXX\",
> \"XXXXXXXXXXXXX\": {
> \"XXXX_XXXXXX\": \"XXXX_XXXXXXX\",
> \"XXXX_XXXXXX_XXXX\": 0
> }
> }
> }
> ]
> }"
> }
>
> The second message is produced after removing format="jsonf" from the message
> field in the template. Note that the escaped characters have gone but the
> quotes around the message field value have not. This causes logstash to
> barf, hence the previous message is the one being used.
> {
> "@timestamp":"2023-09-20T11:16:14.159361+01:00",
> "@version":"1",
> "sysloghost":"XXXXXXXXXXXXXXX",
> "severity":"info",
> "facility":"local6",
> "message":"{
> "XXXX": [
> {
> "XXXXXXXXXXX":0,
> "XXXXXXX":0,
> "XXXXXXXXXXXXX": {
> "XXXXXX": "XXXXXXXXXXXXXXXX",
> "XXXXXXXXXXXXX": {
> "XXXX_XXXXXX": "XXXX_XXXXXXX",
> "XXXX_XXXXXX_XXXX": 0
> }
> }
> }
> ]
> }"
> }
>
> So, how do I get rid of the quotes around the message value?
>
> Cheers,
>
> Sean.
>
> -----Original Message-----
> From: Rainer Gerhards <rgerha...@hq.adiscon.com>
> Sent: 19 September 2023 08:51
> To: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with
> escaped quotations and additional quotations
>
> ----------------------------- PHISHING ALERT
> ----------------------------- This email has been sent from an account
> outside of the BAE Systems network.
>
> Please treat the email with caution, especially if you are requested to click
> on a link or open an attachment.
> For further information on how to spot and report a phishing email please
> access the Global Intranet, then select <Functions> / <IT>.
>
> ----------------------------------------------------------------------
> --------------
>
> It's not easy to guess what rsyslog really sees, but I have one final
> shot. As it looks, msg might already contain json. In that case, do
> not use any special json formatting option. These options exist to
> ensure non-json data (or json data as an inner encapsulation layer)
> will be received as-is.
>
> Rainer
>
> El lun, 18 sept 2023 a las 17:04, Lennon, Sean (UK)
> (<sean.lenn...@baesystems.com>) escribió:
> >
> > This email may contain proprietary information of BAE Systems and/or third
> > parties.
> >
> >
> >
> > Sorry, but for ‘reasons’ I can only give you a severely edited version, I
> > have used debug output from mmexternal first and the received message from
> > logstash second:
> >
> >
> >
> > 1. mexternal debug output – I am satisfied with this.
> > { “msg” :
> > {“messageGroup”:[{“field1”:1,”field2”:2},{“field1”:3,”field2”:4}]}}
> >
> > 2. what logstash receives
> > “message” => “{ \“msg\” :
> > {\“messageGroup\”:[{\“field1\”:1,\”field2\”:2},{\“field1\”:3,\”field2\”:4}]}}”
> >
> >
> >
> >
> >
> > From: Rainer Gerhards <rgerha...@hq.adiscon.com>
> > Sent: 18 September 2023 15:47
> > To: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
> > Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> > Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with
> > escaped quotations and additional quotations
> >
> >
> >
> >
> > PHISHING ALERT
> >
> > This email has been sent from an account outside of the BAE Systems network.
> >
> > Please treat the email with caution, especially if you are requested to
> > click on a link or open an attachment.
> > For further information on how to spot and report a phishing email please
> > access the Global Intranet then select <Functions> / <IT>.
> > If you think this is a phishing email, please report it by using the
> > "Report Phishing" button in Outlook.
> >
> >
> >
> >
> >
> > Output the message with RSYSLOG_DebugFormat template. I need to see which
> > data msg actually has.
> >
> >
> >
> > Rainer
> >
> > Sent from phone, thus brief.
> >
> >
> >
> > Lennon, Sean (UK) <sean.lenn...@baesystems.com> schrieb am Mo., 18. Sept.
> > 2023, 16:41:
> >
> >
> >
> >
> >
> > This email may contain proprietary information of BAE Systems and/or third
> > parties.
> >
> > Thanks for your response Rainer. I don't think it answers my question, I
> > have property fields from the Rsyslog message that are fine, they get
> > formatted correctly, for example 'timereported' or 'syslogseverity-text'.
> > So, the output json for these and others are correct, it's the msg field
> > that is returned from my custom code (using mmexternal) that is the problem.
> >
> > I have created a newer template that is more upto date and looks something
> > similar to this:
> >
> > template(name="json-template" type="list" option.jsonf="on") {
> > property(outname="@timestamp" name="timereported"
> > dataformat="rfc3339" format="jsonf")
> > property(outname="message" name="msg" format="jsonf") }
> >
> > -----Original Message-----
> > From: Rainer Gerhards <rgerha...@hq.adiscon.com>
> > Sent: 18 September 2023 15:26
> > To: rsyslog-users <rsyslog@lists.adiscon.com>
> > Cc: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
> > Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with
> > escaped quotations and additional quotations
> >
> > ----------------------------- PHISHING ALERT
> > ----------------------------- This email has been sent from an account
> > outside of the BAE Systems network.
> >
> > Please treat the email with caution, especially if you are requested to
> > click on a link or open an attachment.
> > For further information on how to spot and report a phishing email please
> > access the Global Intranet, then select <Functions> / <IT>.
> >
> > --------------------------------------------------------------------
> > ----------------
> >
> > Does this example from the rsyslog testbench help?
> >
> > https://github.com/rsyslog/rsyslog/blob/761cb2bc51e3046b242b45994cff
> > 11ff8be3990e/tests/json-nonstring.sh#L4
> >
> > Rainer
> >
> > El lun, 18 sept 2023 a las 15:10, Lennon, Sean (UK) via rsyslog
> > (<rsyslog@lists.adiscon.com>) escribió:
> > >
> > >
> > >
> > >
> > >
> > > This email may contain proprietary information of BAE Systems and/or
> > > third parties.
> > >
> > > This is the one I meant.
> > >
> > > -----Original Message-----
> > > From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of
> > > Lennon, Sean (UK) via rsyslog
> > > Sent: 29 August 2023 17:39
> > > To: rsyslog@lists.adiscon.com
> > > Cc: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
> > > Subject: [rsyslog] rsyslog mmextenal logstash json output with
> > > escaped quotations and additional quotations
> > >
> > > ----------------------------- PHISHING ALERT
> > > ----------------------------- This email has been sent from an account
> > > outside of the BAE Systems network.
> > >
> > > Please treat the email with caution, especially if you are requested to
> > > click on a link or open an attachment.
> > > For further information on how to spot and report a phishing email please
> > > access the Global Intranet, then select <Functions> / <IT>.
> > >
> > > ------------------------------------------------------------------
> > > ----
> > > --------------
> > >
> > > This email may contain proprietary information of BAE Systems and/or
> > > third parties.
> > >
> > > Hi all,
> > >
> > > I've encountered an issue with formatting json output to logstash. I'm
> > > using mmexternal to reformat data received from a remote system, the data
> > > is project specific and needs to be massaged into json for use with
> > > logstash. The intention is to create a json message for logstash with
> > > the mmexternal output being part of that message. I'm able to receive
> > > this json output at logstash but the message field (which contains the
> > > mmexternal output) is encapsulated within double quotes and all json
> > > fields within have escaped double quotes. This means that logstash is
> > > not able to interpret part of the message. If I take the raw output of
> > > the mmextenal code and send it to a omfile then it looks perfectly fine.
> > >
> > > I have asked a more detailed question, on Stackoverflow:
> > > https://stackoverflow.com/questions/77001549/rsyslog-mmextenal-log
> > > stas h-json-output-with-escaped-quotations-and-additional-qu
> > >
> > > What am I missing?
> > >
> > > I appreciate your help.
> > >
> > > Sean
> > >
> > > ******************************************************************
> > > ** This email and any attachments are confidential to the intended
> > > recipient and may also be privileged. If you are not the intended
> > > recipient please delete it from your system and notify the sender.
> > > You should not copy it or use it for any purpose nor disclose or
> > > distribute its contents to any other person.
> > > ******************************************************************
> > > **
> > >
> > > BAE Systems may process information about you that may be subject
> > > to data protection laws. For more information about how we use
> > > your personal information, how we protect your information, our
> > > legal basis for using your information, your rights and who you
> > > can contact, please refer to our Privacy Notice at
> > > www.baesystems.com/en/privacy
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL:
> > > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites
> > > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
> > > THAT.
> > >
> > > _______________________________________________
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> > > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
> > > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T
> > > LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is
a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our
control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
Re: [rsyslog] rsyslog mmextenal logstash json output with escaped quotations and additional quotations
Lennon, Sean (UK) via rsyslog Wed, 20 Sep 2023 06:09:45 -0700
- Re: [rsyslog] rsyslog mmextenal logstas... Lennon, Sean (UK) via rsyslog
- Re: [rsyslog] rsyslog mmextenal logstas... David Lang via rsyslog
- Re: [rsyslog] rsyslog mmextenal logstas... Lennon, Sean (UK) via rsyslog
- Re: [rsyslog] rsyslog mmextenal logstas... David Lang via rsyslog
- Re: [rsyslog] rsyslog mmextenal logstash jso... Simon Lundström via rsyslog
- Re: [rsyslog] rsyslog mmextenal logstas... Simon Lundström via rsyslog
- Re: [rsyslog] rsyslog mmextenal logstash jso... Rainer Gerhards via rsyslog
- Re: [rsyslog] rsyslog mmextenal logstas... Lennon, Sean (UK) via rsyslog
- Re: [rsyslog] rsyslog mmextenal logstas... Simon Lundström via rsyslog
- Re: [rsyslog] rsyslog mmextenal logstas... Lennon, Sean (UK) via rsyslog
- Re: [rsyslog] rsyslog mmextenal logstas... Lennon, Sean (UK) via rsyslog
