Re: [rsyslog] rsyslog mmextenal logstash json output with escaped quotations and additional quotations

Wed, 20 Sep 2023 04:22:23 -0700 

 


This email may contain proprietary information of BAE Systems and/or third 
parties.
 
Rainer/David/Simon,

Thanks for your help, I have done what you have requested/suggested:

The following are redacted versions of the shortest message I have.  I've used 
David's suggestion of using sed to process the messages to remove sensitive 
data, I've opted to change numbers to 0.  I have left some inconsequential 
fields un-redacted for clarity.  These messages were captured using tcpdump as 
suggested by Simon, therefore they are what rsyslog puts out to logstash.

The first message is what I've settled on for the time being, i.e. logstash 
will accept it and we can get logstash to process it to remove the quotes 
around the value for the message field and also remove the escape characters.  
It's a bit of a fudge and not ideal, but it 'works'.
{
    "@timestamp":"2023-09-20T11:06:03.571548+01:00",
    "@version":"1",
    "sysloghost":"XXXXXXXXXXXXXXX",
    "severity":"info",
    "facility":"local6",
    "message":"{ 
        \"XXXX\": [
            {
                \"XXXXXXXXXXX\":0,
                \"XXXXXXX\":0,
                \"XXXXXXXXXXXXX\": {
                    \"XXXXXX\": \"XXXXXXXXXXXXXXXX\",
                    \"XXXXXXXXXXXXX\": {
                        \"XXXX_XXXXXX\": \"XXXX_XXXXXXX\",
                        \"XXXX_XXXXXX_XXXX\": 0
                    }
                }
            }
        ]
    }"
}

The second message is produced after removing format="jsonf" from the message 
field in the template.  Note that the escaped characters have gone but the 
quotes around the message field value have not.  This causes logstash to barf, 
hence the previous message is the one being used.
{
    "@timestamp":"2023-09-20T11:16:14.159361+01:00",
    "@version":"1",
    "sysloghost":"XXXXXXXXXXXXXXX",
    "severity":"info",
    "facility":"local6",
    "message":"{ 
        "XXXX": [
            {
                "XXXXXXXXXXX":0,
                "XXXXXXX":0,
                "XXXXXXXXXXXXX": {
                    "XXXXXX": "XXXXXXXXXXXXXXXX",
                    "XXXXXXXXXXXXX": {
                        "XXXX_XXXXXX": "XXXX_XXXXXXX",
                        "XXXX_XXXXXX_XXXX": 0
                    }
                }
            }
        ]
    }"
}

So, how do I get rid of the quotes around the message value?

Cheers,

Sean.

-----Original Message-----
From: Rainer Gerhards <rgerha...@hq.adiscon.com> 
Sent: 19 September 2023 08:51
To: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
Cc: rsyslog-users <rsyslog@lists.adiscon.com>
Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with escaped 
quotations and additional quotations

-----------------------------  PHISHING ALERT  -----------------------------
This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click 
on a link or open an attachment.
For further information on how to spot and report a phishing email please 
access the Global Intranet, then select <Functions> / <IT>.

------------------------------------------------------------------------------------

It's not easy to guess what rsyslog really sees, but I have one final
shot. As it looks, msg might already contain json. In that case, do
not use any special json formatting option. These options exist to
ensure non-json data (or json data as an inner encapsulation layer)
will be received as-is.

Rainer

El lun, 18 sept 2023 a las 17:04, Lennon, Sean (UK)
(<sean.lenn...@baesystems.com>) escribió:
>
> This email may contain proprietary information of BAE Systems and/or third 
> parties.
>
>
>
> Sorry, but for ‘reasons’ I can only give you a severely edited version, I 
> have used debug output from mmexternal first and the received message from 
> logstash second:
>
>
>
> 1.       mexternal debug output – I am satisfied with this.
> { “msg” : {“messageGroup”:[{“field1”:1,”field2”:2},{“field1”:3,”field2”:4}]}}
>
> 2.       what logstash receives
> “message” => “{ \“msg\” : 
> {\“messageGroup\”:[{\“field1\”:1,\”field2\”:2},{\“field1\”:3,\”field2\”:4}]}}”
>
>
>
>
>
> From: Rainer Gerhards <rgerha...@hq.adiscon.com>
> Sent: 18 September 2023 15:47
> To: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
> Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with escaped 
> quotations and additional quotations
>
>
>
>
> PHISHING ALERT
>
> This email has been sent from an account outside of the BAE Systems network.
>
> Please treat the email with caution, especially if you are requested to click 
> on a link or open an attachment.
> For further information on how to spot and report a phishing email please 
> access the Global Intranet then select <Functions> / <IT>.
> If you think this is a phishing email, please report it by using the "Report 
> Phishing" button in Outlook.
>
>
>
>
>
> Output the message with RSYSLOG_DebugFormat template. I need to see which 
> data msg actually has.
>
>
>
> Rainer
>
> Sent from phone, thus brief.
>
>
>
> Lennon, Sean (UK) <sean.lenn...@baesystems.com> schrieb am Mo., 18. Sept. 
> 2023, 16:41:
>
>
>
>
>
> This email may contain proprietary information of BAE Systems and/or third 
> parties.
>
> Thanks for your response Rainer.  I don't think it answers my question, I 
> have property fields from the Rsyslog message that are fine, they get 
> formatted correctly, for example 'timereported' or 'syslogseverity-text'.  
> So, the output json for these and others are correct, it's the msg field that 
> is returned from my custom code (using mmexternal) that is the problem.
>
> I have created a newer template that is more upto date and looks something 
> similar to this:
>
> template(name="json-template" type="list" option.jsonf="on") {
>         property(outname="@timestamp" name="timereported" 
> dataformat="rfc3339" format="jsonf")
>         property(outname="message" name="msg" format="jsonf")
> }
>
> -----Original Message-----
> From: Rainer Gerhards <rgerha...@hq.adiscon.com>
> Sent: 18 September 2023 15:26
> To: rsyslog-users <rsyslog@lists.adiscon.com>
> Cc: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
> Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with escaped 
> quotations and additional quotations
>
> -----------------------------  PHISHING ALERT  ----------------------------- 
> This email has been sent from an account outside of the BAE Systems network.
>
> Please treat the email with caution, especially if you are requested to click 
> on a link or open an attachment.
> For further information on how to spot and report a phishing email please 
> access the Global Intranet, then select <Functions> / <IT>.
>
> ------------------------------------------------------------------------------------
>
> Does this example from the rsyslog testbench help?
>
> https://github.com/rsyslog/rsyslog/blob/761cb2bc51e3046b242b45994cff11ff8be3990e/tests/json-nonstring.sh#L4
>
> Rainer
>
> El lun, 18 sept 2023 a las 15:10, Lennon, Sean (UK) via rsyslog
> (<rsyslog@lists.adiscon.com>) escribió:
> >
> >
> >
> >
> >
> > This email may contain proprietary information of BAE Systems and/or third 
> > parties.
> >
> > This is the one I meant.
> >
> > -----Original Message-----
> > From: rsyslog <rsyslog-boun...@lists.adiscon.com> On Behalf Of Lennon,
> > Sean (UK) via rsyslog
> > Sent: 29 August 2023 17:39
> > To: rsyslog@lists.adiscon.com
> > Cc: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
> > Subject: [rsyslog] rsyslog mmextenal logstash json output with escaped
> > quotations and additional quotations
> >
> > -----------------------------  PHISHING ALERT  
> > ----------------------------- This email has been sent from an account 
> > outside of the BAE Systems network.
> >
> > Please treat the email with caution, especially if you are requested to 
> > click on a link or open an attachment.
> > For further information on how to spot and report a phishing email please 
> > access the Global Intranet, then select <Functions> / <IT>.
> >
> > ----------------------------------------------------------------------
> > --------------
> >
> > This email may contain proprietary information of BAE Systems and/or third 
> > parties.
> >
> > Hi all,
> >
> > I've encountered an issue with formatting json output to logstash.  I'm 
> > using mmexternal to reformat data received from a remote system, the data 
> > is project specific and needs to be massaged into json for use with 
> > logstash.  The intention is to create a json message for logstash with the 
> > mmexternal output being part of that message.  I'm able to receive this 
> > json output at logstash but the message field (which contains the 
> > mmexternal output) is encapsulated within double quotes and all json fields 
> > within have escaped double quotes.  This means that logstash is not able to 
> > interpret part of the message.  If I take the raw output of the mmextenal 
> > code and send it to a omfile then it looks perfectly fine.
> >
> > I have asked a more detailed question, on Stackoverflow:
> > https://stackoverflow.com/questions/77001549/rsyslog-mmextenal-logstas
> > h-json-output-with-escaped-quotations-and-additional-qu
> >
> > What am I missing?
> >
> > I appreciate your help.
> >
> > Sean
> >
> > ********************************************************************
> > This email and any attachments are confidential to the intended recipient 
> > and may also be privileged. If you are not the intended recipient please 
> > delete it from your system and notify the sender.
> > You should not copy it or use it for any purpose nor disclose or distribute 
> > its contents to any other person.
> > ********************************************************************
> >
> > BAE Systems may process information about you that may be subject to
> > data protection laws. For more information about how we use your
> > personal information, how we protect your information, our legal basis
> > for using your information, your rights and who you can contact,
> > please refer to our Privacy Notice at www.baesystems.com/en/privacy
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: 
> > This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites 
> > beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
> > THAT.
> >
> > _______________________________________________
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
> > WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> > LIKE THAT.
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to