Re: [rsyslog] rsyslog mmextenal logstash json output with escaped quotations and additional quotations

Tue, 19 Sep 2023 00:44:24 -0700 

 


This email may contain proprietary information of BAE Systems and/or third 
parties.
 
Sorry David, please accept my apologies.  I did not mean to come across as 
rude.  I have certain restrictions that prevent me from elaborating too much, 
hence my reluctance to accede to your request.  

What is it specifically that you wish to understand about the data?  The 
example you have is heavily redacted but is still in the form of both what I 
expect to see and what I am getting.  

Anyway, if it helps, this is the flow of data:
1. My rsyslog receives a message from another machine via RELP.  This message 
contains a bespoke encoded payload.
2. My rsyslog sends the message to some bespoke code (c++) I have that is 
called up by mmextenal.
3. The mmextenal loaded code will identify the payload and decode it and mangle 
into a json object.
4. The output of the mmexternal code is in the form of a string: std::cout << 
"{ \"msg\" : " << outputString << " }\n";
5. The outputString is a json object representing the decoded data.
6. This output is received back into rsyslog and is then put through the json 
template mentioned previously and onto logstash.

Now, you'll see step 4 already has the quotes and the escaping I don't want, 
but as I understood this is how rsyslog expects the message to be formatted.  
However, when viewing the debug out for mmexternal the json message looks fine. 
 What is it that I'm not doing correctly?


-----Original Message-----
From: David Lang <da...@lang.hm> 
Sent: 19 September 2023 08:16
To: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
Cc: David Lang <da...@lang.hm>; Lennon, Sean (UK) via rsyslog 
<rsyslog@lists.adiscon.com>; Rainer Gerhards <rgerha...@hq.adiscon.com>
Subject: RE: [rsyslog] rsyslog mmextenal logstash json output with escaped 
quotations and additional quotations

-----------------------------  PHISHING ALERT  ----------------------------- 
This email has been sent from an account outside of the BAE Systems network.

Please treat the email with caution, especially if you are requested to click 
on a link or open an attachment.
For further information on how to spot and report a phishing email please 
access the Global Intranet, then select <Functions> / <IT>.

------------------------------------------------------------------------------------

AS far as I can tell, you haven't provided any information about the user 
defined variable fields

but apparently you know better than I do what information I need, so I won't 
bother you further.

Good luck.

David Lang

On Tue, 19 Sep 2023, Lennon, Sean (UK) wrote:

> David, thanks for your response but providing additional fields will not give 
> you additional information other than additional fields.  I have provided the 
> form and fit of what is happening.  Therefore, I believe what I have provided 
> is sufficient for this discussion.
>
> -----Original Message-----
> From: David Lang <da...@lang.hm>
> Sent: 18 September 2023 20:29
> To: Lennon, Sean (UK) via rsyslog <rsyslog@lists.adiscon.com>
> Cc: Rainer Gerhards <rgerha...@hq.adiscon.com>; Lennon, Sean (UK) 
> <sean.lenn...@baesystems.com>
> Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with 
> escaped quotations and additional quotations
>
> -----------------------------  PHISHING ALERT  ----------------------------- 
> This email has been sent from an account outside of the BAE Systems network.
>
> Please treat the email with caution, especially if you are requested to click 
> on a link or open an attachment.
> For further information on how to spot and report a phishing email please 
> access the Global Intranet, then select <Functions> / <IT>.
>
> ----------------------------------------------------------------------
> --------------
>
> we need to see a lot more about what's created, your editing is hiding too 
> much.
>
> go ahead and mask out the contents, but we need to see all the values in the 
> debug output and their structure (i.e. any json significant characters), 
> change all the words/numbers to garbage if you want.
>
> Another option would be to contact Adiscon and setup a professional services 
> contract so that you can have a NDA rather than having to share the content 
> on a public mailing list for the community to help.
>
> David Lang
>
>  On Mon, 18 Sep 2023, Lennon, Sean (UK) via rsyslog wrote:
>
>> This email may contain proprietary information of BAE Systems and/or third 
>> parties.
>>
>> Sorry, but for ‘reasons’ I can only give you a severely edited version, I 
>> have used debug output from mmexternal first and the received message from 
>> logstash second:
>>
>>
>> 1.       mexternal debug output – I am satisfied with this.
>> { “msg” :
>> {“messageGroup”:[{“field1”:1,”field2”:2},{“field1”:3,”field2”:4}]}}
>>
>> 2.       what logstash receives
>> “message” => “{ \“msg\” : 
>> {\“messageGroup\”:[{\“field1\”:1,\”field2\”:2},{\“field1\”:3,\”field2\”:4}]}}”
>>
>>
>> From: Rainer Gerhards <rgerha...@hq.adiscon.com>
>> Sent: 18 September 2023 15:47
>> To: Lennon, Sean (UK) <sean.lenn...@baesystems.com>
>> Cc: rsyslog-users <rsyslog@lists.adiscon.com>
>> Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with 
>> escaped quotations and additional quotations
>>
>>
>> PHISHING ALERT
>> This email has been sent from an account outside of the BAE Systems network.
>>
>> Please treat the email with caution, especially if you are requested to 
>> click on a link or open an attachment.
>> For further information on how to spot and report a phishing email please 
>> access the Global Intranet then select <Functions> / <IT>.
>> If you think this is a phishing email, please report it by using the "Report 
>> Phishing" button in Outlook.
>>
>>
>> Output the message with RSYSLOG_DebugFormat template. I need to see which 
>> data msg actually has.
>>
>> Rainer
>> Sent from phone, thus brief.
>>
>> Lennon, Sean (UK) 
>> <sean.lenn...@baesystems.com<mailto:sean.lenn...@baesystems.com>> schrieb am 
>> Mo., 18. Sept. 2023, 16:41:
>>
>>
>>
>>
>> This email may contain proprietary information of BAE Systems and/or third 
>> parties.
>>
>> Thanks for your response Rainer.  I don't think it answers my question, I 
>> have property fields from the Rsyslog message that are fine, they get 
>> formatted correctly, for example 'timereported' or 'syslogseverity-text'.  
>> So, the output json for these and others are correct, it's the msg field 
>> that is returned from my custom code (using mmexternal) that is the problem.
>>
>> I have created a newer template that is more upto date and looks something 
>> similar to this:
>>
>> template(name="json-template" type="list" option.jsonf="on") {
>>        property(outname="@timestamp" name="timereported" 
>> dataformat="rfc3339" format="jsonf")
>>        property(outname="message" name="msg" format="jsonf") }
>>
>> -----Original Message-----
>> From: Rainer Gerhards
>> <rgerha...@hq.adiscon.com<mailto:rgerha...@hq.adiscon.com>>
>> Sent: 18 September 2023 15:26
>> To: rsyslog-users
>> <rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>
>> Cc: Lennon, Sean (UK)
>> <sean.lenn...@baesystems.com<mailto:sean.lenn...@baesystems.com>>
>> Subject: Re: [rsyslog] rsyslog mmextenal logstash json output with 
>> escaped quotations and additional quotations
>>
>> -----------------------------  PHISHING ALERT  ----------------------------- 
>> This email has been sent from an account outside of the BAE Systems network.
>>
>> Please treat the email with caution, especially if you are requested to 
>> click on a link or open an attachment.
>> For further information on how to spot and report a phishing email please 
>> access the Global Intranet, then select <Functions> / <IT>.
>>
>> ---------------------------------------------------------------------
>> -
>> --------------
>>
>> Does this example from the rsyslog testbench help?
>>
>> https://github.com/rsyslog/rsyslog/blob/761cb2bc51e3046b242b45994cff1
>> 1
>> ff8be3990e/tests/json-nonstring.sh#L4
>>
>> Rainer
>>
>> El lun, 18 sept 2023 a las 15:10, Lennon, Sean (UK) via rsyslog
>> (<rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>>) escribió:
>>>
>>>
>>>
>>>
>>>
>>> This email may contain proprietary information of BAE Systems and/or third 
>>> parties.
>>>
>>> This is the one I meant.
>>>
>>> -----Original Message-----
>>> From: rsyslog
>>> <rsyslog-boun...@lists.adiscon.com<mailto:rsyslog-boun...@lists.adis
>>> c on.com>> On Behalf Of Lennon, Sean (UK) via rsyslog
>>> Sent: 29 August 2023 17:39
>>> To: rsyslog@lists.adiscon.com<mailto:rsyslog@lists.adiscon.com>
>>> Cc: Lennon, Sean (UK)
>>> <sean.lenn...@baesystems.com<mailto:sean.lenn...@baesystems.com>>
>>> Subject: [rsyslog] rsyslog mmextenal logstash json output with 
>>> escaped quotations and additional quotations
>>>
>>> -----------------------------  PHISHING ALERT  
>>> ----------------------------- This email has been sent from an account 
>>> outside of the BAE Systems network.
>>>
>>> Please treat the email with caution, especially if you are requested to 
>>> click on a link or open an attachment.
>>> For further information on how to spot and report a phishing email please 
>>> access the Global Intranet, then select <Functions> / <IT>.
>>>
>>> --------------------------------------------------------------------
>>> -
>>> -
>>> --------------
>>>
>>> This email may contain proprietary information of BAE Systems and/or third 
>>> parties.
>>>
>>> Hi all,
>>>
>>> I've encountered an issue with formatting json output to logstash.  I'm 
>>> using mmexternal to reformat data received from a remote system, the data 
>>> is project specific and needs to be massaged into json for use with 
>>> logstash.  The intention is to create a json message for logstash with the 
>>> mmexternal output being part of that message.  I'm able to receive this 
>>> json output at logstash but the message field (which contains the 
>>> mmexternal output) is encapsulated within double quotes and all json fields 
>>> within have escaped double quotes.  This means that logstash is not able to 
>>> interpret part of the message.  If I take the raw output of the mmextenal 
>>> code and send it to a omfile then it looks perfectly fine.
>>>
>>> I have asked a more detailed question, on Stackoverflow:
>>> https://stackoverflow.com/questions/77001549/rsyslog-mmextenal-logst
>>> a s h-json-output-with-escaped-quotations-and-additional-qu
>>>
>>> What am I missing?
>>>
>>> I appreciate your help.
>>>
>>> Sean
>>>
>>> ********************************************************************
>>> This email and any attachments are confidential to the intended recipient 
>>> and may also be privileged. If you are not the intended recipient please 
>>> delete it from your system and notify the sender.
>>> You should not copy it or use it for any purpose nor disclose or distribute 
>>> its contents to any other person.
>>> ********************************************************************
>>>
>>> BAE Systems may process information about you that may be subject to 
>>> data protection laws. For more information about how we use your 
>>> personal information, how we protect your information, our legal 
>>> basis for using your information, your rights and who you can 
>>> contact, please refer to our Privacy Notice at 
>>> www.baesystems.com/en/privacy<http://www.baesystems.com/en/privacy>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: 
>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites 
>>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
>>> THAT.
>>>
>>> _______________________________________________
>>> rsyslog mailing list
>>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>>> http://www.rsyslog.com/professional-services/
>>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
>>> LIKE THAT.
>> _______________________________________________
>> rsyslog mailing list
>> https://lists.adiscon.net/mailman/listinfo/rsyslog
>> http://www.rsyslog.com/professional-services/
>> What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE
>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites 
>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
>> THAT.
>
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Reply via email to