On 9/5/19 11:13 AM, Daan Hoogland wrote:
> Wido, makes sense that log4j and logrotate would conflict. Log4j has its
> own rotate functionality.
It does indeed :-)
> I didn't understand before but the binary data is always the beginning of
> the file? Is always nulls?
I found this out yesterday, it's always the beginning of the file. Not
sure what it is though.
On two servers I removed /etc/logrotate.d/cloudstack-agent and I'll
report back in a few days.
Wido
>
> Op do 5 sep. 2019 09:18 schreef Wei ZHOU <ustcweiz...@gmail.com>:
>
>> Hi Wido,
>>
>> I saw this issue in a 4.11.3 platform today.
>>
>> It seems to be caused by file
>>
>> https://github.com/apache/cloudstack/blob/master/agent/conf/cloudstack-agent.logrotate.in
>>
>> Maybe the file /etc/logrotate.d/cloudstack-agent is not needed ( in Ubuntu
>> ?).
>>
>> -Wei
>>
>>
>>
>> On Thu, 5 Sep 2019 at 08:08, Wido den Hollander <w...@widodh.nl> wrote:
>>
>>>
>>>
>>> On 9/3/19 2:55 PM, Daan Hoogland wrote:
>>>> On Tue, Sep 3, 2019 at 2:22 PM Wido den Hollander <w...@widodh.nl>
>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On 9/3/19 9:57 AM, Daan Hoogland wrote:
>>>>>> Can you find/look at the line before in the log. It is probably the
>> one
>>>>>> containing the hindering data. Or otherwise it *might* be a clue
>> where
>>> in
>>>>>> the flow it happens.
>>>>>>
>>>>>
>>>>> Do you have any idea what the easiest way might be?
>>>>>
>>>> In short, no, but.. in totally skewed order of alleged quickness
>>>> 1: I'm afraid it is a line by line analysis. If you are lucky the line
>>>> start with date and class name is not obscured and there is a newline
>>>> entered in the log. That would give you the exact location. when in bad
>>>> luck the binary data contains control chars that erase to beginning of
>>> line
>>>> or so and the best you can find is the previous and next lines.
>>>> 3: code analysis; My first thought is that some binary data is read
>> from
>>> a
>>>> file or a socket and logged as is. but starting this with code analysis
>>> is
>>>> not the quickest way to get to the culprit I imagine.
>>>> 2: Other than line by line log analysis and if you can reproduce the
>>> issue,
>>>> you could try zooming in by playing around with log levels for
>> different
>>>> classes.
>>>>
>>>
>>> I checked, but on all hypervisors I checked the "agent.log" contains
>>> binary data, but all the rotated (.gz) files do not.
>>>
>>> The binary data is at the beginning of the file(s). So it seems like it
>>> is logrotate and maybe a combination of log4j causing this?
>>>
>>> I'm not sure yet, but that seems to be the case.
>>>
>>> Wido
>>>
>>>>
>>>>>
>>>>> I checked agent.log.1.gz and after I decompress is that file is not
>>>>> binary, text only.
>>>>>
>>>> so much for reproducibilty. Hope it was not a buffer overflow hack
>>> attempt.
>>>>
>>>>
>>>>>
>>>>> The binary data only seems to be present in the current file.
>>>>>
>>>>> Wido
>>>>>
>>>>>> On Tue, Sep 3, 2019 at 8:22 AM Wido den Hollander <w...@widodh.nl>
>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 9/2/19 10:18 PM, Wei ZHOU wrote:
>>>>>>>> Hi Wido,
>>>>>>>>
>>>>>>>> I had similar issue (not agent.log). It is probably caused by one
>> or
>>>>>>> few> lines with special characters.
>>>>>>>
>>>>>>> And I'm trying to figure out what causes it :-)
>>>>>>>
>>>>>>>> "grep -a" should work.
>>>>>>>
>>>>>>> I know, but other tools which analyze the logfile might also have
>>>>>>> trouble reading the file due to binary characters.
>>>>>>>
>>>>>>> Wido
>>>>>>>
>>>>>>>>
>>>>>>>> -Wei
>>>>>>>>
>>>>>>>> On Mon, 2 Sep 2019 at 19:35, Wido den Hollander <w...@widodh.nl>
>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I've seen this on multiple occasions with Ubuntu 18.04 (and maybe
>>>>>>>>> 16.04?) hypervisors where according to 'grep' the agent.log is a
>>>>> binary
>>>>>>>>> file:
>>>>>>>>>
>>>>>>>>> root@n06:~# grep security_group
>> /var/log/cloudstack/agent/agent.log
>>>>>>>>> Binary file /var/log/cloudstack/agent/agent.log matches
>>>>>>>>> root@n06:~#
>>>>>>>>>
>>>>>>>>> If I open the file with 'less' I indeed see binary data.
>>>>>>>>>
>>>>>>>>> Tailing the file works just fine.
>>>>>>>>>
>>>>>>>>> Does anybody know where this is coming from? What in the
>> CloudStack
>>>>>>>>> agent causes it to write binary data to the logfile?
>>>>>>>>>
>>>>>>>>> Wido
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>
