Hi Wido, I saw this issue in a 4.11.3 platform today.
It seems to be caused by file https://github.com/apache/cloudstack/blob/master/agent/conf/cloudstack-agent.logrotate.in Maybe the file /etc/logrotate.d/cloudstack-agent is not needed ( in Ubuntu ?). -Wei On Thu, 5 Sep 2019 at 08:08, Wido den Hollander <w...@widodh.nl> wrote: > > > On 9/3/19 2:55 PM, Daan Hoogland wrote: > > On Tue, Sep 3, 2019 at 2:22 PM Wido den Hollander <w...@widodh.nl> > wrote: > > > >> > >> > >> On 9/3/19 9:57 AM, Daan Hoogland wrote: > >>> Can you find/look at the line before in the log. It is probably the one > >>> containing the hindering data. Or otherwise it *might* be a clue where > in > >>> the flow it happens. > >>> > >> > >> Do you have any idea what the easiest way might be? > >> > > In short, no, but.. in totally skewed order of alleged quickness > > 1: I'm afraid it is a line by line analysis. If you are lucky the line > > start with date and class name is not obscured and there is a newline > > entered in the log. That would give you the exact location. when in bad > > luck the binary data contains control chars that erase to beginning of > line > > or so and the best you can find is the previous and next lines. > > 3: code analysis; My first thought is that some binary data is read from > a > > file or a socket and logged as is. but starting this with code analysis > is > > not the quickest way to get to the culprit I imagine. > > 2: Other than line by line log analysis and if you can reproduce the > issue, > > you could try zooming in by playing around with log levels for different > > classes. > > > > I checked, but on all hypervisors I checked the "agent.log" contains > binary data, but all the rotated (.gz) files do not. > > The binary data is at the beginning of the file(s). So it seems like it > is logrotate and maybe a combination of log4j causing this? > > I'm not sure yet, but that seems to be the case. > > Wido > > > > >> > >> I checked agent.log.1.gz and after I decompress is that file is not > >> binary, text only. > >> > > so much for reproducibilty. Hope it was not a buffer overflow hack > attempt. > > > > > >> > >> The binary data only seems to be present in the current file. > >> > >> Wido > >> > >>> On Tue, Sep 3, 2019 at 8:22 AM Wido den Hollander <w...@widodh.nl> > >> wrote: > >>> > >>>> > >>>> > >>>> On 9/2/19 10:18 PM, Wei ZHOU wrote: > >>>>> Hi Wido, > >>>>> > >>>>> I had similar issue (not agent.log). It is probably caused by one or > >>>> few> lines with special characters. > >>>> > >>>> And I'm trying to figure out what causes it :-) > >>>> > >>>>> "grep -a" should work. > >>>> > >>>> I know, but other tools which analyze the logfile might also have > >>>> trouble reading the file due to binary characters. > >>>> > >>>> Wido > >>>> > >>>>> > >>>>> -Wei > >>>>> > >>>>> On Mon, 2 Sep 2019 at 19:35, Wido den Hollander <w...@widodh.nl> > >> wrote: > >>>>> > >>>>>> Hi, > >>>>>> > >>>>>> I've seen this on multiple occasions with Ubuntu 18.04 (and maybe > >>>>>> 16.04?) hypervisors where according to 'grep' the agent.log is a > >> binary > >>>>>> file: > >>>>>> > >>>>>> root@n06:~# grep security_group /var/log/cloudstack/agent/agent.log > >>>>>> Binary file /var/log/cloudstack/agent/agent.log matches > >>>>>> root@n06:~# > >>>>>> > >>>>>> If I open the file with 'less' I indeed see binary data. > >>>>>> > >>>>>> Tailing the file works just fine. > >>>>>> > >>>>>> Does anybody know where this is coming from? What in the CloudStack > >>>>>> agent causes it to write binary data to the logfile? > >>>>>> > >>>>>> Wido > >>>>>> > >>>>> > >>>> > >>>> > >>>> > >>> > >> > > > > >
