Wido, makes sense that log4j and logrotate would conflict. Log4j has its own rotate functionality. I didn't understand before but the binary data is always the beginning of the file? Is always nulls?
Op do 5 sep. 2019 09:18 schreef Wei ZHOU <ustcweiz...@gmail.com>: > Hi Wido, > > I saw this issue in a 4.11.3 platform today. > > It seems to be caused by file > > https://github.com/apache/cloudstack/blob/master/agent/conf/cloudstack-agent.logrotate.in > > Maybe the file /etc/logrotate.d/cloudstack-agent is not needed ( in Ubuntu > ?). > > -Wei > > > > On Thu, 5 Sep 2019 at 08:08, Wido den Hollander <w...@widodh.nl> wrote: > > > > > > > On 9/3/19 2:55 PM, Daan Hoogland wrote: > > > On Tue, Sep 3, 2019 at 2:22 PM Wido den Hollander <w...@widodh.nl> > > wrote: > > > > > >> > > >> > > >> On 9/3/19 9:57 AM, Daan Hoogland wrote: > > >>> Can you find/look at the line before in the log. It is probably the > one > > >>> containing the hindering data. Or otherwise it *might* be a clue > where > > in > > >>> the flow it happens. > > >>> > > >> > > >> Do you have any idea what the easiest way might be? > > >> > > > In short, no, but.. in totally skewed order of alleged quickness > > > 1: I'm afraid it is a line by line analysis. If you are lucky the line > > > start with date and class name is not obscured and there is a newline > > > entered in the log. That would give you the exact location. when in bad > > > luck the binary data contains control chars that erase to beginning of > > line > > > or so and the best you can find is the previous and next lines. > > > 3: code analysis; My first thought is that some binary data is read > from > > a > > > file or a socket and logged as is. but starting this with code analysis > > is > > > not the quickest way to get to the culprit I imagine. > > > 2: Other than line by line log analysis and if you can reproduce the > > issue, > > > you could try zooming in by playing around with log levels for > different > > > classes. > > > > > > > I checked, but on all hypervisors I checked the "agent.log" contains > > binary data, but all the rotated (.gz) files do not. > > > > The binary data is at the beginning of the file(s). So it seems like it > > is logrotate and maybe a combination of log4j causing this? > > > > I'm not sure yet, but that seems to be the case. > > > > Wido > > > > > > > >> > > >> I checked agent.log.1.gz and after I decompress is that file is not > > >> binary, text only. > > >> > > > so much for reproducibilty. Hope it was not a buffer overflow hack > > attempt. > > > > > > > > >> > > >> The binary data only seems to be present in the current file. > > >> > > >> Wido > > >> > > >>> On Tue, Sep 3, 2019 at 8:22 AM Wido den Hollander <w...@widodh.nl> > > >> wrote: > > >>> > > >>>> > > >>>> > > >>>> On 9/2/19 10:18 PM, Wei ZHOU wrote: > > >>>>> Hi Wido, > > >>>>> > > >>>>> I had similar issue (not agent.log). It is probably caused by one > or > > >>>> few> lines with special characters. > > >>>> > > >>>> And I'm trying to figure out what causes it :-) > > >>>> > > >>>>> "grep -a" should work. > > >>>> > > >>>> I know, but other tools which analyze the logfile might also have > > >>>> trouble reading the file due to binary characters. > > >>>> > > >>>> Wido > > >>>> > > >>>>> > > >>>>> -Wei > > >>>>> > > >>>>> On Mon, 2 Sep 2019 at 19:35, Wido den Hollander <w...@widodh.nl> > > >> wrote: > > >>>>> > > >>>>>> Hi, > > >>>>>> > > >>>>>> I've seen this on multiple occasions with Ubuntu 18.04 (and maybe > > >>>>>> 16.04?) hypervisors where according to 'grep' the agent.log is a > > >> binary > > >>>>>> file: > > >>>>>> > > >>>>>> root@n06:~# grep security_group > /var/log/cloudstack/agent/agent.log > > >>>>>> Binary file /var/log/cloudstack/agent/agent.log matches > > >>>>>> root@n06:~# > > >>>>>> > > >>>>>> If I open the file with 'less' I indeed see binary data. > > >>>>>> > > >>>>>> Tailing the file works just fine. > > >>>>>> > > >>>>>> Does anybody know where this is coming from? What in the > CloudStack > > >>>>>> agent causes it to write binary data to the logfile? > > >>>>>> > > >>>>>> Wido > > >>>>>> > > >>>>> > > >>>> > > >>>> > > >>>> > > >>> > > >> > > > > > > > > >
