Sorry for the late reply.Thanks ATM, Colin and others. Indeed applying the
patch took care of most of the memory issues. Now we are tackling the sudden
increase in the number of active threads. We will post more concrete problem
definition shortly.
Regards,Mohammad
On Tuesday, Septemb
Hi Mohammad,
Like ATM said, HDFS-8965 is an important fix in this area. We have
found that it prevents cases where INotify tries to read invalid
sequences of bytes (sometimes because the edit log was truncated or
corrupted; other times because it is in the middle of a write).
HDFS-8964 fixes the
Thanks Aaron and Ravi for stepping in.
We are working on that direction to go to the bottom. The recent fixes may take
care of this.
We will update if we find anything.
Regards,Mohammad
On Wednesday, September 16, 2015 12:59 PM, Aaron T. Myers
wrote:
Hey Mohammad,
Ravi's suggesti
Hey Mohammad,
Ravi's suggestion of getting a heap dump would almost certainly let you get
to the bottom of this, but I'll just throw out there that it sounds like
you may have been hitting https://issues.apache.org/jira/browse/HDFS-8965,
which indeed can be triggered by use of inotify. That's fixe
Hi Mohammad!
Thanks for reporting the issue. Could you please take a heap dump of the NN and
analyze it to see where the memory is being spent?
ThanksRavi
On Tuesday, September 15, 2015 11:53 AM, Mohammad Islam
wrote:
Hi,We were using INotify feature in one of our internal service.
any help or pointer?
On Tuesday, September 15, 2015 11:52 AM, Mohammad Islam
wrote:
Hi,We were using INotify feature in one of our internal service. Looks like it
creates a lot of memory pressure on NN. Memory usage goes very high and remains
the same causing expensive GC.
Did an
