I was thinking we provided the blockId, blockPoolId, path thats enough for
the client to get an ExtendedBlock and they can do the rest?
Agree on the snapshot operations for Inotify.
I'll create a jira ticket.
thanks for the feedback
rahul
On Tue, Jul 5, 2016 at 11:36 AM, Colin McCabe wrote:
>
I think it makes sense to have an AddBlockEvent. It seems like we could
provide something like the block ID, block pool ID, and genstamp, as
well as the inode ID and path of the file which the block was added to.
Clearly, we cannot provide the length, since we don't know how many
bytes the client
Hello Rahul,
I don't believe there is any fundamental difference in the handling of the
atime update for HDFS inotify. I'd expect you to receive an event with
EventType.METADATA and MetadataUpdateEvent.TIMES, and I wouldn't expect
that you'd need to save a new fsimage to trigger the notification.
Sorry for the late reply.Thanks ATM, Colin and others. Indeed applying the
patch took care of most of the memory issues. Now we are tackling the sudden
increase in the number of active threads. We will post more concrete problem
definition shortly.
Regards,Mohammad
On Tuesday, Septemb
Hi Mohammad,
Like ATM said, HDFS-8965 is an important fix in this area. We have
found that it prevents cases where INotify tries to read invalid
sequences of bytes (sometimes because the edit log was truncated or
corrupted; other times because it is in the middle of a write).
HDFS-8964 fixes the
Thanks Aaron and Ravi for stepping in.
We are working on that direction to go to the bottom. The recent fixes may take
care of this.
We will update if we find anything.
Regards,Mohammad
On Wednesday, September 16, 2015 12:59 PM, Aaron T. Myers
wrote:
Hey Mohammad,
Ravi's suggesti
Hey Mohammad,
Ravi's suggestion of getting a heap dump would almost certainly let you get
to the bottom of this, but I'll just throw out there that it sounds like
you may have been hitting https://issues.apache.org/jira/browse/HDFS-8965,
which indeed can be triggered by use of inotify. That's fixe
Hi Mohammad!
Thanks for reporting the issue. Could you please take a heap dump of the NN and
analyze it to see where the memory is being spent?
ThanksRavi
On Tuesday, September 15, 2015 11:53 AM, Mohammad Islam
wrote:
Hi,We were using INotify feature in one of our internal service.
any help or pointer?
On Tuesday, September 15, 2015 11:52 AM, Mohammad Islam
wrote:
Hi,We were using INotify feature in one of our internal service. Looks like it
creates a lot of memory pressure on NN. Memory usage goes very high and remains
the same causing expensive GC.
Did an
