On Thu, Jun 4, 2015 at 2:46 PM, Rahul Shrivastava <rhshr...@gmail.com> wrote: > Hi, > > > Suppose I write a Java client to create a directory on HDFS. Is there a way > to tag this request and get the tagged information on NameNode via > DFSInotifyEventInputStream or otherwise ? > > In short, is there a way to give optional information attached to every > request ? > > I do not want to save this information in File System but just to use this > information via DFSInotifyEventInputStream to relate it back to the client. > > The requirement is attach a UUID with every request on the client and > process this UUID from NameNode. > > thanks > Rahul
Hi Rahul, By definition, inotify reports on things which are "saved" i.e. persisted somehow on the NameNode. If the information were not persisted, we'd run into trouble using inotify in scenarios like shutting down the NameNode and bringing it back up, or failing over between namenodes. Requests do have UUIDs in the form of RPC call IDs. It would be rather difficult to expose this information through the Java API, though. I'm not sure how we would allow the calling code for mkdirs (for example) to get the call ID of the RPC it made. Certainly we don't want to change the return signature of mkdirs, and using thread-local data seems kind of ugly. I'm also not sure if we want to make the call ID part of the inotify API since it's an implementation detail. We had a discussion at Hadoop Summit about ways to use inotify effectively. One thing that came up is that we should add support for determining that an HDFS snapshot had occurred to inotify. This would allow users to do things like: 1. start listening for events 2. take a snapshot named Foo_2015-06-15_13:09 3. get a SnapshotCreateEvent for Foo_2015-06-15_13:09 4. do a full scan on the snapshot and buffer that information somewhere 5. ask for all events that happened since the SnapshotCreateEvent and apply them to the buffered information At the end of #5, you have a fully consistent view of the filesystem. I'm not sure if this is your exact usecase, but perhaps it's helpful. best, Colin
