Hi Chris, thanks a lot for taking the time answering my question. Skimming through BlockPlacementPolicyDefault helped my a lot; I managed to get DatanodeDescriptor(s) by using Host2NodesMap object. The DatanodeDescriptor contains storage info which I was looking for.
Thanks again for your help, I really appreciate it. Best regards, Youssef On Wed, Jan 7, 2015 at 10:21 PM, Chris Nauroth <cnaur...@hortonworks.com> wrote: > Hi Youssef, > > You might want to review the code for BlockPlacementPolicyDefault to get a > sense for how an implementation works. Perhaps you could even start your > own work by copying and modifying that class. This code would show you how > to get ahold of a DatanodeDescriptor and then use it to call > getStorageInfos. > > Hope this helps. > > Chris Nauroth > Hortonworks > http://hortonworks.com/ > > > On Fri, Jan 2, 2015 at 10:12 AM, Youssef Hatem <yousef.ha...@gmail.com> > wrote: > > > Hi, > > > > I am working on implementing a custom placement policy. So I created a > > class that extends BlockPlacementPolicy. I am trying to get a list of all > > DatanodeStorageInfo objects in order to be able to decide which one to > > choose for placement (based on certain criteria), this is the logic of > > chooseTarget method. > > > > Could you please tell me how can I get this list? I tried to do the > > following but I got none of these objects: > > > > DistributedFileSystem fs = (DistributedFileSystem)FileSystem.get(conf); > > //conf is passed in initialize method > > List<DatanodeStorageInfo> storageInfos = new ArrayList<>(); > > for(DatanodeInfo node : > > fs.getDataNodeStats(HdfsConstants.DatanodeReportType.LIVE)){ > > if(node instanceof DatanodeDescriptor){ > > > > > > > storageInfos.addAll(Arrays.asList(((DatanodeDescriptor)node).getStorageInfo())); > > } > > > > It seems though that none of the resulting DatanodeInfo objects are > > actually a DatanodeDescriptor therefor the storageInfos list is always > > empty. > > > > It would be great if you could provide me with more information on how to > > get a list of DatanodeStorageInfo objects. > > > > Thanks a lot in advance, > > Youssef > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. >
