Re: Getting a list of all StorageInfo

[Youssef Hatem]

Hi Chris,

thanks a lot for taking the time answering my question. Skimming through
BlockPlacementPolicyDefault helped my a lot; I managed to get
DatanodeDescriptor(s) by using Host2NodesMap object. The DatanodeDescriptor
contains storage info which I was looking for.

Thanks again for your help, I really appreciate it.

Best regards,
Youssef

On Wed, Jan 7, 2015 at 10:21 PM, Chris Nauroth 
wrote:

> Hi Youssef,
>
> You might want to review the code for BlockPlacementPolicyDefault to get a
> sense for how an implementation works.  Perhaps you could even start your
> own work by copying and modifying that class.  This code would show you how
> to get ahold of a DatanodeDescriptor and then use it to call
> getStorageInfos.
>
> Hope this helps.
>
> Chris Nauroth
> Hortonworks
> http://hortonworks.com/
>
>
> On Fri, Jan 2, 2015 at 10:12 AM, Youssef Hatem 
> wrote:
>
> > Hi,
> >
> > I am working on implementing a custom placement policy. So I created a
> > class that extends BlockPlacementPolicy. I am trying to get a list of all
> > DatanodeStorageInfo objects in order to be able to decide which one to
> > choose for placement (based on certain criteria), this is the logic of
> > chooseTarget method.
> >
> > Could you please tell me how can I get this list? I tried to do the
> > following but I got none of these objects:
> >
> > DistributedFileSystem fs = (DistributedFileSystem)FileSystem.get(conf);
> > //conf is passed in initialize method
> > List storageInfos = new ArrayList<>();
> > for(DatanodeInfo node :
> > fs.getDataNodeStats(HdfsConstants.DatanodeReportType.LIVE)){
> >   if(node instanceof DatanodeDescriptor){
> >
> >
> >
> storageInfos.addAll(Arrays.asList(((DatanodeDescriptor)node).getStorageInfo()));
> >   }
> >
> > It seems though that none of the resulting DatanodeInfo objects are
> > actually a DatanodeDescriptor therefor the storageInfos list is always
> > empty.
> >
> > It would be great if you could provide me with more information on how to
> > get a list of DatanodeStorageInfo objects.
> >
> > Thanks a lot in advance,
> > Youssef
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

[jira] [Created] (HDFS-7593) Data Not replicate in ssd drive

[ViVek Raghuwanshi (JIRA)]

ViVek Raghuwanshi created HDFS-7593:
---

 Summary: Data Not replicate in ssd drive
 Key: HDFS-7593
 URL: https://issues.apache.org/jira/browse/HDFS-7593
 Project: Hadoop HDFS
  Issue Type: Test
  Components: benchmarks
Affects Versions: 2.6.0
Reporter: ViVek Raghuwanshi
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-7594) Add isFileClosed and IsInSafeMode APIs in o.a.h.hdfs.client.HdfsAdmin

[Uma Maheswara Rao G (JIRA)]

Uma Maheswara Rao G created HDFS-7594:
-

 Summary: Add isFileClosed and IsInSafeMode APIs in 
o.a.h.hdfs.client.HdfsAdmin
 Key: HDFS-7594
 URL: https://issues.apache.org/jira/browse/HDFS-7594
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs-client
Affects Versions: 2.6.0, 3.0.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G


DistributedFileSystem has exposed isFileClosed and IsInSafeMode.
Applications like Hbase would use this apis directly from DistributedFileSystem 
class. Now HdfsAdmin public class added and one of the purpose is for exposing 
the apis from DistributedFileSystem/DfsClient for the apps.
So, It would be nice to add this 2 apis also in that public class.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Hadoop-Hdfs-trunk #1999

[Apache Jenkins Server]

See 

[jira] [Created] (HDFS-7595) Remove hftp

[Allen Wittenauer (JIRA)]

Allen Wittenauer created HDFS-7595:
--

 Summary: Remove hftp
 Key: HDFS-7595
 URL: https://issues.apache.org/jira/browse/HDFS-7595
 Project: Hadoop HDFS
  Issue Type: Improvement
Affects Versions: 3.0.0
Reporter: Allen Wittenauer


We should just remove hftp from the 3.x series in order to reduce the code 
footprint.  WebHDFS has effectively replaced it and will have been in a 
supported, stable release for years by the time 3.x ships.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: HDFS 2.6.0 upgrade ends with missing blocks

[Colin P. McCabe]

Hi dlmarion,

In general, any upgrade process we do will consume disk space, because
it's creating hardlinks and a new "current" directory, and so forth.
So upgrading when disk space is very low is a bad idea in any
scenario.  It's certainly a good idea to free up some space before
doing the upgrade.  We should put a note about this somewhere.  We
should also add this check into various Hadoop management software
packages.

Unfortunately, I can't tell you how much space you need exactly to do
an upgrade.  As you guessed, it depends on the local UNIX filesystem
you are using.  It also depends on how you formatted the filesystem.
If you are using ext4, this parameter is controlled by the
bytes-per-inode of the filesystem, multiplied by the number of inodes.
ext4 may also run out of inodes even though there is free space on the
disk (but this is very unlikely when using HDFS with reasonably large
block sizes.)  What I would say is that you need to have at least as
many free inodes as blocks, and also a few kilobytes of space for
writing some new VERSION files and other metadata files.  In general,
I would say leave a generous number of megabytes free to avoid
problems with things like the block scanner filling up your disk
unexpectedly, or the rebalancer filling up your disk during an MR job
or other job.

You mentioned that the DataNode continued to start even though some of
the volumes couldn't be upgraded, due to
dfs.datanode.failed.volumes.tolerated being non-zero.  That is
definitely unfortunate.  Maybe we could add another configuration
named dfs.datanode.failed.upgrade.volumes.tolerated, that took effect
when upgrading.  That way you could specify that any failures were not
acceptable during an upgrade... i.e., they caused the upgrade to fail.
Would this be helpful?  The main downside that I can see for this--
and it is a big one-- is that if there are any existing failed drives,
then the upgrade would not succeed.  I think this would be scary for a
lot of administrators.

best,
Colin
Cloudera


On Wed, Jan 7, 2015 at 5:25 AM,   wrote:
>
> I recently upgraded from CDH 5.1.2 to CDH 5.3.0. I know, contact Cloudera, 
> but this is actually a generic issue. After the upgrade I brought up the DNs 
> and after all of them had checked in I ended up with missing blocks. I 
> tracked this down in the DN logs to an error at startup where the DN is 
> failing to create subdirectories. This happens at 
> BlockPoolSliceStorage.doUpgrade(). It appears that the directory structure 
> has changed with HDFS-6482 and the DN is pre-creating all of the directories 
> at DN startup time. If the disk is near full, then it fails to create the 
> subdirectories because it consumes the remaining space. If the hdfs 
> configuration allows failed drives (dfs.datanode.failed.volumes.tolerated > 
> 0), then the DN will start without the now full disk and report all of the 
> blocks except the ones on the full disk.
>
> I didn't find any type of warning in the Apache release notes. It might be 
> useful for people in a similar situation. For the Cloudera folks on this 
> list, there is no warning or note in your upgrade instructions that I could 
> find either.
>
> Some questions:
>
> 1. How much free space is needed per disk to pre-create the directory 
> structure. Is it dependent on the type of filesystem? I calculated 256MB 
> given my reading of the ticket, but I may have misunderstood something.
>
> 2. Now that block locations are calculated using the block id, are there 
> restrictions on where blocks can be placed? I assume that the location is not 
> verified on a read for backwards compatibility, if that is not true, then 
> someone needs to comment on HDFS-1312 that the older utilities cannot be 
> used. I need to move blocks from the full disks to other locations, I'm 
> looking for any restrictions in doing that.

Re: HDFS 2.6.0 upgrade ends with missing blocks

[dlmarion]

Colin, 

Thanks for the response, understanding the details is important and I think 
some general guidelines would be great. Since my initial email the system 
administrators told me that the drives are not actually full; the filesystems 
by default keep 5% in reserve. We can lower the reserve by 1%, which frees up 
about 30GB on a 3TB drive. We have been able to do just that, restart the DNs, 
and then everything reported in. At this point I can run the balancer and try 
to even things out across the DNs. Running the balancer before the upgrade was 
not really an option since in CDH 5.1.2 it did not work due to HDFS-6621. As a 
side note, I did apply the fix from HDFS-6621 and the balancer still didn't 
work (or I did someting wrong). 

I think adding some properties for the upgrade would be a good idea. I would 
rather see an upgrade fail due to "not enough space" than have an upgrade 
appear to succeed and then end up with missing blocks. From a user perspective, 
the upgrade just failed in a bad way. Regarding existing failed drives, maybe 
the upgrade instructions should have the system administrators unmount them 
first. 

Dave Marion 

- Original Message -

From: "Colin P. McCabe"  
To: hdfs-dev@hadoop.apache.org 
Sent: Thursday, January 8, 2015 2:43:29 PM 
Subject: Re: HDFS 2.6.0 upgrade ends with missing blocks 

Hi dlmarion, 

In general, any upgrade process we do will consume disk space, because 
it's creating hardlinks and a new "current" directory, and so forth. 
So upgrading when disk space is very low is a bad idea in any 
scenario. It's certainly a good idea to free up some space before 
doing the upgrade. We should put a note about this somewhere. We 
should also add this check into various Hadoop management software 
packages. 

Unfortunately, I can't tell you how much space you need exactly to do 
an upgrade. As you guessed, it depends on the local UNIX filesystem 
you are using. It also depends on how you formatted the filesystem. 
If you are using ext4, this parameter is controlled by the 
bytes-per-inode of the filesystem, multiplied by the number of inodes. 
ext4 may also run out of inodes even though there is free space on the 
disk (but this is very unlikely when using HDFS with reasonably large 
block sizes.) What I would say is that you need to have at least as 
many free inodes as blocks, and also a few kilobytes of space for 
writing some new VERSION files and other metadata files. In general, 
I would say leave a generous number of megabytes free to avoid 
problems with things like the block scanner filling up your disk 
unexpectedly, or the rebalancer filling up your disk during an MR job 
or other job. 

You mentioned that the DataNode continued to start even though some of 
the volumes couldn't be upgraded, due to 
dfs.datanode.failed.volumes.tolerated being non-zero. That is 
definitely unfortunate. Maybe we could add another configuration 
named dfs.datanode.failed.upgrade.volumes.tolerated, that took effect 
when upgrading. That way you could specify that any failures were not 
acceptable during an upgrade... i.e., they caused the upgrade to fail. 
Would this be helpful? The main downside that I can see for this-- 
and it is a big one-- is that if there are any existing failed drives, 
then the upgrade would not succeed. I think this would be scary for a 
lot of administrators. 

best, 
Colin 
Cloudera 


On Wed, Jan 7, 2015 at 5:25 AM,  wrote: 
> 
> I recently upgraded from CDH 5.1.2 to CDH 5.3.0. I know, contact Cloudera, 
> but this is actually a generic issue. After the upgrade I brought up the DNs 
> and after all of them had checked in I ended up with missing blocks. I 
> tracked this down in the DN logs to an error at startup where the DN is 
> failing to create subdirectories. This happens at 
> BlockPoolSliceStorage.doUpgrade(). It appears that the directory structure 
> has changed with HDFS-6482 and the DN is pre-creating all of the directories 
> at DN startup time. If the disk is near full, then it fails to create the 
> subdirectories because it consumes the remaining space. If the hdfs 
> configuration allows failed drives (dfs.datanode.failed.volumes.tolerated > 
> 0), then the DN will start without the now full disk and report all of the 
> blocks except the ones on the full disk. 
> 
> I didn't find any type of warning in the Apache release notes. It might be 
> useful for people in a similar situation. For the Cloudera folks on this 
> list, there is no warning or note in your upgrade instructions that I could 
> find either. 
> 
> Some questions: 
> 
> 1. How much free space is needed per disk to pre-create the directory 
> structure. Is it dependent on the type of filesystem? I calculated 256MB 
> given my reading of the ticket, but I may have misunderstood something. 
> 
> 2. Now that block locations are calculated using the block id, are there 
> restrictions on where blocks can be placed? I assume that the location is not 
> verified on a

[jira] [Created] (HDFS-7596) NameNode should prune dead storages from storageMap

[Arpit Agarwal (JIRA)]

Arpit Agarwal created HDFS-7596:
---

 Summary: NameNode should prune dead storages from storageMap
 Key: HDFS-7596
 URL: https://issues.apache.org/jira/browse/HDFS-7596
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.6.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal


The NameNode must be able to prune storages that are no longer reported by the 
DataNode and that have no blocks associated. These stale storages can skew the 
balancer behavior.

Detailed discussion on HDFS-7575.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)