Re: replication in HDFS

(sorry for the delay in replying) Hi Zheng You are right about HDFS RAID. It is used to save space, and is not involved in the file write path. The generation of parity blocks and reducing replication factor happens after a configurable amount of time. What is the design you have in mind? When t

Re: Review Request: DFSClient.getBlockLocations returns BlockLocations with no indication that the corresponding blocks are corrupt

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27/#review29 --- Looks good to me, but this diff depends on a hadoop-common change, right?

Re: Review Request: DFSClient.getBlockLocations returns BlockLocations with no indication that the corresponding blocks are corrupt

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27/#review27 --- http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/had

[jira] Resolved: (HDFS-1171) RaidNode should fix missing blocks directly on Data Node

[ https://issues.apache.org/jira/browse/HDFS-1171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Vadali resolved HDFS-1171. --- Resolution: Invalid Recreating this in Map/Reduce: MAPREDUCE-2150 > RaidNode should

[jira] Created: (HDFS-1472) Refactor DFSck to allow programmatic access to output

Components: tools Reporter: Ramkumar Vadali DFSck prints the list of corrupt files to stdout. This jira proposes that it write to a PrintStream object that is passed to the constructor. This will allow components like RAID to programmatically get a list of corrupt files. -- This message is

[jira] Created: (HDFS-1461) Refactor hdfs.server.datanode.BlockSender

Reporter: Ramkumar Vadali BlockSender provides the functionality to send a block to a data node. But the current implementation requires the source of the block to be a data node. The RAID contrib project needs the functionality of sending a block to a data node, but cannot use

[jira] Resolved: (HDFS-1453) Need a command line option in RaidShell to fix blocks using raid

[ https://issues.apache.org/jira/browse/HDFS-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramkumar Vadali resolved HDFS-1453. --- Resolution: Invalid RAID is a MR project, will reopen this under MR. > Need a command l

[jira] Created: (HDFS-1453) Need a command line option in RaidShell to fix blocks using raid

: Improvement Components: contrib/raid Reporter: Ramkumar Vadali RaidShell currently has an option to recover a file and return the path to the recovered file. The administrator can then rename the recovered file to the damaged file. The problem with this is that the file metadata is

[jira] Created: (HDFS-1257) Race condition introduced by HADOOP-5124

: Ramkumar Vadali HADOOP-5124 provided some improvements to FSNamesystem#recentInvalidateSets. But it introduced unprotected access to the data structure recentInvalidateSets. Specifically, FSNamesystem.computeInvalidateWork accesses recentInvalidateSets without read-lock protection. If there is

[jira] Created: (HDFS-1175) HAR files used for RAID parity need to have configurable partfile size

Type: Improvement Components: contrib/raid Affects Versions: 0.20.1 Reporter: Ramkumar Vadali Priority: Minor RAID parity files are merged into HAR archives periodically. This is required to reduce the number of files that the NameNode has to track. The number

[jira] Created: (HDFS-1171) RaidNode should fix missing blocks directly on Data Node

: contrib/raid Affects Versions: 0.20.1 Reporter: Ramkumar Vadali RaidNode currently does not fix missing blocks. The missing blocks have to be fixed manually. This task proposes that recovery be more automated: 1. RaidNode periodically fetches a list of corrupt files from the