[jira] [Reopened] (HDFS-2515) start-all.sh namenode createSocketAddr
[ https://issues.apache.org/jira/browse/HDFS-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] 程国强 reopened HDFS-2515: --- still not work > start-all.sh namenode createSocketAddr > -- > > Key: HDFS-2515 > URL: https://issues.apache.org/jira/browse/HDFS-2515 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.20.205.0 > Environment: centos >Reporter: 程国强 > Original Estimate: 5h > Remaining Estimate: 5h > > 2011-10-28 10:52:00,083 ERROR > org.apache.hadoop.hdfs.server.namenode.NameNode: > java.lang.IllegalArgumentException: Does not contain a valid host:port > authority: file:/// > at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:184) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:198) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:228) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:262) > at org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:497) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1268) > at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1277) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-3307) when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some file whose file name has some special messy code(乱码)
[
https://issues.apache.org/jira/browse/HDFS-3307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yixiaohua reopened HDFS-3307:
-
> when save FSImage ，HDFS（ or SecondaryNameNode or FSImage）can't handle some
> file whose file name has some special messy code(乱码)
> -
>
> Key: HDFS-3307
> URL: https://issues.apache.org/jira/browse/HDFS-3307
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.20.1
> Environment: SUSE LINUX
>Reporter: yixiaohua
> Attachments: FSImage.java
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> this the log information of the exception from the SecondaryNameNode:
> 2012-03-28 00:48:42,553 ERROR
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
> java.io.IOException: Found lease for
> non-existent file
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/@???
> ??tor.qzone.qq.com/keypart-00174
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFilesUnderConstruction(FSImage.java:1211)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:959)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:589)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$000(SecondaryNameNode.java:473)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:350)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:314)
> at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:225)
> at java.lang.Thread.run(Thread.java:619)
> this is the log information about the file from namenode:
> 2012-03-28 00:32:26,528 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
> ip=/10.131.16.34cmd=create
> src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174 dst=null
> perm=boss:boss:rw-r--r--
> 2012-03-28 00:37:42,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
> NameSystem.allocateBlock:
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174.
> blk_2751836614265659170_184668759
> 2012-03-28 00:37:42,696 INFO org.apache.hadoop.hdfs.StateChange: DIR*
> NameSystem.completeFile: file
> /user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174 is closed by
> DFSClient_attempt_201203271849_0016_r_000174_0
> 2012-03-28 00:37:50,315 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=boss,boss
> ip=/10.131.16.34cmd=rename
> src=/user/boss/pgv/fission/task16/split/_temporary/_attempt_201203271849_0016_r_000174_0/
> @?tor.qzone.qq.com/keypart-00174
> dst=/user/boss/pgv/fission/task16/split/ @?
> tor.qzone.qq.com/keypart-00174 perm=boss:boss:rw-r--r--
> after check the code that save FSImage,I found there are a problem that maybe
> a bug of HDFS Code,I past below:
> -this is the saveFSImage method in FSImage.java, I make some
> mark at the problem code
> /**
>* Save the contents of the FS image to the file.
>*/
> void saveFSImage(File newFile) throws IOException {
> FSNamesystem fsNamesys = FSNamesystem.getFSNamesystem();
> FSDirectory fsDir = fsNamesys.dir;
> long startTime = FSNamesystem.now();
> //
> // Write out data
> //
> DataOutputStream out = new DataOutputStream(
> new BufferedOutputStream(
> new
> FileOutputStream(newFile)));
> try {
> .
>
> // save the rest of the nodes
> saveImage(strbuf, 0, fsDir.rootDir, out);--problem
> fsNamesys.saveFilesUnderConstruction(out);--problem
> detail is below
> strbuf = null;
> } finally {
> out.close();
> }
> LOG.info("Image file of size " + newFile.length() + " saved in "
> + (FSNamesystem.now() - sta
[jira] [Reopened] (HDFS-2012) Recurring failure of TestBalancer on branch-0.22
[ https://issues.apache.org/jira/browse/HDFS-2012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-2012: --- I am reopening it now as Jenkins just failed and we got a clear log for the issue. > Recurring failure of TestBalancer on branch-0.22 > > > Key: HDFS-2012 > URL: https://issues.apache.org/jira/browse/HDFS-2012 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer, test >Affects Versions: 0.22.0 >Reporter: Aaron T. Myers >Priority: Blocker > Fix For: 0.22.0 > > > This has been failing on Hudson for the last two builds and fails on my local > box as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2395) webhdfs api's should return a root element in the json response
[ https://issues.apache.org/jira/browse/HDFS-2395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reopened HDFS-2395: -- Bug was erroneously marked "Duplicate" instead of "Fixed". Fixing... > webhdfs api's should return a root element in the json response > --- > > Key: HDFS-2395 > URL: https://issues.apache.org/jira/browse/HDFS-2395 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.205.0 >Reporter: Arpit Gupta >Assignee: Tsz Wo (Nicholas), SZE >Priority: Critical > Fix For: 0.20.205.0, 0.24.0 > > Attachments: h2395_20111003.patch, h2395_20111004.patch, > h2395_20111004_0.20s.patch, h2395_20111004b.patch, h2395_20111004c.patch, > h2395_20111004c_0.20s.patch, h2395_20111004d.patch, > h2395_20111004d_0.20s.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2404) webhdfs liststatus json response is not correct
[ https://issues.apache.org/jira/browse/HDFS-2404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reopened HDFS-2404: -- Reopened for trunk fix. > webhdfs liststatus json response is not correct > --- > > Key: HDFS-2404 > URL: https://issues.apache.org/jira/browse/HDFS-2404 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.205.0, 0.23.0 >Reporter: Arpit Gupta >Assignee: Suresh Srinivas > Fix For: 0.20.205.0 > > Attachments: HDFS-2404.20.txt, HDFS-2404.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-208) name node should warn if only one dir is listed in dfs.name.dir
[ https://issues.apache.org/jira/browse/HDFS-208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reopened HDFS-208: -- Re-opening, seems like an easy, valuable improvement. > name node should warn if only one dir is listed in dfs.name.dir > --- > > Key: HDFS-208 > URL: https://issues.apache.org/jira/browse/HDFS-208 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Allen Wittenauer >Priority: Minor > Labels: newbie > Fix For: 0.24.0 > > > The name node should warn that corruption may occur if only one directory is > listed in the dfs.name.dir setting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1943) fail to start datanode while start-dfs.sh is executed by root user
[ https://issues.apache.org/jira/browse/HDFS-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reopened HDFS-1943: -- Reopened for back-port to 0.20.205.1, per request from Roman Shaposhnik. > fail to start datanode while start-dfs.sh is executed by root user > -- > > Key: HDFS-1943 > URL: https://issues.apache.org/jira/browse/HDFS-1943 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.20.205.0, 0.22.0, 0.23.0 >Reporter: Wei Yongjun >Assignee: Wei Yongjun >Priority: Blocker > Fix For: 0.22.0, 0.23.0 > > Attachments: HDFS-1943.patch > > > When start-dfs.sh is run by root user, we got the following error message: > # start-dfs.sh > Starting namenodes on [localhost ] > localhost: namenode running as process 2556. Stop it first. > localhost: starting datanode, logging to > /usr/hadoop/hadoop-common-0.23.0-SNAPSHOT/bin/../logs/hadoop-root-datanode-cspf01.out > localhost: Unrecognized option: -jvm > localhost: Could not create the Java virtual machine. > The -jvm options should be passed to jsvc when we starting a secure > datanode, but it still passed to java when start-dfs.sh is run by root > while secure datanode is disabled. This is a bug of bin/hdfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2547) Design doc is wrong about default block placement policy.
[ https://issues.apache.org/jira/browse/HDFS-2547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J reopened HDFS-2547: --- > Design doc is wrong about default block placement policy. > - > > Key: HDFS-2547 > URL: https://issues.apache.org/jira/browse/HDFS-2547 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.20.1 >Reporter: Harsh J >Assignee: Harsh J >Priority: Trivial > Fix For: 0.24.0 > > Attachments: HDFS-2547.patch > > > bq. For the common case, when the replication factor is three, HDFS's > placement policy is to put one replica on one node in the local rack, another > on a node in a different (remote) rack, and the last on a different node in > the same *remote* rack. > Should actually be: "and the last on a different node in the same *local* > rack." -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[
https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur reopened HDFS-2316:
--
we are using an enum {FILE, DIRECTORY, SYMLINK} for the FileStatus, but the
constructor of FileStatus, for symlinks, takes a isDir boolean.
This means we are losing the isFile/isDir information in the case of symlinks
but it seems that the HDFS implementation keeps that.
If this is the case, we should add that info either as a new enum set or
instead using enum using 2 booleans.
> webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
> --
>
> Key: HDFS-2316
> URL: https://issues.apache.org/jira/browse/HDFS-2316
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf,
> WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s
>
>
> We current have hftp for accessing HDFS over HTTP. However, hftp is a
> read-only FileSystem and does not provide "write" accesses.
> In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem
> implementation for accessing HDFS over HTTP. The is the umbrella JIRA for
> the tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2552) Add WebHdfs Forrest doc
[ https://issues.apache.org/jira/browse/HDFS-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur reopened HDFS-2552: -- Nicholas, the issue about spec-ing case sensitivity of param names and values in the query string has not been addressed. This is a BIG issue to ensure interoperability. If the spec states that the param names & values are case sensitive as per spec I'm good. If the client/server implementation choses to accept case insensitive it is fine, but it should always emit as per spec case. > Add WebHdfs Forrest doc > --- > > Key: HDFS-2552 > URL: https://issues.apache.org/jira/browse/HDFS-2552 > Project: Hadoop HDFS > Issue Type: Task > Components: documentation >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.20.205.1, 0.20.206.0, 0.24.0, 0.23.1 > > Attachments: h2552_2015.patch, h2552_2015b.patch, > h2552_2015b.pdf, h2552_2017.patch, h2552_2017.pdf, > h2552_2021.patch, h2552_2021.pdf, h2552_2021_0.20s.patch, > h2552_2021_0.20s.pdf > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2257) HftpFileSystem should implement getDelegationTokens
[
https://issues.apache.org/jira/browse/HDFS-2257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth reopened HDFS-2257:
--
{{getDelegationToken}} was deprecated in favor of {{getDelegationTokens}} -
likely along with the introduction of ViewFileSystem.
Hftp does not have an implementation of getDelegationTokens. ViewFs cannot
implement getDelegationToken.
The TokenCache in MapReduce currently ends up making one call to
getDelegationToken and another to getDelegationTokens - likely causing
unnecessary token caching on the Namenode?
> HftpFileSystem should implement getDelegationTokens
> ---
>
> Key: HDFS-2257
> URL: https://issues.apache.org/jira/browse/HDFS-2257
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Siddharth Seth
>Assignee: Daryn Sharp
> Attachments: HDFS-2257.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1374) TestBlockRecovery#testZeroLenReplicas fails
[
https://issues.apache.org/jira/browse/HDFS-1374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko reopened HDFS-1374:
---
Reopening to change resolution.
> TestBlockRecovery#testZeroLenReplicas fails
> ---
>
> Key: HDFS-1374
> URL: https://issues.apache.org/jira/browse/HDFS-1374
> Project: Hadoop HDFS
> Issue Type: Test
> Components: test
>Affects Versions: 0.22.0
>Reporter: Eli Collins
> Fix For: 0.22.0
>
>
> {noformat}
> Testcase: testZeroLenReplicas took 10.577 sec
> Caused an ERROR
> Wanted but not invoked:
> datanodeProtocol.commitBlockSynchronization(
> blk_1000_2000,
> 3000,
> 0,
> true,
> true,
> []
> );
> -> at
> org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery.testZeroLenReplicas(TestBlockRecovery.java:402)
> However, there were other interactions with this mock:
> -> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.handshake(DataNode.java:519)
> Wanted but not invoked:
> datanodeProtocol.commitBlockSynchronization(
> blk_1000_2000,
> 3000,
> 0,
> true,
> true,
> []
> );
> -> at
> org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery.testZeroLenReplicas(TestBlockRecovery.java:402)
> However, there were other interactions with this mock:
> -> at
> org.apache.hadoop.hdfs.server.datanode.DataNode.handshake(DataNode.java:519)
> at
> org.apache.hadoop.hdfs.server.datanode.TestBlockRecovery.testZeroLenReplicas(TestBlockRecovery.java:402)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1617) CLONE to COMMON - Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file
[ https://issues.apache.org/jira/browse/HDFS-1617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-1617: --- > CLONE to COMMON - Batch the calls in DataStorage to > FileUtil.createHardLink(), so we call it once per directory instead of once > per file > > > Key: HDFS-1617 > URL: https://issues.apache.org/jira/browse/HDFS-1617 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: 0.20.2 >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.22.0 > > > It was a bit of a puzzle why we can do a full scan of a disk in about 30 > seconds during FSDir() or getVolumeMap(), but the same disk took 11 minutes > to do Upgrade replication via hardlinks. It turns out that the > org.apache.hadoop.fs.FileUtil.createHardLink() method does an outcall to > Runtime.getRuntime().exec(), to utilize native filesystem hardlink > capability. So it is forking a full-weight external process, and we call it > on each individual file to be replicated. > As a simple check on the possible cost of this approach, I built a Perl test > script (under Linux on a production-class datanode). Perl also uses a > compiled and optimized p-code engine, and it has both native support for > hardlinks and the ability to do "exec". > - A simple script to create 256,000 files in a directory tree organized like > the Datanode, took 10 seconds to run. > - Replicating that directory tree using hardlinks, the same way as the > Datanode, took 12 seconds using native hardlink support. > - The same replication using outcalls to exec, one per file, took 256 > seconds! > - Batching the calls, and doing 'exec' once per directory instead of once > per file, took 16 seconds. > Obviously, your mileage will vary based on the number of blocks per volume. > A volume with less than about 4000 blocks will have only 65 directories. A > volume with more than 4K and less than about 250K blocks will have 4200 > directories (more or less). And there are two files per block (the data file > and the .meta file). So the average number of files per directory may vary > from 2:1 to 500:1. A node with 50K blocks and four volumes will have 25K > files per volume, or an average of about 6:1. So this change may be expected > to take it down from, say, 12 minutes per volume to 2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1496) TestStorageRestore is failing after HDFS-903 fix
[ https://issues.apache.org/jira/browse/HDFS-1496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-1496: --- > TestStorageRestore is failing after HDFS-903 fix > > > Key: HDFS-1496 > URL: https://issues.apache.org/jira/browse/HDFS-1496 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.22.0, 0.23.0 >Reporter: Konstantin Boudnik >Assignee: Hairong Kuang >Priority: Blocker > Fix For: 0.22.0 > > Attachments: HDFS-1496.sh, HDFS-1496.sh, HDFS-1496.sh > > > TestStorageRestore seems to be failing after HDFS-903 commit. Running git > bisect confirms it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1084) TestDFSShell fails in trunk.
[
https://issues.apache.org/jira/browse/HDFS-1084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko reopened HDFS-1084:
---
> TestDFSShell fails in trunk.
>
>
> Key: HDFS-1084
> URL: https://issues.apache.org/jira/browse/HDFS-1084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 0.22.0
>Reporter: Konstantin Shvachko
>Assignee: Po Cheung
>Priority: Blocker
> Fix For: 0.22.0
>
>
> {{TestDFSShell.testFilePermissions()}} fails on an assert attached below. I
> see it on my Linux box. Don't see it failing with Hudson, and the same test
> runs fine in 0.21 branch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2287) TestParallelRead has a small off-by-one bug
[ https://issues.apache.org/jira/browse/HDFS-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko reopened HDFS-2287: --- Second assert need to be fixed. > TestParallelRead has a small off-by-one bug > --- > > Key: HDFS-2287 > URL: https://issues.apache.org/jira/browse/HDFS-2287 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Fix For: 0.24.0 > > Attachments: hdfs-2287.txt > > > Noticed this bug when I was running TestParallelRead - a simple off-by-one > error in some internal bounds checking. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2316) webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP
[ https://issues.apache.org/jira/browse/HDFS-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reopened HDFS-2316: -- re-opening to accomodate request to fix also in 0.22.0. > webhdfs: a complete FileSystem implementation for accessing HDFS over HTTP > -- > > Key: HDFS-2316 > URL: https://issues.apache.org/jira/browse/HDFS-2316 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Labels: critical-0.22.0 > Fix For: 0.23.1, 1.0.0 > > Attachments: WebHdfsAPI20111020.pdf, WebHdfsAPI2003.pdf, > WebHdfsAPI2011.pdf, test-webhdfs, test-webhdfs-0.20s > > > We current have hftp for accessing HDFS over HTTP. However, hftp is a > read-only FileSystem and does not provide "write" accesses. > In HDFS-2284, we propose to have webhdfs for providing a complete FileSystem > implementation for accessing HDFS over HTTP. The is the umbrella JIRA for > the tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2616) Change DatanodeProtocol#sendHeartbeat to return HeartbeatResponse
[ https://issues.apache.org/jira/browse/HDFS-2616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reopened HDFS-2616: --- > Change DatanodeProtocol#sendHeartbeat to return HeartbeatResponse > - > > Key: HDFS-2616 > URL: https://issues.apache.org/jira/browse/HDFS-2616 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node, name-node >Affects Versions: HA branch (HDFS-1623) >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, > HDFS-2616.txt, HDFS-2616.txt, HDFS-2616.txt, hdfs-2616-addendum.txt > > > DatanodeProtocol#sendHeartbeat() returns DatanodeCommand[]. This jira > proposes changing it to to return HeartbeatResponse that has > DatanodeCommand[]. This allows adding other information that can be returned > by the namenode to the datanode, instead of having to only return > DatanodeCommand[]. For relevant discussion see HDFS-1972. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2663) Optional parameters are not handled correctly
[ https://issues.apache.org/jira/browse/HDFS-2663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HDFS-2663: --- The last patch had some unnecessary changes. Will attach the right patch. > Optional parameters are not handled correctly > - > > Key: HDFS-2663 > URL: https://issues.apache.org/jira/browse/HDFS-2663 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: 0.24.0 > > Attachments: HDFS-2663.txt, HDFS-2663.txt > > > Some of the parameters are not defined as optional. Also optional parameters > are not handled correctly. The converters should look for if the optional > parameter is present and only then get or set the parameter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-97) DFS should detect slow links(nodes) and avoid them
[ https://issues.apache.org/jira/browse/HDFS-97?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J reopened HDFS-97: - Oh well, didn't notice the 'read' issue too. We cover writes with that, not reads. Reopening. > DFS should detect slow links(nodes) and avoid them > -- > > Key: HDFS-97 > URL: https://issues.apache.org/jira/browse/HDFS-97 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Runping Qi > > The current DFS does not detect slow links (nodes). > Thus, when a node or its network link is slow, it may affect the overall > system performance significantly. > Specifically, when a map job needs to read data from such a node, it may > progress 10X slower. > And when a DFS data node pipeline consists of such a node, the write > performance degrades significantly. > This may lead to some long tails for map/reduce jobs. We have experienced > such behaviors quite often. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2764) TestBackupNode is failing
[ https://issues.apache.org/jira/browse/HDFS-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reopened HDFS-2764: --- I'm seeing this reliably locally on the latest change as of today and one one of my jenkins jobs running an older build. > TestBackupNode is failing > - > > Key: HDFS-2764 > URL: https://issues.apache.org/jira/browse/HDFS-2764 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > > Looks like it has been for a few days. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2802) Support for RW/RO snapshots in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Mankude reopened HDFS-2802: I (We) are well aware of hdfs-233. This jira was opened to provide comprehensive snapshot solution (both RW/RO support) for HDFS. > Support for RW/RO snapshots in HDFS > --- > > Key: HDFS-2802 > URL: https://issues.apache.org/jira/browse/HDFS-2802 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Hari Mankude >Assignee: Hari Mankude > Fix For: 0.24.0 > > > Snapshots are point in time images of parts of the filesystem or the entire > filesystem. Snapshots can be a read-only or a read-write point in time copy > of the filesystem. There are several use cases for snapshots in HDFS. I will > post a detailed write-up soon with with more information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2379) 0.20: Allow block reports to proceed without holding FSDataset lock
[ https://issues.apache.org/jira/browse/HDFS-2379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reopened HDFS-2379: -- There's been a request to include this in 1.0.1, but the patch doesn't apply; it conflicts with another patch on several files. Todd, would you be able to look at making the patch work in branch-1.0, please? > 0.20: Allow block reports to proceed without holding FSDataset lock > --- > > Key: HDFS-2379 > URL: https://issues.apache.org/jira/browse/HDFS-2379 > Project: Hadoop HDFS > Issue Type: Bug > Components: data-node >Affects Versions: 1.1.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 1.1.0 > > Attachments: hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt, > hdfs-2379.txt, hdfs-2379.txt, hdfs-2379.txt > > > As disks are getting larger and more plentiful, we're seeing DNs with > multiple millions of blocks on a single machine. When page cache space is > tight, block reports can take multiple minutes to generate. Currently, during > the scanning of the data directories to generate a report, the FSVolumeSet > lock is held. This causes writes and reads to block, timeout, etc, causing > big problems especially for clients like HBase. > This JIRA is to explore some of the ideas originally discussed in HADOOP-4584 > for the 0.20.20x series. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2922) HA: close out operation categories
[
https://issues.apache.org/jira/browse/HDFS-2922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon reopened HDFS-2922:
---
This seems to have broken a bunch of tests since getDatanodeReport now throws
an error on the standby, so MiniDFSCluster won't start an HA cluster anymore.
I'm going to revert for now to get tests passing again on branch
> HA: close out operation categories
> --
>
> Key: HDFS-2922
> URL: https://issues.apache.org/jira/browse/HDFS-2922
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Eli Collins
>Assignee: Eli Collins
> Attachments: hdfs-2922.txt, hdfs-2922.txt
>
>
> We need to close out the NN operations categories.
> The following operations should be left as is, ie not failover, as it's
> reasonable to call these on a standby, and we just need to update the TODO
> with a comment:
> - {{setSafeMode}} (Might want to force the standby out of safemode)
> - {{restoreFailedStorage}} (Might want to tell the standby to restore the
> shared edits dir)
> - {{saveNamespace}}, {{metaSave}} (Could imagine calling these on a standby
> eg in a recovery scenario)
> - {{refreshNodes}} (Decommissioning needs to refresh the standby)
> The following operations should be checked for READ, as neither should need
> to be called on standby, will failover unless stale reads are enabled:
> - {{getTransactionID}}, {{getEditLogManifest}} (we don't checkoint the
> standby)
> The following operations should be checked for WRITE, as they should not be
> called on a standby, ie should always failover:
> - {{finalizeUpgrade}}, {{distributedUpgradeProgress}} (should not be able to
> upgrade the standby)
> - {{setBalancerBandwidth}} (balancer should failover)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2802) Support for RW/RO snapshots in HDFS
[ https://issues.apache.org/jira/browse/HDFS-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HDFS-2802: --- > Support for RW/RO snapshots in HDFS > --- > > Key: HDFS-2802 > URL: https://issues.apache.org/jira/browse/HDFS-2802 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Reporter: Hari Mankude >Assignee: Hari Mankude > Fix For: 0.24.0 > > > Snapshots are point in time images of parts of the filesystem or the entire > filesystem. Snapshots can be a read-only or a read-write point in time copy > of the filesystem. There are several use cases for snapshots in HDFS. I will > post a detailed write-up soon with with more information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2948) HA: NN throws NPE during shutdown if it fails to startup
[
https://issues.apache.org/jira/browse/HDFS-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon reopened HDFS-2948:
---
Oops, this also needs to be fixed on the SBN it appears. Mind if I just commit
the same null check to the SBN under this same JIRA issue?
{code}
diff --git
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
b/hadoop-hdfs-project/hadoop-hd
index 4e628ce..5c09512 100644
---
a/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
+++
b/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
@@ -994,7 +994,9 @@ public class NameNode {
@Override
public void stopStandbyServices() throws IOException {
// TODO(HA): Are we guaranteed to be the only active here?
- namesystem.stopStandbyServices();
+ if (namesystem != null) {
+namesystem.stopStandbyServices();
+ }
}
@Override
{code}
> HA: NN throws NPE during shutdown if it fails to startup
>
>
> Key: HDFS-2948
> URL: https://issues.apache.org/jira/browse/HDFS-2948
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Minor
> Fix For: HA branch (HDFS-1623)
>
> Attachments: hdfs-2948.txt
>
>
> Last night's nightly build had a bunch of NPEs thrown in NameNode.stop. Not
> sure which patch introduced the issue, but the problem is that
> NameNode.stop() is called if an exception is thrown during startup. If the
> exception is thrown before the namesystem is created, then
> NameNode.namesystem is null, and {{namesystem.stop}} throws NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2979) HA: Balancer should use logical uri for creating failover proxy with HA enabled.
[ https://issues.apache.org/jira/browse/HDFS-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reopened HDFS-2979: --- This patch seems to have made some tests fail (I noticed with TestFileAppend2). It seems to be trying to connect to 127.0.0.1:0 instead of the correct port. > HA: Balancer should use logical uri for creating failover proxy with HA > enabled. > > > Key: HDFS-2979 > URL: https://issues.apache.org/jira/browse/HDFS-2979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer, hdfs client >Affects Versions: HA branch (HDFS-1623) >Reporter: Uma Maheswara Rao G >Assignee: Aaron T. Myers > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-2979-HDFS-1623.patch, HDFS-2979-HDFS-1623.patch, > HDFS-2979-HDFS-1623.patch, HDFS-2979.patch > > > Presently Balancer uses real URI for creating the failover proxy. > Since the failover proxy checks for uri consistency, we should pass logical > uri for creating failover proxy instead of instead of real URI. Presently > will work only with default port. > java.io.IOException: Port 49832 specified in URI hdfs://127.0.0.1:49832 but > host '127.0.0.1' is a logical (HA) namenode and does not use port information. > at > org.apache.hadoop.hdfs.HAUtil.getFailoverProxyProviderClass(HAUtil.java:224) > at org.apache.hadoop.hdfs.HAUtil.createFailoverProxy(HAUtil.java:247) > at > org.apache.hadoop.hdfs.server.balancer.NameNodeConnector.(NameNodeConnector.java:80) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:1401) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2609) DataNode.getDNRegistrationByMachineName can probably be removed or simplified
[
https://issues.apache.org/jira/browse/HDFS-2609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon reopened HDFS-2609:
---
> DataNode.getDNRegistrationByMachineName can probably be removed or simplified
> -
>
> Key: HDFS-2609
> URL: https://issues.apache.org/jira/browse/HDFS-2609
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node
>Affects Versions: 0.23.0
>Reporter: Todd Lipcon
>Assignee: Eli Collins
>
> I noticed this while working on HDFS-1971: The
> {{getDNRegistrationByMachineName}} iterates over block pools to return a
> given block pool's registration object based on its {{machineName}} field.
> But, the machine name for every BPOfferService is identical - they're always
> constructed by just calling {{DataNode.getName}}. All of the call sites for
> this function are from tests, as well. So, maybe it's not necessary, or at
> least it might be able to be simplified or moved to a test method.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-3094) add -nonInteractive and -force option to namenode -format command
[ https://issues.apache.org/jira/browse/HDFS-3094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon reopened HDFS-3094: --- Ah, sorry, forgot about branch-1. I'll look at that later tonight or tomorrow, as well as the docs. Reopening so it doesn't get lost > add -nonInteractive and -force option to namenode -format command > - > > Key: HDFS-3094 > URL: https://issues.apache.org/jira/browse/HDFS-3094 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 0.24.0, 1.0.2 >Reporter: Arpit Gupta >Assignee: Arpit Gupta > Fix For: 2.0.0 > > Attachments: HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, > HDFS-3094.branch-1.0.patch, HDFS-3094.branch-1.0.patch, HDFS-3094.patch, > HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, HDFS-3094.patch, > HDFS-3094.patch, HDFS-3094.patch > > > Currently the bin/hadoop namenode -format prompts the user for a Y/N to setup > the directories in the local file system. > -force : namenode formats the directories without prompting > -nonInterActive : namenode format will return with an exit code of 1 if the > dir exists. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2764) TestBackupNode is racy
[ https://issues.apache.org/jira/browse/HDFS-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reopened HDFS-2764: -- Re-opening issue to commit this to the 0.23 branch, since the test might fail spuriously there as well. > TestBackupNode is racy > -- > > Key: HDFS-2764 > URL: https://issues.apache.org/jira/browse/HDFS-2764 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node, test >Affects Versions: 0.24.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 0.24.0 > > Attachments: HDFS-2764.patch, HDFS-2764.patch > > > TestBackupNode#waitCheckpointDone can spuriously fail because of a race. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2162) Merge NameNode roles into NodeType.
[
https://issues.apache.org/jira/browse/HDFS-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers reopened HDFS-2162:
--
> Merge NameNode roles into NodeType.
> ---
>
> Key: HDFS-2162
> URL: https://issues.apache.org/jira/browse/HDFS-2162
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
>Affects Versions: 0.23.0
>Reporter: Suresh Srinivas
>Assignee: Suresh Srinivas
>Priority: Minor
>
> Currently Namenode has {{NamenodeRole}} with roles NAMENODE, BACKUP and
> CHECKPOINT. {{NodeType}} has node types NAME_NODE and DATA_NODE. Merge
> NamenodeRole into NodeType.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-3014) FSEditLogOp and its subclasses should have toString() method
[ https://issues.apache.org/jira/browse/HDFS-3014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reopened HDFS-3014: -- > FSEditLogOp and its subclasses should have toString() method > > > Key: HDFS-3014 > URL: https://issues.apache.org/jira/browse/HDFS-3014 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Sho Shimauchi >Assignee: Sho Shimauchi > Fix For: 0.24.0 > > Attachments: HDFS-3014.txt, HDFS-3014.txt > > > FSEditLogOp and its subclasses should have toString() method. > It's helpful for investigating editlog related issues. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2979) HA: Balancer should use logical uri for creating failover proxy with HA enabled.
[ https://issues.apache.org/jira/browse/HDFS-2979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reopened HDFS-2979: -- On second thought, I'm going to revert this change and fix the issue with NN conf mutation by modifying the test TestBalancerWithHANameNodes test case. The conf mutation issue should be fixed in a more general way. > HA: Balancer should use logical uri for creating failover proxy with HA > enabled. > > > Key: HDFS-2979 > URL: https://issues.apache.org/jira/browse/HDFS-2979 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: balancer, hdfs client >Affects Versions: HA branch (HDFS-1623) >Reporter: Uma Maheswara Rao G >Assignee: Aaron T. Myers > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-2979-HDFS-1623.patch, HDFS-2979-HDFS-1623.patch, > HDFS-2979-HDFS-1623.patch, HDFS-2979.patch > > > Presently Balancer uses real URI for creating the failover proxy. > Since the failover proxy checks for uri consistency, we should pass logical > uri for creating failover proxy instead of instead of real URI. Presently > will work only with default port. > java.io.IOException: Port 49832 specified in URI hdfs://127.0.0.1:49832 but > host '127.0.0.1' is a logical (HA) namenode and does not use port information. > at > org.apache.hadoop.hdfs.HAUtil.getFailoverProxyProviderClass(HAUtil.java:224) > at org.apache.hadoop.hdfs.HAUtil.createFailoverProxy(HAUtil.java:247) > at > org.apache.hadoop.hdfs.server.balancer.NameNodeConnector.(NameNodeConnector.java:80) > at > org.apache.hadoop.hdfs.server.balancer.Balancer.run(Balancer.java:1401) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-1378) Edit log replay should track and report file offsets in case of errors
[ https://issues.apache.org/jira/browse/HDFS-1378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe reopened HDFS-1378: Assignee: Colin Patrick McCabe (was: Aaron T. Myers) I'd like to port this to branch-1 so that we can have better error messages there. It should be a trivial port. Any objections? > Edit log replay should track and report file offsets in case of errors > -- > > Key: HDFS-1378 > URL: https://issues.apache.org/jira/browse/HDFS-1378 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Colin Patrick McCabe > Fix For: 0.23.0 > > Attachments: hdfs-1378-branch20.txt, hdfs-1378.0.patch, > hdfs-1378.1.patch, hdfs-1378.2.txt > > > Occasionally there are bugs or operational mistakes that result in corrupt > edit logs which I end up having to repair by hand. In these cases it would be > very handy to have the error message also print out the file offsets of the > last several edit log opcodes so it's easier to find the right place to edit > in the OP_INVALID marker. We could also use this facility to provide a rough > estimate of how far along edit log replay the NN is during startup (handy > when a 2NN has died and replay takes a while) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2765) TestNameEditsConfigs is incorrectly swallowing IOE
[ https://issues.apache.org/jira/browse/HDFS-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reopened HDFS-2765: -- Reopening for back-port to branch-2. > TestNameEditsConfigs is incorrectly swallowing IOE > -- > > Key: HDFS-2765 > URL: https://issues.apache.org/jira/browse/HDFS-2765 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-2765.patch > > > The final portion of this test case is swallowing an IOE and in so doing > appearing to succeed, although it should not be succeeding as-written. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2385) Support delegation token renewal in webhdfs
[ https://issues.apache.org/jira/browse/HDFS-2385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE reopened HDFS-2385: -- Thanks Matt for merging this to 205. Reopen since trunk still needs a patch. > Support delegation token renewal in webhdfs > --- > > Key: HDFS-2385 > URL: https://issues.apache.org/jira/browse/HDFS-2385 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.20.205.0 > > Attachments: h2358_20111001_0.20s.patch, h2385_20111004_0.20s.patch, > h2385_20111004b_0.20s.patch, h2385_20111004b_amendment2_0.20s.patch, > h2385_20111004b_amendment_0.20s.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2493) Remove reference to FSNamesystem in blockmanagement classes
[ https://issues.apache.org/jira/browse/HDFS-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE reopened HDFS-2493: -- It turns out that 0.23 needs a slightly different patch. > Remove reference to FSNamesystem in blockmanagement classes > --- > > Key: HDFS-2493 > URL: https://issues.apache.org/jira/browse/HDFS-2493 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.23.0, 0.24.0 > > Attachments: h2493_20111022.patch, h2493_20111022b.patch, > h2493_20111028.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2416) distcp with a webhdfs uri on a secure cluster fails
[ https://issues.apache.org/jira/browse/HDFS-2416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE reopened HDFS-2416: -- > Merged to 0.23. It turns out that 0.23 cannot be compiled after the merge. Reverted the 0.23 commit. > distcp with a webhdfs uri on a secure cluster fails > --- > > Key: HDFS-2416 > URL: https://issues.apache.org/jira/browse/HDFS-2416 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: 0.20.205.0 >Reporter: Arpit Gupta >Assignee: Jitendra Nath Pandey > Fix For: 0.20.205.1, 0.20.206.0, 0.23.0, 0.24.0 > > Attachments: HDFS-2416-branch-0.20-security.6.patch, > HDFS-2416-branch-0.20-security.7.patch, > HDFS-2416-branch-0.20-security.8.patch, HDFS-2416-branch-0.20-security.patch, > HDFS-2416-trunk.patch, HDFS-2416-trunk.patch, > HDFS-2419-branch-0.20-security.patch, HDFS-2419-branch-0.20-security.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-989) Flush/sync do not work on Hadoop LocalFileSystem
[ https://issues.apache.org/jira/browse/HDFS-989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reopened HDFS-989: -- > Flush/sync do not work on Hadoop LocalFileSystem > > > Key: HDFS-989 > URL: https://issues.apache.org/jira/browse/HDFS-989 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.20.1 >Reporter: Nathan Marz > > They seem to be no-ops. This is really easy to reproduce, just open a file > using FileSystem/getLocal(new Configuration()), write data to the output > stream, and then try to flush/sync. I also tried creating the output stream > with a buffer size of 1, but that had no effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2528) webhdfs rest call to a secure dn fails when a token is sent
[
https://issues.apache.org/jira/browse/HDFS-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE reopened HDFS-2528:
--
> webhdfs rest call to a secure dn fails when a token is sent
> ---
>
> Key: HDFS-2528
> URL: https://issues.apache.org/jira/browse/HDFS-2528
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Affects Versions: 0.20.205.0
>Reporter: Arpit Gupta
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.20.205.1, 0.20.206.0, 0.24.0, 0.23.1
>
> Attachments: h2528_2001.patch, h2528_2001_0.20s.patch,
> h2528_2001b.patch, h2528_2001b_0.20s.patch, h2528_2002.patch,
> h2528_2002_0.20s.patch, h2528_2003.patch, h2528_2003_0.20s.patch,
> h2528_2003_0.20s.patch
>
>
> curl -L -u : --negotiate -i
> "http://NN:50070/webhdfs/v1/tmp/webhdfs_data/file_small_data.txt?op=OPEN";
> the following exception is thrown by the datanode when the redirect happens.
> {"RemoteException":{"exception":"IOException","javaClassName":"java.io.IOException","message":"Call
> to failed on local exception: java.io.IOException:
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]"}}
> Interestingly when using ./bin/hadoop with a webhdfs path we are able to cat
> or tail a file successfully.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2827) Cannot save namespace after renaming a directory above a file with an open lease
[
https://issues.apache.org/jira/browse/HDFS-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G reopened HDFS-2827:
---
> Cannot save namespace after renaming a directory above a file with an open
> lease
>
>
> Key: HDFS-2827
> URL: https://issues.apache.org/jira/browse/HDFS-2827
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.24.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
> Fix For: 0.24.0, 0.23.1
>
> Attachments: HDFS-2827-test.patch, HDFS-2827.patch
>
>
> When i execute the following operations and wait for checkpoint to complete.
> fs.mkdirs(new Path("/test1"));
> FSDataOutputStream create = fs.create(new Path("/test/abc.txt")); //dont close
> fs.rename(new Path("/test/"), new Path("/test1/"));
> Check-pointing is failing with the following exception.
> 2012-01-23 15:03:14,204 ERROR namenode.FSImage (FSImage.java:run(795)) -
> Unable to save image for
> E:\HDFS-1623\hadoop-hdfs-project\hadoop-hdfs\build\test\data\dfs\name3
> java.io.IOException: saveLeases found path /test1/est/abc.txt but no matching
> entry in namespace.[/test1/est/abc.txt]
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.saveFilesUnderConstruction(FSNamesystem.java:4336)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImageFormat$Saver.save(FSImageFormat.java:588)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:761)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:789)
> at java.lang.Thread.run(Unknown Source)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-776) Fix exception handling in Balancer
[
https://issues.apache.org/jira/browse/HDFS-776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE reopened HDFS-776:
-
I somehow set the resolution to "Unresolved". Reopen to fix it.
> Fix exception handling in Balancer
> --
>
> Key: HDFS-776
> URL: https://issues.apache.org/jira/browse/HDFS-776
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: balancer
>Reporter: Owen O'Malley
>Assignee: Uma Maheswara Rao G
>Priority: Critical
> Fix For: 0.24.0, 0.23.2
>
> Attachments: HDFS-776.0.patch, HDFS-776.patch, HDFS-776.patch,
> HDFS-776.patch, HDFS-776.patch
>
>
> The Balancer's AccessKeyUpdater handles exceptions badly. In particular:
> 1. Catching Exception too low. The wrapper around setKeys should only catch
> IOException.
> 2. InterruptedException is ignored. It should be caught at the top level and
> exit run.
> 3. Throwable is not caught. It should be caught at the top level and kill the
> Balancer server process.
> {code}
> class AccessKeyUpdater implements Runnable {
> public void run() {
> while (shouldRun) {
> try {
> accessTokenHandler.setKeys(namenode.getAccessKeys());
> } catch (Exception e) {
> LOG.error(StringUtils.stringifyException(e));
> }
> try {
> Thread.sleep(keyUpdaterInterval);
> } catch (InterruptedException ie) {
> }
> }
> }
> }
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2944) Typo in hdfs-default.xml causes dfs.client.block.write.replace-datanode-on-failure.enable to be mistakenly disabled
[
https://issues.apache.org/jira/browse/HDFS-2944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE reopened HDFS-2944:
--
Reopen for committing it to 0.23.
> Typo in hdfs-default.xml causes
> dfs.client.block.write.replace-datanode-on-failure.enable to be mistakenly
> disabled
> ---
>
> Key: HDFS-2944
> URL: https://issues.apache.org/jira/browse/HDFS-2944
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 0.24.0
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Fix For: 0.24.0
>
> Attachments: HDFS-2944.patch
>
>
> Here, "ture" should be "true" if we really do want this feature enabled by
> default.
> {code}
>
> dfs.client.block.write.replace-datanode-on-failure.enable
> ture
> ...
>
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-3075) Add mechanism to restore the removed storage directories
[ https://issues.apache.org/jira/browse/HDFS-3075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE reopened HDFS-3075: -- @Uma, you are right that HADOOP-4885 already has fixed this. So this one is a backport. Will revise the title. @Eli, this is not a dupe of HDFS-2781. > Add mechanism to restore the removed storage directories > > > Key: HDFS-3075 > URL: https://issues.apache.org/jira/browse/HDFS-3075 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0, 1.1.0 >Reporter: Brandon Li >Assignee: Brandon Li > > When a storage directory is inaccessible, namenode removes it from the valid > storage dir list to a removedStorageDirs list. Those storage directories will > not be restored when they become healthy again. > The proposed solution is to restore the previous failed directories at the > beginning of checkpointing, say, rollEdits, by copying necessary metadata > files from healthy directory to unhealthy ones. In this way, whenever a > failed storage directory is recovered by the administrator, he/she can > immediately force a checkpointing to restored a failed directory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2768) BackupNode stop can not close proxy connections because it is not a proxy instance.
[ https://issues.apache.org/jira/browse/HDFS-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G reopened HDFS-2768: --- Seems this changes has been reverted with HA merge. > BackupNode stop can not close proxy connections because it is not a proxy > instance. > --- > > Key: HDFS-2768 > URL: https://issues.apache.org/jira/browse/HDFS-2768 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 2.0.0 > > Attachments: HDFS-2768.patch, HDFS-2768.patch > > > Observe this from BackupNode tests: > java.lang.IllegalArgumentException: not a proxy instance > at java.lang.reflect.Proxy.getInvocationHandler(Unknown Source) > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) > at > org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:355) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testBackupNode(TestBackupNode.java:241) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
