[jira] [Created] (HDFS-12117) HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST Interface
Wellington Chevreuil created HDFS-12117: --- Summary: HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST Interface Key: HDFS-12117 URL: https://issues.apache.org/jira/browse/HDFS-12117 Project: Hadoop HDFS Issue Type: New Feature Components: httpfs Affects Versions: 3.0.0-alpha3 Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil Currently, HttpFS is lacking implementation for SNAPSHOT related methods from WebHDFS REST interface as defined by WebHDFS documentation [WebHDFS documentation|https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Snapshot_Operations] I would like to work on this implementation, following the existing design approach already implemented by other WebHDFS methods on current HttpFS project, so I'll be proposing an initial patch soon for reviews. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12118) Ozone: Ozone shell: Add more testing for volume shell commands
Yiqun Lin created HDFS-12118: Summary: Ozone: Ozone shell: Add more testing for volume shell commands Key: HDFS-12118 URL: https://issues.apache.org/jira/browse/HDFS-12118 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone, tools Affects Versions: HDFS-7240 Reporter: Yiqun Lin Assignee: Yiqun Lin Currently there are missing enough testing to cover all the ozone command. Now we have to test this in a manual way to see if commands run okay as expected. There will be lots of test cases we should add for all the volume, bucket and key commands. In order to easy reviewed, I'd like to separate this work to three subtasks. This JIRA is a good start for implementing for volume commands. Plan to add the unit test to test following command and its available options: * infoVolume * updateVolume * listVolume * createVolume -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12119) Inconsistent "Number of Under-Replicated Blocks" shown on HDFS web UI and fsck report
liuyiyang created HDFS-12119: Summary: Inconsistent "Number of Under-Replicated Blocks" shown on HDFS web UI and fsck report Key: HDFS-12119 URL: https://issues.apache.org/jira/browse/HDFS-12119 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.6.0 Reporter: liuyiyang Sometimes the information "Number of Under-Replicated Blocks" shown on NameNode web UI is inconsistent with the "Under-replicated blocks" information shown in fsck report. It's easy to reproduce such a case as follows: 1、In a cluster with DN1(rack0)、DN2(rack1) and DN3(rack2) which stores a lot of blocks, the replication factor is set to 2; 2、Re-allocate racks as DN1(rack0)、DN2(rack1) and DN3(rack1) ; 3、Restart HDFS daemons. Then you can find inconsistent "Number of Under-Replicated Blocks" on web ui and "Under-replicated blocks" in fsck result. I dug into the source code and found that "Number of Under-Replicated Blocks" on web ui consists of blocks that have less than target number of replicas and blocks that have the right number of replicas, but which the block manager felt were badly distributed. In fsck result, "Under-replicated blocks" are the blocks that have less than target number of replicas, and "Mis-replicated blocks" are the blocks that are badly distributed. So the Under-Replicated Blocks info on web UI and fsck result may be inconsistent. Since Under-Replicated Blocks means higer missing risk for blocks, when threre is no blocks that have less than target replicas but a lot of blocks that have the right number of blocks but are badlly distributed, the "Number of Under-Replicated Blocks" on web UI will be same as number of "Mis-replicated blocks", which is misleading for users. It would be clear to make "Number of Under-Replicated Blocks" on web UI be consistent with "Under-replicated blocks" in fsck result. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12120) Use new block for pre-RollingUpgrade files` append requests
Vinayakumar B created HDFS-12120: Summary: Use new block for pre-RollingUpgrade files` append requests Key: HDFS-12120 URL: https://issues.apache.org/jira/browse/HDFS-12120 Project: Hadoop HDFS Issue Type: Bug Reporter: Vinayakumar B After the RollingUpgrade prepare, append on pre-RU files will re-open the same last block and makes changes to it (appending extra data, changing genstamp etc). These changes to the block will not be tracked in Datanodes (either in trash or via hardlinks) This creates problem if RollingUpgrade.Rollback is called. Since block state and size both changed, after rollback block will be marked corrupted. To avoid this, first time append on pre-RU files can be forced to write to new block itself. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11944) TestAclsEndToEnd#testCreateEncryptionZone failing very frequently.
[
https://issues.apache.org/jira/browse/HDFS-11944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah resolved HDFS-11944.
---
Resolution: Invalid
Release Note:
It was not a flaky test.
Fixed it via HADOOP-14563
> TestAclsEndToEnd#testCreateEncryptionZone failing very frequently.
> --
>
> Key: HDFS-11944
> URL: https://issues.apache.org/jira/browse/HDFS-11944
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: encryption, test
>Affects Versions: 2.8.0
>Reporter: Rushabh S Shah
>
> TestAclsEndToEnd#testCreateEncryptionZone is failing v frequently.
> The way test is written makes very hard to debug.
> Ideally each test case should test only one behavior.
> But in this test case, it reset the dfs state many times in same test case.
> It fails with the following stack trace.
> {noformat}
> Tests run: 9, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 35.17 sec <<<
> FAILURE! - in org.apache.hadoop.hdfs.TestAclsEndToEnd
> testCreateEncryptionZone(org.apache.hadoop.hdfs.TestAclsEndToEnd) Time
> elapsed: 3.844 sec <<< FAILURE!
> java.lang.AssertionError: Allowed zone creation of zone with blacklisted
> GENERATE_EEK
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at
> org.apache.hadoop.hdfs.TestAclsEndToEnd.testCreateEncryptionZone(TestAclsEndToEnd.java:753)
> Results :
> Failed tests:
> TestAclsEndToEnd.testCreateEncryptionZone:753 Allowed zone creation of zone
> with blacklisted GENERATE_EEK
> {noformat}
> It failed in the following pre-commits.
> [HDFS-11885
> precommit|https://issues.apache.org/jira/browse/HDFS-11885?focusedCommentId=16040117&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16040117]
> [HDFS-11804
> precommit|https://issues.apache.org/jira/browse/HDFS-11804?focusedCommentId=16039872&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16039872]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/461/ [Jul 10, 2017 10:53:13 AM] (stevel) HADOOP-14634. Remove jline from main Hadoop pom.xml. Contributed by Ray [Jul 10, 2017 9:34:58 PM] (Arun Suresh) YARN-6776. Refactor ApplicaitonMasterService to move actual processing [Jul 11, 2017 12:48:27 AM] (jitendra) HADOOP-10829. Iteration on CredentialProviderFactory.serviceLoader is [Jul 11, 2017 4:30:13 AM] (aajisaka) HADOOP-14638. Replace commons-logging APIs with slf4j in StreamPumper. -1 overall The following subsystems voted -1: findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: FindBugs : module:hadoop-hdfs-project/hadoop-hdfs-client Possible exposure of partially initialized object in org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At DFSClient.java:object in org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At DFSClient.java:[line 2888] org.apache.hadoop.hdfs.server.protocol.SlowDiskReports.equals(Object) makes inefficient use of keySet iterator instead of entrySet iterator At SlowDiskReports.java:keySet iterator instead of entrySet iterator At SlowDiskReports.java:[line 105] FindBugs : module:hadoop-hdfs-project/hadoop-hdfs Possible null pointer dereference in org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to return value of called method Dereferenced at JournalNode.java:org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to return value of called method Dereferenced at JournalNode.java:[line 302] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setClusterId(String) unconditionally sets the field clusterId At HdfsServerConstants.java:clusterId At HdfsServerConstants.java:[line 193] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForce(int) unconditionally sets the field force At HdfsServerConstants.java:force At HdfsServerConstants.java:[line 217] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForceFormat(boolean) unconditionally sets the field isForceFormat At HdfsServerConstants.java:isForceFormat At HdfsServerConstants.java:[line 229] org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setInteractiveFormat(boolean) unconditionally sets the field isInteractiveFormat At HdfsServerConstants.java:isInteractiveFormat At HdfsServerConstants.java:[line 237] Possible null pointer dereference in org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, int, HardLink, boolean, File, List) due to return value of called method Dereferenced at DataStorage.java:org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, int, HardLink, boolean, File, List) due to return value of called method Dereferenced at DataStorage.java:[line 1339] Possible null pointer dereference in org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String, long) due to return value of called method Dereferenced at NNStorageRetentionManager.java:org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String, long) due to return value of called method Dereferenced at NNStorageRetentionManager.java:[line 258] Possible null pointer dereference in org.apache.hadoop.hdfs.server.namenode.NNUpgradeUtil$1.visitFile(Path, BasicFileAttributes) due to return value of called method Dereferenced at NNUpgradeUtil.java:org.apache.hadoop.hdfs.server.namenode.NNUpgradeUtil$1.visitFile(Path, BasicFileAttributes) due to return value of called method Dereferenced at NNUpgradeUtil.java:[line 133] Useless condition:argv.length >= 1 at this point At DFSAdmin.java:[line 2085] Useless condition:numBlocks == -1 at this point At ImageLoaderCurrent.java:[line 727] FindBugs : module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Useless object stored in variable removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:removedNullContainers of method org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List) At NodeStatusUpdaterImpl.java:[line 642] org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeVeryOldStoppedContainersFromCache() makes inefficient use of keySet iterator instead of entrySet iterator At NodeStatusUpdaterImpl.java:keySet iterator instead of entrySet iterator At
[jira] [Created] (HDFS-12122) TestEditLogTailer is flaky
John Zhuge created HDFS-12122:
-
Summary: TestEditLogTailer is flaky
Key: HDFS-12122
URL: https://issues.apache.org/jira/browse/HDFS-12122
Project: Hadoop HDFS
Issue Type: Test
Components: namenode
Affects Versions: 3.0.0-alpha4
Reporter: John Zhuge
Priority: Minor
https://builds.apache.org/job/PreCommit-HDFS-Build/20225/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
{noformat}
Tests run: 12, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 46.901 sec <<<
FAILURE! - in org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer
testTriggersLogRollsForAllStandbyNN[0](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
Time elapsed: 10.728 sec <<< FAILURE!
java.lang.AssertionError: Test resulted in an unexpected exit
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1957)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1944)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1937)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testTriggersLogRollsForAllStandbyNN(TestEditLogTailer.java:245)
testNN1TriggersLogRolls[1](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
Time elapsed: 0.394 sec <<< FAILURE!
java.lang.AssertionError: Test resulted in an unexpected exit
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1957)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1944)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1937)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRolls(TestEditLogTailer.java:201)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testNN1TriggersLogRolls(TestEditLogTailer.java:154)
testNN2TriggersLogRolls[1](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
Time elapsed: 1.287 sec <<< FAILURE!
java.lang.AssertionError: Test resulted in an unexpected exit
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1957)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1944)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1937)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRolls(TestEditLogTailer.java:201)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testNN2TriggersLogRolls(TestEditLogTailer.java:159)
testNN0TriggersLogRolls[1](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
Time elapsed: 1.286 sec <<< FAILURE!
java.lang.AssertionError: Test resulted in an unexpected exit
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1957)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1944)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1937)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testStandbyTriggersLogRolls(TestEditLogTailer.java:201)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testNN0TriggersLogRolls(TestEditLogTailer.java:149)
testTriggersLogRollsForAllStandbyNN[1](org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer)
Time elapsed: 10.469 sec <<< FAILURE!
java.lang.AssertionError: Test resulted in an unexpected exit
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1957)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1944)
at
org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1937)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestEditLogTailer.testTriggersLogRollsForAllStandbyNN(TestEditLogTailer.java:245)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12123) Ozone: OzoneClient: Abstraction of OzoneClient and default implementation
Nandakumar created HDFS-12123:
-
Summary: Ozone: OzoneClient: Abstraction of OzoneClient and
default implementation
Key: HDFS-12123
URL: https://issues.apache.org/jira/browse/HDFS-12123
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ozone
Reporter: Nandakumar
Assignee: Nandakumar
{{OzoneClient}} interface defines all the client operations supported by Ozone.
{{OzoneClientImpl}} will have the default implementation, it should connects to
KSM, SCM and DataNode through RPC to execute client calls.
Similarly we should have a client implementation which implements
{{OzoneClient}} and uses REST protocol to execute client calls.
This will provide lots of flexibility to Ozone applications, when applications
are running inside the cluster, they can use RPC protocol, but when running
from outside the cluster, the same applications can speak REST protocol to
communicate with Ozone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12121) Audit logging of unsuccessful namesystem calls should not be done while holding fslock.
[
https://issues.apache.org/jira/browse/HDFS-12121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rushabh S Shah resolved HDFS-12121.
---
Resolution: Duplicate
Dup of HDFS-11246
> Audit logging of unsuccessful namesystem calls should not be done while
> holding fslock.
> ---
>
> Key: HDFS-12121
> URL: https://issues.apache.org/jira/browse/HDFS-12121
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
>
> Currently we are audit logging the unsuccessful namesystem calls while
> holding fslock.
> Following is just one example.
> {code:title=FSNamesystem.java|borderStyle=solid}
> HdfsFileStatus getFileInfo(final String src, boolean resolveLink)
> throws IOException {
> final String operationName = "getfileinfo";
> checkOperation(OperationCategory.READ);
> HdfsFileStatus stat = null;
> readLock();
> try {
> checkOperation(OperationCategory.READ);
> stat = FSDirStatAndListingOp.getFileInfo(dir, src, resolveLink);
> } catch (AccessControlException e) {
> logAuditEvent(false, operationName, src); //< audit logging while
> holding fs read lock.
> throw e;
> } finally {
> readUnlock(operationName);
> }
> logAuditEvent(true, operationName, src);
> return stat;
> }
> {code}
> We should avoid doing this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-9590) NPE in Storage$StorageDirectory#unlock()
[
https://issues.apache.org/jira/browse/HDFS-9590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen resolved HDFS-9590.
-
Resolution: Cannot Reproduce
Haven't seen this since, closing...
> NPE in Storage$StorageDirectory#unlock()
>
>
> Key: HDFS-9590
> URL: https://issues.apache.org/jira/browse/HDFS-9590
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Xiao Chen
>Assignee: Xiao Chen
> Attachments: HDFS-9590.01.patch
>
>
> The code looks to be possible to have race conditions in multiple-threaded
> runs.
> {code}
> public void unlock() throws IOException {
> if (this.lock == null)
> return;
> this.lock.release();
> lock.channel().close();
> lock = null;
> }
> {code}
> This is called in a handful of places, and I don't see any protection. Shall
> we add some synchronization mechanism? Not sure if I missed any design
> assumptions here.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12121) Audit logging of unsuccessful should not be done while holding fslock.
Rushabh S Shah created HDFS-12121:
-
Summary: Audit logging of unsuccessful should not be done while
holding fslock.
Key: HDFS-12121
URL: https://issues.apache.org/jira/browse/HDFS-12121
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Rushabh S Shah
Assignee: Rushabh S Shah
Currently we are audit logging the unsuccessful calls while holding fslock.
Following is just one example.
{code:title=FSNamesystem.java|borderStyle=solid}
HdfsFileStatus getFileInfo(final String src, boolean resolveLink)
throws IOException {
final String operationName = "getfileinfo";
checkOperation(OperationCategory.READ);
HdfsFileStatus stat = null;
readLock();
try {
checkOperation(OperationCategory.READ);
stat = FSDirStatAndListingOp.getFileInfo(dir, src, resolveLink);
} catch (AccessControlException e) {
logAuditEvent(false, operationName, src); //< audit logging while
holding fs read lock.
throw e;
} finally {
readUnlock(operationName);
}
logAuditEvent(true, operationName, src);
return stat;
}
{code}
We should avoid doing this.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-12124) Namenode should throw a RetryStartFileException exception instead of making a synchronous call to kms incase of cache miss.
Rushabh S Shah created HDFS-12124: - Summary: Namenode should throw a RetryStartFileException exception instead of making a synchronous call to kms incase of cache miss. Key: HDFS-12124 URL: https://issues.apache.org/jira/browse/HDFS-12124 Project: Hadoop HDFS Issue Type: Bug Components: kms Affects Versions: 2.7.0 Reporter: Rushabh S Shah Assignee: Rushabh S Shah The namenode should throw a RetryStartFileException exception instead of making a synchronous call to kms incase of cache miss. Other kms issues (like kms getting stalled, issues talking to underlying kms) will block all the namenode handler threads. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/372/ [Jul 10, 2017 9:34:58 PM] (Arun Suresh) YARN-6776. Refactor ApplicaitonMasterService to move actual processing [Jul 11, 2017 12:48:27 AM] (jitendra) HADOOP-10829. Iteration on CredentialProviderFactory.serviceLoader is [Jul 11, 2017 4:30:13 AM] (aajisaka) HADOOP-14638. Replace commons-logging APIs with slf4j in StreamPumper. [Jul 11, 2017 9:22:44 AM] (sunilg) YARN-6714. IllegalStateException while handling APP_ATTEMPT_REMOVED [Jul 11, 2017 12:40:11 PM] (yqlin) HDFS-12085. Reconfigure namenode heartbeat interval fails if the -1 overall The following subsystems voted -1: compile mvninstall unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestEncryptionZonesWithKMS hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy hadoop.hdfs.server.namenode.TestDecommissioningStatus hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.hdfs.server.datanode.fsdataset.impl.TestSpaceReservation hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.containermanager.TestContainerManager hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService hadoop.yarn.server.TestDiskFailures hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.api.impl.TestNMClient hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService hadoop.mapreduce.TestMRJobClient hadoop.tools.TestDistCpSystem Timed out junit tests : org.apache.hadoop.hdfs.TestLeaseRecovery2 org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA mvninstall: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/372/artifact/out/patch-mvninstall-root.txt [620K] compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/372/artifact/out/patch-compile-root.txt [20K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/372/artifact/out/patch-compile-root.txt [20K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/372/artifact/out/patch-compile-root.txt [20K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/372/artifact/out/patch-unit-hadoop-assemblies.txt [4.0K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/372/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [688K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/372/artifact/out/patch-unit-hadoop-yarn-p
[jira] [Created] (HDFS-12125) Document the missing -removePolicy command of ec
Wenxin He created HDFS-12125: Summary: Document the missing -removePolicy command of ec Key: HDFS-12125 URL: https://issues.apache.org/jira/browse/HDFS-12125 Project: Hadoop HDFS Issue Type: Bug Components: documentation, erasure-coding Affects Versions: 3.0.0-alpha4 Reporter: Wenxin He Assignee: Wenxin He Document the missing command -removePolicy in HDFSErasureCoding.md and HDFSCommands.md and regroup the ec commands to improve the user experience. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-12109) "fs" java.net.UnknownHostException when HA NameNode is used
[ https://issues.apache.org/jira/browse/HDFS-12109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luigi Di Fraia resolved HDFS-12109. --- Resolution: Not A Bug > "fs" java.net.UnknownHostException when HA NameNode is used > --- > > Key: HDFS-12109 > URL: https://issues.apache.org/jira/browse/HDFS-12109 > Project: Hadoop HDFS > Issue Type: Bug > Components: fs >Affects Versions: 2.8.0 > Environment: [hadoop@namenode01 ~]$ cat /etc/redhat-release > CentOS Linux release 7.3.1611 (Core) > [hadoop@namenode01 ~]$ uname -a > Linux namenode01 3.10.0-514.10.2.el7.x86_64 #1 SMP Fri Mar 3 00:04:05 UTC > 2017 x86_64 x86_64 x86_64 GNU/Linux > [hadoop@namenode01 ~]$ java -version > java version "1.8.0_131" > Java(TM) SE Runtime Environment (build 1.8.0_131-b11) > Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode) >Reporter: Luigi Di Fraia > > After setting up an HA NameNode configuration, the following invocation of > "fs" fails: > [hadoop@namenode01 ~]$ /usr/local/hadoop/bin/hdfs dfs -ls / > -ls: java.net.UnknownHostException: saccluster > It works if properties are defined as per below: > /usr/local/hadoop/bin/hdfs dfs -Ddfs.nameservices=saccluster > -Ddfs.client.failover.proxy.provider.saccluster=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider > -Ddfs.ha.namenodes.saccluster=namenode01,namenode02 > -Ddfs.namenode.rpc-address.saccluster.namenode01=namenode01:8020 > -Ddfs.namenode.rpc-address.saccluster.namenode02=namenode02:8020 -ls / > These properties are defined in /usr/local/hadoop/etc/hadoop/hdfs-site.xml as > per below: > > dfs.nameservices > saccluster > > > dfs.ha.namenodes.saccluster > namenode01,namenode02 > > > dfs.namenode.rpc-address.saccluster.namenode01 > namenode01:8020 > > > dfs.namenode.rpc-address.saccluster.namenode02 > namenode02:8020 > > > dfs.namenode.http-address.saccluster.namenode01 > namenode01:50070 > > > dfs.namenode.http-address.saccluster.namenode02 > namenode02:50070 > > > dfs.namenode.shared.edits.dir > > qjournal://namenode01:8485;namenode02:8485;datanode01:8485/saccluster > > > dfs.client.failover.proxy.provider.mycluster > > org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider > > In /usr/local/hadoop/etc/hadoop/core-site.xml the default FS is defined as > per below: > > fs.defaultFS > hdfs://saccluster > > In /usr/local/hadoop/etc/hadoop/hadoop-env.sh the following export is defined: > export HADOOP_CONF_DIR="/usr/local/hadoop/etc/hadoop" > Is "fs" trying to read these properties from somewhere else, such as a > separate client configuration file? > Apologies if I am missing something obvious here. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
