[jira] [Created] (HDFS-10884) [SPS]: Add block movement tracker to track the completion of block movement future tasks at DN
Rakesh R created HDFS-10884: --- Summary: [SPS]: Add block movement tracker to track the completion of block movement future tasks at DN Key: HDFS-10884 URL: https://issues.apache.org/jira/browse/HDFS-10884 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Rakesh R Assignee: Rakesh R Presently [StoragePolicySatisfyWorker#processBlockMovingTasks()|https://github.com/apache/hadoop/blob/HDFS-10285/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/StoragePolicySatisfyWorker.java#L147] function act as a blocking call. The idea of this jira is to implement a mechanism to track these movements async so that would allow other movement while processing the previous one. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10885) Mover tool should not be allowed to run when Storage Policy Satisfier is on
Wei Zhou created HDFS-10885: --- Summary: Mover tool should not be allowed to run when Storage Policy Satisfier is on Key: HDFS-10885 URL: https://issues.apache.org/jira/browse/HDFS-10885 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Wei Zhou Assignee: Wei Zhou These two can not work at the same time to avoid conflicts and fight with each other. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/ [Sep 20, 2016 7:03:31 AM] (jianhe) YARN-3140. Improve locks in AbstractCSQueue/LeafQueue/ParentQueue. [Sep 21, 2016 1:34:48 PM] (kai.zheng) HDFS-10861. Refactor StripeReaders and use ECChunk version decode API. [Sep 20, 2016 2:15:06 PM] (jlowe) YARN-5655. TestContainerManagerSecurity#testNMTokens is asserting. [Sep 20, 2016 6:21:01 PM] (wangda) Addendum patch for fix javadocs failure which is caused by YARN-3141. [Sep 20, 2016 7:27:17 PM] (Arun Suresh) YARN-5656. Fix ReservationACLsTestBase. (Sean Po via asuresh) [Sep 20, 2016 8:20:01 PM] (liuml07) HADOOP-13601. Fix a log message typo in [Sep 20, 2016 11:56:52 PM] (xiao) HDFS-10879. TestEncryptionZonesWithKMS#testReadWrite fails [Sep 21, 2016 12:20:50 AM] (wangda) YARN-4591. YARN Web UIs should provide a robots.txt. (Sidharta Seethana [Sep 21, 2016 1:35:25 AM] (iwasakims) HDFS-9333. Some tests using MiniDFSCluster errored complaining port in -1 overall The following subsystems voted -1: asflicense mvnsite unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.security.authentication.util.TestZKSignerSecretProvider hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/diff-compile-javac-root.txt [168K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/diff-checkstyle-root.txt [16M] mvnsite: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/patch-mvnsite-root.txt [112K] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/diff-patch-pylint.txt [16K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/diff-patch-shellcheck.txt [20K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/diff-patch-shelldocs.txt [12K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/patch-unit-hadoop-common-project_hadoop-auth.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [148K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [268K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/171/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
HADOOP-13636 and io.bytes.per.checksum
I'm getting told off for using the deprecated option: io.bytes.per.checksum https://issues.apache.org/jira/browse/HADOOP-13636 Except: I'm not. FileSystem.getServerDefaults() is, which is used by Trash to work out where to delete things. It strikes me that the system is inconsitent: HdfsConfiguration is deprecating a property that everything else is happy to use; I see it in four places in production, and various tests, plus core-default.xml Is it really deprecated? If so, are there any volunteers to remove it from the codebase, while pulling up the default value into core-default? otherwise: how about the complaint is turned off?
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/101/ [Sep 21, 2016 1:34:48 PM] (kai.zheng) HDFS-10861. Refactor StripeReaders and use ECChunk version decode API. [Sep 20, 2016 2:15:06 PM] (jlowe) YARN-5655. TestContainerManagerSecurity#testNMTokens is asserting. [Sep 20, 2016 6:21:01 PM] (wangda) Addendum patch for fix javadocs failure which is caused by YARN-3141. [Sep 20, 2016 7:27:17 PM] (Arun Suresh) YARN-5656. Fix ReservationACLsTestBase. (Sean Po via asuresh) [Sep 20, 2016 8:20:01 PM] (liuml07) HADOOP-13601. Fix a log message typo in [Sep 20, 2016 11:56:52 PM] (xiao) HDFS-10879. TestEncryptionZonesWithKMS#testReadWrite fails [Sep 21, 2016 12:20:50 AM] (wangda) YARN-4591. YARN Web UIs should provide a robots.txt. (Sidharta Seethana [Sep 21, 2016 1:35:25 AM] (iwasakims) HDFS-9333. Some tests using MiniDFSCluster errored complaining port in -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.ipc.TestRPCWaitForProxy hadoop.hdfs.TestWriteReadStripedFile hadoop.hdfs.server.namenode.TestDecommissioningStatus hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.client.api.impl.TestNMClient hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache org.apache.hadoop.hdfs.TestErasureCodeBenchmarkThroughput org.apache.hadoop.mapred.TestMRIntermediateDataEncryption org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers org.apache.hadoop.mapred.TestMROpportunisticMaps compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/101/artifact/out/patch-compile-root.txt [308K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/101/artifact/out/patch-compile-root.txt [308K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/101/artifact/out/patch-compile-root.txt [308K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/101/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt [120K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/101/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [200K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/101/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [52
[jira] [Created] (HDFS-10886) Replace "fs.default.name" with "fs.defaultFS" in viewfs
Brahma Reddy Battula created HDFS-10886: --- Summary: Replace "fs.default.name" with "fs.defaultFS" in viewfs Key: HDFS-10886 URL: https://issues.apache.org/jira/browse/HDFS-10886 Project: Hadoop HDFS Issue Type: Bug Components: documentation, federation Reporter: Brahma Reddy Battula Priority: Minor As we given two sections,update in *New World – Federation and ViewFs* not in *The Old World (Prior to Federation)* section. "fs.default.name" is deprecated, we should use "fs.defaultFS" -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-10365) FullBlockReports retransmission delays NN startup time in large cluster.
[
https://issues.apache.org/jira/browse/HDFS-10365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chackaravarthy resolved HDFS-10365.
---
Resolution: Not A Bug
Closing this jira as the issue was resolved after increasing
{{dfs.blockreport.initialDelay}} value. Thanks for all the inputs.
> FullBlockReports retransmission delays NN startup time in large cluster.
>
>
> Key: HDFS-10365
> URL: https://issues.apache.org/jira/browse/HDFS-10365
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 2.6.0
> Environment: version - hadoop-2.6.0 (hdp-2.2)
> DN - 1200 nodes
>Reporter: Chackaravarthy
>Priority: Critical
>
> Whenever NN is restarted, it takes huge time for NN to come back to stable
> state. i.e. Last contact time remains more than 1 or 2 mins continuously for
> around 3 to 4 hours. This is mainly because most of the DN's getting timeout
> (60s) in blockReport (FBR) rpc call and then it keep sending FBR again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-10887) Provide admin/debug tool to dump block map
Yongjun Zhang created HDFS-10887:
Summary: Provide admin/debug tool to dump block map
Key: HDFS-10887
URL: https://issues.apache.org/jira/browse/HDFS-10887
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs, namenode
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
>From time to time, when NN restarts, we see
{code}
"The reported blocks X needs additional Y blocks to reach the threshold 0.9990
of total blocks Z. Safe mode will be turned off automatically.
{code}
We'd wonder what these blocks that still need block reports are, and what DNs
they could possibly be located, what happened to these DNs.
This jira to to propose a new admin or debug tool to dump the block map info
with the blocks that have fewer than minRepl replicas.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
