[jira] [Created] (HDFS-11165) ErasureCodeBenchmarkThroughput report "out of heap memory" during execution
SammiChen created HDFS-11165: Summary: ErasureCodeBenchmarkThroughput report "out of heap memory" during execution Key: HDFS-11165 URL: https://issues.apache.org/jira/browse/HDFS-11165 Project: Hadoop HDFS Issue Type: Bug Reporter: SammiChen When run hadoop jar ./share/hadoop/hdfs/hadoop-hdfs-3.0.0-alpha2-SNAPSHOT-tests.jar org.apache.hadoop.hdfs.ErasureCodeBenchmarkThroughput read 5000 ec 5 stf, following error happens, Exception in thread "main" java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hadoop.hdfs.ErasureCodeBenchmarkThroughput.doBenchmark(ErasureCodeBenchmarkThroughput.java:136) at org.apache.hadoop.hdfs.ErasureCodeBenchmarkThroughput.benchmark(ErasureCodeBenchmarkThroughput.java:161) at org.apache.hadoop.hdfs.ErasureCodeBenchmarkThroughput.run(ErasureCodeBenchmarkThroughput.java:261) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76) at org.apache.hadoop.hdfs.ErasureCodeBenchmarkThroughput.main(ErasureCodeBenchmarkThroughput.java:422) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:239) at org.apache.hadoop.util.RunJar.main(RunJar.java:153) Caused by: java.lang.OutOfMemoryError: Java heap space -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11166) Doc for GET_BLOCK_LOCATIONS is missing in webhdfs API
Weiwei Yang created HDFS-11166: -- Summary: Doc for GET_BLOCK_LOCATIONS is missing in webhdfs API Key: HDFS-11166 URL: https://issues.apache.org/jira/browse/HDFS-11166 Project: Hadoop HDFS Issue Type: Bug Components: documentation, webhdfs Affects Versions: 2.7.3 Reporter: Weiwei Yang Assignee: Weiwei Yang Add GET_BLOCK_LOCATIONS in webhdfs API docs, currently it is missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11167) IBM java can not start secure datanode with HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server"
Pan Yuxuan created HDFS-11167:
-
Summary: IBM java can not start secure datanode with
HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server"
Key: HDFS-11167
URL: https://issues.apache.org/jira/browse/HDFS-11167
Project: Hadoop HDFS
Issue Type: Bug
Components: scripts
Affects Versions: 3.0.0-alpha1
Environment: IBM PowerPC
Reporter: Pan Yuxuan
Assignee: Pan Yuxuan
When we run secure datanode on IBM PowerPC with IBM JDK, the jsvc wrong with
error
{{noformat}}
jsvc error: Invalid JVM name specified server
{{noformat}}
This is because of the HDFS_DATANODE_SECURE_EXTRA_OPTS "-jvm server".
For IBM JDK it is enough to run it without -jvm server.
So I think we can check if IBM jdk in hdfs-config.sh before setting
HDFS_DATANODE_SECURE_EXTRA_OPTS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/ [Nov 21, 2016 2:04:52 PM] (brahma) HDFS-11101. TestDFSShell#testMoveWithTargetPortEmpty fails [Nov 21, 2016 4:13:24 PM] (kihwal) HDFS-10966. Enhance Dispatcher logic on deciding when to give up a [Nov 21, 2016 4:18:17 PM] (stevel) HADOOP-12705 Upgrade Jackson 2.2.3 to 2.7.8 [Nov 21, 2016 4:36:08 PM] (jlowe) YARN-5859. [Nov 21, 2016 4:40:57 PM] (rohithsharmaks) MAPREDUCE-6793. io.sort.factor code default and mapred-default.xml [Nov 22, 2016 1:20:31 AM] (aajisaka) YARN-5713. Update jackson from 1.9.13 to 2.x in hadoop-yarn. -1 overall The following subsystems voted -1: asflicense findbugs unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped hadoop.hdfs.server.datanode.checker.TestThrottledAsyncChecker hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/diff-compile-javac-root.txt [172K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/diff-patch-shellcheck.txt [28K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/whitespace-tabs.txt [1.3M] findbugs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [4.0K] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [264K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [316K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt [124K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/233/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/163/ [Nov 21, 2016 2:04:52 PM] (brahma) HDFS-11101. TestDFSShell#testMoveWithTargetPortEmpty fails [Nov 21, 2016 4:13:24 PM] (kihwal) HDFS-10966. Enhance Dispatcher logic on deciding when to give up a [Nov 21, 2016 4:18:17 PM] (stevel) HADOOP-12705 Upgrade Jackson 2.2.3 to 2.7.8 [Nov 21, 2016 4:36:08 PM] (jlowe) YARN-5859. [Nov 21, 2016 4:40:57 PM] (rohithsharmaks) MAPREDUCE-6793. io.sort.factor code default and mapred-default.xml [Nov 22, 2016 1:20:31 AM] (aajisaka) YARN-5713. Update jackson from 1.9.13 to 2.x in hadoop-yarn. [Nov 22, 2016 9:19:15 AM] (rohithsharmaks) YARN-5865. Retrospect updateApplicationPriority api to handle state -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.TestRollingUpgrade hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/163/artifact/out/patch-compile-root.txt [176K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/163/artifact/out/patch-compile-root.txt [176K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/163/artifact/out/patch-compile-root.txt [176K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/163/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [200K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/163/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/163/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/163/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt [20K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/163/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [88K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/163/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_
[jira] [Created] (HDFS-11168) Bump Netty 4 version
Tsz Wo Nicholas Sze created HDFS-11168: -- Summary: Bump Netty 4 version Key: HDFS-11168 URL: https://issues.apache.org/jira/browse/HDFS-11168 Project: Hadoop HDFS Issue Type: Bug Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze The current Netty 4 version is 4.1.0.Beta5. We should bump it to a non-beta version. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11169) GetBlockLocations returns a block when offset > filesize and file only has 1 block
David Tucker created HDFS-11169: --- Summary: GetBlockLocations returns a block when offset > filesize and file only has 1 block Key: HDFS-11169 URL: https://issues.apache.org/jira/browse/HDFS-11169 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.7.1 Environment: HDP 2.5, Ambari 2.4 Reporter: David Tucker Priority: Minor Start with a fresh deployment of HDFS. 1. Create a file. 2. AddBlock the file. 3. Call GetBlockLocations. Expectation: 0 blocks are returned because the only added block is incomplete. Observation: 1 block is returned. This only seems to occur when 1 block is in play (i.e. if you write a block and call AddBlock again, GetBlockLocations seems to behave as expected). This seems to be related to HDFS-513. I suspect the following line needs revision: https://github.com/apache/hadoop/blob/4484b48498b2ab2a40a404c487c7a4e875df10dc/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L1062 I believe it should be >= instead of >: if (nrBlocks >= 0 && curBlk == nrBlocks) // offset >= end of file -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
[jira] [Created] (HDFS-11170) Refactor create API in filesystem public class to support assign parameter through builder
SammiChen created HDFS-11170: Summary: Refactor create API in filesystem public class to support assign parameter through builder Key: HDFS-11170 URL: https://issues.apache.org/jira/browse/HDFS-11170 Project: Hadoop HDFS Issue Type: Improvement Reporter: SammiChen Assignee: SammiChen FileSystem class supports multiple create functions to help user create file. Some create functions has many parameters, it's hard for user to exactly remember these parameters and their orders. This task is to add builder based create functions to help user more easily create file. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
