[jira] [Created] (HDFS-11263) ClassCastException when we use Bzipcodec for Fsimage compression
Brahma Reddy Battula created HDFS-11263: --- Summary: ClassCastException when we use Bzipcodec for Fsimage compression Key: HDFS-11263 URL: https://issues.apache.org/jira/browse/HDFS-11263 Project: Hadoop HDFS Issue Type: Bug Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Critical *Trace* java.lang.ClassCastException: org.apache.hadoop.io.compress.BZip2Codec$BZip2CompressionOutputStream cannot be cast to org.apache.hadoop.io.compress.CompressorStream at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Saver.flushSectionOutputStream(FSImageFormatProtobuf.java:420) at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Saver.commitSection(FSImageFormatProtobuf.java:405) at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Saver.saveNameSystemSection(FSImageFormatProtobuf.java:583) at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Saver.saveInternal(FSImageFormatProtobuf.java:494) at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Saver.save(FSImageFormatProtobuf.java:431) at org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:913) at org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:964) at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/ [Dec 19, 2016 6:14:03 AM] (sunilg) YARN-5650. Render Application Timeout value in web UI. Contributed by -1 overall The following subsystems voted -1: asflicense unit The following subsystems voted -1 but were configured to be filtered/ignored: cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/diff-compile-cc-root.txt [4.0K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/diff-compile-javac-root.txt [164K] checkstyle: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/diff-checkstyle-root.txt [16M] pylint: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/diff-patch-pylint.txt [20K] shellcheck: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/diff-patch-shellcheck.txt [28K] shelldocs: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/diff-patch-shelldocs.txt [16K] whitespace: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/whitespace-eol.txt [11M] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/whitespace-tabs.txt [1.3M] javadoc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/diff-javadoc-javadoc-root.txt [2.2M] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [148K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [12K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [56K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [316K] asflicense: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/260/artifact/out/patch-asflicense-problems.txt [4.0K] Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/ [Dec 19, 2016 6:14:03 AM] (sunilg) YARN-5650. Render Application Timeout value in web UI. Contributed by -1 overall The following subsystems voted -1: compile unit The following subsystems voted -1 but were configured to be filtered/ignored: cc javac The following subsystems are considered long running: (runtime bigger than 1h 0m 0s) unit Specific tests: Failed junit tests : hadoop.hdfs.TestEncryptedTransfer hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer hadoop.hdfs.web.TestWebHdfsTimeouts hadoop.yarn.server.nodemanager.recovery.TestNMLeveldbStateStoreService hadoop.yarn.server.nodemanager.TestNodeManagerShutdown hadoop.yarn.server.timeline.TestRollingLevelDB hadoop.yarn.server.timeline.TestTimelineDataManager hadoop.yarn.server.timeline.TestLeveldbTimelineStore hadoop.yarn.server.timeline.webapp.TestTimelineWebServices hadoop.yarn.server.timeline.recovery.TestLeveldbTimelineStateStore hadoop.yarn.server.timeline.TestRollingLevelDBTimelineStore hadoop.yarn.server.applicationhistoryservice.TestApplicationHistoryServer hadoop.yarn.server.timelineservice.storage.common.TestRowKeys hadoop.yarn.server.timelineservice.storage.common.TestKeyConverters hadoop.yarn.server.timelineservice.storage.common.TestSeparator hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore hadoop.yarn.server.resourcemanager.TestRMRestart hadoop.yarn.server.resourcemanager.TestResourceTrackerService hadoop.yarn.server.TestMiniYarnClusterNodeUtilization hadoop.yarn.server.TestContainerManagerSecurity hadoop.yarn.server.timeline.TestLevelDBCacheTimelineStore hadoop.yarn.server.timeline.TestOverrideTimelineStoreYarnClient hadoop.yarn.server.timeline.TestEntityGroupFSTimelineStore hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRunCompaction hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowRun hadoop.yarn.server.timelineservice.storage.TestPhoenixOfflineAggregationWriterImpl hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage hadoop.yarn.server.timelineservice.storage.flow.TestHBaseStorageFlowActivity hadoop.yarn.applications.distributedshell.TestDistributedShell hadoop.mapred.TestShuffleHandler hadoop.mapreduce.v2.hs.TestHistoryServerLeveldbStateStoreService Timed out junit tests : org.apache.hadoop.hdfs.server.datanode.TestFsDatasetCache compile: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/artifact/out/patch-compile-root.txt [156K] cc: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/artifact/out/patch-compile-root.txt [156K] javac: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/artifact/out/patch-compile-root.txt [156K] unit: https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt [200K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt [52K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt [24K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt [76K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt [316K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt [28K] https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/191/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase-tests.txt [20K]
Unnecessary FsDatasetImpl lock in FsVolumeImpl#getDfsUsed?
Hello, I am trying to understand the lock in FsVolumeImpl#getDfsUsed(). As many of you know, a busy datanode can occasionally block on the call to FsVolumeImpl#getDfsUsed(). This method is used in a number of places, especially used in storage reports in heartbeat messages. There have been some attempt to make this situation better, for example HDFS-7999 stops a writer if it blocks FsDatasetImpl for too long, preventing heartbeat messages. However, when I looked at the code closer, I started to wonder if FsVolumeImpl#getDfsUsed() needs FsDatasetImpl lock at all. Because the BlockPoolSlice hash table bpSlices is a ConcurrentHashMap so iteration over this object doesn’t need a lock. It also seems to be an overkill because the operation does not modify BlockPoolSlice object at all. I do not see extra benefits locking on FsDatasetImpl object. Thoughts? Wei-Chiu Chuang -- A very happy Hadoop contributor
[jira] [Created] (HDFS-11264) [SPS]: Double checks to ensure that SPS/Mover are not running together
Wei Zhou created HDFS-11264: --- Summary: [SPS]: Double checks to ensure that SPS/Mover are not running together Key: HDFS-11264 URL: https://issues.apache.org/jira/browse/HDFS-11264 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Wei Zhou Assignee: Wei Zhou As discussed in HDFS-10885, double checks needed to insure SPS/Mover not running together, otherwise it may cause some issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
