Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

2020-10-07 Thread Karthik Palanisamy
Thank you Arpit Agarwal for notifying me on this user mail. Yes, heap pressure is introduced due to block-layout migration. Right, high heap usage is only during the upgrade, and once the upgrade is done then heap usage back to normal. Have experienced this issue from many clusters(20+) but only

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2020-10-07 Thread Apache Jenkins Server
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/288/ [Oct 6, 2020 4:07:54 PM] (noreply) HADOOP-17125. Use snappy-java in SnappyCodec (#2297) [Oct 6, 2020 6:18:08 PM] (Jim Brennan) YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type

[jira] [Resolved] (HDFS-15253) Set default throttle value on dfs.image.transfer.bandwidthPerSec

2020-10-07 Thread Rakesh Radhakrishnan (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh Radhakrishnan resolved HDFS-15253. - Fix Version/s: 3.4.0 Resolution: Fixed > Set default throttle value on dfs

Re: [E] Re: Increased DN heap usage during Hadoop 3 upgrade

2020-10-07 Thread Kihwal Lee
We haven't experienced anything like that up to 2.8. We are still in the process of stabilizing 2.10 as we upgrade some of the bigger clusters. We will know soon how 2.10 datanodes behave under heavy load and storage utilization. If you are seeing a significant change, it might be something post-2

reviewers needed: HADOOP-16830 Add public IOStatistics API

2020-10-07 Thread Steve Loughran
Hi, Can I get some reviews of this PR https://github.com/apache/hadoop/pull/2323 It adds a new API, IOStatisticsSource, for any class to act as a source of a static or dynamic IOStatistics set of counters/gauges/min/max/mean stats The intent is to allow applications to collect statistics on stre

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2020-10-07 Thread Apache Jenkins Server
For more details, see https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/79/ [Oct 6, 2020 6:46:08 PM] (Jim Brennan) YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne -1 overall The following subsystems vote