[jira] [Created] (HDFS-15219) DFS Client will stuck when ResponseProcessor.run throw Error

2020-03-11 Thread zhengchenyu (Jira)
zhengchenyu created HDFS-15219: -- Summary: DFS Client will stuck when ResponseProcessor.run throw Error Key: HDFS-15219 URL: https://issues.apache.org/jira/browse/HDFS-15219 Project: Hadoop HDFS

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2020-03-11 Thread Apache Jenkins Server
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1436/ [Mar 11, 2020 8:39:10 PM] (github) Hadoop 16890. Change in expiry calculation for MSI token provider. - To unsubscribe, e-mail: hdfs-dev

Re: [DISCUSS] Accelerate Hadoop dependency updates

2020-03-11 Thread Wei-Chiu Chuang
FWIW we are updating guava in Spark and Hive at Cloudera. Don't know which Apache version are they going to land, but we'll upstream them for sure. The guava change is debatable. It's not as critical as others. There are critical vulnerabilities in other dependencies that we have no way but to upd

Re: [DISCUSS] Accelerate Hadoop dependency updates

2020-03-11 Thread Igor Dvorzhak
Generally I'm for updating dependencies, but I think that Hadoop should stick with semantic versioning and do not make major and minor dependency updates in subminor releases. For example, Hadoop 3.2.1 updated Guava to 27.0-jre, and because of this Spark 3.0 stuck with Hadoop 3.2.0 - they use Hiv

[VOTE] Release Apache Hadoop Thirdparty 1.0.0 - RC1

2020-03-11 Thread Vinayakumar B
Hi folks, Thanks to everyone's help on this release. I have re-created a release candidate (RC1) for Apache Hadoop Thirdparty 1.0.0. RC Release artifacts are available at : http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC1/ Maven artifacts are available in staging repo:

Re: data node reads blocks sequentially from disk

2020-03-11 Thread Todd Lipcon
We also issue explicit readahead via fadvise since 2011 or so, so the typical io sizes hitting the device are large enough to max out the throughput, at least for typical spinning disks. Todd On Wed, Mar 11, 2020, 9:46 AM Kihwal Lee wrote: > When Datanode was initially designed, Linux AIO was s

Re: [VOTE] Release Apache Hadoop Thirdparty 1.0.0

2020-03-11 Thread Vinayakumar B
Hello everyone. RC0 has been canceled. Since issues mentioned above are already fixed now, soon I will create RC1 and re-create VOTE thread. Thanks for trying out RC, -Vinay On Tue, Mar 3, 2020 at 11:41 AM Ayush Saxena wrote: > Hi Vinay > Thanx for driving the release. > Verified checksums a

Re: data node reads blocks sequentially from disk

2020-03-11 Thread Kihwal Lee
When Datanode was initially designed, Linux AIO was still early in its adoption. Kernel support was there and the libraries were almost there. No java support, of course. We would have to write a lot of native code for it and use JNI. Also, AIO means bypassing kernel page cache since you are doing

Re: data node reads blocks sequentially from disk

2020-03-11 Thread Wei-Chiu Chuang
Hi David, We talked a bit about a similar topic on DataNode sockets a while back. Any feedback on the DataNode disk access? On Thu, Mar 5, 2020 at 4:16 PM Mania Abdi wrote: > Hello everyone > > I have a question regarding HDFS, data node code version 2.7.2. I have > posted my question as Jira is

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2020-03-11 Thread Apache Jenkins Server
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1435/ [Mar 10, 2020 2:17:45 PM] (ericp) YARN-942. [Mar 10, 2020 3:07:46 PM] (snemeth) YARN-10168. FS-CS Converter: tool doesn't handle min/max resource [Mar 10, 2020 3:35:04 PM] (snemeth) YARN-10002. Code cleanu

Apache Hadoop qbt Report: branch2.10+JDK7 on Linux/x86

2020-03-11 Thread Apache Jenkins Server
For more details, see https://builds.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86/621/ [Mar 10, 2020 3:48:49 PM] (ericp) YARN-942. -1 overall The following subsystems voted -1: asflicense findbugs hadolint pathlen unit xml The following subsystems voted -1 but were configured