RE: Listing large directories via WebHDFS

JFI, HADOOP-12502 introduced RemoteIterator at client side which is not committed through. --Brahma Reddy Battula -Original Message- From: Andrew Wang [mailto:andrew.w...@cloudera.com] Sent: 20 October 2016 05:48 To: Zhe Zhang Cc: Xiao Chen; hdfs-dev@hadoop.apache.org Subject: Re: Lis

[jira] [Created] (HDFS-11039) Expose more configuration properties to hdfs-default.xml

Yi Liu created HDFS-11039: - Summary: Expose more configuration properties to hdfs-default.xml Key: HDFS-11039 URL: https://issues.apache.org/jira/browse/HDFS-11039 Project: Hadoop HDFS Issue Type: Im

[jira] [Created] (HDFS-11038) DiskBalancer: support running multiple commands under one setup of disk balancer

Xiaobing Zhou created HDFS-11038: Summary: DiskBalancer: support running multiple commands under one setup of disk balancer Key: HDFS-11038 URL: https://issues.apache.org/jira/browse/HDFS-11038 Projec

[jira] [Created] (HDFS-11037) DiskBalancer: redirect stdout/stderr stream for easy tests

Xiaobing Zhou created HDFS-11037: Summary: DiskBalancer: redirect stdout/stderr stream for easy tests Key: HDFS-11037 URL: https://issues.apache.org/jira/browse/HDFS-11037 Project: Hadoop HDFS

[jira] [Created] (HDFS-11036) Ozone : reuse Xceiver connection

Chen Liang created HDFS-11036: - Summary: Ozone : reuse Xceiver connection Key: HDFS-11036 URL: https://issues.apache.org/jira/browse/HDFS-11036 Project: Hadoop HDFS Issue Type: Sub-task

Re: Listing large directories via WebHDFS

If the issue is just "hadoop fs -ls -R /", one thing we can look into is making the Globber use the listStatus API that returns a RemoteIterator rather than a FileStatus[]. That'll use the client-side pagination Xiao mentioned for WebHDFS/HttpFS (though this is currently not in a 2.x release). The

Re: Listing large directories via WebHDFS

Thanks Xiao! Seems like server-side throttling are still vulnerable to abusing users issuing large listing requests. Once such a request is scheduled, it will keep listing potentially millions of files without having to go through IPC/RPC queue again. It does have to compete for fsn lock though, t

Re: Listing large directories via WebHDFS

Hi Zhe, Per my understanding, the runner in webhdfs goes to NamenodeWebHdfsMethods , w

Listing large directories via WebHDFS

Hi, The regular HDFS client (DistributedFileSystem) throttles the workload of listing large directories by dividing the work into batches, something like below: {code} // fetch the first batch of entries in the directory DirectoryListing thisListing = dfs.listPaths( src, HdfsFileSt

Apache Hadoop qbt Report: trunk+JDK8 on Linux/ppc64le

For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-ppc/129/ [Oct 18, 2016 6:02:52 PM] (varunsaxena) YARN-5743. [Atsv2] Publish queue name and RMAppMetrics to ATS (Rohith [Oct 18, 2016 6:06:47 PM] (xgong) YARN-5718. TimelineClient (and other places in YARN) shouldn'

[jira] [Created] (HDFS-11035) Better documentation for maintenace mode and upgrade domain

Wei-Chiu Chuang created HDFS-11035: -- Summary: Better documentation for maintenace mode and upgrade domain Key: HDFS-11035 URL: https://issues.apache.org/jira/browse/HDFS-11035 Project: Hadoop HDFS

[jira] [Created] (HDFS-11034) Provide a command line tool to clear decommissioned DataNode information from the NameNode without restarting.

Chris Nauroth created HDFS-11034: Summary: Provide a command line tool to clear decommissioned DataNode information from the NameNode without restarting. Key: HDFS-11034 URL: https://issues.apache.org/jira/browse/

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/199/ [Oct 18, 2016 8:51:08 AM] (kai.zheng) HDFS-10920. TestStorageMover#testNoSpaceDisk is failing intermittently. [Oct 18, 2016 6:02:52 PM] (varunsaxena) YARN-5743. [Atsv2] Publish queue name and RMAppMetrics

[DISCUSS] HADOOP-13603 - Remove package line length checkstyle rule

All, I would like to start a discussion on the possibility of removing the package line length checkstyle rule (HADOOP-13603 ). While working on various aspects of YARN container runtimes, all of my pre-commit jobs would fail as the package line

[jira] [Created] (HDFS-11033) Add documents for native raw eraure coder in XOR codes

SammiChen created HDFS-11033: Summary: Add documents for native raw eraure coder in XOR codes Key: HDFS-11033 URL: https://issues.apache.org/jira/browse/HDFS-11033 Project: Hadoop HDFS Issue Type

[jira] [Created] (HDFS-11032) [SPS]: Handling of block movement failure at the coordinator datanode

Rakesh R created HDFS-11032: --- Summary: [SPS]: Handling of block movement failure at the coordinator datanode Key: HDFS-11032 URL: https://issues.apache.org/jira/browse/HDFS-11032 Project: Hadoop HDFS