Re: [DISCUSS] Merge Storage Policy Satisfier (SPS) [HDFS-10285] feature branch to trunk

2017-08-18 Thread Gangumalla, Uma
Hi Andrew, >Great to hear. It'd be nice to define which use cases are met by the current >version of SPS, and which will be handled after the merge. After the discussions in JIRA, we planned to support recursive API as well. The primary use cases we planned was for Hbase. Please check next point

[jira] [Resolved] (HDFS-12244) Ozone: the static cache provided by ContainerCache does not work in Unit tests

2017-08-18 Thread Mukul Kumar Singh (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mukul Kumar Singh resolved HDFS-12244. -- Resolution: Duplicate Assignee: Mukul Kumar Singh (was: Anu Engineer) Fi

Re: Branch merges and 3.0.0-beta1 scope

2017-08-18 Thread Vinod Kumar Vavilapalli
Andrew, Each of the branches below have been created more than a year ago (!) and have been consistently worked upon and are now finally seeing the light of the day. When they are "few weeks” away, pushing them out by 7 *more* months just doesn’t make sense. While I deeply appreciate the push

Branch merges and 3.0.0-beta1 scope

2017-08-18 Thread Andrew Wang
Hi folks, As you might have seen, we've had a number of branch merges floated this past week targeted for 3.0.0-beta1, which is planned for about a month from now. In total, I'm currently tracking these branches: YARN-2915: YARN federation (recently merged) HADOOP-13345: S3Guard (currently being

[jira] [Created] (HDFS-12325) SFTPFileSystem operations should restore cwd

2017-08-18 Thread Chen Liang (JIRA)
Chen Liang created HDFS-12325: - Summary: SFTPFileSystem operations should restore cwd Key: HDFS-12325 URL: https://issues.apache.org/jira/browse/HDFS-12325 Project: Hadoop HDFS Issue Type: Bug

[jira] [Created] (HDFS-12324) Ozone: TestQueryNode#testStaleNodesCount is failing.

2017-08-18 Thread Xiaoyu Yao (JIRA)
Xiaoyu Yao created HDFS-12324: - Summary: Ozone: TestQueryNode#testStaleNodesCount is failing. Key: HDFS-12324 URL: https://issues.apache.org/jira/browse/HDFS-12324 Project: Hadoop HDFS Issue Type

Re: [DISCUSS] Merge YARN resource profile (YARN-3926) branch into trunk

2017-08-18 Thread Wangda Tan
Hi Andrew, This feature can be disabled, It is off by default. The major change of existing code path it Resource YARN PB record related implementations, we did lots of tests around this regarding to performances and safety of these changes, so far so good (please refer to my email regarding to pe

Re: [DISCUSS] Merge yarn-native-services branch into trunk

2017-08-18 Thread Andrew Wang
Hi Jian, thanks for the reply, On Thu, Aug 17, 2017 at 1:03 PM, Jian He wrote: > Thanks Andrew for the comments. Answers below: > > - There are no new APIs added in YARN/Hadoop core. In fact, all the new > code are running outside of existing system and they are optional and > require users to e

Re: [DISCUSS] Merge YARN resource profile (YARN-3926) branch into trunk

2017-08-18 Thread Andrew Wang
Hi Wangda, Can this feature be disabled? Is it on or off by default? We're 1 month from the target release for beta1, so I don't want to introduce risk to existing code paths. TSv2 and S3Guard and YARN Federation are all okay in that regard. I'm also not clear on what work is remaining, there are

[jira] [Created] (HDFS-12323) NameNode terminates after full GC thinking QJM unresponsive if full GC is longer than timeout

2017-08-18 Thread Erik Krogen (JIRA)
Erik Krogen created HDFS-12323: -- Summary: NameNode terminates after full GC thinking QJM unresponsive if full GC is longer than timeout Key: HDFS-12323 URL: https://issues.apache.org/jira/browse/HDFS-12323

[jira] [Created] (HDFS-12322) [branch-2] Add storage type demand to into DFSNetworkTopology#chooseRandom.

2017-08-18 Thread Chen Liang (JIRA)
Chen Liang created HDFS-12322: - Summary: [branch-2] Add storage type demand to into DFSNetworkTopology#chooseRandom. Key: HDFS-12322 URL: https://issues.apache.org/jira/browse/HDFS-12322 Project: Hadoop H

[jira] [Created] (HDFS-12321) Ozone : debug cli: add support to load user-provided SQL query

2017-08-18 Thread Chen Liang (JIRA)
Chen Liang created HDFS-12321: - Summary: Ozone : debug cli: add support to load user-provided SQL query Key: HDFS-12321 URL: https://issues.apache.org/jira/browse/HDFS-12321 Project: Hadoop HDFS

[jira] [Created] (HDFS-12320) Add quantiles for transactions batched in Journal sync

2017-08-18 Thread Hanisha Koneru (JIRA)
Hanisha Koneru created HDFS-12320: - Summary: Add quantiles for transactions batched in Journal sync Key: HDFS-12320 URL: https://issues.apache.org/jira/browse/HDFS-12320 Project: Hadoop HDFS

[jira] [Resolved] (HDFS-10531) Add EC policy and storage policy related usage summarization function to dfs du command

2017-08-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-10531. Resolution: Duplicate > Add EC policy and storage policy related usage summarization function to dfs

[jira] [Reopened] (HDFS-10531) Add EC policy and storage policy related usage summarization function to dfs du command

2017-08-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HDFS-10531: > Add EC policy and storage policy related usage summarization function to dfs > du command > -

[jira] [Resolved] (HDFS-8387) Erasure Coding: Revisit the long and int datatypes usage in striping logic

2017-08-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-8387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-8387. --- Resolution: Delivered Thanks Rakesh for working on this, resolving per above recommendation. > Erasure

Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk

2017-08-18 Thread Sangjin Lee
Kudos to Vrushali and the team for getting ready for this large and important feature! I know a huge team effort went into this. I look forward to seeing this merged. I'd like to ask one piece of due diligence. Could you please inspect rigorously to ensure that when disabled Timeline Service v.2 d

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-08-18 Thread Apache Jenkins Server
For more details, see https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/496/ [Aug 17, 2017 8:05:19 AM] (jzhuge) HADOOP-14560. Make HttpServer2 backlog size configurable. Contributed by [Aug 17, 2017 9:37:15 AM] (sunilg) YARN-3254. HealthReport should include disk full information.

[jira] [Created] (HDFS-12319) DirectoryScanner will throw IllegalStateException when Multiple BP's are present

2017-08-18 Thread Brahma Reddy Battula (JIRA)
Brahma Reddy Battula created HDFS-12319: --- Summary: DirectoryScanner will throw IllegalStateException when Multiple BP's are present Key: HDFS-12319 URL: https://issues.apache.org/jira/browse/HDFS-12319

[jira] [Created] (HDFS-12318) Fix IOException condition for openInfo in DFSInputStream

2017-08-18 Thread legend (JIRA)
legend created HDFS-12318: - Summary: Fix IOException condition for openInfo in DFSInputStream Key: HDFS-12318 URL: https://issues.apache.org/jira/browse/HDFS-12318 Project: Hadoop HDFS Issue Type: Bu