[jira] [Resolved] (HDFS-12029) Data node process crashes after kernel upgrade

[Nandakumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nandakumar resolved HDFS-12029.
---
Resolution: Not A Bug

>  Data node process crashes after kernel upgrade
> ---
>
> Key: HDFS-12029
> URL: https://issues.apache.org/jira/browse/HDFS-12029
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: datanode
>Reporter: Anu Engineer
>Assignee: Nandakumar
>Priority: Blocker
>
>  We have seen that when Linux kernel is upgraded to address a specific CVE 
>  ( https://access.redhat.com/security/vulnerabilities/stackguard ) it might 
> cause a datanode crash.
> We have observed this issue while upgrading from 3.10.0-514.6.2 to 
> 3.10.0-514.21.2 versions of the kernel.
> Original kernel fix is here -- 
> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1be7107fbe18eed3e319a6c3e83c78254b693acb
> Datanode fails with the following stack trace, 
> {noformat}
> # 
> # A fatal error has been detected by the Java Runtime Environment: 
> # 
> # SIGBUS (0x7) at pc=0x7f458d078b7c, pid=13214, tid=139936990349120 
> # 
> # JRE version: (8.0_40-b25) (build ) 
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.40-b25 mixed mode 
> linux-amd64 compressed oops) 
> # Problematic frame: 
> # j java.lang.Object.()V+0 
> # 
> # Failed to write core dump. Core dumps have been disabled. To enable core 
> dumping, try "ulimit -c unlimited" before starting Java again 
> # 
> # An error report file with more information is saved as: 
> # /tmp/hs_err_pid13214.log 
> # 
> # If you would like to submit a bug report, please visit: 
> # http://bugreport.java.com/bugreport/crash.jsp 
> # 
> {noformat}
> The root cause is a failure in jsvc. If we pass a greater than 1MB value as 
> the stack size argument, this can be mitigated.  Something like:
> {code}
> exec "$JSVC" \
> -Xss2m
> org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter "$@"
> {code}
> This JIRA tracks potential fixes for this problem. We don't have data on how 
> this impacts other applications that run on datanode as this might impact 
> datanodes memory usage.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Running HDFS from source broken since HDFS-11596

[Lars Francke]

Thanks John, that was helpful. I see that you're using the hadoop-dist
directory while the wiki points directly to the project folders (e.g.
hadoop-hdfs-project etc.).

The former works the latter doesn't. So I guess it's a matter of updating
the wiki.


On Thu, Jul 20, 2017 at 9:09 AM, John Zhuge  wrote:

> Hi Lars,
>
> I am able to run pseudo-distributed mode from a dev tree. Here is the
> wiki: https://hadoop.apache.org/docs/current/hadoop-
> project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation
> .
>
> Check out my script pseudo_dist
>  to
> start/stop a pseudo-distributed cluster.
>
> Here are the steps:
>
>1. mvn install -DskipTests -DskipShade -Dmaven.javadoc.skip -Pdist
>-Dtar
>2. pseudo_dist start ~/hadoop-sanity-tests/config/insecure/
>3. test_env hdfs dfs -ls /tmp
>
> Thanks,
>
> On Wed, Jul 19, 2017 at 11:49 PM, Lars Francke 
> wrote:
>
>> I've already asked in 
>> but haven't gotten a reply so far so I thought I'd bump it here.
>>
>> The issue replaces the compile time dependency of the various HDFS
>> projects
>> to hdfs-client with a "provided" dependency.
>>
>> Unfortunately that means that HDFS cannot be run anymore from source as is
>> documented in the Wiki (<
>> https://wiki.apache.org/hadoop/HowToSetupYourDevelopmentEnvironment>) and
>> as used to be possible before the patch. This is because the hdfs client
>> classes (e.g. ClientProtocol is the first one that HDFS complains about
>> during startup) are not in the classpath anymore.
>>
>> I wonder how all of you are running Hadoop these days from source? I've
>> always followed the Wiki instructions but maybe they are out of date and
>> there's a better way?
>>
>> Thanks,
>> Lars
>>
>
>
>
> --
> John
>

[jira] [Created] (HDFS-12178) Ozone: OzoneClient: Handling SCM container creationFlag at client side

[Nandakumar (JIRA)]

Nandakumar created HDFS-12178:
-

 Summary: Ozone: OzoneClient: Handling SCM container creationFlag 
at client side
 Key: HDFS-12178
 URL: https://issues.apache.org/jira/browse/HDFS-12178
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Nandakumar
Assignee: Nandakumar


SCM BlockManager provisions a pool of containers upon block creation request, 
only one container is returned with the creationFlag to the client. The other 
container provisioned in the same batch will not have this flag. This jira is 
to handle this scenario at client side, until  HDFS-11888 is fixed.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12179) Ozone: Corona: Adding corona as part of hdfs command

[Nandakumar (JIRA)]

Nandakumar created HDFS-12179:
-

 Summary: Ozone: Corona: Adding corona as part of hdfs command
 Key: HDFS-12179
 URL: https://issues.apache.org/jira/browse/HDFS-12179
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
 Environment: Corona should be added as part of {{hdfs}} command, so 
that we can launch it through {{bin/hdfs corona}}
Reporter: Nandakumar
Assignee: Nandakumar






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12180) Ozone: Corona: Add stats and progress bar to corona

[Nandakumar (JIRA)]

Nandakumar created HDFS-12180:
-

 Summary: Ozone: Corona: Add stats and progress bar to corona
 Key: HDFS-12180
 URL: https://issues.apache.org/jira/browse/HDFS-12180
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Nandakumar
Assignee: Nandakumar


More statistics to be added to corona run summary and also progress bar which 
shows the status of run.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12181) Ozone:TestContainerReplicationManager fails because LogCapturer fails to find expected log messages

[Mukul Kumar Singh (JIRA)]

Mukul Kumar Singh created HDFS-12181:


 Summary: Ozone:TestContainerReplicationManager fails because 
LogCapturer fails to find expected log messages
 Key: HDFS-12181
 URL: https://issues.apache.org/jira/browse/HDFS-12181
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Affects Versions: HDFS-7240
Reporter: Mukul Kumar Singh
Assignee: Mukul Kumar Singh
 Fix For: HDFS-7240


TestContainerReplicationManager fails consistently because the following lines 
are marked at info level. Tests are looking for the log entries in the 
logcapturer and failing.
{code}
  if (pool.getLastProcessedTime() + this.containerProcessingLag <
  Time.monotonicNow()) {
 ->  LOG.debug("Adding pool {} to container processing queue", pool
.getPoolName());
InProgressPool inProgressPool =  new InProgressPool(maxPoolWait,
pool, this.nodeManager, this.poolManager, this.commandQueue,
this.executorService);
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12182) BlockManager.metSave does not distinguish between "under replicated" and "corrupt" blocks

[Wellington Chevreuil (JIRA)]

Wellington Chevreuil created HDFS-12182:
---

 Summary: BlockManager.metSave does not distinguish between "under 
replicated" and "corrupt" blocks
 Key: HDFS-12182
 URL: https://issues.apache.org/jira/browse/HDFS-12182
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs
Reporter: Wellington Chevreuil
Priority: Trivial
 Fix For: 3.0.0-alpha3


Currently, *BlockManager.metaSave* method (which is called by "-metasave" dfs 
CLI command) reports both "under replicated" and "corrupt" blocks under same 
metric *Metasave: Blocks waiting for reconstruction:* as shown on below code 
snippet:

{noformat}
   synchronized (neededReconstruction) {
  out.println("Metasave: Blocks waiting for reconstruction: "
  + neededReconstruction.size());
  for (Block block : neededReconstruction) {
dumpBlockMeta(block, out);
  }
}
{noformat}

*neededReconstruction* is an instance of *LowRedundancyBlocks*, which actually 
wraps 5 priority queues currently. 4 of these queues store different under 
replicated scenarios, but the 5th one is dedicated for corrupt blocks. 

Thus, metasave report may suggest some corrupt blocks are just under 
replicated. This can be misleading for admins and operators trying to track 
block corruption issues, and/or other issues related to *BlockManager* metrics.

I would like to propose a patch with trivial changes that would report corrupt 
blocks separately.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Kihwal Lee]

Thanks for driving the next 2.8 release, Junping. While I was committing a 
blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
2.8.2.
Thanks,Kihwal

On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du  
wrote:

Hi all,
    Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
released today which is a special security release. Now, we should work towards 
2.8.2 release which aim for production deployment. The focus obviously is to 
fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We 
currently have 13 blocker/critical issues, and 10 of them are Patch Available.

  I plan to cut an RC in a month - target for releasing before end of Aug., to 
give enough time for outstanding blocker / critical issues. Will start moving 
out any tickets that are not blockers and/or won't fit the timeline. For 
progress of releasing effort, please refer our release wiki [2].

  Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
[2] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release


From: Vinod Kumar Vavilapalli 
Sent: Thursday, July 20, 2017 1:05 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released

Hi all,

The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
page: http://hadoop.apache.org/releases.html#Download
This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus 
security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.

Please note that 2.8.x release line continues to be not yet ready for 
production use. Critical issues are being ironed out via testing and downstream 
adoption. Production users should wait for a subsequent release in the 2.8.x 
line.

Thanks
+Vinod


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Jason Lowe]

+1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that are 
in branch-2.8.  There also are a lot of JIRAs that claim they are fixed in 
2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on recent 
activity in branch-2.8 would solve both of these issues, and we'd only need to 
move the handful of JIRAs that have marked themselves correctly as fixed in 
2.8.3 to be fixed in 2.8.2.

Jason
 

On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
 wrote:
 

 Thanks for driving the next 2.8 release, Junping. While I was committing a 
blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
2.8.2.
Thanks,Kihwal

On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du  
wrote:

Hi all,
    Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
released today which is a special security release. Now, we should work towards 
2.8.2 release which aim for production deployment. The focus obviously is to 
fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We 
currently have 13 blocker/critical issues, and 10 of them are Patch Available.

  I plan to cut an RC in a month - target for releasing before end of Aug., to 
give enough time for outstanding blocker / critical issues. Will start moving 
out any tickets that are not blockers and/or won't fit the timeline. For 
progress of releasing effort, please refer our release wiki [2].

  Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
[2] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release


From: Vinod Kumar Vavilapalli 
Sent: Thursday, July 20, 2017 1:05 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released

Hi all,

The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
page: http://hadoop.apache.org/releases.html#Download
This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus 
security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.

Please note that 2.8.x release line continues to be not yet ready for 
production use. Critical issues are being ironed out via testing and downstream 
adoption. Production users should wait for a subsequent release in the 2.8.x 
line.

Thanks
+Vinod


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

   

[jira] [Created] (HDFS-12183) Ozone: KSM : Use proper defaults for block client address

[Anu Engineer (JIRA)]

Anu Engineer created HDFS-12183:
---

 Summary: Ozone: KSM :  Use proper defaults for block client address
 Key: HDFS-12183
 URL: https://issues.apache.org/jira/browse/HDFS-12183
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Affects Versions: HDFS-7240
Reporter: Anu Engineer
Priority: Trivial
 Fix For: HDFS-7240


The setting *ozone.scm.block.client.address* allows extra flexibility but it is 
set to the *ozone.scm.client.address* all the time when we deploy the cluster.

Currently, KSM will fail to startup if *ozone.scm.block.client.address* is not 
configured. This JIRA suggests that we use *ozone.scm.client.address* as the 
default for the *ozone.scm.block.client.address*, so we don't have to 
explicitly configure *ozone.scm.block.client.address*.

Change involves modifying *getScmAddressForBlockClients* and the passing the 
default parameter to *getHostNameFromConfigKeys* and 
*getPortNumberFromConfigKeys*



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)

[sanjay Radia]

Erik
  Great stuff. 
BTW did you build on top of the “simulated data nodes” in HDFS which has a way 
to storing only the length of data (but not real data)? That work allowed 
supplementing  with a matching editsLog for the NN. Your approach of using a 
real image has the advantage of being able to replay traces from audit logs.
(Ref 
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java)

thanks

sanjay
> On Jul 20, 2017, at 10:42 AM, Erik Krogen  
> wrote:
> 
> forking off of the 2.7.4 release thread to answer this question about
> Dynamometer
> 
> Dynamometer is a tool developed at LinkedIn for scale testing HDFS,
> specifically the NameNode. We have been using it for some time now and have
> recently been making some enhancements to ease of use and reproducibility.
> We hope to post a blog post sometime in the not-too-distant future, and
> also to open source it. I can provide some details here given that we have
> been leveraging it as part of our 2.7.4 release / upgrade process (in
> addition to previous upgrades).
> 
> The basic idea is to get full-scale black-box testing of the HDFS NN while
> using significantly less (~10%) hardware than a real cluster of that size
> would require. We use real NN images from our at-scale clusters paired with
> some logic to fake out DNs into thinking they are storing data when they
> are not, allowing us to stuff more DNs onto each machine. Since we use a
> real image, we can replay real traces (collected from audit logs) to
> compare actual production performance vs. performance on this simulated
> cluster (with additional tuning, different version, etc.). We leverage YARN
> to manage setting up this cluster and to replay the traces.
> 
> Happy to answer questions.
> 
> Erik
> 
> On Wed, Jul 19, 2017 at 5:05 PM, Konstantin Shvachko 
> wrote:
> 
>> Hi Tianyi,
>> 
>> Glad you are interested in Dynamometer. Erik (CC-ed) is actively working
>> on this project right now, I'll let him elaborate.
>> Erik, you should probably respond on Apache dev list, as I think it could
>> be interesting for other people as well, asince we planned to open source
>> it. You can fork the "About 2.7.4 Release" thread with a new subject and
>> give some details about Dynamometer there.
>> 
>> Thanks,
>> --Konstantin
>> 
>> On Wed, Jul 19, 2017 at 1:40 AM, 何天一  wrote:
>> 
>>> Hi, Shavachko.
>>> 
>>> You mentioned an internal tool called Dynamometer to test NameNode
>>> performance earlier in the 2.7.4 release thread.
>>> I wonder if you could share some ideas behind the tool. Or is there a
>>> plan to bring Dynamometer to open source community?
>>> 
>>> Thanks.
>>> 
>>> BR,
>>> Tianyi
>>> 
>>> On Fri, Jul 14, 2017 at 8:45 AM Konstantin Shvachko 
>>> wrote:
>>> 
 Hi everybody.
 
 We have been doing some internal testing of Hadoop 2.7.4. The testing is
 going well.
 Did not find any major issues on our workloads.
 Used an internal tool called Dynamometer to check NameNode performance on
 real cluster traces. Good.
 Overall test cluster performance looks good.
 Some more testing is still going on.
 
 I plan to build an RC next week. If there are no objection.
 
 Thanks,
 --Konst
 
 On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <
 shv.had...@gmail.com>
 wrote:
 
> Hey guys.
> 
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
> 
> I looked through nightly Jenkins build results for 2.7.4 both on Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I
 filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
> 
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the results.
> 
> There is a plan to enable BigTop for 2.7.4 testing.
> 
> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release.
> Thank you everybody for contributing to this effort.
> 
> Regards,
> --Konstantin
> 
> 
> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka 
> wrote:
> 
>> Sure.
>> If you want to edit the wiki, please tell me your ASF confluence
 account.
>> 
>> -Akira
>> 
>> On 2017/05/30 15:31, Rohith Sharma K S wrote:
>> 
>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These
 will
>>> solve RM HA unstability issues.
>>> https://issues.apache.org/jira/browse/YARN-5333
>>> https://issues.apache.org/jira/browse/YARN-5988
>>> https://issues.apache.org/jira

[jira] [Created] (HDFS-12184) Avoid redundant ancestor metadata traversals for listStatus child entries.

[Daryn Sharp (JIRA)]

Daryn Sharp created HDFS-12184:
--

 Summary: Avoid redundant ancestor metadata traversals for 
listStatus child entries.
 Key: HDFS-12184
 URL: https://issues.apache.org/jira/browse/HDFS-12184
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Affects Versions: 2.7.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp


Creating a file status requires metadata for EC, EZ, storage policies, etc that 
are computed by traversing up the ancestor inodes.  List status will incur the 
same penalties for all child entries when one traversal is enough.  The penalty 
for large directories is not trivial.  Storage policies  short-circuit the full 
traversal.  The other lookups should too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Thanks for suggestions, Jason and Kihwal!
+1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
cannot be abandoned/replaced (suspect all branches are read-only now), I will 
manually merge all commits that not landed on 2.8.2 yet.

Thanks,

Junping

From: Jason Lowe 
Sent: Friday, July 21, 2017 8:17 AM
To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

+1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that are 
in branch-2.8.  There also are a lot of JIRAs that claim they are fixed in 
2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on recent 
activity in branch-2.8 would solve both of these issues, and we'd only need to 
move the handful of JIRAs that have marked themselves correctly as fixed in 
2.8.3 to be fixed in 2.8.2.

Jason


On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
 wrote:


 Thanks for driving the next 2.8 release, Junping. While I was committing a 
blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
2.8.2.
Thanks,Kihwal

On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du  
wrote:

Hi all,
Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
released today which is a special security release. Now, we should work towards 
2.8.2 release which aim for production deployment. The focus obviously is to 
fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We 
currently have 13 blocker/critical issues, and 10 of them are Patch Available.

  I plan to cut an RC in a month - target for releasing before end of Aug., to 
give enough time for outstanding blocker / critical issues. Will start moving 
out any tickets that are not blockers and/or won't fit the timeline. For 
progress of releasing effort, please refer our release wiki [2].

  Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
[2] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release


From: Vinod Kumar Vavilapalli 
Sent: Thursday, July 20, 2017 1:05 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released

Hi all,

The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
page: http://hadoop.apache.org/releases.html#Download
This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus 
security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.

Please note that 2.8.x release line continues to be not yet ready for 
production use. Critical issues are being ironed out via testing and downstream 
adoption. Production users should wait for a subsequent release in the 2.8.x 
line.

Thanks
+Vinod


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org




-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12185) Ozone: Fix Leaking in TestXceiverClientManager

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HDFS-12185:
-

 Summary: Ozone: Fix Leaking in TestXceiverClientManager
 Key: HDFS-12185
 URL: https://issues.apache.org/jira/browse/HDFS-12185
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS-7240
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


Multiple MiniOzoneCluster were spin up during tests but only last one is 
shutdown. That is causing leaking and IntelliJ run OOM after 3 continuous run.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12186) Add INodeAttributeProvider startup progress into HDFS Web UI

[Wei-Chiu Chuang (JIRA)]

Wei-Chiu Chuang created HDFS-12186:
--

 Summary: Add INodeAttributeProvider startup progress into HDFS Web 
UI 
 Key: HDFS-12186
 URL: https://issues.apache.org/jira/browse/HDFS-12186
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: ui
Reporter: Wei-Chiu Chuang


For a cluster of substantial size, INodeAttributeProvider may take a long time. 
We saw a large cluster where part of file system ACLs is managed by Apache 
Sentry, and NameNode took a few minutes for the Sentry HDFS NameNode Plugin to 
initialize. I suppose the same issue can arise for Apache Ranger and other 
INodeAttributeProviders implementation.

It would be nice to add an extra row in NameNode Web UI startup progress, in 
addition to "Loading fsimage", "Loading edits", "Saving checkpoint" and "Safe 
mode", to give a better visibility what NameNode is doing.

In addition, there might also be a need to add a similar row into Web UI for 
loading NameNode plugins. So filing this jira to invite more discussion.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Vinod Kumar Vavilapalli]

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du  wrote:
> 
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
> 
> Thanks,
> 
> Junping
> 
> From: Jason Lowe 
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
> 
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
> 
> Jason
> 
> 
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
>  wrote:
> 
> 
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
> 
> On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du  
> wrote:
> 
> Hi all,
>Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
> released today which is a special security release. Now, we should work 
> towards 2.8.2 release which aim for production deployment. The focus 
> obviously is to fix blocker/critical issues [2], bug-fixes and *no* features 
> / improvements. We currently have 13 blocker/critical issues, and 10 of them 
> are Patch Available.
> 
>  I plan to cut an RC in a month - target for releasing before end of Aug., to 
> give enough time for outstanding blocker / critical issues. Will start moving 
> out any tickets that are not blockers and/or won't fit the timeline. For 
> progress of releasing effort, please refer our release wiki [2].
> 
>  Please share thoughts if you have any. Thanks!
> 
> Thanks,
> 
> Junping
> 
> [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
> [2] 2.8 Release wiki: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
> 
> 
> From: Vinod Kumar Vavilapalli 
> Sent: Thursday, July 20, 2017 1:05 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released
> 
> Hi all,
> 
> The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
> page: http://hadoop.apache.org/releases.html#Download
> This is a security release in the 2.8.0 release line. It consists of 2.8.0 
> plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.
> 
> Please note that 2.8.x release line continues to be not yet ready for 
> production use. Critical issues are being ironed out via testing and 
> downstream adoption. Production users should wait for a subsequent release in 
> the 2.8.x line.
> 
> Thanks
> +Vinod
> 
> 
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12187) Ozone : add support to DEBUG CLI for ksm.db

[Chen Liang (JIRA)]

Chen Liang created HDFS-12187:
-

 Summary: Ozone : add support to DEBUG CLI for ksm.db
 Key: HDFS-12187
 URL: https://issues.apache.org/jira/browse/HDFS-12187
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Chen Liang
Assignee: Chen Liang


This JIRA adds the ability to convert ksm meta data file (ksm.db) into sqlite 
db.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli 
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du  wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
>
> Thanks,
>
> Junping
> 
> From: Jason Lowe 
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
>
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
>
> Jason
>
>
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
>  wrote:
>
>
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
>
> On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du  
> wrote:
>
> Hi all,
>Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
> released today which is a special security release. Now, we should work 
> towards 2.8.2 release which aim for production deployment. The focus 
> obviously is to fix blocker/critical issues [2], bug-fixes and *no* features 
> / improvements. We currently have 13 blocker/critical issues, and 10 of them 
> are Patch Available.
>
>  I plan to cut an RC in a month - target for releasing before end of Aug., to 
> give enough time for outstanding blocker / critical issues. Will start moving 
> out any tickets that are not blockers and/or won't fit the timeline. For 
> progress of releasing effort, please refer our release wiki [2].
>
>  Please share thoughts if you have any. Thanks!
>
> Thanks,
>
> Junping
>
> [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
> [2] 2.8 Release wiki: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
> 
> From: Vinod Kumar Vavilapalli 
> Sent: Thursday, July 20, 2017 1:05 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released
>
> Hi all,
>
> The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
> page: http://hadoop.apache.org/releases.html#Download
> This is a security release in the 2.8.0 release line. It consists of 2.8.0 
> plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.
>
> Please note that 2.8.x release line continues to be not yet ready for 
> production use. Critical issues are being ironed out via testing and 
> downstream adoption. Production users should wait for a subsequent release in 
> the 2.8.x line.
>
> Thanks
> +Vinod
>
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Junping Du]

Already get back from Daniel who is from ASF INFRA team, I plan to do following 
operations on next Monday morning:
1. Drop current branch-2.8.2 and recut branch-2.8.2 from branch-2.8
2. Drop abandoned branch-2.8.1 and rename branch-2.8.1-private to branch-2.8.1 
where we just released 2.8.1 from.
I will also adjust fix version on all affected JIRA accordingly.

If you have any concerns on above operations, please raise it before the end of 
this Sunday (7/23).


Thanks,

Junping


From: Junping Du 
Sent: Friday, July 21, 2017 2:29 PM
To: Vinod Kumar Vavilapalli
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Make sense, just raise: https://issues.apache.org/jira/browse/INFRA-14669

Thanks,

Junping

From: Vinod Kumar Vavilapalli 
Sent: Friday, July 21, 2017 12:31 PM
To: Junping Du
Cc: Kihwal Lee; common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org; Jason Lowe
Subject: Re: Apache Hadoop 2.8.2 Release Plan

Junping,

If we are looking at a month, I’d not rebranch branch-2.8.2 right now given how 
these things go. We can just continue to commit on branch-2.8 for now.

I also think we should just follow up with ASF INFRA and clean up the branches
 - Delete branch-2.8.2 so that we can recreate it afresh a little later.
 - branch-2.8.1 is also stale and it should be deleted. branch-2.8.1-private 
should be renamed to branch-2.8.1

Thanks
+Vinod

> On Jul 21, 2017, at 11:23 AM, Junping Du  wrote:
>
> Thanks for suggestions, Jason and Kihwal!
> +1 on releasing 2.8.2 on latest branch-2.8 too. Practically, if branch-2.8.2 
> cannot be abandoned/replaced (suspect all branches are read-only now), I will 
> manually merge all commits that not landed on 2.8.2 yet.
>
> Thanks,
>
> Junping
> 
> From: Jason Lowe 
> Sent: Friday, July 21, 2017 8:17 AM
> To: Kihwal Lee; Junping Du; common-...@hadoop.apache.org; 
> hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org; 
> yarn-...@hadoop.apache.org
> Subject: Re: Apache Hadoop 2.8.2 Release Plan
>
> +1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
> Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that 
> are in branch-2.8.  There also are a lot of JIRAs that claim they are fixed 
> in 2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on 
> recent activity in branch-2.8 would solve both of these issues, and we'd only 
> need to move the handful of JIRAs that have marked themselves correctly as 
> fixed in 2.8.3 to be fixed in 2.8.2.
>
> Jason
>
>
>On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
>  wrote:
>
>
> Thanks for driving the next 2.8 release, Junping. While I was committing a 
> blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
> missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
> 2.8.2.
> Thanks,Kihwal
>
> On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du  
> wrote:
>
> Hi all,
>Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
> released today which is a special security release. Now, we should work 
> towards 2.8.2 release which aim for production deployment. The focus 
> obviously is to fix blocker/critical issues [2], bug-fixes and *no* features 
> / improvements. We currently have 13 blocker/critical issues, and 10 of them 
> are Patch Available.
>
>  I plan to cut an RC in a month - target for releasing before end of Aug., to 
> give enough time for outstanding blocker / critical issues. Will start moving 
> out any tickets that are not blockers and/or won't fit the timeline. For 
> progress of releasing effort, please refer our release wiki [2].
>
>  Please share thoughts if you have any. Thanks!
>
> Thanks,
>
> Junping
>
> [1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
> [2] 2.8 Release wiki: 
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
> 
> From: Vinod Kumar Vavilapalli 
> Sent: Thursday, July 20, 2017 1:05 PM
> To: gene...@hadoop.apache.org
> Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released
>
> Hi all,
>
> The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
> page: http://hadoop.apache.org/releases.html#Download
> This is a security release in the 2.8.0 release line. It consists of 2.8.0 
> plus security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.
>
> Please note that 2.8.x release line continues to be not yet ready for 
> production use. Critical issues are being ironed out via testing and 
> downstream adoption. Production users should wait for a subsequent release in 
> the 2.8.x line.
>
> Thanks
> +Vinod
>
>
> 

Re: LinkedIn Dynamometer Tool (was About 2.7.4 Release)

[Erik Krogen]

Hi Sanjay,

Actually I was not aware of that work… This seems to be a better way of 
achieving some of the same things we do externally to the DN process. I will 
look into reimplementing some parts on top of this; seems it should just 
require some very small extensions to DataNodeCluster. Thank you very much for 
the pointer!

Erik

On 7/21/17, 11:01 AM, "sanjay Radia"  wrote:

Erik
  Great stuff. 
BTW did you build on top of the “simulated data nodes” in HDFS which has a 
way to storing only the length of data (but not real data)? That work allowed 
supplementing  with a matching editsLog for the NN. Your approach of using a 
real image has the advantage of being able to replay traces from audit logs.
(Ref 
https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DataNodeCluster.java)

thanks

sanjay
> On Jul 20, 2017, at 10:42 AM, Erik Krogen  
wrote:
> 
> forking off of the 2.7.4 release thread to answer this question about
> Dynamometer
> 
> Dynamometer is a tool developed at LinkedIn for scale testing HDFS,
> specifically the NameNode. We have been using it for some time now and 
have
> recently been making some enhancements to ease of use and reproducibility.
> We hope to post a blog post sometime in the not-too-distant future, and
> also to open source it. I can provide some details here given that we have
> been leveraging it as part of our 2.7.4 release / upgrade process (in
> addition to previous upgrades).
> 
> The basic idea is to get full-scale black-box testing of the HDFS NN while
> using significantly less (~10%) hardware than a real cluster of that size
> would require. We use real NN images from our at-scale clusters paired 
with
> some logic to fake out DNs into thinking they are storing data when they
> are not, allowing us to stuff more DNs onto each machine. Since we use a
> real image, we can replay real traces (collected from audit logs) to
> compare actual production performance vs. performance on this simulated
> cluster (with additional tuning, different version, etc.). We leverage 
YARN
> to manage setting up this cluster and to replay the traces.
> 
> Happy to answer questions.
> 
> Erik
> 
> On Wed, Jul 19, 2017 at 5:05 PM, Konstantin Shvachko 

> wrote:
> 
>> Hi Tianyi,
>> 
>> Glad you are interested in Dynamometer. Erik (CC-ed) is actively working
>> on this project right now, I'll let him elaborate.
>> Erik, you should probably respond on Apache dev list, as I think it could
>> be interesting for other people as well, asince we planned to open source
>> it. You can fork the "About 2.7.4 Release" thread with a new subject and
>> give some details about Dynamometer there.
>> 
>> Thanks,
>> --Konstantin
>> 
>> On Wed, Jul 19, 2017 at 1:40 AM, 何天一  wrote:
>> 
>>> Hi, Shavachko.
>>> 
>>> You mentioned an internal tool called Dynamometer to test NameNode
>>> performance earlier in the 2.7.4 release thread.
>>> I wonder if you could share some ideas behind the tool. Or is there a
>>> plan to bring Dynamometer to open source community?
>>> 
>>> Thanks.
>>> 
>>> BR,
>>> Tianyi
>>> 
>>> On Fri, Jul 14, 2017 at 8:45 AM Konstantin Shvachko 

>>> wrote:
>>> 
 Hi everybody.
 
 We have been doing some internal testing of Hadoop 2.7.4. The testing 
is
 going well.
 Did not find any major issues on our workloads.
 Used an internal tool called Dynamometer to check NameNode performance 
on
 real cluster traces. Good.
 Overall test cluster performance looks good.
 Some more testing is still going on.
 
 I plan to build an RC next week. If there are no objection.
 
 Thanks,
 --Konst
 
 On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <
 shv.had...@gmail.com>
 wrote:
 
> Hey guys.
> 
> An update on 2.7.4 progress.
> We are down to 4 blockers. There is some work remaining on those.
> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814
> Would be good if people could follow up on review comments.
> 
> I looked through nightly Jenkins build results for 2.7.4 both on 
Apache
> Jenkins and internal.
> Some test fail intermittently, but there no consistent failures. I
 filed
> HDFS-11985 to track some of them.
> https://issues.apache.org/jira/browse/HDFS-11985
> I do not currently consider these failures as blockers. LMK if some of
> them are.
> 
> We started internal testing of branch-2.7 on one of our smallish (100+
> nodes) test clusters.
> Will update on the resu

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/471/

[Jul 20, 2017 8:15:46 AM] (haibochen) YARN-6685. Add job count in to SLS JSON 
input format. (Yufei Gu via
[Jul 20, 2017 9:03:04 AM] (jlowe) YARN-6837. Null LocalResource visibility or 
resource type can crash the
[Jul 20, 2017 2:12:21 PM] (kihwal) HADOOP-14659. UGI getShortUserName does not 
need to search the Subject.
[Jul 20, 2017 2:47:25 PM] (shv) HDFS-11472. Fix inconsistent replica size after 
a data pipeline failure.
[Jul 20, 2017 6:23:13 PM] (weichiu) HADOOP-14557. Document HADOOP-8143 (Change 
distcp to have -pb on by
[Jul 20, 2017 8:45:09 PM] (jzhuge) HDFS-12166. Do not deprecate HTTPFS_TEMP. 
Contributed by John Zhuge.




-1 overall


The following subsystems voted -1:
findbugs unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs-client 
   Possible exposure of partially initialized object in 
org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At 
DFSClient.java:object in 
org.apache.hadoop.hdfs.DFSClient.initThreadsNumForStripedReads(int) At 
DFSClient.java:[line 2888] 
   org.apache.hadoop.hdfs.server.protocol.SlowDiskReports.equals(Object) 
makes inefficient use of keySet iterator instead of entrySet iterator At 
SlowDiskReports.java:keySet iterator instead of entrySet iterator At 
SlowDiskReports.java:[line 105] 

FindBugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus() due to 
return value of called method Dereferenced at 
JournalNode.java:org.apache.hadoop.hdfs.qjournal.server.JournalNode.getJournalsStatus()
 due to return value of called method Dereferenced at JournalNode.java:[line 
302] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setClusterId(String)
 unconditionally sets the field clusterId At HdfsServerConstants.java:clusterId 
At HdfsServerConstants.java:[line 193] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForce(int)
 unconditionally sets the field force At HdfsServerConstants.java:force At 
HdfsServerConstants.java:[line 217] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setForceFormat(boolean)
 unconditionally sets the field isForceFormat At 
HdfsServerConstants.java:isForceFormat At HdfsServerConstants.java:[line 229] 
   
org.apache.hadoop.hdfs.server.common.HdfsServerConstants$StartupOption.setInteractiveFormat(boolean)
 unconditionally sets the field isInteractiveFormat At 
HdfsServerConstants.java:isInteractiveFormat At HdfsServerConstants.java:[line 
237] 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File, File, 
int, HardLink, boolean, File, List) due to return value of called method 
Dereferenced at 
DataStorage.java:org.apache.hadoop.hdfs.server.datanode.DataStorage.linkBlocksHelper(File,
 File, int, HardLink, boolean, File, List) due to return value of called method 
Dereferenced at DataStorage.java:[line 1339] 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String,
 long) due to return value of called method Dereferenced at 
NNStorageRetentionManager.java:org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldLegacyOIVImages(String,
 long) due to return value of called method Dereferenced at 
NNStorageRetentionManager.java:[line 258] 
   Possible null pointer dereference in 
org.apache.hadoop.hdfs.server.namenode.NNUpgradeUtil$1.visitFile(Path, 
BasicFileAttributes) due to return value of called method Dereferenced at 
NNUpgradeUtil.java:org.apache.hadoop.hdfs.server.namenode.NNUpgradeUtil$1.visitFile(Path,
 BasicFileAttributes) due to return value of called method Dereferenced at 
NNUpgradeUtil.java:[line 133] 
   Useless condition:argv.length >= 1 at this point At DFSAdmin.java:[line 
2096] 
   Useless condition:numBlocks == -1 at this point At 
ImageLoaderCurrent.java:[line 727] 

FindBugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 
   Useless object stored in variable removedNullContainers of method 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
 At NodeStatusUpdaterImpl.java:removedNullContainers of method 
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.removeOrTrackCompletedContainersFromContext(List)
 At NodeStatusUpdaterImpl.java:[line 642] 
   
org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpda

Re: Pre-Commit build is failing

[Konstantin Shvachko]

+ d...@yetus.apache.org

Guys, could you please take a look. Seems like Yetus problem with
pre-commit build for branch-2.7.

Thanks,
--Konstantin

On Thu, Jul 20, 2017 at 7:19 PM, Brahma Reddy Battula <
brahmareddy.batt...@huawei.com> wrote:

> Looks this problem is in only branc-2.7..
>
>
> --Brahma Reddy Battula
>
> From: Brahma Reddy Battula
> Sent: 21 July 2017 09:36
> To: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org
> Subject: Pre-Commit build is failing
> Importance: High
>
> Looks pre-commit build is failing with following error.
>
>
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh:
> line 87: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/patch-dryrun.log: No such file or directory
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh:
> line 98: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/patch-dryrun.log: No such file or directory
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh:
> line 87: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/patch-dryrun.log: No such file or directory
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/apache-yetus-a444ed1/precommit/core.d/00-yetuslib.sh:
> line 98: /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
> Build/patchprocess/patch-dryrun.log: No such file or directory
>
>
>
> Reference :
>
> https://builds.apache.org/view/PreCommit%20Builds/job/
> PreCommit-HDFS-Build/20362/console
>
>
>
>
> --Brahma Reddy Battula
>
>

Re: Pre-Commit build is failing

[Allen Wittenauer]

> On Jul 21, 2017, at 5:46 PM, Konstantin Shvachko  wrote:
> 
> + d...@yetus.apache.org
> 
> Guys, could you please take a look. Seems like Yetus problem with
> pre-commit build for branch-2.7.


branch-2.7 is missing stuff in .gitignore.
-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Pre-Commit build is failing

[Konstantin Shvachko]

What stuff? Is there a jira?
It did work like a week ago. Is it a new Yetus requirement.
Anyways I can commit a change to fix the build on our side.
Just need to know what is missing.

Thanks,
--Konst

On Fri, Jul 21, 2017 at 5:50 PM, Allen Wittenauer 
wrote:

>
> > On Jul 21, 2017, at 5:46 PM, Konstantin Shvachko 
> wrote:
> >
> > + d...@yetus.apache.org
> >
> > Guys, could you please take a look. Seems like Yetus problem with
> > pre-commit build for branch-2.7.
>
>
> branch-2.7 is missing stuff in .gitignore.

[jira] [Created] (HDFS-12188) TestDecommissioningStatus#testDecommissionStatus fails intermittently

[Brahma Reddy Battula (JIRA)]

Brahma Reddy Battula created HDFS-12188:
---

 Summary: TestDecommissioningStatus#testDecommissionStatus fails 
intermittently
 Key: HDFS-12188
 URL: https://issues.apache.org/jira/browse/HDFS-12188
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Reporter: Brahma Reddy Battula


{noformat}
java.lang.AssertionError: Unexpected num under-replicated blocks expected:<3> 
but was:<4>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at 
org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus.checkDecommissionStatus(TestDecommissioningStatus.java:144)
at 
org.apache.hadoop.hdfs.server.namenode.TestDecommissioningStatus.testDecommissionStatus(TestDecommissioningStatus.java:240)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12189) TestPread#testPreadFailureWithChangedBlockLocations fails intermittently

[Brahma Reddy Battula (JIRA)]

Brahma Reddy Battula created HDFS-12189:
---

 Summary: TestPread#testPreadFailureWithChangedBlockLocations fails 
intermittently
 Key: HDFS-12189
 URL: https://issues.apache.org/jira/browse/HDFS-12189
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Reporter: Brahma Reddy Battula


{{noformat}}
java.lang.ArrayIndexOutOfBoundsException: 1
at 
org.apache.hadoop.hdfs.TestPread.doPreadTestWithChangedLocations(TestPread.java:656)
at 
org.apache.hadoop.hdfs.TestPread.testPreadFailureWithChangedBlockLocations(TestPread.java:566)
{{noformat}}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12190) Enable 'hdfs dfs -stat' to display access time

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HDFS-12190:


 Summary: Enable 'hdfs dfs -stat' to display access time
 Key: HDFS-12190
 URL: https://issues.apache.org/jira/browse/HDFS-12190
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs, shell
Reporter: Yongjun Zhang


"hdfs dfs -stat" currently only can show modification time of a file but not 
access time. Sometimes it's useful to show access time. 





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org