Re: Looking to a Hadoop 3 release

[Dmitry Sivachenko]

> On 19 Feb 2016, at 01:35, Andrew Wang  wrote:
> 
> Hi all,
> 
> Reviving this thread. I've seen renewed interest in a trunk release since
> HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
> shell script rewrite, and many other improvements, I think it's time to
> revisit Hadoop 3.0 release plans.
> 


Hello,

any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes out?

Thanks!

Re: Looking to a Hadoop 3 release

[Steve Loughran]

> On 19 Feb 2016, at 11:27, Dmitry Sivachenko  wrote:
> 
> 
>> On 19 Feb 2016, at 01:35, Andrew Wang  wrote:
>> 
>> Hi all,
>> 
>> Reviving this thread. I've seen renewed interest in a trunk release since
>> HDFS erasure coding has not yet made it to branch-2. Along with JDK8, the
>> shell script rewrite, and many other improvements, I think it's time to
>> revisit Hadoop 3.0 release plans.
>> 
> 

It's time to start ... I suspect it'll take a while to stabilise. I look 
forward to the new shell scripts already

One thing I do want there is for all the alpha releases to make clear that 
there are no compatibility policies here; protocols may change and there is no 
requirement of the first 3.x release to be compatible with all the 3.0.x 
alphas. That's something we missed out on the 2.0.x-alpha process, or at least 
not repeated often enough.

> 
> Hello,
> 
> any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes out?
> 
> Thanks!
> 
> 

sounds like a good time for a status update on the FB work —and anything people 
can do to test it would be appreciated by all. That includes testing on ipv4 
systems, and especially, IPv4/v6 systems with Kerberos turned on and both MIT 
and AD kerberos servers. At the same time, IPv6 support ought to be something 
that could be added in.


I don't have any opinions on timescale, but

+1 to anything related to classpath isolation
+1 to a careful bump of versions of dependencies.
+1 to fixing the outstanding Java 8 migration issues, especially the big Jersey 
patch that's just been updated.
+1 to switching to JIRA-created release notes

Having been doing the slider releases recently, it's clear to me that you can 
do a lot in automating the release process itself. All those steps in the 
release runbook can be turned into targets in a special ant release.xml build 
file, calling maven, gpg, etc.

I think doing something like this for 3.0 will significantly benefit both the 
release phase here but the future releases

This is the slider one: 
https://github.com/apache/incubator-slider/blob/develop/bin/release.xml

It doesn't replace maven, instead it choreographs that along with all the other 
steps: signing and checksumming artifacts, publishing them, voting

it includes
 -refusing to release if the git repo is modified
 -making the various git branch/tag/push operations
 -issuing the various mvn versions:update commands
 -signing
 -publishing via asf SVN 
 -using GET calls too verify the artifacts made it
 -generating the vote and vote result emails (it even counts the votes)

I recommend this is included as part of the release process. It does make a 
difference; we can now cut new releases with no human intervention other than 
editing a properties file and running different targets as the process goes 
through its release and vote phases.

-Steve

[jira] [Created] (HADOOP-12827) WebHdfs socket timeouts should be configurable

[Austin Donnelly (JIRA)]

Austin Donnelly created HADOOP-12827:


 Summary: WebHdfs socket timeouts should be configurable
 Key: HADOOP-12827
 URL: https://issues.apache.org/jira/browse/HADOOP-12827
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
 Environment: all
Reporter: Austin Donnelly
Assignee: Austin Donnelly


WebHdfs client connections use sockets with fixed timeouts of 60 seconds to 
connect, and 60 seconds for reads.
This is a problem because I am trying to use WebHdfs to access an archive 
storage system which can take minutes to hours to return the requested data 
over WebHdfs.

The fix is to add new configuration file options to allow these 60s defaults to 
be customised in hdfs-site.xml.

If the new configuration options are not present, the behavior is unchanged 
from before.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] hadoop pull request: Branched to archive the Avro RPC work. On tru...

[arif505]

GitHub user arif505 opened a pull request:

https://github.com/apache/hadoop/pull/77

Branched to archive the Avro RPC work. On trunk Avro RPC is removed a…

…fter creating this branch.

git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/HADOOP-6659@1214008 
13f79535-47bb-0310-9956-ffa450edef68

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/hadoop HADOOP-6659

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hadoop/pull/77.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #77


commit 42f957c887c0bf390f103b9b1a0d881090c9ab53
Author: Suresh Srinivas 
Date:   2011-12-14T00:44:58Z

Branched to archive the Avro RPC work. On trunk Avro RPC is removed after 
creating this branch.


git-svn-id: 
https://svn.apache.org/repos/asf/hadoop/common/branches/HADOOP-6659@1214008 
13f79535-47bb-0310-9956-ffa450edef68




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Looking to a Hadoop 3 release

[Ravi Prakash]

+1 for the plan to start cutting 3.x alpha releases. Thanks for the
initiative Andrew!

On Fri, Feb 19, 2016 at 6:19 AM, Steve Loughran 
wrote:

>
> > On 19 Feb 2016, at 11:27, Dmitry Sivachenko  wrote:
> >
> >
> >> On 19 Feb 2016, at 01:35, Andrew Wang  wrote:
> >>
> >> Hi all,
> >>
> >> Reviving this thread. I've seen renewed interest in a trunk release
> since
> >> HDFS erasure coding has not yet made it to branch-2. Along with JDK8,
> the
> >> shell script rewrite, and many other improvements, I think it's time to
> >> revisit Hadoop 3.0 release plans.
> >>
> >
>
> It's time to start ... I suspect it'll take a while to stabilise. I look
> forward to the new shell scripts already
>
> One thing I do want there is for all the alpha releases to make clear that
> there are no compatibility policies here; protocols may change and there is
> no requirement of the first 3.x release to be compatible with all the 3.0.x
> alphas. That's something we missed out on the 2.0.x-alpha process, or at
> least not repeated often enough.
>
> >
> > Hello,
> >
> > any chance IPv6 support (HADOOP-11890) will be finished before 3.0 comes
> out?
> >
> > Thanks!
> >
> >
>
> sounds like a good time for a status update on the FB work —and anything
> people can do to test it would be appreciated by all. That includes testing
> on ipv4 systems, and especially, IPv4/v6 systems with Kerberos turned on
> and both MIT and AD kerberos servers. At the same time, IPv6 support ought
> to be something that could be added in.
>
>
> I don't have any opinions on timescale, but
>
> +1 to anything related to classpath isolation
> +1 to a careful bump of versions of dependencies.
> +1 to fixing the outstanding Java 8 migration issues, especially the big
> Jersey patch that's just been updated.
> +1 to switching to JIRA-created release notes
>
> Having been doing the slider releases recently, it's clear to me that you
> can do a lot in automating the release process itself. All those steps in
> the release runbook can be turned into targets in a special ant release.xml
> build file, calling maven, gpg, etc.
>
> I think doing something like this for 3.0 will significantly benefit both
> the release phase here but the future releases
>
> This is the slider one:
> https://github.com/apache/incubator-slider/blob/develop/bin/release.xml
>
> It doesn't replace maven, instead it choreographs that along with all the
> other steps: signing and checksumming artifacts, publishing them, voting
>
> it includes
>  -refusing to release if the git repo is modified
>  -making the various git branch/tag/push operations
>  -issuing the various mvn versions:update commands
>  -signing
>  -publishing via asf SVN
>  -using GET calls too verify the artifacts made it
>  -generating the vote and vote result emails (it even counts the votes)
>
> I recommend this is included as part of the release process. It does make
> a difference; we can now cut new releases with no human intervention other
> than editing a properties file and running different targets as the process
> goes through its release and vote phases.
>
> -Steve

Re: Replacing Commons-httpclient and bumping httpclient version

[Wei-Chiu Chuang]

Thanks every one for the feedbacks and attention to the related patches for 
replacing cnmmons-httpclient.

The second part of my question is how do people feel about bumping httpclient 
version? httpclient 4.2.5 used by current Hadoop also has a few security 
vulnerabilities. Fortunately in this case, we can easily bump its version to 
address the security vulnerabilities.
This refers to HADOOP-12767 
 (update apache httpclient 
version to the latest 4.5 for security)

Thanks again,
Wei-Chiu Chuang
A very happy Clouderan

> On Feb 18, 2016, at 6:50 PM, Brahma Reddy Battula 
>  wrote:
> 
> Thanks Wei-Chiu Chuang for initiating discussion here.
> 
> I'm +1 too to clean up dependency on commons-httpclient.
> 
> -Original Message-
> From: Masatake Iwasaki [mailto:iwasak...@oss.nttdata.co.jp] 
> Sent: 17 February 2016 22:52
> To: common-dev@hadoop.apache.org
> Subject: Re: Replacing Commons-httpclient and bumping httpclient version
> 
> Thanks for the suggestion, Wei-Chiu Chuang.
> 
> I'm +1 too to clean up dependency on commons-httpclient.
> 
> Your suggestion reminded me of HADOOP-12552 which seems to depends on 
> HADOOP-12710 and HADOOP-12711 now.
> I will revisit it.
> 
> Masatake Iwasaki
> 
> On 2/17/16 03:59, Colin P. McCabe wrote:
>> +1 for updating the dependencies in trunk.
>> 
>> best,
>> Colin
>> 
>> On Tue, Feb 16, 2016 at 9:20 AM, Wei-Chiu Chuang  
>> wrote:
>>> Fellow Hadoop developers,
>>> 
>>> Hadoop codebase depends on commons-httpclient, and its latest version, 
>>> 3.1.2, is EOL nearly 5 years ago. But because its API is not compatible 
>>> with its successor, httpclient 4, the community seem to have been reluctant 
>>> to upgrade.
>>> However, a lot of evidence indicates that commons-httpclient has a number 
>>> of security vulnerabilities which are never addressed, including 
>>> CVE-2012-6153. To make Hadoop less susceptible to existing and future 
>>> vulnerabilities, we should seriously consider replacing commons-httpclient 
>>> with httpclient 4.x.
>>> 
>>> There are a few Hadoop JIRAs that have patches available to address that, 
>>> but they really need more attention to get them committed:
>>> HADOOP-10105  (remove 
>>> httpclient dependency) is the umbrella JIRA for all.
>>> Other efforts includes HADOOP-11613 
>>>  (Remove httpclient 
>>> dependency from hadoop-azure), HADOOP-11614 
>>>  (Remove httpclient 
>>> dependency from hadoop-openstack), HADOOP-12710 
>>>  (Remove dependency on 
>>> commons-httpclient for TestHttpServerLogs), HADOOP-12711 
>>>  (Remove dependency on 
>>> commons-httpclient for ServletUtil). I’d also like to urge the community to 
>>> reject patches that imports commons-httpclient in the future.
>>> 
>>> Additionally, Hadoop trunk depends on httpclient 4.2.5, which is known to 
>>> suffer from several security vulnerabilities as well, including 
>>> CVE-2012-6153, CVE-2011-4461, CVE-2014-3577, CVE-2015-5262. HADOOP-12767 
>>>  (update apache 
>>> httpclient version to the latest 4.5 for security) has a patch that bumps 
>>> the version to 4.5.1. But I’d like to ask the community whether we should 
>>> do it or not, and the implication of bump the latest version.
>>> 
>>> Best regards,
>>> Wei-Chiu Chuang
>>> A very happy Clouderan
>>> 
> 

Re: Looking to a Hadoop 3 release

[Yongjun Zhang]

Thanks Andrew for initiating the effort!

+1 on pushing 3.x with extended alpha cycle, and continuing the more stable
2.x releases.

--Yongjun

On Thu, Feb 18, 2016 at 5:58 PM, Andrew Wang 
wrote:

> Hi Kai,
>
> Sure, I'm open to it. It's a new major release, so we're allowed to make
> these kinds of big changes. The idea behind the extended alpha cycle is
> that downstreams can give us feedback. This way if we do anything too
> radical, we can address it in the next alpha and have downstreams re-test.
>
> Best,
> Andrew
>
> On Thu, Feb 18, 2016 at 5:23 PM, Zheng, Kai  wrote:
>
> > Thanks Andrew for driving this. Wonder if it's a good chance for
> > HADOOP-12579 (Deprecate and remove WriteableRPCEngine) to be in. Note
> it's
> > not an incompatible change, but feel better to be done in the major
> release.
> >
> > Regards,
> > Kai
> >
> > -Original Message-
> > From: Andrew Wang [mailto:andrew.w...@cloudera.com]
> > Sent: Friday, February 19, 2016 7:04 AM
> > To: hdfs-...@hadoop.apache.org; Kihwal Lee 
> > Cc: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org;
> > yarn-...@hadoop.apache.org
> > Subject: Re: Looking to a Hadoop 3 release
> >
> > Hi Kihwal,
> >
> > I think there's still value in continuing the 2.x releases. 3.x comes
> with
> > the incompatible bump to a JDK8 runtime, and also the fact that 3.x won't
> > be beta or GA for some number of months. In the meanwhile, it'd be good
> to
> > keep putting out regular, stable 2.x releases.
> >
> > Best,
> > Andrew
> >
> >
> > On Thu, Feb 18, 2016 at 2:50 PM, Kihwal Lee  >
> > wrote:
> >
> > > Moving Hadoop 3 forward sounds fine. If EC is one of the main
> > > motivations, are we getting rid of branch-2.8?
> > >
> > > Kihwal
> > >
> > >   From: Andrew Wang 
> > >  To: "common-dev@hadoop.apache.org" 
> > > Cc: "yarn-...@hadoop.apache.org" ; "
> > > mapreduce-...@hadoop.apache.org" ;
> > > hdfs-dev 
> > >  Sent: Thursday, February 18, 2016 4:35 PM
> > >  Subject: Re: Looking to a Hadoop 3 release
> > >
> > > Hi all,
> > >
> > > Reviving this thread. I've seen renewed interest in a trunk release
> > > since HDFS erasure coding has not yet made it to branch-2. Along with
> > > JDK8, the shell script rewrite, and many other improvements, I think
> > > it's time to revisit Hadoop 3.0 release plans.
> > >
> > > My overall plan is still the same as in my original email: a series of
> > > regular alpha releases leading up to beta and GA. Alpha releases make
> > > it easier for downstreams to integrate with our code, and making them
> > > regular means features can be included when they are ready.
> > >
> > > I know there are some incompatible changes waiting in the wings (i.e.
> > > HDFS-6984 making FileStatus a PB rather than Writable, some of
> > > HADOOP-9991 bumping dependency versions) that would be good to get in.
> > > If you have changes like this, please set the target version to 3.0.0
> > > and mark them "Incompatible". We can use this JIRA query to track:
> > >
> > >
> > > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%2
> > > 0HDFS%2C%20YARN%2C%20MAPREDUCE)%20and%20%22Target%20Version%2Fs%22%20%
> > > 3D%20%223.0.0%22%20and%20resolution%3D%22unresolved%22%20and%20%22Hado
> > > op%20Flags%22%3D%22Incompatible%20change%22%20order%20by%20priority
> > >
> > > There's some release-related stuff that needs to be sorted out
> > > (namely, the new CHANGES.txt and release note generation from Yetus),
> > > but I'd tentatively like to roll the first alpha a month out, so third
> > > week of March.
> > >
> > > Best,
> > > Andrew
> > >
> > > On Mon, Mar 9, 2015 at 7:23 PM, Raymie Stata 
> > wrote:
> > >
> > > > Avoiding the use of JDK8 language features (and, presumably, APIs)
> > > > means you've abandoned #1, i.e., you haven't (really) bumped the JDK
> > > > source version to JDK8.
> > > >
> > > > Also, note that releasing from trunk is a way of achieving #3, it's
> > > > not a way of abandoning it.
> > > >
> > > >
> > > >
> > > > On Mon, Mar 9, 2015 at 7:10 PM, Andrew Wang
> > > > 
> > > > wrote:
> > > > > Hi Raymie,
> > > > >
> > > > > Konst proposed just releasing off of trunk rather than cutting a
> > > > branch-2,
> > > > > and there was general agreement there. So, consider #3 abandoned.
> > > > > 1&2
> > > can
> > > > > be achieved at the same time, we just need to avoid using JDK8
> > > > > language features in trunk so things can be backported.
> > > > >
> > > > > Best,
> > > > > Andrew
> > > > >
> > > > > On Mon, Mar 9, 2015 at 7:01 PM, Raymie Stata
> > > > > 
> > > > wrote:
> > > > >
> > > > >> In this (and the related threads), I see the following three
> > > > requirements:
> > > > >>
> > > > >> 1. "Bump the source JDK version to JDK8" (ie, drop JDK7 support).
> > > > >>
> > > > >> 2. "We'll still be releasing 2.x releases for a while, with
> > > > >> similar feature sets as 3.x."
> > > > >>
> > > > >> 3. Avoid the "risk of split-brain behavior" by "minimize
> > > > >> backporting headaches. Pulling t

Build failed in Jenkins: Hadoop-common-trunk-Java8 #1103

[Apache Jenkins Server]

See 

Changes:

[junping_du] Support additional compression levels for GzipCodec. Contributed 
by Ravi

--
[...truncated 5501 lines...]
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.244 sec - in 
org.apache.hadoop.io.erasurecode.coder.TestXORCoder
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.223 sec - in 
org.apache.hadoop.io.erasurecode.coder.TestHHXORErasureCoder
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestSequenceFileSync
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.764 sec - in 
org.apache.hadoop.io.TestSequenceFileSync
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestVersionedWritable
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.163 sec - in 
org.apache.hadoop.io.TestVersionedWritable
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestWritable
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.306 sec - in 
org.apache.hadoop.io.TestWritable
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestBloomMapFile
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.286 sec - in 
org.apache.hadoop.io.TestBloomMapFile
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestSequenceFileAppend
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.775 sec - in 
org.apache.hadoop.io.TestSequenceFileAppend
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestEnumSetWritable
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.441 sec - in 
org.apache.hadoop.io.TestEnumSetWritable
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestMapWritable
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.18 sec - in 
org.apache.hadoop.io.TestMapWritable
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestBooleanWritable
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.175 sec - in 
org.apache.hadoop.io.TestBooleanWritable
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestBytesWritable
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.182 sec - in 
org.apache.hadoop.io.TestBytesWritable
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestSequenceFile
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.461 sec - in 
org.apache.hadoop.io.TestSequenceFile
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestTextNonUTF8
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.18 sec - in 
org.apache.hadoop.io.TestTextNonUTF8
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestObjectWritableProtos
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.279 sec - in 
org.apache.hadoop.io.TestObjectWritableProtos
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.TestDefaultStringifier
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.368 sec - in 
org.apache.hadoop.io.TestDefaultStringifier
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.retry.TestRetryProxy
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.219 sec - in 
org.apache.hadoop.io.retry.TestRetryProxy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.retry.TestDefaultRetryPolicy
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.348 sec - in 
org.apache.hadoop.io.retry.TestDefaultRetryPolicy
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; 
support was removed in 8.0
Running org.apache.hadoop.io.retry.TestFailoverProxy
Tests run: 9, Failures

[jira] [Created] (HADOOP-12829) StatisticsDataReferenceCleaner swallos interrupt exceptions

[Gregory Chanan (JIRA)]

Gregory Chanan created HADOOP-12829:
---

 Summary: StatisticsDataReferenceCleaner swallos interrupt 
exceptions
 Key: HADOOP-12829
 URL: https://issues.apache.org/jira/browse/HADOOP-12829
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.6.4, 2.8.0, 2.7.3
Reporter: Gregory Chanan
Assignee: Gregory Chanan


The StatisticsDataReferenceCleaner, implemented in HADOOP-12107 swallows 
interrupt exceptions.  Over in Solr/Sentry land, we run thread leak checkers on 
our test code, which passed before this change and fails after it.  Here's a 
sample report:

{code}
1 thread leaked from SUITE scope at 
org.apache.solr.handler.TestSecureReplicationHandler: 
   1) Thread[id=16, 
name=org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner, 
state=WAITING, group=TGRP-TestSecureReplicationHandler]
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at 
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3040)
at java.lang.Thread.run(Thread.java:745)
{code}

And here's an indication that the interrupt is being ignored:
{code}
25209 T16 oahf.FileSystem$Statistics$StatisticsDataReferenceCleaner.run WARN 
exception in the cleaner thread but it will continue to run 
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at 
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3040)
at java.lang.Thread.run(Thread.java:745)
{code}

This is inconsistent with how other long-running threads in hadoop, i.e. 
PeerCache respond to being interrupted.

The argument for doing this in HADOOP-12107 is given as 
(https://issues.apache.org/jira/browse/HADOOP-12107?focusedCommentId=14598397&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14598397):
{quote}
Cleaner#run
Catch and log InterruptedException in the while loop, such that thread does not 
die on a spurious wakeup. It's safe since it's a daemon thread.
{quote}

I'm unclear on what "spurious wakeup" means and it is not mentioned in 
https://docs.oracle.com/javase/tutorial/essential/concurrency/interrupt.html:
{quote}
A thread sends an interrupt by invoking interrupt on the Thread object for the 
thread to be interrupted. For the interrupt mechanism to work correctly, the 
interrupted thread must support its own interruption.
{quote}

So, I believe this thread should respect interruption.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Hadoop-common-trunk-Java8 #1104

[Apache Jenkins Server]

See 

Build failed in Jenkins: Hadoop-Common-trunk #2407

[Apache Jenkins Server]

See 

Changes:

[jing9] HDFS-9818. Correctly handle EC reconstruction work caused by not enough

--
[...truncated 8789 lines...]
Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating 

Generating