[jira] [Created] (HADOOP-14964) backport HADOOP-12756 to branch-2

[Genmao Yu (JIRA)]

Genmao Yu created HADOOP-14964:
--

 Summary: backport HADOOP-12756 to branch-2
 Key: HADOOP-14964
 URL: https://issues.apache.org/jira/browse/HADOOP-14964
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs/oss
Reporter: Genmao Yu






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14965) s3a input stream "normal" fadvise mode to be adaptive

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-14965:
---

 Summary: s3a input stream "normal" fadvise mode to be adaptive
 Key: HADOOP-14965
 URL: https://issues.apache.org/jira/browse/HADOOP-14965
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Steve Loughran


HADOOP-14535 added seek optimisation to wasb, but rather than require the 
caller to declare sequential vs random, it works out for itself.

# defaults to sequential, lazy seek
# if the caller ever seeks backwards, switches to random IO.

This means that on the use pattern of columnar stores: of go to end of file, 
read summary, then go to columns and work forwards, will switch to random IO 
after that first seek back (cost: one aborted HTTP connection)/.

Where this should benefit the most is in downstream apps where you are working 
with different data sources in the same object store/running of the same app 
config, but have different read patterns. I'm seeing exactly this in some of my 
spark tests, where it's near impossible to set things up so that .gz files are 
read sequentially, but ORC data is read in random IO

I propose the "normal" fadvise => adaptive, sequential==sequential always, 
random => random from the outset.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14966) Handle JDK-8071638 for hadoop-common

[Bibin A Chundatt (JIRA)]

Bibin A Chundatt created HADOOP-14966:
-

 Summary: Handle JDK-8071638 for hadoop-common
 Key: HADOOP-14966
 URL: https://issues.apache.org/jira/browse/HADOOP-14966
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.8.1, 3.0.0-beta1
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt
Priority: Blocker


Impact modules
-- YARN nodemanger cache clean up
-- Mapreduce Log/History cleaner

Will being adding jira in YARN & MAPREDUCE to track the same





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Do we still have nightly (or even weekly) unit test run for Hadoop projects?

[Sean Busbey]

Here's the email from last night to common-dev@hadoop:

https://s.apache.org/ARe1

On Wed, Oct 18, 2017 at 10:42 PM, Akira Ajisaka  wrote:

> Yes, qbt runs nightly and it sends e-mail to dev lists.
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/
>
> Regards,
> Akira
>
>
> On 2017/10/19 7:54, Wangda Tan wrote:
>
>> Hi,
>>
>> Do we still have nightly (or even weekly) unit test run for Hadoop
>> projects? I couldn't find it on Jenkins dashboard and I haven't seen
>> reports set to dev lists for a while.
>>
>> Thanks,
>> Wangda
>>
>>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>


-- 
busbey

[jira] [Created] (HADOOP-14967) Use jetty CORS filter for web interface

[Eric Yang (JIRA)]

Eric Yang created HADOOP-14967:
--

 Summary: Use jetty CORS filter for web interface
 Key: HADOOP-14967
 URL: https://issues.apache.org/jira/browse/HADOOP-14967
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Eric Yang


Hadoop CORS filter only prevent iframe from embedding Hadoop UI.  It would be 
nice to use standard jetty CORS filter to improve control of CORS filtering.  A 
standard approach is to add this section of code in web.xml:

{code}

 
   cross-origin
   org.eclipse.jetty.servlets.CrossOriginFilter
   
   allowedOrigins
   *
   
   
   allowedMethods
   *
   
   
   allowedHeaders
   *
   
 
 
 cross-origin
 /*
 

{code}

and pom.xml:

{code}

org.eclipse.jetty
jetty-servlets
${jetty.version}

{code}

Hadoop web application are written with embedding Jetty.  This is most likely 
translate to a callable class to initialize web filter using standard jetty web 
filter, when {{hadoop.http.cross-origin.enabled}} is set to true.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/

[Oct 18, 2017 10:06:30 PM] (junping_du) HADOOP-14958. Fix source-level 
compatibility after HADOOP-11252.




-1 overall


The following subsystems voted -1:
unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.net.TestDNS 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.hdfs.TestReadStripedFileWithMissingBlocks 
   hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler 

Timed out junit tests :

   
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-compile-javac-root.txt
  [284K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [148K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [380K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [40K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/562/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14968) YarnClientImpl.getRootQueueInfos() should not do a recursive call to rmClient.getQueueInfo()

[Daniel Templeton (JIRA)]

Daniel Templeton created HADOOP-14968:
-

 Summary: YarnClientImpl.getRootQueueInfos() should not do a 
recursive call to rmClient.getQueueInfo()
 Key: HADOOP-14968
 URL: https://issues.apache.org/jira/browse/HADOOP-14968
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 3.0.0-beta1
Reporter: Daniel Templeton
Priority: Minor


{code}
QueueInfo rootQueue =
rmClient.getQueueInfo(getQueueInfoRequest(ROOT, false, true, true))
  .getQueueInfo();
getChildQueues(rootQueue, queues, false);
{code}

The final parameter to {{getQueueInfoRequest()}} should match the final 
parameter to {{getChildQueues()}}.  They should both be false in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Do we still have nightly (or even weekly) unit test run for Hadoop projects?

[Wangda Tan]

Gotcha, thanks!

- Wangda

On Thu, Oct 19, 2017 at 7:25 AM, Sean Busbey  wrote:

> Here's the email from last night to common-dev@hadoop:
>
> https://s.apache.org/ARe1
>
> On Wed, Oct 18, 2017 at 10:42 PM, Akira Ajisaka 
> wrote:
>
>> Yes, qbt runs nightly and it sends e-mail to dev lists.
>> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/
>>
>> Regards,
>> Akira
>>
>>
>> On 2017/10/19 7:54, Wangda Tan wrote:
>>
>>> Hi,
>>>
>>> Do we still have nightly (or even weekly) unit test run for Hadoop
>>> projects? I couldn't find it on Jenkins dashboard and I haven't seen
>>> reports set to dev lists for a while.
>>>
>>> Thanks,
>>> Wangda
>>>
>>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>
>
> --
> busbey
>

[jira] [Created] (HADOOP-14969) Improve diagnostics in secure DataNode startup

[Ajay Kumar (JIRA)]

Ajay Kumar created HADOOP-14969:
---

 Summary: Improve diagnostics in secure DataNode startup
 Key: HADOOP-14969
 URL: https://issues.apache.org/jira/browse/HADOOP-14969
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Ajay Kumar
Assignee: Ajay Kumar


When DN secure mode configuration is incorrect, it throws the following 
exception from Datanode#checkSecureConfig
{code}
  private static void checkSecureConfig(DNConf dnConf, Configuration conf,
  SecureResources resources) throws RuntimeException {
if (!UserGroupInformation.isSecurityEnabled()) {
  return;
}
...
throw new RuntimeException("Cannot start secure DataNode without " +
  "configuring either privileged resources or SASL RPC data transfer " +
  "protection and SSL for HTTP.  Using privileged resources in " +
  "combination with SASL RPC data transfer protection is not supported.");
{code}
The DN should print more useful diagnostics as to what exactly what went wrong.
Also when starting secure DN with resources then the startup scripts should 
launch the SecureDataNodeStarter class. If no SASL is configured and 
SecureDataNodeStarter is not used, then we could mention that too.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [Update] Apache Hadoop 2.8.2 Release Status

[Junping Du]

A quick update: the last patch (YARN-7230) for docker container support in 2.8 
just get committed yesterday. Now there is no left blocker/critical issues for 
2.8.2 and I checked all landed commits are matching with JIRA's fix version. 
With kicking off a new RC build, I will publish RC bits for vote once the build 
process get finished. In the mean time, please hold on any commits to 
branch-2.8.2 unless it really belongs to a blocker and please ping me ahead. 

Thanks all for your patience!

Thanks,

Junping

From: Junping Du 
Sent: Friday, September 22, 2017 5:57 PM
To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Cc: Shane Kumpf; Miklos Szegedi; Varun Vasudev
Subject: [Update] Apache Hadoop 2.8.2 Release Status

Hi folks,
 I would like to give you a quick update on 2.8.2 release status:

- First release candidate (RC0) is published over the last weekend, but several 
docker container blockers (bugs, documents, etc.)
 are reported so we decided to cancel the RC0 for vote.

- New coming release blockers (for docker container support) are YARN-7034 
(just committed), YARN-6623, YARN-6930 and YARN-7230.
Shane, Miklos and Varun are actively working on this. Appreciate the effort 
here!

- I will kick off new release candidate (RC1) once these blockers are resolved.

To all committers, branch-2.8.2 is still open for blocker/critical issues 
landing, but for major/minor/trivial issues, please commit to branch-2.8 and 
marked the fixed version as 2.8.3.

Thanks all for heads up. Have a good weekend!


Thanks,

Junping


From: Junping Du 
Sent: Tuesday, September 5, 2017 2:57 PM
To: larry mccay; Steve Loughran
Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

I assume the quiet over the holiday means we agreed to move forward without 
taking HADOOP-14439 into 2.8.2.
There is a new release building (docker based) issue could be related to 
HADOOP-14474 where we removed oracle java 7 installer due to recent download 
address/contract change by Oracle. The build refuse to work - report as 
JAVA_HOME issue, but hard coded my local java home in create-release or 
Dockerfile doesn't help so we may need to add java 7 installation back (no 
matter Oracle JDK 7 or openJDK 7).
Filed HADOOP-14842 with more details to track as blocker for 2.8.2.

Thanks,

Junping

From: Junping Du 
Sent: Friday, September 1, 2017 12:37 PM
To: larry mccay; Steve Loughran
Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

This issue (HADOOP-14439) is out of my radar given it is marked as Minor 
priority. If my understanding is correct, here is a trade-off between security 
and backward compatibility. IMO, priority of security is generally higher than 
backward compatibility especially 2.8.0 is still non-production release.
I think we should skip this for 2.8.2 in case it doesn't break compatibility 
from 2.7.x. Thoughts?

Thanks,

Junping

From: larry mccay 
Sent: Friday, September 1, 2017 10:55 AM
To: Steve Loughran
Cc: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org
Subject: Re: Apache Hadoop 2.8.2 Release Plan

If we do "fix" this in 2.8.2 we should seriously consider not doing so in
3.0.
This is a very poor practice.

I can see an argument for backward compatibility in 2.8.x line though.

On Fri, Sep 1, 2017 at 1:41 PM, Steve Loughran 
wrote:

> One thing we need to consider is
>
> HADOOP-14439: regression: secret stripping from S3x URIs breaks some
> downstream code
>
> Hadoop 2.8 has a best-effort attempt to strip out secrets from the
> toString() value of an s3a or s3n path where someone has embedded them in
> the URI; this has caused problems in some uses, specifically: when people
> use secrets this way (bad) and assume that you can round trip paths to
> string and back
>
> Should we fix this? If so, Hadoop 2.8.2 is the time to do it
>
>
> > On 1 Sep 2017, at 11:14, Junping Du  wrote:
> >
> > HADOOP-14814 get committed and HADOOP-9747 get push out to 2.8.3, so we
> are clean on blocker/critical issues now.
> > I finish practice of going through JACC report and no more incompatible
> public API changes get found between 2.8.2 and 2.7.4. Also I check commit
> history and fixed 10+ commits which are missing from branch-2.8.2 for some
> reason. So, the current branch-2.8.2 should be good to go for RC stage, and
> I will kick off our first RC tomorrow.
> > In the meanwhile, please don't land any commits to branch-2.8.2 since
> now. If some issues really belong to blocker, please ping me on the JIRA

[VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Junping Du]

Hi folks,
 I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.

 Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and 
will be the latest stable/production release for Apache Hadoop - it includes 
315 new fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical 
issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC1

  The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
66c47f2a01ad9637879e95f80c41f798373828fb

  The maven artifacts are available via 
repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1064

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 10/24/2017 6pm PST time.

Thanks,

Junping

[jira] [Resolved] (HADOOP-13767) Aliyun Connection broken when idle then 1 minutes or build than 3 hours

[Genmao Yu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Genmao Yu resolved HADOOP-13767.

Resolution: Won't Fix

> Aliyun Connection broken when idle then 1 minutes or build than 3 hours
> ---
>
> Key: HADOOP-13767
> URL: https://issues.apache.org/jira/browse/HADOOP-13767
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: fs, fs/oss
>Affects Versions: 3.0.0-alpha2
>Reporter: Genmao Yu
>Assignee: Genmao Yu
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org