Re: Heads up: branch-2.1-beta

[Arun C Murthy]

Which one are you talking about Ralph? This doesn't show up on any blocker list.

http://s.apache.org/hadoop-blocker-bugs

Arun


On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:

> Not trying to be a pain, but I am trying to get clarification. The protocol 
> buffer support is still broken. Do you intend to release 2.1 with that 
> unfixed?
> 
> 
> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla  wrote:
> 
>> Re-posting here to the wider audience:
>> 
>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>> all suggestions so far. It would be great if people could take another look
>> at it. I ll iterate fast on any comments so we get this in by the time rest
>> of the code pieces are committed.
>> 
>> Thanks
>> Karthik
>> 
>> 
>> 
>> 
>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain  wrote:
>> 
>>> Just curious of your procedures. Given that there is at least one blocker
>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>> anyway?
>>> 
>>> 
>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur  wrote:
>>> 
 If the intention is to get the release out in time for the Hadoop Summit
>>> we
 have a very tight schedule.
 
 Because the release vote runs for 7 days, we should have an RC latest
 Monday afternoon, and we should encourage folks to verify & vote ASAP, so
 if we need to cut a new RC we can do it on Tuesday. Another thing to
 consider is that if the changes on an RC are corrections that do not
>>> affect
 code, we could agree on not reseting the voting period clock if we need
>>> to
 cut a new RC (ie doc, build, notes changes).
 
 Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>> are
 YARN-752, MAPREDUCE-5171 & YARN-787.
 
 The first 2 are already +1ed, the last one needs to be reviewed.
 
 I have not committed the first 2 ones yet because I don't want to disrupt
 things for the folks doing QA.
 
 Arun, as you are coordinating the work for this release, please do commit
 them or give me the go ahead and I'll commit.
 
 Also, it  would be great if you can review YARN-787 (as per discussions,
 the changes on the milli-slot calculations do not affect the current
 calculations, that would be left for MAPREDUCE-5311 to do).
 
 I'll be checking my email over the weekend and I can take care of some
 stuff if needed (while the monkeys sleep).
 
 Thx
 
 
 
 On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur >>> wrote:
 
> Following is a revisited assessment of JIRAs I would like to get in the
> 2.1 release:
> 
> From the 1st group I think all 3 should make.
> 
> From the 2nd group I think YARN-791 should make it for sure and ideally
> MAPREDUCE-5130.
> 
> From the 3rd group, I don't think this JIRA will make it.
> 
> From the 4th group, we don't need to worry about this or 2.1
> 
> Thanks
> 
> Alejandro
> 
> --
> JIRAs that are in shape to make it to 2.1
> 
> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> for requested nodes
> 
> impact: behavior change
> 
> status: patch avail, +1ed.
> 
> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
> 
> impact: Addition to MRAM HTTP API
> 
> status: patch avail, +1ed, needs to be committed
> 
> * YARN-787: Remove resource min from Yarn client API
> 
> impact: Yarn client API change
> 
> status: patch avail, needs to be reviewed. (the calculation of
>>> slot-millis
> is not affected, the MIN is taken from conf for now)
> 
> --
> JIRAs that require minor work to make it to 2.1
> 
> * YARN-521: Augment AM - RM client module to be able to request
>>> containers
> only at specific locations
> 
> impact: AMRM client API change
> 
> status: patch not avail yet (requires YARN-752)
> 
> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>> with
> /nodes REST API
> 
> impact: Yarn client API & proto change
> 
> status: patch avail, review in progress
> 
> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
> 
> impact: behavior change
> 
> status: patch avail but some tests are failing
> 
> --
> JIRAs that require significant work to make it to 2.1 and may not make
>>> it
> 
> * YARN-649: Make container logs available over HTTP in plain text
> 
> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> does not change API)
> 
> status: patch avail, review in progress
> 
> --
>

Re: Heads up: branch-2.1-beta

[Arun C Murthy]

On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> 
> Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
> YARN-752, MAPREDUCE-5171 & YARN-787.

I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it right 
now.

I'm ok with YARN-752 & MAPREDUCE-5171 going in, but don't consider either a 
blocker.

Suresh - will HDFS-4777 go in soon too?

Vinod - How about YARN-386?

thanks,
Arun

> 
> Thx
> 
> 
> 
> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur wrote:
> 
>> Following is a revisited assessment of JIRAs I would like to get in the
>> 2.1 release:
>> 
>> From the 1st group I think all 3 should make.
>> 
>> From the 2nd group I think YARN-791 should make it for sure and ideally
>> MAPREDUCE-5130.
>> 
>> From the 3rd group, I don't think this JIRA will make it.
>> 
>> From the 4th group, we don't need to worry about this or 2.1
>> 
>> Thanks
>> 
>> Alejandro
>> 
>> --
>> JIRAs that are in shape to make it to 2.1
>> 
>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>> for requested nodes
>> 
>> impact: behavior change
>> 
>> status: patch avail, +1ed.
>> 
>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>> 
>> impact: Addition to MRAM HTTP API
>> 
>> status: patch avail, +1ed, needs to be committed
>> 
>> * YARN-787: Remove resource min from Yarn client API
>> 
>> impact: Yarn client API change
>> 
>> status: patch avail, needs to be reviewed. (the calculation of slot-millis
>> is not affected, the MIN is taken from conf for now)
>> 
>> --
>> JIRAs that require minor work to make it to 2.1
>> 
>> * YARN-521: Augment AM - RM client module to be able to request containers
>> only at specific locations
>> 
>> impact: AMRM client API change
>> 
>> status: patch not avail yet (requires YARN-752)
>> 
>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
>> /nodes REST API
>> 
>> impact: Yarn client API & proto change
>> 
>> status: patch avail, review in progress
>> 
>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>> 
>> impact: behavior change
>> 
>> status: patch avail but some tests are failing
>> 
>> --
>> JIRAs that require significant work to make it to 2.1 and may not make it
>> 
>> * YARN-649: Make container logs available over HTTP in plain text
>> 
>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>> does not change API)
>> 
>> status: patch avail, review in progress
>> 
>> --
>> JIRAs that don't need to make it to 2.1
>> 
>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>> counter constants
>> 
>> impact: behavior change
>> 
>> status: per discussion we should first add memory-millis and vcores-millis
>> 
>> --
>> 
>> 
>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik  wrote:
>> 
>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy 
>>> wrote:
 
 On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> 
> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
> based on Hadoop 2.0.x codeline we plan to start running the same
>>> battery
> of integration tests on the branch-2.1-beta.
> 
> We plan to simply file JIRAs if anything gets detected and I will also
> publish the URL of the Jenkins job once it gets created.
 
 Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>> Blocker priority to catch attention.
>>> 
>>> The build is up and running (and all green on all of the 9 Linux
>>> platforms!):
>>>http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>> 
>>> The immediate benefit here is that we get to see that the
>>> build is ok on all these Linuxes and all anybody can easily
>>> install packaged Hadoop 2.1.0 nightly builds.
>>> 
>>> Starting from next week, I'll start running regular tests
>>> on these bits and will keep you guys posted!
>>> 
>>> Thanks,
>>> Roman.
>>> 
>> 
>> 
>> 
>> --
>> Alejandro
>> 
> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy]

Karthik,

 My apologies - I'll definitely spend time on this *today*.

Arun

On Jun 15, 2013, at 4:18 PM, Karthik Kambatla wrote:

> Re-posting here to the wider audience:
> 
> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
> all suggestions so far. It would be great if people could take another look
> at it. I ll iterate fast on any comments so we get this in by the time rest
> of the code pieces are committed.
> 
> Thanks
> Karthik
> 
> 
> 
> 
> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain  wrote:
> 
>> Just curious of your procedures. Given that there is at least one blocker
>> JIRA out there that has yet to be fully resolved, do you intend to release
>> anyway?
>> 
>> 
>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur  wrote:
>> 
>>> If the intention is to get the release out in time for the Hadoop Summit
>> we
>>> have a very tight schedule.
>>> 
>>> Because the release vote runs for 7 days, we should have an RC latest
>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>> consider is that if the changes on an RC are corrections that do not
>> affect
>>> code, we could agree on not reseting the voting period clock if we need
>> to
>>> cut a new RC (ie doc, build, notes changes).
>>> 
>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>> are
>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>> 
>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>> 
>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>> things for the folks doing QA.
>>> 
>>> Arun, as you are coordinating the work for this release, please do commit
>>> them or give me the go ahead and I'll commit.
>>> 
>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>> the changes on the milli-slot calculations do not affect the current
>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>> 
>>> I'll be checking my email over the weekend and I can take care of some
>>> stuff if needed (while the monkeys sleep).
>>> 
>>> Thx
>>> 
>>> 
>>> 
>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur >> wrote:
>>> 
 Following is a revisited assessment of JIRAs I would like to get in the
 2.1 release:
 
 From the 1st group I think all 3 should make.
 
 From the 2nd group I think YARN-791 should make it for sure and ideally
 MAPREDUCE-5130.
 
 From the 3rd group, I don't think this JIRA will make it.
 
 From the 4th group, we don't need to worry about this or 2.1
 
 Thanks
 
 Alejandro
 
 --
 JIRAs that are in shape to make it to 2.1
 
 * YARN-752: In AMRMClient, automatically add corresponding rack requests
 for requested nodes
 
 impact: behavior change
 
 status: patch avail, +1ed.
 
 * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
 
 impact: Addition to MRAM HTTP API
 
 status: patch avail, +1ed, needs to be committed
 
 * YARN-787: Remove resource min from Yarn client API
 
 impact: Yarn client API change
 
 status: patch avail, needs to be reviewed. (the calculation of
>> slot-millis
 is not affected, the MIN is taken from conf for now)
 
 --
 JIRAs that require minor work to make it to 2.1
 
 * YARN-521: Augment AM - RM client module to be able to request
>> containers
 only at specific locations
 
 impact: AMRM client API change
 
 status: patch not avail yet (requires YARN-752)
 
 * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>> with
 /nodes REST API
 
 impact: Yarn client API & proto change
 
 status: patch avail, review in progress
 
 * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
 
 impact: behavior change
 
 status: patch avail but some tests are failing
 
 --
 JIRAs that require significant work to make it to 2.1 and may not make
>> it
 
 * YARN-649: Make container logs available over HTTP in plain text
 
 impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
 does not change API)
 
 status: patch avail, review in progress
 
 --
 JIRAs that don't need to make it to 2.1
 
 * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
 counter constants
 
 impact: behavior change
 
 status: per discussion we should first add memory-millis and
>> vcores-millis
 
 --
 
 
 On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik 
>> wrote:
 

Build failed in Jenkins: Hadoop-Hdfs-trunk #1432

[Apache Jenkins Server]

See 

Changes:

[cnauroth] HADOOP-9632. TestShellCommandFencer will fail if there is a 'host' 
machine in the network. Contributed by Chuan Liu.

[vinodkv] YARN-693. Modified RM to send NMTokens on allocate call so that AMs 
can then use them for authentication with NMs. Contributed by Omkar Vinit Joshi.

[acmurthy] MAPREDUCE-5192. Allow for alternate resolutions of 
TaskCompletionEvents. Contributed by Chris Douglas.

[vinodkv] YARN-781. Exposing LOGDIR in all containers' environment which should 
be used by containers for logging purposes. Contributed by Jian He.

--
[...truncated 10298 lines...]
Running org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
Tests run: 49, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.408 sec
Running org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.401 sec
Running org.apache.hadoop.hdfs.web.resources.TestParam
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.515 sec
Running org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.855 sec
Running org.apache.hadoop.hdfs.web.TestOffsetUrlInputStream
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.166 sec
Running org.apache.hadoop.hdfs.web.TestWebHDFS
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 103.935 sec
Running org.apache.hadoop.hdfs.web.TestWebHdfsUrl
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.286 sec
Running org.apache.hadoop.hdfs.web.TestJsonUtil
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.171 sec
Running org.apache.hadoop.hdfs.web.TestWebHdfsTimeouts
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.974 sec
Running org.apache.hadoop.hdfs.web.TestAuthFilter
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.405 sec
Running org.apache.hadoop.hdfs.TestConnCache
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.105 sec
Running org.apache.hadoop.hdfs.TestDFSClientRetries
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 141.596 sec
Running org.apache.hadoop.hdfs.TestListPathServlet
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.276 sec
Running org.apache.hadoop.hdfs.TestParallelShortCircuitRead
Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.164 sec
Running org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 117.414 sec
Running org.apache.hadoop.hdfs.TestFileCreationEmpty
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.201 sec
Running org.apache.hadoop.hdfs.TestSetrepIncreasing
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.776 sec
Running org.apache.hadoop.hdfs.TestEncryptedTransfer
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 83.979 sec
Running org.apache.hadoop.hdfs.TestDFSUpgrade
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.754 sec
Running org.apache.hadoop.hdfs.TestCrcCorruption
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 28.874 sec
Running org.apache.hadoop.hdfs.TestHFlush
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 23.975 sec
Running org.apache.hadoop.hdfs.TestFileAppendRestart
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.671 sec
Running org.apache.hadoop.hdfs.TestDatanodeReport
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.455 sec
Running org.apache.hadoop.hdfs.TestShortCircuitLocalRead
Tests run: 10, Failures: 0, Errors: 0, Skipped: 10, Time elapsed: 0.193 sec
Running org.apache.hadoop.hdfs.TestFileInputStreamCache
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.202 sec
Running org.apache.hadoop.hdfs.TestRestartDFS
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.338 sec
Running org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.833 sec
Running org.apache.hadoop.hdfs.TestDFSRemove
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.908 sec
Running org.apache.hadoop.hdfs.TestHDFSTrash
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.447 sec
Running org.apache.hadoop.hdfs.TestClientReportBadBlock
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 54.292 sec
Running org.apache.hadoop.hdfs.TestQuota
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.693 sec
Running org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.501 sec
Running org.apache.hadoop.hdfs.TestDatanodeRegistration
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.283 sec
Running org.apache.hadoop.hdfs.TestAbandonBlock
Tests run: 2, Failures: 0, Er

Hadoop-Hdfs-trunk - Build # 1432 - Still Failing

[Apache Jenkins Server]

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1432/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 10491 lines...]
[WARNING] The POM for org.eclipse.m2e:lifecycle-mapping:jar:1.0.0 is missing, 
no dependency information available
[WARNING] Failed to retrieve plugin descriptor for 
org.eclipse.m2e:lifecycle-mapping:1.0.0: Plugin 
org.eclipse.m2e:lifecycle-mapping:1.0.0 or one of its dependencies could not be 
resolved: Failed to read artifact descriptor for 
org.eclipse.m2e:lifecycle-mapping:jar:1.0.0
[INFO] 
[INFO] --- maven-clean-plugin:2.4.1:clean (default-clean) @ hadoop-hdfs-project 
---
[INFO] Deleting 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/target
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-hdfs-project 
---
[INFO] Executing tasks

main:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/target/test-dir
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-source-plugin:2.1.2:test-jar-no-fork (hadoop-java-sources) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.0:enforce (dist-enforce) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-site-plugin:3.0:attach-descriptor (attach-descriptor) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- maven-javadoc-plugin:2.8.1:jar (module-javadocs) @ 
hadoop-hdfs-project ---
[INFO] Not executing Javadoc as the project is not a Java classpath-capable 
package
[INFO] 
[INFO] --- maven-checkstyle-plugin:2.6:checkstyle (default-cli) @ 
hadoop-hdfs-project ---
[INFO] 
[INFO] --- findbugs-maven-plugin:2.3.2:findbugs (default-cli) @ 
hadoop-hdfs-project ---
[INFO] ** FindBugsMojo execute ***
[INFO] canGenerate is false
[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE 
[1:31:42.499s]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS Project  SUCCESS [1.734s]
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 1:31:45.095s
[INFO] Finished at: Sun Jun 16 13:05:29 UTC 2013
[INFO] Final Memory: 38M/299M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on 
project hadoop-hdfs: ExecutionException; nested exception is 
java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked 
VM terminated without saying properly goodbye. VM crash or System.exit called ? 
-> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Build step 'Execute shell' marked build as failure
Archiving artifacts
Updating MAPREDUCE-5192
Updating YARN-781
Updating YARN-693
Updating HADOOP-9632
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
No tests ran.

Re: Heads up: branch-2.1-beta

[Arun C Murthy]

On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:

> 
> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
>> 
>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
>> YARN-752, MAPREDUCE-5171 & YARN-787.
> 
> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it right 
> now.

I committed YARN-787. Thanks.

Arun

Re: Block deletion after benchmarks

[Harsh J]

Eitan,

I don't completely get your question. TestDFSIO is a test that will
create several files for testing the IO and then delete it at the end
of the test.

Block deletions in HDFS is an asynchronous process. File deletions are
instantaneous (as a transaction in the namespace) but the identified
block's deletions are progressively done over DN heartbeats and are
throttled (to avoid a storm of deletes from affecting DN memory
usage). You can look at dfs.namenode.invalidate.work.pct.per.iteration
in 
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/hdfs-default.xml
to control this to make it go faster, but am not sure I got your
question right. The test just uses FS APIs, the FS just has a
different data (not file) deletion behavior - the Test isn't
responsible for that.

On Tue, Jun 11, 2013 at 4:03 AM, Eitan Rosenfeld  wrote:
> Hi all,
>
> In my two-datanode cluster, I require that file operations on the
> underlying filesystem take place in the same order.  Essentially, I wish
> for blocks to be created, written, and/or deleted deterministically across
> datanodes.
>
> However, this is not the case towards the end of the TestDFSIO benchmark.
> Several blocks are deleted, but each datanode performs this deletion at a
> *different time* relative to the last few blocks being written.
>
> What component is initiating the block deletion at the end of the
> benchmark?
>
> (It seems to be the Replication Monitor, but I'm unclear on what causes the
> Replication Monitor to suddenly run and delete blocks at the end of the
> benchmark).  I am using Hadoop 1.0.4.
>
> Thank you,
> Eitan Rosenfeld



-- 
Harsh J

Re: dfs.datanode.socket.reuse.keepalive

[Harsh J]

Hi Colin,

Do we have a JIRA already for this? Is it
https://issues.apache.org/jira/browse/HDFS-4307?

On Mon, Jun 10, 2013 at 11:05 PM, Todd Lipcon  wrote:
> +1 for dropping the client side expiry down to something like 1-2 seconds.
> I'd rather do that than up the server side, since the server side resource
> (DN threads) is likely to be more contended.
>
> -Todd
>
> On Fri, Jun 7, 2013 at 4:29 PM, Colin McCabe  wrote:
>
>> Hi all,
>>
>> HDFS-941 added dfs.datanode.socket.reuse.keepalive.  This allows
>> DataXceiver worker threads in the DataNode to linger for a second or
>> two after finishing a request, in case the client wants to send
>> another request.  On the client side, HDFS-941 added a SocketCache, so
>> that subsequent client requests could reuse the same socket.  Sockets
>> were closed purely by an LRU eviction policy.
>>
>> Later, HDFS-3373 added a minimum expiration time to the SocketCache,
>> and added a thread which periodically closed old sockets.
>>
>> However, the default timeout for SocketCache (which is now called
>> PeerCache) is much longer than the DN would possibly keep the socket
>> open.  Specifically, dfs.client.socketcache.expiryMsec defaults to 2 *
>> 60 * 1000 (2 minutes), whereas dfs.datanode.socket.reuse.keepalive
>> defaults to 1000 (1 second).
>>
>> I'm not sure why we have such a big disparity here.  It seems like
>> this will inevitably lead to clients trying to use sockets which have
>> gone stale, because the server closes them way before the client
>> expires them.  Unless I'm missing something, we should probably either
>> lengthen the keepalive, or shorten the socket cache expiry, or both.
>>
>> thoughts?
>> Colin
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera



--
Harsh J

[jira] [Created] (HDFS-4908) Reduce snapshot inode memory usage

[Tsz Wo (Nicholas), SZE (JIRA)]

Tsz Wo (Nicholas), SZE created HDFS-4908:


 Summary: Reduce snapshot inode memory usage
 Key: HDFS-4908
 URL: https://issues.apache.org/jira/browse/HDFS-4908
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: namenode, snapshots
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE


Snapshots currently use INodeFile and INodeDirectory for storing previous 
file/dir states as snapshot inodes.  However, INodeFile and INodeDirectory have 
some fields not used by snapshot inodes such as parent reference, inode id, 
block/children reference.  The memory footprint could be reduced by having some 
specific classes for snapshot inodes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur]

Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
around noon today (SUN noon PST).

What is your take on YARN-791 & MAPREDUCE-5130?



On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy  wrote:

>
> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
>
> >
> > On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> >>
> >> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
> are
> >> YARN-752, MAPREDUCE-5171 & YARN-787.
> >
> > I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
> right now.
>
> I committed YARN-787. Thanks.
>
> Arun
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Arun C Murthy]

Alejandro,

 Can you please take a look at MAPREDUCE-5327? This is related to YARN-787.

thanks,
Arun

On Jun 16, 2013, at 8:56 AM, Alejandro Abdelnur wrote:

> Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
> around noon today (SUN noon PST).
> 
> What is your take on YARN-791 & MAPREDUCE-5130?
> 
> 
> 
> On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy  wrote:
> 
>> 
>> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
>> 
>>> 
>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
 
 Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>> are
 YARN-752, MAPREDUCE-5171 & YARN-787.
>>> 
>>> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
>> right now.
>> 
>> I committed YARN-787. Thanks.
>> 
>> Arun
>> 
>> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy]

HDFS-4866 was committed a couple of days ago - hence isn't showing up in my 
filters.

Can you please file a new jira for the linker issues?

thanks,
Arun

On Jun 16, 2013, at 10:16 AM, Ralph Castain wrote:

> HDFS-4866 is marked as a blocker. A patch was submitted and applied, but it 
> only fixes compilation. As I marked on the ticket, the results remain 
> unusable due to linker issues.
> 
> 
> On Jun 16, 2013, at 5:33 AM, Arun C Murthy  wrote:
> 
>> Which one are you talking about Ralph? This doesn't show up on any blocker 
>> list.
>> 
>> http://s.apache.org/hadoop-blocker-bugs
>> 
>> Arun
>> 
>> 
>> On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:
>> 
>>> Not trying to be a pain, but I am trying to get clarification. The protocol 
>>> buffer support is still broken. Do you intend to release 2.1 with that 
>>> unfixed?
>>> 
>>> 
>>> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla  wrote:
>>> 
 Re-posting here to the wider audience:
 
 HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
 all suggestions so far. It would be great if people could take another look
 at it. I ll iterate fast on any comments so we get this in by the time rest
 of the code pieces are committed.
 
 Thanks
 Karthik
 
 
 
 
 On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain  wrote:
 
> Just curious of your procedures. Given that there is at least one blocker
> JIRA out there that has yet to be fully resolved, do you intend to release
> anyway?
> 
> 
> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur  wrote:
> 
>> If the intention is to get the release out in time for the Hadoop Summit
> we
>> have a very tight schedule.
>> 
>> Because the release vote runs for 7 days, we should have an RC latest
>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>> consider is that if the changes on an RC are corrections that do not
> affect
>> code, we could agree on not reseting the voting period clock if we need
> to
>> cut a new RC (ie doc, build, notes changes).
>> 
>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
> are
>> YARN-752, MAPREDUCE-5171 & YARN-787.
>> 
>> The first 2 are already +1ed, the last one needs to be reviewed.
>> 
>> I have not committed the first 2 ones yet because I don't want to disrupt
>> things for the folks doing QA.
>> 
>> Arun, as you are coordinating the work for this release, please do commit
>> them or give me the go ahead and I'll commit.
>> 
>> Also, it  would be great if you can review YARN-787 (as per discussions,
>> the changes on the milli-slot calculations do not affect the current
>> calculations, that would be left for MAPREDUCE-5311 to do).
>> 
>> I'll be checking my email over the weekend and I can take care of some
>> stuff if needed (while the monkeys sleep).
>> 
>> Thx
>> 
>> 
>> 
>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur > wrote:
>> 
>>> Following is a revisited assessment of JIRAs I would like to get in the
>>> 2.1 release:
>>> 
>>> From the 1st group I think all 3 should make.
>>> 
>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>> MAPREDUCE-5130.
>>> 
>>> From the 3rd group, I don't think this JIRA will make it.
>>> 
>>> From the 4th group, we don't need to worry about this or 2.1
>>> 
>>> Thanks
>>> 
>>> Alejandro
>>> 
>>> --
>>> JIRAs that are in shape to make it to 2.1
>>> 
>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>> for requested nodes
>>> 
>>> impact: behavior change
>>> 
>>> status: patch avail, +1ed.
>>> 
>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>> 
>>> impact: Addition to MRAM HTTP API
>>> 
>>> status: patch avail, +1ed, needs to be committed
>>> 
>>> * YARN-787: Remove resource min from Yarn client API
>>> 
>>> impact: Yarn client API change
>>> 
>>> status: patch avail, needs to be reviewed. (the calculation of
> slot-millis
>>> is not affected, the MIN is taken from conf for now)
>>> 
>>> --
>>> JIRAs that require minor work to make it to 2.1
>>> 
>>> * YARN-521: Augment AM - RM client module to be able to request
> containers
>>> only at specific locations
>>> 
>>> impact: AMRM client API change
>>> 
>>> status: patch not avail yet (requires YARN-752)
>>> 
>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
> with
>>> /nodes REST API
>>> 
>>> impact:

[jira] [Created] (HDFS-4909) Protocol buffer support compiles under C, but fails to link due to duplicate symbols

[Ralph Castain (JIRA)]

Ralph Castain created HDFS-4909:
---

 Summary: Protocol buffer support compiles under C, but fails to 
link due to duplicate symbols
 Key: HDFS-4909
 URL: https://issues.apache.org/jira/browse/HDFS-4909
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: datanode, journal-node, namenode
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Ralph Castain
Priority: Blocker
 Fix For: 3.0.0, 2.1.0-beta


The revised protocol buffer support seems to be compiling for me when using the 
protobuf-c cross-compiler. However, I still cannot construct a library of the 
results. This may be a Hadoop issue, or could be an issue with the protobuf-c 
cross-compiler. What I see are a bunch of these when attempting to link the 
resulting .o files:

/home/common/hadoop/hadoop-common/foo/obj/DatanodeProtocol.pb-c.o: In function 
`hadoop_hdfsreport_bad_blocks_request_proto_init':
DatanodeProtocol.pb-c.c.text+0x2bb4): multiple definition of 
`hadoop_hdfsreport_bad_blocks_request_proto_init'
/home/common/hadoop/hadoop-common/foo/obj/ClientNamenodeProtocol.pb-c.o:ClientNamenodeProtocol.pb-c.c.text+0x277d):
 first defined here

>From what I can see, this is caused by the

package hadoop.hdfs;

line in the .proto files, when combined with the later

import "hdfs.proto";

This appears to bring a complete copy of the hdfs.proto file into the source 
code, which then recompiles it - leading to the duplicate symbols.

I have attached an updated pcreate.pl script that illustrates the problem. 
Excluding the following .proto files allows all to be successfully built and 
linked:

DatanodeProtocol
ClientNamenodeProtocol
QJournalProtocol

HTH
Ralph


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur]

Arun, actually YARN-787 fixed the problem. Run those tests from trunk and
branch-2.1-beta HEADs without issues. Without YARN-787 they fail.

Thx


On Sun, Jun 16, 2013 at 12:03 PM, Arun C Murthy  wrote:

> Alejandro,
>
>  Can you please take a look at MAPREDUCE-5327? This is related to YARN-787.
>
> thanks,
> Arun
>
> On Jun 16, 2013, at 8:56 AM, Alejandro Abdelnur wrote:
>
> > Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
> > around noon today (SUN noon PST).
> >
> > What is your take on YARN-791 & MAPREDUCE-5130?
> >
> >
> >
> > On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy 
> wrote:
> >
> >>
> >> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
> >>
> >>>
> >>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> 
>  Of the JIRAs in my laundry list for 2.1 the ones I would really want
> in
> >> are
>  YARN-752, MAPREDUCE-5171 & YARN-787.
> >>>
> >>> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
> >> right now.
> >>
> >> I committed YARN-787. Thanks.
> >>
> >> Arun
> >>
> >>
> >
> >
> > --
> > Alejandro
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Arun C Murthy]

Responses inline:

On Jun 16, 2013, at 1:04 PM, Roman Shaposhnik wrote:

> But there's a bit of bad news too (or at least the news that need to
> be triaged). At
> this point I don't know whether the Hadoop code is to blame or the
> tests/components
> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>   1. HDFS append integration tests failed:
>
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>   It seems that there's some incompatibility between the client code
>   that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>   and the current Hadoop 2.1.0.

This is well known, we need to recompile against hadoop-2.1.0-beta.

> 
>2. Quite a few Sqoop tests ended up failing because of what seems
> like AM not
>realizing that one of the tasks exited and waiting for it to
> timeout. In the end
>the task is getting killed like this:
> AttemptID:attempt_1371348647940_0030_m_00_2 Timed out
> after 600 secsContainer killed by the ApplicationMaster.
> but it takes a VERY long time (on the task side the log is
> attached bellow).

Essentially, this is a Sqoop error - we could investigate why it took 600s, but 
doesn't look like a blocker to me.

We'll probably need another 2.1.1-beta anyway...

> 
> 3. There's a couple of Hive tests (out of more than a dozen) that
> failed in a pretty odd way
>  (scroll to the very bottom of every page to see the excpetion):
> 
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
> 
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
> What's peculiar here is that nothing has failed *before* or
> *after* these particular
> tests. Hence I don't think that the state of the cluster
> deployment is to blame.

Both errors seemed to be related to Hive unit tests failing since 
MiniHDFSCluster didn't come up:

Job Submission failed with exception 
'org.apache.hadoop.ipc.RemoteException(File 
/user/jenkins/.staging/job_1371348647940_0308/job.split could only be 
replicated to 0 nodes instead of minReplication (=1).  There are 4 datanode(s) 
running and no node(s) are excluded in this operation.


> 
>  4. All of the Mahout tests failed with the following:
>  
> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>  This odd, and as I said -- all I know at this point is that
> the very same
>   tests running the very same Mahout pass with Hadoop 2.0.5-alpha.

Again, this is related to fact that we need to recompile Mahout against 
2.1.0-beta - in particular this was due to the compatibility work done via 
MAPREDUCE-5156 (part of MAPREDUCE-5108).


Arun

Re: Heads up: branch-2.1-beta

[Arun C Murthy]

Roman,

 Is there a chance you can run the tests with the full stack built against 
branch-2.1-beta and help us know where we are?

thanks,
Arun

On Jun 16, 2013, at 4:50 PM, Arun C Murthy wrote:

> Responses inline:
> 
> On Jun 16, 2013, at 1:04 PM, Roman Shaposhnik wrote:
> 
>> But there's a bit of bad news too (or at least the news that need to
>> be triaged). At
>> this point I don't know whether the Hadoop code is to blame or the
>> tests/components
>> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>>   1. HDFS append integration tests failed:
>>
>> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>>   It seems that there's some incompatibility between the client code
>>   that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 
>> release)
>>   and the current Hadoop 2.1.0.
> 
> This is well known, we need to recompile against hadoop-2.1.0-beta.
> 
>> 
>>2. Quite a few Sqoop tests ended up failing because of what seems
>> like AM not
>>realizing that one of the tasks exited and waiting for it to
>> timeout. In the end
>>the task is getting killed like this:
>> AttemptID:attempt_1371348647940_0030_m_00_2 Timed out
>> after 600 secsContainer killed by the ApplicationMaster.
>> but it takes a VERY long time (on the task side the log is
>> attached bellow).
> 
> Essentially, this is a Sqoop error - we could investigate why it took 600s, 
> but doesn't look like a blocker to me.
> 
> We'll probably need another 2.1.1-beta anyway...
> 
>> 
>> 3. There's a couple of Hive tests (out of more than a dozen) that
>> failed in a pretty odd way
>>  (scroll to the very bottom of every page to see the excpetion):
>> 
>> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
>> 
>> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
>> What's peculiar here is that nothing has failed *before* or
>> *after* these particular
>> tests. Hence I don't think that the state of the cluster
>> deployment is to blame.
> 
> Both errors seemed to be related to Hive unit tests failing since 
> MiniHDFSCluster didn't come up:
> 
> Job Submission failed with exception 
> 'org.apache.hadoop.ipc.RemoteException(File 
> /user/jenkins/.staging/job_1371348647940_0308/job.split could only be 
> replicated to 0 nodes instead of minReplication (=1).  There are 4 
> datanode(s) running and no node(s) are excluded in this operation.
> 
> 
>> 
>>  4. All of the Mahout tests failed with the following:
>>  
>> http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>>  This odd, and as I said -- all I know at this point is that
>> the very same
>>   tests running the very same Mahout pass with Hadoop 2.0.5-alpha.
> 
> Again, this is related to fact that we need to recompile Mahout against 
> 2.1.0-beta - in particular this was due to the compatibility work done via 
> MAPREDUCE-5156 (part of MAPREDUCE-5108).
> 
> 
> Arun
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

[jira] [Created] (HDFS-4910) TestPermission failed in branch-2

[Chuan Liu (JIRA)]

Chuan Liu created HDFS-4910:
---

 Summary: TestPermission failed in branch-2
 Key: HDFS-4910
 URL: https://issues.apache.org/jira/browse/HDFS-4910
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: Chuan Liu
Assignee: Chuan Liu


TestPermission.testSymlinkPermissions unit test introduced in HDFS-4765 has a 
regression in branch-2. The test case is failing on both Windows and Linux.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira