I want to revise the source codes, which version is proper?

[Li Shengmei]

Hi, all

 I want to do some experiments with hadoop. I may revise the source
codes of hadoop. Which version is stable for my work?

 

Thanks,

May

 

Build failed in Jenkins: Hadoop-Common-trunk #511

[Apache Jenkins Server]

See 

Changes:

[tucu] MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov 
via tucu)

[eli] HADOOP-8075. Lower native-hadoop library log from info to debug. 
Contributed by Hızır Sefa İrken

[atm] MAPREDUCE-4577. HDFS-3672 broke 
TestCombineFileInputFormat.testMissingBlocks() test. Contributed by Aaron T. 
Myers.

[tucu] MAPREDUCE-4068. Jars in lib subdirectory of the submittable JAR are not 
added to the classpath (rkanter via tucu)

[atm] HADOOP-8721. ZKFC should not retry 45 times when attempting a graceful 
fence during a failover. Contributed by Vinayakumar B.

[atm] HDFS-3835. Long-lived 2NN cannot perform a checkpoint if security is 
enabled and the NN restarts with outstanding delegation tokens. Contributed by 
Aaron T. Myers.

[jitendra] HDFS-3819. Should check whether invalidate work percentage default 
value is not greater than 1.0f. Contributed by Jing Zhao.

[suresh] Moving HDFS-2686 and HDFS-3832 to branch-2 section in CHANGES.txt

[suresh] HDFS-3832. Remove protocol methods related to DistributedUpgrade. 
Contributed by Suresh Srinivas.

[eli] HDFS-3837. Fix DataNode.recoverBlock findbugs warning. Contributed by Eli 
Collins

[eli] HADOOP-8720. TestLocalFileSystem should use test root subdirectory. 
Contributed by Vlad Rozov

[eli] HDFS-3830. test_libhdfs_threaded: use forceNewInstance. Contributed by 
Colin Patrick McCabe

[suresh] HDFS-3817. Avoid printing SafeModeException stack trace in RPC server. 
Contributed by Brandon Li.

--
[...truncated 27185 lines...]
[DEBUG]   (s) debug = false
[DEBUG]   (s) effort = Default
[DEBUG]   (s) failOnError = true
[DEBUG]   (s) findbugsXmlOutput = false
[DEBUG]   (s) findbugsXmlOutputDirectory = 

[DEBUG]   (s) fork = true
[DEBUG]   (s) includeTests = false
[DEBUG]   (s) localRepository =id: local
  url: file:///home/jenkins/.m2/repository/
   layout: none

[DEBUG]   (s) maxHeap = 512
[DEBUG]   (s) nested = false
[DEBUG]   (s) outputDirectory = 

[DEBUG]   (s) outputEncoding = UTF-8
[DEBUG]   (s) pluginArtifacts = 
[org.codehaus.mojo:findbugs-maven-plugin:maven-plugin:2.3.2:, 
com.google.code.findbugs:bcel:jar:1.3.9:compile, 
org.codehaus.gmaven:gmaven-mojo:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-api:jar:1.3:compile, 
org.codehaus.gmaven.feature:gmaven-feature-api:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-1.5:jar:1.3:compile, 
org.codehaus.gmaven.feature:gmaven-feature-support:jar:1.3:compile, 
org.codehaus.groovy:groovy-all-minimal:jar:1.5.8:compile, 
org.apache.ant:ant:jar:1.7.1:compile, 
org.apache.ant:ant-launcher:jar:1.7.1:compile, jline:jline:jar:0.9.94:compile, 
org.codehaus.plexus:plexus-interpolation:jar:1.1:compile, 
org.codehaus.gmaven:gmaven-plugin:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-loader:jar:1.3:compile, 
org.codehaus.gmaven.runtime:gmaven-runtime-support:jar:1.3:compile, 
org.sonatype.gshell:gshell-io:jar:2.0:compile, 
com.thoughtworks.qdox:qdox:jar:1.10:compile, 
org.apache.maven.shared:file-management:jar:1.2.1:compile, 
org.apache.maven.shared:maven-shared-io:jar:1.1:compile, 
commons-lang:commons-lang:jar:2.4:compile, 
org.slf4j:slf4j-api:jar:1.5.10:compile, 
org.sonatype.gossip:gossip:jar:1.2:compile, 
org.apache.maven.reporting:maven-reporting-impl:jar:2.1:compile, 
commons-validator:commons-validator:jar:1.2.0:compile, 
commons-beanutils:commons-beanutils:jar:1.7.0:compile, 
commons-digester:commons-digester:jar:1.6:compile, 
commons-logging:commons-logging:jar:1.0.4:compile, oro:oro:jar:2.0.8:compile, 
xml-apis:xml-apis:jar:1.0.b2:compile, 
org.codehaus.groovy:groovy-all:jar:1.7.4:compile, 
org.apache.maven.reporting:maven-reporting-api:jar:3.0:compile, 
org.apache.maven.doxia:doxia-core:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-logging-api:jar:1.1.3:compile, 
xerces:xercesImpl:jar:2.9.1:compile, 
commons-httpclient:commons-httpclient:jar:3.1:compile, 
commons-codec:commons-codec:jar:1.2:compile, 
org.apache.maven.doxia:doxia-sink-api:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-decoration-model:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-site-renderer:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-module-xhtml:jar:1.1.3:compile, 
org.apache.maven.doxia:doxia-module-fml:jar:1.1.3:compile, 
org.codehaus.plexus:plexus-i18n:jar:1.0-beta-7:compile, 
org.codehaus.plexus:plexus-velocity:jar:1.1.7:compile, 
org.apache.velocity:velocity:jar:1.5:compile, 
commons-collections:commons-collections:jar:3.2:compile, 
org.apache.maven.shared:maven-doxia-tools:jar:1.2.1:compile, 
commons-io:commons-io:jar:1.4:compile, 
com.google.code.findbugs:findbugs-ant:jar:1.3.9:compile, 
com.google.code.findbugs:findbugs:jar:1.3.9:compile, 
com.google.code.find

Re: How to get TaskId from ContainerId or ApplicationId or Request in Hadoop 0.23??

[Robert Evans]

There really is no way.  The RM also has no knowledge of map tasks vs
reduce tasks nor should it know.

--Bobby

On 8/22/12 8:23 PM, "Shekhar Gupta"  wrote:

>In ResourceManager, is there any way to findout if the assigned container
>is going to execute a mapping task or a reduce task? I can access objects
>Container, Application and Request in ResourceManager, can I somehow get
>TaskId by using any of these objects?? Please let me know a way.
>
>Thanks.

[jira] [Created] (HADOOP-8723) Remove tests and tests-sources jars from classpath

[Jason Lowe (JIRA)]

Jason Lowe created HADOOP-8723:
--

 Summary: Remove tests and tests-sources jars from classpath
 Key: HADOOP-8723
 URL: https://issues.apache.org/jira/browse/HADOOP-8723
 Project: Hadoop Common
  Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Jason Lowe


Currently {{hadoop-config.sh}} is including any tests and tests-sources jars in 
the default classpath, as those jars are shipped in the dist tarball next to 
the main jars and the script is globbing everything in that directory.

The tests and tests-sources jars aren't required to run Hadoop, but they can 
cause breakage when inadvertently picked up.  See HDFS-3831 as an example.  
Ideally we should not be adding these jars to the classpath.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: I want to revise the source codes, which version is proper?

[Vinod Kumar Vavilapalli]

See http://hadoop.apache.org/common/releases.html for versions and their 
current status. You can checkout the source code from one of the links at 
http://hadoop.apache.org/common/version_control.html.

HTH,
+Vinod

On Aug 23, 2012, at 1:56 AM, Li Shengmei wrote:

> Hi, all
> 
> I want to do some experiments with hadoop. I may revise the source
> codes of hadoop. Which version is stable for my work?
> 
> 
> 
> Thanks,
> 
> May
> 
> 
> 

Re: How to get TaskId from ContainerId or ApplicationId or Request in Hadoop 0.23??

[Vinod Kumar Vavilapalli]

Moving to yarn-dev, as that is the right place for this discussion.

Can you let us know more about what  you are trying to accomplish? Working with 
MapReduce over YARN or your own YARN application?

If you are working with MR over YARN, you have to note that 
TaskId/TaskAttemptID/Map/Reduce tasks are MapReduce concepts. So 
ResourceManager has no idea about them. MapReduce ApplicationMaster is the 
place where you can obtain more such MR specific information.

If it is the later, your question is moot.

HTH,
+Vinod

On Aug 22, 2012, at 6:23 PM, Shekhar Gupta wrote:

> In ResourceManager, is there any way to findout if the assigned container
> is going to execute a mapping task or a reduce task? I can access objects
> Container, Application and Request in ResourceManager, can I somehow get
> TaskId by using any of these objects?? Please let me know a way.
> 
> Thanks.

Re: How to get TaskId from ContainerId or ApplicationId or Request in Hadoop 0.23??

[Shekhar Gupta]

Thanks Vinod. Yes it's the second case where I am working with MR over yarn.

When I run a job then for a specific Machine I am trying to compute that
how much time the machine takes to execute mapping tasks and reduce tasks.
What I am doing now is that I compute time difference between container
assignment and container released steps. But I can't determine if that
container was assigned for a mapping or reduce task. I'll try to look into
ApplicationMaster if I can get this information.



On Thu, Aug 23, 2012 at 11:27 AM, Vinod Kumar Vavilapalli <
vino...@hortonworks.com> wrote:

>
> Moving to yarn-dev, as that is the right place for this discussion.
>
> Can you let us know more about what  you are trying to accomplish? Working
> with MapReduce over YARN or your own YARN application?
>
> If you are working with MR over YARN, you have to note that
> TaskId/TaskAttemptID/Map/Reduce tasks are MapReduce concepts. So
> ResourceManager has no idea about them. MapReduce ApplicationMaster is the
> place where you can obtain more such MR specific information.
>
> If it is the later, your question is moot.
>
> HTH,
> +Vinod
>
> On Aug 22, 2012, at 6:23 PM, Shekhar Gupta wrote:
>
> > In ResourceManager, is there any way to findout if the assigned container
> > is going to execute a mapping task or a reduce task? I can access objects
> > Container, Application and Request in ResourceManager, can I somehow get
> > TaskId by using any of these objects?? Please let me know a way.
> >
> > Thanks.
>
>

[jira] [Created] (HADOOP-8724) Add improved APIs for globbing

[Robert Joseph Evans (JIRA)]

Robert Joseph Evans created HADOOP-8724:
---

 Summary: Add improved APIs for globbing
 Key: HADOOP-8724
 URL: https://issues.apache.org/jira/browse/HADOOP-8724
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Robert Joseph Evans
Assignee: Robert Joseph Evans


After the discussion on HADOOP-8709 it was decided that we need better APIs for 
globbing to remove some of the inconsistencies with other APIs.  Inorder to 
maintain backwards compatibility we should deprecate the existing APIs and add 
in new ones.


See HADOOP-8709 for more information about exactly how those APIs should look 
and behave.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-7942) enabling clover coverage reports fails hadoop unit test compilation

[Matt Foley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matt Foley resolved HADOOP-7942.


Resolution: Fixed

Seems like no one has observed a need for this in other branches in six months. 
 Resolving as part of 1.1.0 release cleanup.

> enabling clover coverage reports fails hadoop unit test compilation
> ---
>
> Key: HADOOP-7942
> URL: https://issues.apache.org/jira/browse/HADOOP-7942
> Project: Hadoop Common
>  Issue Type: Test
>Affects Versions: 1.1.0
> Environment: 
> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-1-Code-Coverage/13/console
>Reporter: Giridharan Kesavan
>Assignee: Jitendra Nath Pandey
> Fix For: 1.1.0
>
> Attachments: HADOOP-7942.branch-1.patch
>
>
> enabling clover reports fails compiling the following junit tests.
> link to the console output of jerkins :
> https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-1-Code-Coverage/13/console
> {noformat}
> [javac] 
> /tmp/clover50695626838999169.tmp/org/apache/hadoop/security/TestUserGroupInformation.java:224:
>  cannot find symbol
> ..
> [javac] 
> /tmp/clover50695626838999169.tmp/org/apache/hadoop/security/TestUserGroupInformation.java:225:
>  cannot find symbol
> ..
>  [javac] 
> /tmp/clover50695626838999169.tmp/org/apache/hadoop/security/TestJobCredentials.java:67:
>  cannot find symbol
> [javac] symbol  : class T 
> ..
> [javac] 
> /tmp/clover50695626838999169.tmp/org/apache/hadoop/security/TestJobCredentials.java:68:
>  cannot find symbol
> [javac] symbol  : class T
> .
> [javac] 
> /tmp/clover50695626838999169.tmp/org/apache/hadoop/fs/TestFileSystem.java:653:
>  cannot find symbol
> [javac] symbol  : class T
> .
> [javac] ^
> [javac] Note: Some input files use or override a deprecated API.
> [javac] Note: Recompile with -Xlint:deprecation for details.
> [javac] 5 errors
> [javac] 63 warnings
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8725) MR is broken when security is off

[Daryn Sharp (JIRA)]

Daryn Sharp created HADOOP-8725:
---

 Summary: MR is broken when security is off
 Key: HADOOP-8725
 URL: https://issues.apache.org/jira/browse/HADOOP-8725
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 0.23.3, 2.1.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Blocker


HADOOP-8225 broke MR when security is off.  MR was changed to stop re-reading 
the credentials that UGI had already read, and to stop putting those tokens 
back into the UGI where they already were.  UGI only reads a credentials file 
when security is enabled, but MR uses tokens (ie. job token) even when security 
is disabled...

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: New to Hadoop

[Chandrashekhar Kotekar]

Hi,

You dont need to do any course for Hadoop. I think first you read
documents, books and understand the architecture, concepts in Hadoop. Then
start using Hadoop and write some programs in it. Once you know basics then
you can contribute to Hadoop but dont jump directly into contributing code,
rather first understand Hadoop.

All the best.


Regards,
Chandrash3khar K0tekar
Mobile - 9766632117



On Fri, Aug 24, 2012 at 3:58 AM, Subodhini  wrote:

> Hello,
>
> I want to contribute to the apache hadoop project. I am absolutely new to
> Hadoop. Can you guys please guide me which is the best place to start for
> hadoop and what are the prerequisites courses in computer science for
> hadoop?
>
> I have work experience in Java programming language and hold a background
> in Computer Science.
>
> Regards,
> Subodhini
>

Does anyone do related work on the log of hadoop?

[Li Shengmei]

Hi, all

 I want to do some study on the log of hadoop. I intend to konw how
hadoop record the IO in the logs. 

Does anyone do related work and give some suggestions? 

Thanks,

May