Build failed in Jenkins: Hadoop-Common-trunk #427

2012-06-03 Thread Apache Jenkins Server 
See 

Changes:

[umamahesh] HDFS-2025. Go Back to File View link is not working in tail.jsp. 
Contributed by Ashish and Sravan.

[eli] MAPREDUCE-4307. TeraInputFormat calls FileSystem.getDefaultBlockSize() 
without a Path - Failure when using ViewFileSystem. Contributed by Ahmed Radwan

--
[...truncated 22724 lines...]
Setting project property: project.packaging -> jar
Setting project property: project.build.directory -> 

Setting project property: project.build.outputDirectory -> 

Setting project property: project.build.testOutputDirectory -> 

Setting project property: project.build.sourceDirectory -> 

Setting project property: project.build.testSourceDirectory -> 

Setting project property: localRepository ->id: local
  url: file:///home/jenkins/.m2/repository/
   layout: none
Setting project property: settings.localRepository -> 
/home/jenkins/.m2/repository
Setting project property: org.apache.hadoop:hadoop-annotations:jar -> 

Setting project property: jdiff:jdiff:jar -> 
/home/jenkins/.m2/repository/jdiff/jdiff/1.0.9/jdiff-1.0.9.jar
Setting project property: com.google.guava:guava:jar -> 
/home/jenkins/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar
Setting project property: com.google.code.findbugs:jsr305:jar -> 
/home/jenkins/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar
Setting project property: commons-cli:commons-cli:jar -> 
/home/jenkins/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar
Setting project property: org.apache.commons:commons-math:jar -> 
/home/jenkins/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar
Setting project property: xmlenc:xmlenc:jar -> 
/home/jenkins/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar
Setting project property: commons-httpclient:commons-httpclient:jar -> 
/home/jenkins/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar
Setting project property: commons-codec:commons-codec:jar -> 
/home/jenkins/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar
Setting project property: commons-net:commons-net:jar -> 
/home/jenkins/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar
Setting project property: commons-io:commons-io:jar -> 
/home/jenkins/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar
Setting project property: javax.servlet:servlet-api:jar -> 
/home/jenkins/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar
Setting project property: org.mortbay.jetty:jetty:jar -> 
/home/jenkins/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar
Setting project property: org.mortbay.jetty:jetty-util:jar -> 
/home/jenkins/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar
Setting project property: asm:asm:jar -> 
/home/jenkins/.m2/repository/asm/asm/3.2/asm-3.2.jar
Setting project property: com.sun.jersey:jersey-core:jar -> 
/home/jenkins/.m2/repository/com/sun/jersey/jersey-core/1.8/jersey-core-1.8.jar
Setting project property: com.sun.jersey:jersey-json:jar -> 
/home/jenkins/.m2/repository/com/sun/jersey/jersey-json/1.8/jersey-json-1.8.jar
Setting project property: org.codehaus.jettison:jettison:jar -> 
/home/jenkins/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar
Setting project property: stax:stax-api:jar -> 
/home/jenkins/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar
Setting project property: com.sun.xml.bind:jaxb-impl:jar -> 
/home/jenkins/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar
Setting project property: javax.xml.bind:jaxb-api:jar -> 
/home/jenkins/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar
Setting project property: javax.activation:activation:jar -> 
/home/jenkins/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar
Setting project property: org.codehaus.jackson:jackson-core-asl:jar -> 
/home/jenkins/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar
Setting project property: org.codehaus.jackson:jackson-jaxrs:jar -> 
/home/jenkins/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.8/jackson-jaxrs-1.8.8.jar
Setting project property: org.codehaus.jackso

Build failed in Jenkins: Hadoop-Common-0.23-Build #272

2012-06-03 Thread Apache Jenkins Server 
See 

--
[...truncated 12431 lines...]
  [javadoc] Loading source files for package org.apache.hadoop.fs.ftp...
  [javadoc] Loading source files for package org.apache.hadoop.fs.kfs...
  [javadoc] Loading source files for package org.apache.hadoop.fs.local...
  [javadoc] Loading source files for package org.apache.hadoop.fs.permission...
  [javadoc] Loading source files for package org.apache.hadoop.fs.s3...
  [javadoc] Loading source files for package org.apache.hadoop.fs.s3native...
  [javadoc] Loading source files for package org.apache.hadoop.fs.shell...
  [javadoc] Loading source files for package org.apache.hadoop.fs.viewfs...
  [javadoc] Loading source files for package org.apache.hadoop.http...
  [javadoc] Loading source files for package org.apache.hadoop.http.lib...
  [javadoc] Loading source files for package org.apache.hadoop.io...
  [javadoc] Loading source files for package org.apache.hadoop.io.compress...
  [javadoc] Loading source files for package 
org.apache.hadoop.io.compress.bzip2...
  [javadoc] Loading source files for package 
org.apache.hadoop.io.compress.lz4...
  [javadoc] Loading source files for package 
org.apache.hadoop.io.compress.snappy...
  [javadoc] Loading source files for package 
org.apache.hadoop.io.compress.zlib...
  [javadoc] Loading source files for package org.apache.hadoop.io.file.tfile...
  [javadoc] Loading source files for package org.apache.hadoop.io.nativeio...
  [javadoc] Loading source files for package org.apache.hadoop.io.retry...
  [javadoc] Loading source files for package org.apache.hadoop.io.serializer...
  [javadoc] Loading source files for package 
org.apache.hadoop.io.serializer.avro...
  [javadoc] Loading source files for package org.apache.hadoop.ipc...
  [javadoc] Loading source files for package org.apache.hadoop.ipc.metrics...
  [javadoc] Loading source files for package org.apache.hadoop.jmx...
  [javadoc] Loading source files for package org.apache.hadoop.log...
  [javadoc] Loading source files for package org.apache.hadoop.log.metrics...
  [javadoc] Loading source files for package org.apache.hadoop.metrics...
  [javadoc] Loading source files for package org.apache.hadoop.metrics.file...
  [javadoc] Loading source files for package 
org.apache.hadoop.metrics.ganglia...
  [javadoc] Loading source files for package org.apache.hadoop.metrics.jvm...
  [javadoc] Loading source files for package org.apache.hadoop.metrics.spi...
  [javadoc] Loading source files for package org.apache.hadoop.metrics.util...
  [javadoc] Loading source files for package org.apache.hadoop.metrics2...
  [javadoc] Loading source files for package 
org.apache.hadoop.metrics2.annotation...
  [javadoc] Loading source files for package 
org.apache.hadoop.metrics2.filter...
  [javadoc] Loading source files for package org.apache.hadoop.metrics2.impl...
  [javadoc] Loading source files for package org.apache.hadoop.metrics2.lib...
  [javadoc] Loading source files for package org.apache.hadoop.metrics2.sink...
  [javadoc] Loading source files for package 
org.apache.hadoop.metrics2.sink.ganglia...
  [javadoc] Loading source files for package 
org.apache.hadoop.metrics2.source...
  [javadoc] Loading source files for package org.apache.hadoop.metrics2.util...
  [javadoc] Loading source files for package org.apache.hadoop.net...
  [javadoc] Loading source files for package org.apache.hadoop.record...
  [javadoc] Loading source files for package 
org.apache.hadoop.record.compiler...
  [javadoc] Loading source files for package 
org.apache.hadoop.record.compiler.ant...
  [javadoc] Loading source files for package 
org.apache.hadoop.record.compiler.generated...
  [javadoc] Loading source files for package org.apache.hadoop.record.meta...
  [javadoc] Loading source files for package org.apache.hadoop.security...
  [javadoc] Loading source files for package 
org.apache.hadoop.security.authorize...
  [javadoc] Loading source files for package org.apache.hadoop.security.token...
  [javadoc] Loading source files for package 
org.apache.hadoop.security.token.delegation...
  [javadoc] Loading source files for package org.apache.hadoop.tools...
  [javadoc] Loading source files for package org.apache.hadoop.util...
  [javadoc] Loading source files for package org.apache.hadoop.util.bloom...
  [javadoc] Loading source files for package org.apache.hadoop.util.hash...
  [javadoc] 2 errors
 [xslt] Processing 

 to 

 [xslt] Loading stylesheet 
/home/jenkins/tools/findbugs/latest/src/xsl/default.xsl
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.6:run (pre-dist) @ hadoop-common ---
[INFO] Executing tasks

main:
[INFO] Executed ta

[jira] [Created] (HADOOP-8468) Umbrella of enhancements to support different failure and locality topologies

2012-06-03 Thread Junping Du (JIRA) 
Junping Du created HADOOP-8468:
--

 Summary: Umbrella of enhancements to support different failure and 
locality topologies
 Key: HADOOP-8468
 URL: https://issues.apache.org/jira/browse/HADOOP-8468
 Project: Hadoop Common
  Issue Type: Bug
  Components: ha, io
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du
Priority: Critical


The current hadoop network topology (described in some previous issues like: 
Hadoop-692) works well in classic three-tiers network when it comes out. 
However, it does not take into account other failure models or changes in the 
infrastructure that can affect network bandwidth efficiency like: 
virtualization. 
Virtualized platform has following genes that shouldn't been ignored by hadoop 
topology in scheduling tasks, placing replica, do balancing or fetching block 
for reading: 
1. VMs on the same physical host are affected by the same hardware failure. In 
order to match the reliability of a physical deployment, replication of data 
across two virtual machines on the same host should be avoided.
2. The network between VMs on the same physical host has higher throughput and 
lower latency and does not consume any physical switch bandwidth.
Thus, we propose to make hadoop network topology extend-able and introduce a 
new level in the hierarchical topology, a node group level, which maps well 
onto an infrastructure that is based on a virtualized environment.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8469) Make NetworkTopology class pluggable and support user specified topology class

2012-06-03 Thread Junping Du (JIRA) 
Junping Du created HADOOP-8469:
--

 Summary: Make NetworkTopology class pluggable and support user 
specified topology class
 Key: HADOOP-8469
 URL: https://issues.apache.org/jira/browse/HADOOP-8469
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du


The class NetworkTopology is where the three-layer hierarchical topology is 
modeled in the current code base and is instantiated directly by the 
DatanodeManager and Balancer.
To support alternative topologies, changes were make the topology class 
pluggable, that is to support using a user specified topology class specified 
in the Hadoop configuration file core-defaul.xml. The user specified topology 
class is instantiated using reflection in the same manner as other customizable 
classes in Hadoop. If no use specified topology class is found, the fallback is 
to use the NetworkTopology to preserve current behavior. To make it possible to 
reuse code in NetworkTopology several minor changes were made to make the class 
more extensible. The NetworkTopology class is currently annotated with 
@InterfaceAudience.LimitedPrivate({"HDFS", "MapReduce"}) and 
@InterfaceStability.Unstable.
The proposed changes in NetworkTopology listed below
1. Some fields were changes from private to protected
2. Added some protected methods so that sub classes could override behavior
3. Added a new method,isNodeGroupAware,to NetworkTopology
4. The inner class InnerNode was made a package protected class to it would be 
easier to subclass

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8470) Implementation of 4-layer subclass of NetworkTopology (NetworkTopologyWithNodeGroup)

2012-06-03 Thread Junping Du (JIRA) 
Junping Du created HADOOP-8470:
--

 Summary: Implementation of 4-layer subclass of NetworkTopology 
(NetworkTopologyWithNodeGroup)
 Key: HADOOP-8470
 URL: https://issues.apache.org/jira/browse/HADOOP-8470
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du


To support the four-layer hierarchical topology shown in attached figure as a 
subclass of NetworkTopology, NetworkTopologyWithNodeGroup was developed along 
with unit tests. NetworkTopologyWithNodeGroup overriding the methods add, 
remove, and pseudoSortByDistance were the most relevant to support the 
four-layer topology. The method seudoSortByDistance selects the nodes to use 
for reading data and sorts the nodes in sequence of node-local, 
nodegroup-local, rack- local, rack–off. Another slightly change to 
seudoSortByDistance is to support cases of separation data node and node 
manager: if the reader cannot be found in NetworkTopology tree (formed by data 
nodes only), then it will try to sort according to reader's sibling node in the 
tree.
The distance calculation changes the weights from 0 (local), 2 (rack- local), 4 
(rack-off) to: 0 (local), 2 (nodegroup-local), 4 (rack-local), 6 (rack-off).
The additional node group layer should be specified in the topology script or 
table mapping, e.g. input 10.1.1.1, output: /rack1/nodegroup1
A subclass on InnerNode, InnerNodeWithNodeGroup, was also needed to support 
NetworkTopologyWithNodeGroup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8471) Make ReplicaPlacementPolicyDefault extensible for reuse code in subclass

2012-06-03 Thread Junping Du (JIRA) 
Junping Du created HADOOP-8471:
--

 Summary: Make ReplicaPlacementPolicyDefault extensible for reuse 
code in subclass
 Key: HADOOP-8471
 URL: https://issues.apache.org/jira/browse/HADOOP-8471
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: ha, io
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du


ReplicaPlacementPolicy is already a pluggable component in Hadoop. A user 
specified ReplicaPlacementPolicy can be specified in the hdfs-site.xml 
configuration under the key "dfs.block.replicator.classname". However, to make 
it possible to reuse code in ReplicaPlacementPolicyDefault a few of its methods 
were changed from private to protected. ReplicaPlacementPolicy and 
BlockPlacementPolicyDefault are currently annotated with 
@InterfaceAudience.Private.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8472) Implementation of ReplicaPlacementPolicyNodeGroup to support 4-layer network topology

2012-06-03 Thread Junping Du (JIRA) 
Junping Du created HADOOP-8472:
--

 Summary: Implementation of ReplicaPlacementPolicyNodeGroup to 
support 4-layer network topology
 Key: HADOOP-8472
 URL: https://issues.apache.org/jira/browse/HADOOP-8472
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: ha, io
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du


A subclass of ReplicaPlacementPolicyDefault, ReplicaPlacementPolicyNodeGroup 
was developed along with unit tests to support the four-layer hierarchical 
topology.
The replica placement strategy used in ReplicaPlacementPolicyNodeGroup 
virtualization is almost the same as the original one. The differences are:
1. The 3rd replica will be off node group of the 2nd replica
2. If there is no local node available, the 1st replica will be placed on a 
node in the local node group.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8473) Update Balancer to support new NetworkTopology with NodeGroup

2012-06-03 Thread Junping Du (JIRA) 
Junping Du created HADOOP-8473:
--

 Summary: Update Balancer to support new NetworkTopology with 
NodeGroup
 Key: HADOOP-8473
 URL: https://issues.apache.org/jira/browse/HADOOP-8473
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: util
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du


Since the Balancer is a Hadoop Tool, it was updated to be directly aware of 
four-layer hierarchy instead of creating an alternative Balancer 
implementation. To accommodate extensibility, a new protected method, 
doChooseNodesForCustomFaultDomain is now called from the existing chooseNodes 
method so that a subclass of the Balancer could customize the balancer 
algotirhm for other failure and locality topologies. An alternative option is 
to encapsulate the algorithm used for the four-layer hierarchy into a 
collaborating strategy class.
The key changes introduced to support a four-layer hierarchy were to override 
the algorithm of choosing  pairs for balancing. Unit tests were 
created to test the new algorithm.
The algorithm now makes sure to choose the target and source node on the same 
node group for balancing as the first priority. Then the overall balancing 
policy is: first doing balancing between nodes within the same nodegroup then 
the same rack and off rack at last. Also, we need to check no duplicated 
replicas live in the same node group after balancing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8474) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology

2012-06-03 Thread Junping Du (JIRA) 
Junping Du created HADOOP-8474:
--

 Summary: Make locatlity in YARN's container assignment and task 
scheduling pluggable for other deployment topology
 Key: HADOOP-8474
 URL: https://issues.apache.org/jira/browse/HADOOP-8474
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du


There are several classes in YARN’s container assignment and task scheduling 
algorithms that relate to data locality which were updated to give preference 
to running a container on other locality besides node-local and rack-local 
(like nodegroup-local). This propose to make these data structure/algorithms 
pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class 
ScheduledRequests was made a package level class to it would be easier to 
create a subclass, ScheduledRequestsWithNodeGroup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8475) 4-layer topology (with NodeGroup layer) implementation of Container Assignment and Task Scheduling (for YARN)

2012-06-03 Thread Junping Du (JIRA) 
Junping Du created HADOOP-8475:
--

 Summary: 4-layer topology (with NodeGroup layer) implementation of 
Container Assignment and Task Scheduling (for YARN)
 Key: HADOOP-8475
 URL: https://issues.apache.org/jira/browse/HADOOP-8475
 Project: Hadoop Common
  Issue Type: Sub-task
Affects Versions: 2.0.0-alpha, 1.0.0
Reporter: Junping Du
Assignee: Junping Du


There are several classes in YARN’s container assignment and task scheduling 
algorithms that related to data locality which were updated to give preference 
to running a container on the same nodegroup. This section summarized the 
changes in the patch that provides a new implementation to support a four-layer 
hierarchy.
When the ApplicationMaster makes a resource allocation request to the scheduler 
of ResourceManager, it will add the node group to the list of attributes in the 
ResourceRequest. The parameters of the resource request will change from 
 to .
After receiving the ResoureRequest the RM scheduler will assign containers for 
requests in the sequence of data-local, nodegroup-local, rack-local and 
off-switch.Then, ApplicationMaster schedules tasks on allocated containers in 
sequence of data- local, nodegroup-local, rack-local and off-switch.
In terms of code changes made to YARN task scheduling, we updated the class 
ContainerRequestEvent so that applications can requests for containers can 
include anodegroup. In RM schedulers, FifoScheduler and CapacityScheduler were 
updated. For the FifoScheduler, the changes were in the method 
assignContainers. For the Capacity Scheduler the method assignContainersOnNode 
in the class of LeafQueue was updated. In both changes a new method, 
assignNodeGroupLocalContainers() was added in between the assignment data-local 
and rack-local.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira