[jira] Created: (HDFS-1666) HdfsProxy breakes HDFS trunk build

[Konstantin Boudnik (JIRA)]

HdfsProxy breakes HDFS trunk build
--

 Key: HDFS-1666
 URL: https://issues.apache.org/jira/browse/HDFS-1666
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: contrib/hdfsproxy
Affects Versions: 0.23.0
Reporter: Konstantin Boudnik
Priority: Critical


two test cases were failing for a number of builds (see attached logs)

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Fwd: Hadoop-Hdfs-trunk - Build # 591 - Still Failing

[Konstantin Boudnik]

I have took a look at the test and I should note that the way they written if
kinda misleading. For instance the message we are seeing in the Hudson says 
  expected:<403> but was:<200>

where's the reality is that the expected was <200> and actual value was <403>.
Basically the order of assert calls is reversed in a number of places. While
this isn't a cause of the failure it is confuses the analysis.

That's be great to see a maintainer of this component to take a look at the
failures so we can eventually have a green HDFS bulld.

I have opened https://issues.apache.org/jira/browse/HDFS-1666 to track it

Cos

On Thu, Feb 24, 2011 at 10:14AM, Todd Lipcon wrote:
> Can someone familiar with hdfsproxy look into this consistent unit test
> failure? People voted in support of keeping this contrib, but it would be
> easier to be satisfied with that decision if someone stepped up to fix these
> tests that have been failing for quite some time.
> 
> -Todd
> 
> -- Forwarded message --
> From: Apache Hudson Server 
> Date: Thu, Feb 24, 2011 at 4:36 AM
> Subject: Hadoop-Hdfs-trunk - Build # 591 - Still Failing
> To: hdfs-dev@hadoop.apache.org
> 
> 
> See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk/591/
> 
> ###
> ## LAST 60 LINES OF THE CONSOLE
> ###
> [...truncated 719693 lines...]
>[mkdir] Created dir:
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target
> [echo]  Including clover.jar in the war file ...
> [cactifywar] Analyzing war:
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/hdfsproxy-2.0-test.war
> [cactifywar] Building war:
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/test.war
> 
> cactifywar:
> 
> test-cactus:
> [echo]  Free Ports: startup-57271 / http-57272 / https-57273
> [echo] Please take a deep breath while Cargo gets the Tomcat for running
> the servlet tests...
>[mkdir] Created dir:
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/tomcat-config
>[mkdir] Created dir:
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/tomcat-config/conf
>[mkdir] Created dir:
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/tomcat-config/webapps
>[mkdir] Created dir:
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/tomcat-config/temp
>[mkdir] Created dir:
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/logs
>[mkdir] Created dir:
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/reports
> [copy] Copying 1 file to
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/tomcat-config/conf
> [copy] Copying 1 file to
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/tomcat-config/conf
> [copy] Copying 1 file to
> /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/tomcat-config/conf
>   [cactus] -
>   [cactus] Running tests against Tomcat 5.x @ http://localhost:57272
>   [cactus] -
>   [cactus] Deploying
> [/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/test.war]
> to
> [/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/contrib/hdfsproxy/target/tomcat-config/webapps]...
>   [cactus] Tomcat 5.x starting...
> Server [Apache-Coyote/1.1] started
>   [cactus] WARNING: multiple versions of ant detected in path for junit
>   [cactus]
>  
> jar:file:/homes/hudson/tools/ant/latest/lib/ant.jar!/org/apache/tools/ant/Project.class
>   [cactus]  and
> jar:file:/homes/hudson/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
>   [cactus] Running org.apache.hadoop.hdfsproxy.TestAuthorizationFilter
>   [cactus] Tests run: 4, Failures: 2, Errors: 0, Time elapsed: 0.454 sec
>   [cactus] Test org.apache.hadoop.hdfsproxy.TestAuthorizationFilter FAILED
>   [cactus] Running org.apache.hadoop.hdfsproxy.TestLdapIpDirFilter
>   [cactus] Tomcat 5.x started on port [57272]
>   [cactus] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.32 sec
>   [cactus] Running org.apache.hadoop.hdfsproxy.TestProxyFilter
>   [cactus] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.347 sec
>   [cactus] Running org.apache.hadoop.hdfsproxy.TestProxyForwardServlet
>   [cactus] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 0.307 sec
>   [cactus] Running org.apache.hadoop.hdfsproxy.TestProxyUtil
>   [cactus] Tests r

[jira] Created: (HDFS-1667) Consider redesign of block report processing

[Matt Foley (JIRA)]

Consider redesign of block report processing


 Key: HDFS-1667
 URL: https://issues.apache.org/jira/browse/HDFS-1667
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: name-node
Affects Versions: 0.22.0
Reporter: Matt Foley
Assignee: Matt Foley


The current implementation of block reporting has the datanode send the entire 
list of blocks resident on that node in every report, hourly.  
BlockManager.processReport in the namenode runs each block report through 
reportDiff() first, to build four linked lists of replicas for different 
dispositions, then processes each list.  During that process, every block 
belonging to the node (even the unchanged blocks) are removed and re-linked in 
that node's blocklist.  The entire process happens under the global 
FSNamesystem write lock, so there is essentially zero concurrency.  It takes 
about 90 milliseconds to process a single 50,000-replica block report in the 
namenode, during which no other read or write activities can occur.

There are several opportunities for improvement in this design.  In order of 
probable benefit, they include:
1. Change the datanode to send a differential report.  This saves the namenode 
from having to do most of the work in reportDiff(), and avoids the need to 
re-link all the unchanged blocks during the "diff" calculation.
2. Keep the linked lists of "to do" work, but modify reportDiff() so that it 
can be done under a read lock instead of the write lock.  Then only the 
processing of the lists needs to be under the write lock.  Since the number of 
blocks changed is usually much smaller than the number unchanged, this should 
improve concurrency.
3. Eliminate the linked lists and just immediately process each changed block 
as it is read from the block report.  The work on HDFS-1295 indicates that this 
might save a large fraction of the total block processing time at scale, due to 
the much smaller number of objects created and garbage collected during 
processing of hundreds of millions of blocks.
4. As a sidelight to #3, remove linked list use from BlockManager.addBlock().  
It currently uses linked lists as an argument to processReportedBlock() even 
though addBlock() only processes a single replica on each call.  This should be 
replaced with a call to immediately process the block instead of enqueuing it.



-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira