[jira] [Created] (HDFS-4720) Misleading warning message in WebhdfsFileSystem when trying to check whether path exist using webhdfs url

2013-04-21 Thread Yu Li (JIRA)
Yu Li created HDFS-4720:
---

 Summary: Misleading warning message in WebhdfsFileSystem when 
trying to check whether path exist using webhdfs url
 Key: HDFS-4720
 URL: https://issues.apache.org/jira/browse/HDFS-4720
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 1.1.2, 1.1.1
Reporter: Yu Li
Assignee: Yu Li
Priority: Minor


When we trying to check whether the target path exists in HDFS through webhdfs, 
if the given path to check doesn't exist, we will always observe warning 
message like:
===
13/04/21 04:38:01 WARN web.WebHdfsFileSystem: Original exception is
org.apache.hadoop.ipc.RemoteException: user = biadmin, proxyUser = null, path = 
/testWebhdfs
at 
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:114)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:294)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$500(WebHdfsFileSystem.java:103)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.getResponse(WebHdfsFileSystem.java:552)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$Runner.run(WebHdfsFileSystem.java:473)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:404)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:573)
at 
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:584)
at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:768)
===

while actually FileNotFoundException should be expected when the operation is 
GETFILESTATUS and target path doesn't exist. The fact that RemoteException 
didn't include the real exception class(FileNotFoundException) in its toString 
method even make the message more misleading, since from the message user won't 
know what the warning is about

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


Jenkins build is back to stable : Hadoop-Hdfs-0.23-Build #588

2013-04-21 Thread Apache Jenkins Server
See 



Hadoop-Hdfs-trunk - Build # 1379 - Still Failing

2013-04-21 Thread Apache Jenkins Server
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1379/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 10220 lines...]
Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.874 sec
Running org.apache.hadoop.fs.viewfs.TestViewFsAtHdfsRoot
Tests run: 42, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.96 sec
Running org.apache.hadoop.fs.viewfs.TestViewFileSystemHdfs
Tests run: 39, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.262 sec
Running org.apache.hadoop.fs.viewfs.TestViewFsFileStatusHdfs
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.032 sec
Running org.apache.hadoop.fs.permission.TestStickyBit
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.048 sec
Running org.apache.hadoop.fs.loadGenerator.TestLoadGenerator
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.349 sec
Running org.apache.hadoop.fs.TestFcHdfsSymlink
Tests run: 69, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.585 sec
Running org.apache.hadoop.fs.TestFcHdfsCreateMkdir
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.326 sec
Running org.apache.hadoop.fs.TestHDFSFileContextMainOperations
Tests run: 60, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.086 sec
Running org.apache.hadoop.fs.TestVolumeId
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.067 sec

Results :

Failed tests:   
testReduceReplFactorDueToRejoinRespectsRackPolicy(org.apache.hadoop.hdfs.server.blockmanagement.TestBlocksWithNotEnoughRacks):
 Test resulted in an unexpected exit

Tests run: 1782, Failures: 1, Errors: 0, Skipped: 34

[INFO] 
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Hadoop HDFS  FAILURE 
[1:21:30.052s]
[INFO] Apache Hadoop HttpFS .. SKIPPED
[INFO] Apache Hadoop HDFS BookKeeper Journal . SKIPPED
[INFO] Apache Hadoop HDFS Project  SKIPPED
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 1:21:30.822s
[INFO] Finished at: Sun Apr 21 12:55:17 UTC 2013
[INFO] Final Memory: 23M/519M
[INFO] 
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on 
project hadoop-hdfs: There are test failures.
[ERROR] 
[ERROR] Please refer to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Hdfs-trunk/trunk/hadoop-hdfs-project/hadoop-hdfs/target/surefire-reports
 for the individual test results.
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
Build step 'Execute shell' marked build as failure
Archiving artifacts
Updating MAPREDUCE-5066
Updating YARN-534
Updating YARN-594
Updating HADOOP-9443
Sending e-mails to: hdfs-dev@hadoop.apache.org
Email was triggered for: Failure
Sending email for trigger: Failure



###
## FAILED TESTS (if any) 
##
No tests ran.

Build failed in Jenkins: Hadoop-Hdfs-trunk #1379

2013-04-21 Thread Apache Jenkins Server
See 

Changes:

[bikas] YARN-594. Update test and add comments in YARN-534 (Jian He via bikas)

[suresh] HADOOP-9443. Port winutils static code analysis change to trunk. 
Contributed by Chuan Liu.

[acmurthy] MAPREDUCE-5066. Added a timeout for the job.end.notification.url. 
Contributed by Ivan Mitic.

--
[...truncated 10027 lines...]
Running org.apache.hadoop.hdfs.protocolPB.TestPBHelper
Tests run: 24, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.507 sec
Running org.apache.hadoop.hdfs.TestDatanodeDeath
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 136.208 sec
Running org.apache.hadoop.hdfs.TestLeaseRecovery
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.555 sec
Running org.apache.hadoop.hdfs.TestLargeBlock
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 42.772 sec
Running org.apache.hadoop.hdfs.protocol.datatransfer.TestPacketReceiver
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.35 sec
Running org.apache.hadoop.hdfs.protocol.TestLayoutVersion
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.06 sec
Running org.apache.hadoop.hdfs.protocol.TestExtendedBlock
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.056 sec
Running org.apache.hadoop.hdfs.TestHDFSServerPorts
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.948 sec
Running org.apache.hadoop.hdfs.TestFileCreationDelete
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.575 sec
Running org.apache.hadoop.hdfs.TestDFSMkdirs
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.886 sec
Running org.apache.hadoop.hdfs.TestDistributedFileSystem
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 38.242 sec
Running org.apache.hadoop.hdfs.TestDecommission
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 122.645 sec
Running org.apache.hadoop.hdfs.TestLeaseRecovery2
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 61.298 sec
Running org.apache.hadoop.hdfs.TestFileStatus
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.16 sec
Running org.apache.hadoop.hdfs.TestBlockMissingException
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.2 sec
Running org.apache.hadoop.hdfs.TestDataTransferKeepalive
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.836 sec
Running org.apache.hadoop.hdfs.TestLeaseRenewer
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.02 sec
Running org.apache.hadoop.hdfs.TestFileAppend
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.908 sec
Running org.apache.hadoop.hdfs.TestDatanodeConfig
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.228 sec
Running org.apache.hadoop.hdfs.TestDisableConnCache
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.137 sec
Running org.apache.hadoop.hdfs.web.TestWebHdfsTokens
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.775 sec
Running org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
Tests run: 49, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 21.106 sec
Running org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs
Tests run: 50, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.34 sec
Running org.apache.hadoop.hdfs.web.resources.TestParam
Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.512 sec
Running org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.608 sec
Running org.apache.hadoop.hdfs.web.TestOffsetUrlInputStream
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.164 sec
Running org.apache.hadoop.hdfs.web.TestWebHDFS
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 104.15 sec
Running org.apache.hadoop.hdfs.web.TestWebHdfsUrl
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.269 sec
Running org.apache.hadoop.hdfs.web.TestJsonUtil
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.169 sec
Running org.apache.hadoop.hdfs.web.TestAuthFilter
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.389 sec
Running org.apache.hadoop.hdfs.TestConnCache
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.886 sec
Running org.apache.hadoop.hdfs.TestDFSClientRetries
Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 139.348 sec
Running org.apache.hadoop.hdfs.TestListPathServlet
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.166 sec
Running org.apache.hadoop.hdfs.TestParallelShortCircuitRead
Tests run: 4, Failures: 0, Errors: 0, Skipped: 4, Time elapsed: 0.162 sec
Running org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 115.789 sec
Running org.apache.hadoop.hdfs.TestFileCreationEmpty
Tests run: 1, Failures: 0,

[jira] [Created] (HDFS-4721) Speed up lease/block recovery when DN fails and a block goes into recovery

2013-04-21 Thread Varun Sharma (JIRA)
Varun Sharma created HDFS-4721:
--

 Summary: Speed up lease/block recovery when DN fails and a block 
goes into recovery
 Key: HDFS-4721
 URL: https://issues.apache.org/jira/browse/HDFS-4721
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Affects Versions: 2.0.3-alpha
Reporter: Varun Sharma


This was observed while doing HBase WAL recovery. HBase uses append to write to 
its write ahead log. So initially the pipeline is setup as

DN1 --> DN2 --> DN3

This WAL needs to be read when DN1 fails since it houses the HBase regionserver 
for the WAL.

HBase first recovers the lease on the WAL file. During recovery, we choose DN1 
as the primary DN to do the recovery even though DN1 has failed and is not 
heartbeating any more.

Avoiding the stale DN1 would speed up recovery and reduce hbase MTTR. There are 
two options.
a) Ride on HDFS 3703 and if stale node detection is turned on, we do not choose 
stale datanodes (typically not heart beated for 20-30 seconds) as primary DN(s)
b) We sort the replicas in order of last heart beat and always pick the ones 
which gave the most recent heart beat

Going to the dead datanode increases lease + block recovery since the block 
goes into UNDER_RECOVERY state even though no one is recovering it actively. 
Please let me know if this makes sense. If yes, whether we should move forward 
with a) or b).

Thanks


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Created] (HDFS-4722) TestGetConf#testFederation times out on Windows

2013-04-21 Thread Ivan Mitic (JIRA)
Ivan Mitic created HDFS-4722:


 Summary: TestGetConf#testFederation times out on Windows
 Key: HDFS-4722
 URL: https://issues.apache.org/jira/browse/HDFS-4722
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 3.0.0
Reporter: Ivan Mitic
Assignee: Ivan Mitic


Test times out on the below stack:

{code}
java.lang.Exception: test timed out after 1 milliseconds
at java.net.Inet4AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:849)
at java.net.InetAddress.getAddressFromNameService(InetAddress.java:1202)
at java.net.InetAddress.getAllByName0(InetAddress.java:1153)
at java.net.InetAddress.getAllByName(InetAddress.java:1083)
at java.net.InetAddress.getAllByName(InetAddress.java:1019)
at java.net.InetAddress.getByName(InetAddress.java:969)
at 
org.apache.hadoop.security.SecurityUtil$StandardHostResolver.getByName(SecurityUtil.java:543)
at 
org.apache.hadoop.security.SecurityUtil.getByName(SecurityUtil.java:530)
at 
org.apache.hadoop.net.NetUtils.createSocketAddrForHost(NetUtils.java:232)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:212)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:160)
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:149)
at 
org.apache.hadoop.hdfs.DFSUtil.getAddressesForNameserviceId(DFSUtil.java:483)
at org.apache.hadoop.hdfs.DFSUtil.getAddresses(DFSUtil.java:466)
at 
org.apache.hadoop.hdfs.DFSUtil.getNNServiceRpcAddresses(DFSUtil.java:592)
at 
org.apache.hadoop.hdfs.tools.TestGetConf.getAddressListFromConf(TestGetConf.java:109)
at 
org.apache.hadoop.hdfs.tools.TestGetConf.verifyAddresses(TestGetConf.java:209)
at 
org.apache.hadoop.hdfs.tools.TestGetConf.testFederation(TestGetConf.java:313)
{code} 


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira


[jira] [Created] (HDFS-4723) Occasional failure in TestDFSClientRetries#testGetFileChecksum because the number of available xcievers is set too low

2013-04-21 Thread Andrew Purtell (JIRA)
Andrew Purtell created HDFS-4723:


 Summary: Occasional failure in 
TestDFSClientRetries#testGetFileChecksum because the number of available 
xcievers is set too low
 Key: HDFS-4723
 URL: https://issues.apache.org/jira/browse/HDFS-4723
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 3.0.0, 2.0.4-alpha
Reporter: Andrew Purtell


Occasional failure in TestDFSClientRetries#testGetFileChecksum because the 
number of available xcievers is set too low. 

{noformat}
2013-04-21 18:48:28,273 WARN  datanode.DataNode 
(DataXceiverServer.java:run(161)) - 127.0.0.1:37608:DataXceiverServer: 
java.io.IOException: Xceiver count 3 exceeds the limit of concurrent xcievers: 2
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:143)
at java.lang.Thread.run(Thread.java:662)
2013-04-21 18:48:28,274 INFO  datanode.DataNode 
(DataXceiver.java:writeBlock(453)) - Datanode 2 got response for connect ack  
from downstream datanode with firstbadlink as 127.0.0.1:37608
2013-04-21 18:48:28,276 INFO  datanode.DataNode 
(DataXceiver.java:writeBlock(491)) - Datanode 2 forwarding connect ack to 
upstream firstbadlink is 127.0.0.1:37608
2013-04-21 18:48:28,276 ERROR datanode.DataNode 
(DataXceiver.java:writeBlock(477)) - 
DataNode{data=FSDataset{dirpath='[/home/ec2-user/jenkins/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3/current,
 
/home/ec2-user/jenkins/workspace/hadoop/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data4/current]'},
 localName='127.0.0.1:33298', 
storageID='DS-1506063529-10.174.86.97-33298-1366570107286', 
xmitsInProgress=0}:Exception transfering block 
BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071 to mirror 
127.0.0.1:37608: java.io.EOFException: Premature EOF: no length prefix available
2013-04-21 18:48:28,276 INFO  hdfs.DFSClient 
(DFSOutputStream.java:createBlockOutputStream(1105)) - Exception in 
createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink as 127.0.0.1:37608
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1096)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1019)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464)
2013-04-21 18:48:28,276 INFO  datanode.DataNode 
(DataXceiver.java:writeBlock(537)) - opWriteBlock 
BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071 received 
exception java.io.EOFException: Premature EOF: no length prefix available
2013-04-21 18:48:28,277 INFO  datanode.DataNode 
(BlockReceiver.java:receiveBlock(674)) - Exception for 
BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071
java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:414)
at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:644)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:506)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:65)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
at java.lang.Thread.run(Thread.java:662)
2013-04-21 18:48:28,277 INFO  hdfs.DFSClient 
(DFSOutputStream.java:nextBlockOutputStream(1022)) - Abandoning 
BP-2121022065-10.174.86.97-1366570107029:blk_6876843860808656778_1071
2013-04-21 18:48:28,277 ERROR datanode.DataNode (DataXceiver.java:run(223)) - 
127.0.0.1:33298:DataXceiver error processing WRITE_BLOCK operation  src: 
/127.0.0.1:55182 dest: /127.0.0.1:33298
java.io.EOFException: Premature EOF: no length prefix available
at 
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1340)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:448)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:98)
at 
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:65)
at 
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:219)
at java.lang.Thread.run(Thread.java:662)
2013-04-21 18:48:28,277 INF