date:20101122

Build failed in Hudson: Hadoop-Hdfs-trunk #497

2010-11-22 Thread Apache Hudson Server

See 

--
[...truncated 955209 lines...]
[junit] at java.lang.Thread.run(Thread.java:619)
[junit] 2010-11-22 15:43:30,209 WARN  datanode.DataNode 
(DataNode.java:checkDiskError(828)) - checkDiskError: exception: 
[junit] java.io.IOException: Connection reset by peer
[junit] at sun.nio.ch.FileDispatcher.write0(Native Method)
[junit] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
[junit] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
[junit] at sun.nio.ch.IOUtil.write(IOUtil.java:75)
[junit] at 
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
[junit] at 
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:60)
[junit] at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
[junit] at 
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:151)
[junit] at 
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:112)
[junit] at 
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:105)
[junit] at 
java.io.DataOutputStream.writeShort(DataOutputStream.java:150)
[junit] at 
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:543)
[junit] at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.write_aroundBody0(BlockReceiver.java:932)
[junit] at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.write_aroundBody1$advice(BlockReceiver.java:160)
[junit] at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:932)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit] 2010-11-22 15:43:30,210 INFO  datanode.DataNode 
(BlockReceiver.java:run(956)) - PacketResponder blk_4859461139722731885_1001 2 
Exception java.io.IOException: Connection reset by peer
[junit] at sun.nio.ch.FileDispatcher.write0(Native Method)
[junit] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
[junit] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104)
[junit] at sun.nio.ch.IOUtil.write(IOUtil.java:75)
[junit] at 
sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334)
[junit] at 
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:60)
[junit] at 
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
[junit] at 
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:151)
[junit] at 
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:112)
[junit] at 
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:105)
[junit] at 
java.io.DataOutputStream.writeShort(DataOutputStream.java:150)
[junit] at 
org.apache.hadoop.hdfs.protocol.DataTransferProtocol$PipelineAck.write(DataTransferProtocol.java:543)
[junit] at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.write_aroundBody0(BlockReceiver.java:932)
[junit] at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.write_aroundBody1$advice(BlockReceiver.java:160)
[junit] at 
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.run(BlockReceiver.java:932)
[junit] at java.lang.Thread.run(Thread.java:619)
[junit] 
[junit] 2010-11-22 15:43:30,210 INFO  datanode.DataTransferProtocolAspects 
(DataTransferProtocolAspects.aj:ajc$afterReturning$org_apache_hadoop_hdfs_server_datanode_DataTransferProtocolAspects$1$8f59fdd7(51))
 - FI: receiverOp WRITE_BLOCK, datanode=127.0.0.1:37925
[junit] 2010-11-22 15:43:30,210 INFO  datanode.DataTransferProtocolAspects 
(DataTransferProtocolAspects.aj:ajc$before$org_apache_hadoop_hdfs_server_datanode_DataTransferProtocolAspects$3$3251489(73))
 - FI: receiverOpWriteBlock
[junit] 2010-11-22 15:43:30,211 INFO  datanode.DataNode 
(DataXceiver.java:opWriteBlock(257)) - Receiving block 
blk_4859461139722731885_1001 src: /127.0.0.1:53325 dest: /127.0.0.1:37925
[junit] 2010-11-22 15:43:30,211 INFO  datanode.DataNode 
(FSDataset.java:recoverClose(1243)) - Recover failed close 
blk_4859461139722731885_1001
[junit] 2010-11-22 15:43:30,211 INFO  datanode.DataNode 
(BlockReceiver.java:run(972)) - PacketResponder 2 for block 
blk_4859461139722731885_1001 terminating
[junit] 2010-11-22 15:43:30,226 INFO  datanode.DataNode 
(DataXceiver.java:opWriteBlock(414)) - writeBlock blk_4859461139722731885_1001 
received exception java.io.IOException: Interrupted receiveBlock
[junit] 2010-11-22 15:43:30,238 INFO  datanode.BlockReceiverAspects 
(BlockReceiverAspects.aj:ajc$after$org_apache_hadoop_hdfs_server_datanode_BlockReceiverAspects$9$725950a6(220))
 -

[jira] Created: (HDFS-1514) project.version in aop.xml is out of sync with build.xml

2010-11-22 Thread Patrick Kling (JIRA)

project.version in aop.xml is out of sync with build.xml


 Key: HDFS-1514
 URL: https://issues.apache.org/jira/browse/HDFS-1514
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Patrick Kling


project.version in aop.xml is set to 0.22.0-SNAPSHOT whereas version in 
build.xml is set to 0.23.0-SNAPSHOT. This causes ant test-patch to fail when 
using a local maven repository.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: Review Request: Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)

2010-11-22 Thread Patrick Kling


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/18/
---

(Updated 2010-11-22 13:47:46.894388)


Review request for hadoop-hdfs.


Changes
---

Fixed javadoc warnings.


Summary
---

Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)

As discussed in HDFS-, it would be beneficial for tools such as the RAID 
block fixer and RAID FSCK to have access to listCorruptFileBlocks via the 
DistributedFileSystem (rather than having to parse Servlet output, which could 
present a performance problem).

For further details, see https://issues.apache.org/jira/browse/HDFS-1482


This addresses bug HDFS-1482.
https://issues.apache.org/jira/browse/HDFS-1482


Diffs (updated)
-

  
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/fs/Hdfs.java
 1036663 
  
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DFSClient.java
 1036663 
  
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
 1036663 
  
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/protocol/ClientProtocol.java
 1036663 
  
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
 1036663 
  
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
 1036663 
  
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestCorruptFilesJsp.java
 1036663 
  
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
 1036663 
  
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestListCorruptFileBlocks.java
 1036663 
  
http://svn.apache.org/repos/asf/hadoop/hdfs/trunk/src/webapps/hdfs/corrupt_files.jsp
 1036663 

Diff: https://reviews.apache.org/r/18/diff


Testing
---

Unit tests (including new test case in TestListCorruptFileBlocks)


Thanks,

Patrick

HDFS benchmarks

2010-11-22 Thread Thanh Do

Hi all,

Is there any benchmarks for HDFS available)
(measuring read/write throughput, latency and such).
It would be great if somebody point me to any source.

Thanks much
Thanh

Re: HDFS benchmarks

2010-11-22 Thread Tao Xie

TestDFSIO may be what you what.

2010/11/23 Thanh Do 

> Hi all,
>
> Is there any benchmarks for HDFS available)
> (measuring read/write throughput, latency and such).
> It would be great if somebody point me to any source.
>
> Thanks much
> Thanh
>

Build failed in Hudson: Hadoop-Hdfs-trunk-Commit #465

2010-11-22 Thread Apache Hudson Server

See 

Changes:

[eli] HDFS-1513. Fix a number of warnings. Contributed by Eli Collins

--
[...truncated 3893 lines...]
[junit] 2010-11-23 02:05:30,340 INFO  util.GSet 
(BlocksMap.java:computeCapacity(86)) - capacity  = 2^21 = 2097152 entries
[junit] 2010-11-23 02:05:30,340 INFO  util.GSet 
(LightWeightGSet.java:(82)) - recommended=2097152, actual=2097152
[junit] 2010-11-23 02:05:30,344 INFO  namenode.FSNamesystem 
(FSNamesystem.java:setConfigurationParameters(460)) - fsOwner=hudson
[junit] 2010-11-23 02:05:30,344 INFO  namenode.FSNamesystem 
(FSNamesystem.java:setConfigurationParameters(466)) - supergroup=supergroup
[junit] 2010-11-23 02:05:30,345 INFO  namenode.FSNamesystem 
(FSNamesystem.java:setConfigurationParameters(467)) - isPermissionEnabled=false
[junit] 2010-11-23 02:05:30,345 INFO  namenode.FSNamesystem 
(FSNamesystem.java:setConfigurationParameters(508)) - isBlockTokenEnabled=false 
blockKeyUpdateInterval=0 min(s), blockTokenLifetime=0 min(s)
[junit] 2010-11-23 02:05:30,346 INFO  metrics.FSNamesystemMetrics 
(FSNamesystemMetrics.java:(80)) - Initializing FSNamesystemMetrics using 
context object:org.apache.hadoop.metrics.spi.NullContext
[junit] 2010-11-23 02:05:30,346 INFO  namenode.FSNamesystem 
(FSNamesystem.java:registerMBean(4469)) - Registered FSNamesystemStatusMBean
[junit] 2010-11-23 02:05:30,348 INFO  namenode.NameNode 
(FSDirectory.java:(125)) - Caching file names occuring more than 10 times 
[junit] 2010-11-23 02:05:30,348 WARN  common.Util 
(Util.java:stringAsURI(63)) - Path 

 should be specified as a URI in configuration files. Please update hdfs 
configuration.
[junit] 2010-11-23 02:05:30,348 WARN  common.Util 
(Util.java:stringAsURI(63)) - Path 

 should be specified as a URI in configuration files. Please update hdfs 
configuration.
[junit] 2010-11-23 02:05:30,350 INFO  common.Storage 
(FSImage.java:loadFSImage(1175)) - Number of files = 1
[junit] 2010-11-23 02:05:30,350 INFO  common.Storage 
(FSImage.java:loadFilesUnderConstruction(1755)) - Number of files under 
construction = 0
[junit] 2010-11-23 02:05:30,350 INFO  common.Storage 
(FSImage.java:loadFSImage(1289)) - Image file of size 113 loaded in 0 seconds.
[junit] 2010-11-23 02:05:30,351 INFO  common.Storage 
(FSEditLogLoader.java:loadFSEdits(61)) - Edits file 

 of size 4 edits # 0 loaded in 0 seconds.
[junit] 2010-11-23 02:05:30,398 INFO  namenode.NameCache 
(NameCache.java:initialized(143)) - initialized with 0 entries 0 lookups
[junit] 2010-11-23 02:05:30,399 INFO  namenode.FSNamesystem 
(FSNamesystem.java:initialize(309)) - Finished loading FSImage in 61 msecs
[junit] 2010-11-23 02:05:30,399 INFO  util.HostsFileReader 
(HostsFileReader.java:refresh(85)) - Refreshing hosts (include/exclude) list
[junit] 2010-11-23 02:05:30,400 DEBUG namenode.TestNNLeaseRecovery 
(TestNNLeaseRecovery.java:__CLR3_0_2cpnfswvqb(253)) - Running 
__CLR3_0_2cpnfswvqb
[junit] 2010-11-23 02:05:30,401 INFO  namenode.FSEditLog 
(FSEditLog.java:printStatistics(631)) - Number of transactions: 0 Total time 
for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of 
syncs: 0 SyncTimes(ms): 0 
[junit] 2010-11-23 02:05:30,401 DEBUG namenode.FSNamesystem 
(EditLogFileOutputStream.java:preallocate(173)) - Preallocating Edit log, 
current size 4
[junit] 2010-11-23 02:05:30,402 DEBUG namenode.FSNamesystem 
(EditLogFileOutputStream.java:preallocate(180)) - Edit log size is now 1049092 
written 512 bytes  at offset 1048580
[junit] 2010-11-23 02:05:30,408 INFO  namenode.FSNamesystem 
(FSNamesystem.java:commitBlockSynchronization(2326)) - 
commitBlockSynchronization(lastblock=Mock for BlockInfoUnderConstruction, 
hashCode: 20245380, newgenerationstamp=2002, newlength=273487234, 
newtargets=[null], closeFile=true, deleteBlock=false)
[junit] 2010-11-23 02:05:30,425 INFO  jvm.JvmMetrics 
(JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with 
processName=NameNode, sessionId=null - already initialized
[junit] 2010-11-23 02:05:30,425 INFO  metrics.NameNodeMetrics 
(NameNodeMetrics.java:(113)) - Initializing NameNodeMeterics using 
context object:org.apache.hadoop.metrics.spi.NullContext
[junit] 2010-11-23 02:05:30,426 WARN  common.Util 
(Util.java:stringAsURI(63)) - Path 

 should be specified as a URI in configuration files. Please update hdfs 
configuration.
[junit] 2010-11-23 02:05:30,426 WARN  common.Util 
(Util.java:st

[jira] Created: (HDFS-1515) Test append and quotas

2010-11-22 Thread Eli Collins (JIRA)

Test append and quotas 
---

 Key: HDFS-1515
 URL: https://issues.apache.org/jira/browse/HDFS-1515
 Project: Hadoop HDFS
  Issue Type: Test
  Components: test
Affects Versions: 0.22.0, 0.23.0
Reporter: Eli Collins
Assignee: Eli Collins
 Fix For: 0.22.0, 0.23.0


There is no test coverage for quotas and append. Let's add a test to TestQuota 
that covers that quotas are updated correctly when appending to a file.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Build failed in Hudson: Hadoop-Hdfs-trunk-Commit #466

2010-11-22 Thread Apache Hudson Server

See 

Changes:

[hairong] HDFS-1481. NameNode should validate fsimage before rolling. 
Contributed by Hairong Kuang.

--
[...truncated 3893 lines...]
[junit] 2010-11-23 07:19:15,492 INFO  util.GSet 
(BlocksMap.java:computeCapacity(86)) - capacity  = 2^21 = 2097152 entries
[junit] 2010-11-23 07:19:15,492 INFO  util.GSet 
(LightWeightGSet.java:(82)) - recommended=2097152, actual=2097152
[junit] 2010-11-23 07:19:15,495 INFO  namenode.FSNamesystem 
(FSNamesystem.java:setConfigurationParameters(460)) - fsOwner=hudson
[junit] 2010-11-23 07:19:15,496 INFO  namenode.FSNamesystem 
(FSNamesystem.java:setConfigurationParameters(466)) - supergroup=supergroup
[junit] 2010-11-23 07:19:15,496 INFO  namenode.FSNamesystem 
(FSNamesystem.java:setConfigurationParameters(467)) - isPermissionEnabled=false
[junit] 2010-11-23 07:19:15,496 INFO  namenode.FSNamesystem 
(FSNamesystem.java:setConfigurationParameters(508)) - isBlockTokenEnabled=false 
blockKeyUpdateInterval=0 min(s), blockTokenLifetime=0 min(s)
[junit] 2010-11-23 07:19:15,497 INFO  metrics.FSNamesystemMetrics 
(FSNamesystemMetrics.java:(80)) - Initializing FSNamesystemMetrics using 
context object:org.apache.hadoop.metrics.spi.NullContext
[junit] 2010-11-23 07:19:15,497 INFO  namenode.FSNamesystem 
(FSNamesystem.java:registerMBean(4469)) - Registered FSNamesystemStatusMBean
[junit] 2010-11-23 07:19:15,499 INFO  namenode.NameNode 
(FSDirectory.java:(125)) - Caching file names occuring more than 10 times 
[junit] 2010-11-23 07:19:15,499 WARN  common.Util 
(Util.java:stringAsURI(63)) - Path 

 should be specified as a URI in configuration files. Please update hdfs 
configuration.
[junit] 2010-11-23 07:19:15,500 WARN  common.Util 
(Util.java:stringAsURI(63)) - Path 

 should be specified as a URI in configuration files. Please update hdfs 
configuration.
[junit] 2010-11-23 07:19:15,501 INFO  common.Storage 
(FSImage.java:loadFSImage(1175)) - Number of files = 1
[junit] 2010-11-23 07:19:15,501 INFO  common.Storage 
(FSImage.java:loadFilesUnderConstruction(1755)) - Number of files under 
construction = 0
[junit] 2010-11-23 07:19:15,502 INFO  common.Storage 
(FSImage.java:loadFSImage(1289)) - Image file of size 113 loaded in 0 seconds.
[junit] 2010-11-23 07:19:15,614 INFO  common.Storage 
(FSEditLogLoader.java:loadFSEdits(61)) - Edits file 

 of size 4 edits # 0 loaded in 0 seconds.
[junit] 2010-11-23 07:19:15,615 INFO  namenode.NameCache 
(NameCache.java:initialized(143)) - initialized with 0 entries 0 lookups
[junit] 2010-11-23 07:19:15,615 INFO  namenode.FSNamesystem 
(FSNamesystem.java:initialize(309)) - Finished loading FSImage in 126 msecs
[junit] 2010-11-23 07:19:15,616 INFO  util.HostsFileReader 
(HostsFileReader.java:refresh(85)) - Refreshing hosts (include/exclude) list
[junit] 2010-11-23 07:19:15,616 DEBUG namenode.TestNNLeaseRecovery 
(TestNNLeaseRecovery.java:__CLR3_0_2cpnfswvqn(253)) - Running 
__CLR3_0_2cpnfswvqn
[junit] 2010-11-23 07:19:15,618 INFO  namenode.FSEditLog 
(FSEditLog.java:printStatistics(631)) - Number of transactions: 0 Total time 
for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of 
syncs: 0 SyncTimes(ms): 0 
[junit] 2010-11-23 07:19:15,618 DEBUG namenode.FSNamesystem 
(EditLogFileOutputStream.java:preallocate(173)) - Preallocating Edit log, 
current size 4
[junit] 2010-11-23 07:19:15,619 DEBUG namenode.FSNamesystem 
(EditLogFileOutputStream.java:preallocate(180)) - Edit log size is now 1049092 
written 512 bytes  at offset 1048580
[junit] 2010-11-23 07:19:15,626 INFO  namenode.FSNamesystem 
(FSNamesystem.java:commitBlockSynchronization(2326)) - 
commitBlockSynchronization(lastblock=Mock for BlockInfoUnderConstruction, 
hashCode: 5683514, newgenerationstamp=2002, newlength=273487234, 
newtargets=[null], closeFile=true, deleteBlock=false)
[junit] 2010-11-23 07:19:15,646 INFO  jvm.JvmMetrics 
(JvmMetrics.java:init(71)) - Cannot initialize JVM Metrics with 
processName=NameNode, sessionId=null - already initialized
[junit] 2010-11-23 07:19:15,647 INFO  metrics.NameNodeMetrics 
(NameNodeMetrics.java:(113)) - Initializing NameNodeMeterics using 
context object:org.apache.hadoop.metrics.spi.NullContext
[junit] 2010-11-23 07:19:15,647 WARN  common.Util 
(Util.java:stringAsURI(63)) - Path 

 should be specified as a URI in configuration files. Please update hdfs 
configuration.
[junit] 2010-11-23 07:19:15,648 W

Build failed in Hudson: Hadoop-Hdfs-trunk #497

[jira] Created: (HDFS-1514) project.version in aop.xml is out of sync with build.xml

Re: Review Request: Add listCorruptFileBlocks to DistributedFileSystem (and ClientProtocol)

HDFS benchmarks

Re: HDFS benchmarks

Build failed in Hudson: Hadoop-Hdfs-trunk-Commit #465

[jira] Created: (HDFS-1515) Test append and quotas

Build failed in Hudson: Hadoop-Hdfs-trunk-Commit #466

8 matches

Site Navigation

Mail list logo

Footer information