See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk/528/
################################################################################### ########################## LAST 60 LINES OF THE CONSOLE ########################### [...truncated 841373 lines...] [junit] 2010-12-21 13:23:19,658 WARN datanode.FSDatasetAsyncDiskService (FSDatasetAsyncDiskService.java:shutdown(130)) - AsyncDiskService has already shut down. [junit] 2010-12-21 13:23:19,658 INFO hdfs.MiniDFSCluster (MiniDFSCluster.java:shutdownDataNodes(786)) - Shutting down DataNode 0 [junit] 2010-12-21 13:23:19,760 INFO ipc.Server (Server.java:stop(1611)) - Stopping server on 35416 [junit] 2010-12-21 13:23:19,760 INFO ipc.Server (Server.java:run(1444)) - IPC Server handler 0 on 35416: exiting [junit] 2010-12-21 13:23:19,760 INFO ipc.Server (Server.java:run(675)) - Stopping IPC Server Responder [junit] 2010-12-21 13:23:19,760 INFO datanode.DataNode (DataNode.java:shutdown(771)) - Waiting for threadgroup to exit, active threads is 1 [junit] 2010-12-21 13:23:19,760 INFO ipc.Server (Server.java:run(475)) - Stopping IPC Server listener on 35416 [junit] 2010-12-21 13:23:19,760 WARN datanode.DataNode (DataXceiverServer.java:run(141)) - DatanodeRegistration(127.0.0.1:50831, storageID=DS-1946037341-127.0.1.1-50831-1292937788838, infoPort=36930, ipcPort=35416):DataXceiveServer: java.nio.channels.AsynchronousCloseException [junit] at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185) [junit] at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) [junit] at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84) [junit] at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:134) [junit] at java.lang.Thread.run(Thread.java:619) [junit] [junit] 2010-12-21 13:23:19,763 INFO datanode.DataNode (DataNode.java:shutdown(771)) - Waiting for threadgroup to exit, active threads is 0 [junit] 2010-12-21 13:23:19,863 INFO datanode.DataBlockScanner (DataBlockScanner.java:run(622)) - Exiting DataBlockScanner thread. [junit] 2010-12-21 13:23:19,863 INFO datanode.DataNode (DataNode.java:run(1445)) - DatanodeRegistration(127.0.0.1:50831, storageID=DS-1946037341-127.0.1.1-50831-1292937788838, infoPort=36930, ipcPort=35416):Finishing DataNode in: FSDataset{dirpath='/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build-fi/test/data/dfs/data/data1/current/finalized,/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build-fi/test/data/dfs/data/data2/current/finalized'} [junit] 2010-12-21 13:23:19,864 INFO ipc.Server (Server.java:stop(1611)) - Stopping server on 35416 [junit] 2010-12-21 13:23:19,864 INFO datanode.DataNode (DataNode.java:shutdown(771)) - Waiting for threadgroup to exit, active threads is 0 [junit] 2010-12-21 13:23:19,864 INFO datanode.FSDatasetAsyncDiskService (FSDatasetAsyncDiskService.java:shutdown(133)) - Shutting down all async disk service threads... [junit] 2010-12-21 13:23:19,864 INFO datanode.FSDatasetAsyncDiskService (FSDatasetAsyncDiskService.java:shutdown(142)) - All async disk service threads have been shut down. [junit] 2010-12-21 13:23:19,865 WARN datanode.FSDatasetAsyncDiskService (FSDatasetAsyncDiskService.java:shutdown(130)) - AsyncDiskService has already shut down. [junit] 2010-12-21 13:23:19,969 WARN namenode.DecommissionManager (DecommissionManager.java:run(70)) - Monitor interrupted: java.lang.InterruptedException: sleep interrupted [junit] 2010-12-21 13:23:19,969 INFO namenode.FSEditLog (FSEditLog.java:printStatistics(631)) - Number of transactions: 6 Total time for transactions(ms): 1Number of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 7 3 [junit] 2010-12-21 13:23:19,969 WARN namenode.FSNamesystem (FSNamesystem.java:run(2822)) - ReplicationMonitor thread received InterruptedException.java.lang.InterruptedException: sleep interrupted [junit] 2010-12-21 13:23:19,970 INFO ipc.Server (Server.java:stop(1611)) - Stopping server on 34872 [junit] 2010-12-21 13:23:19,971 INFO ipc.Server (Server.java:run(1444)) - IPC Server handler 0 on 34872: exiting [junit] 2010-12-21 13:23:19,971 INFO ipc.Server (Server.java:run(1444)) - IPC Server handler 6 on 34872: exiting [junit] 2010-12-21 13:23:19,971 INFO ipc.Server (Server.java:run(1444)) - IPC Server handler 7 on 34872: exiting [junit] 2010-12-21 13:23:19,971 INFO ipc.Server (Server.java:run(1444)) - IPC Server handler 2 on 34872: exiting [junit] 2010-12-21 13:23:19,971 INFO ipc.Server (Server.java:run(1444)) - IPC Server handler 9 on 34872: exiting [junit] 2010-12-21 13:23:19,971 INFO ipc.Server (Server.java:run(1444)) - IPC Server handler 5 on 34872: exiting [junit] 2010-12-21 13:23:19,971 INFO ipc.Server (Server.java:run(1444)) - IPC Server handler 4 on 34872: exiting [junit] 2010-12-21 13:23:19,971 INFO ipc.Server (Server.java:run(1444)) - IPC Server handler 1 on 34872: exiting [junit] 2010-12-21 13:23:19,971 INFO ipc.Server (Server.java:run(1444)) - IPC Server handler 3 on 34872: exiting [junit] 2010-12-21 13:23:19,971 INFO ipc.Server (Server.java:run(1444)) - IPC Server handler 8 on 34872: exiting [junit] 2010-12-21 13:23:19,973 INFO ipc.Server (Server.java:run(675)) - Stopping IPC Server Responder [junit] 2010-12-21 13:23:19,973 INFO ipc.Server (Server.java:run(475)) - Stopping IPC Server listener on 34872 [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 36.233 sec checkfailure: run-test-hdfs-all-withtestcaseonly: run-test-hdfs: BUILD FAILED /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build.xml:724: Tests failed! Total time: 107 minutes 12 seconds [FINDBUGS] Skipping publisher since build result is FAILURE Publishing Javadoc Archiving artifacts Recording test results Recording fingerprints Publishing Clover coverage report... No Clover report will be published due to a Build Failure Email was triggered for: Failure Sending email for trigger: Failure ################################################################################### ############################## FAILED TESTS (if any) ############################## 2 tests failed. FAILED: org.apache.hadoop.hdfs.TestFileConcurrentReader.testUnfinishedBlockCRCErrorNormalTransferVerySmallWrite Error Message: Cannot run program "du": java.io.IOException: error=24, Too many open files Stack Trace: java.io.IOException: Cannot run program "du": java.io.IOException: error=24, Too many open files at java.lang.ProcessBuilder.start(ProcessBuilder.java:459) at org.apache.hadoop.util.Shell.runCommand(Shell.java:201) at org.apache.hadoop.util.Shell.run(Shell.java:183) at org.apache.hadoop.fs.DU.<init>(DU.java:57) at org.apache.hadoop.fs.DU.<init>(DU.java:67) at org.apache.hadoop.hdfs.server.datanode.FSDataset$FSVolume.<init>(FSDataset.java:342) at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:873) at org.apache.hadoop.hdfs.server.datanode.DataNode.initFsDataSet(DataNode.java:395) at org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(DataNode.java:500) at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:281) at org.apache.hadoop.hdfs.server.datanode.DataNode.<init>(DataNode.java:263) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1561) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1504) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1471) at org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:630) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:464) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:186) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:71) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:178) at org.apache.hadoop.hdfs.TestFileConcurrentReader.init(TestFileConcurrentReader.java:88) at org.apache.hadoop.hdfs.TestFileConcurrentReader.runTestUnfinishedBlockCRCError(TestFileConcurrentReader.java:315) at org.apache.hadoop.hdfs.TestFileConcurrentReader.runTestUnfinishedBlockCRCError(TestFileConcurrentReader.java:302) at org.apache.hadoop.hdfs.TestFileConcurrentReader.__CLR3_0_2wjxr3fqwv(TestFileConcurrentReader.java:290) at org.apache.hadoop.hdfs.TestFileConcurrentReader.testUnfinishedBlockCRCErrorNormalTransferVerySmallWrite(TestFileConcurrentReader.java:289) Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files at java.lang.UNIXProcess.<init>(UNIXProcess.java:148) at java.lang.ProcessImpl.start(ProcessImpl.java:65) at java.lang.ProcessBuilder.start(ProcessBuilder.java:452) FAILED: org.apache.hadoop.hdfs.server.namenode.TestStorageRestore.testStorageRestore Error Message: Image file /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/secondary/current/fsimage is corrupt with MD5 checksum of 49b1c766a3abae340a8a9182865daa83 but expecting d9c3bd1a2d0a232f264d907016ecb0b8 Stack Trace: java.io.IOException: Image file /grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/secondary/current/fsimage is corrupt with MD5 checksum of 49b1c766a3abae340a8a9182865daa83 but expecting d9c3bd1a2d0a232f264d907016ecb0b8 at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:1063) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.doMerge(SecondaryNameNode.java:702) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode$CheckpointStorage.access$500(SecondaryNameNode.java:600) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doMerge(SecondaryNameNode.java:477) at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:438) at org.apache.hadoop.hdfs.server.namenode.TestStorageRestore.__CLR3_0_2dn2tm4tjd(TestStorageRestore.java:316) at org.apache.hadoop.hdfs.server.namenode.TestStorageRestore.testStorageRestore(TestStorageRestore.java:286)