Need help, About one issue on NameNode & DataNode.

2011-01-02 Thread mac fang 
Hi, guys,

Since i am quite new in using the Hadoop, currently i face one problem about
the NameNode HA. Below is the case

1. We are using Hadoop 0.21
2. We manage about 1 NameNode, 1 Backup node and 100 datanodes

Current issue:
if the namenode crashed, it switched to the backup node. However we had to
restart all 100 datanode. This cost us about 5-10 mins to restart all
DataNodes.

Any suggestion to resolve this, perhaps we can avoid restarting the
datanoes, how to?


Thanks in advanced.

regards
macf

Hadoop-Hdfs-trunk - Build # 540 - Still Failing

2011-01-02 Thread Apache Hudson Server 
See https://hudson.apache.org/hudson/job/Hadoop-Hdfs-trunk/540/

###
## LAST 60 LINES OF THE CONSOLE 
###
[...truncated 576817 lines...]
[junit] 2011-01-02 12:19:20,006 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:run(198)) - Deleted block 
blk_3094852499616718116_1016 at file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir55/blk_3094852499616718116
[junit] 2011-01-02 12:19:20,007 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:deleteAsync(152)) - Scheduling block 
blk_3336629607501324374_1025 file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir62/blk_3336629607501324374
 for deletion
[junit] 2011-01-02 12:19:20,007 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:deleteAsync(152)) - Scheduling block 
blk_3503705794993884282_1054 file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir20/blk_3503705794993884282
 for deletion
[junit] 2011-01-02 12:19:20,007 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:run(198)) - Deleted block 
blk_3336629607501324374_1025 at file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir62/blk_3336629607501324374
[junit] 2011-01-02 12:19:20,007 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:deleteAsync(152)) - Scheduling block 
blk_3548177645357240189_1076 file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data2/current/finalized/subdir63/blk_3548177645357240189
 for deletion
[junit] 2011-01-02 12:19:20,007 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:run(198)) - Deleted block 
blk_3503705794993884282_1054 at file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir20/blk_3503705794993884282
[junit] 2011-01-02 12:19:20,007 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:deleteAsync(152)) - Scheduling block 
blk_3966046540798531510_1020 file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir58/blk_3966046540798531510
 for deletion
[junit] 2011-01-02 12:19:20,008 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:deleteAsync(152)) - Scheduling block 
blk_4197480359425474997_1062 file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir27/blk_4197480359425474997
 for deletion
[junit] 2011-01-02 12:19:20,008 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:run(198)) - Deleted block 
blk_3966046540798531510_1020 at file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir58/blk_3966046540798531510
[junit] 2011-01-02 12:19:20,008 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:deleteAsync(152)) - Scheduling block 
blk_4696538338555801224_1075 file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir37/blk_4696538338555801224
 for deletion
[junit] 2011-01-02 12:19:20,008 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:run(198)) - Deleted block 
blk_4197480359425474997_1062 at file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir27/blk_4197480359425474997
[junit] 2011-01-02 12:19:20,008 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:deleteAsync(152)) - Scheduling block 
blk_4718036001403563385_1024 file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data2/current/finalized/subdir22/blk_4718036001403563385
 for deletion
[junit] 2011-01-02 12:19:20,008 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:run(198)) - Deleted block 
blk_4696538338555801224_1075 at file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir37/blk_4696538338555801224
[junit] 2011-01-02 12:19:20,008 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:deleteAsync(152)) - Scheduling block 
blk_5081289600839753379_1012 file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data2/current/finalized/subdir12/blk_5081289600839753379
 for deletion
[junit] 2011-01-02 12:19:20,008 INFO  datanode.DataNode 
(FSDatasetAsyncDiskService.java:deleteAsync(152)) - Scheduling block 
blk_5367648145032749599_1067 file 
/grid/0/hudson/hudson-slave/workspace/Hadoop-Hdfs-trunk/trunk/build/test/data/dfs/data/data1/current/finalized/subdir30/blk_5367648145032749599
 for deletion
[j

[jira] Resolved: (HDFS-1563) create(file, true) appears to be violating atomicity

2011-01-02 Thread dhruba borthakur (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HDFS-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

dhruba borthakur resolved HDFS-1563.


Resolution: Won't Fix

I vote that we keep the semantics of the create(overwrite==true) the same as it 
is now. Please reopen this JIRA if you strongly feel otherwise.

> create(file, true) appears to be violating atomicity
> 
>
> Key: HDFS-1563
> URL: https://issues.apache.org/jira/browse/HDFS-1563
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: name-node
>Affects Versions: 0.20.1
>Reporter: Guanghao Shen
>Assignee: dhruba borthakur
> Attachments: unittest.diff
>
>
> Will upload a unittest to reveal this bug.
> In a word, when a thread is doing create(file, true) on existing file, there 
> are chances that another thread will get 'false' for exists(file) during the 
> period.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.