Fwd: about how the hdfs choose datanodes to store the files

Hi: all! we know that the hdfs divide a large file into several blocks(with each 64mb, 3 replications default). and once the metadata in the namenode are modified, there goes a thread dataStreamer to transport the blocks to the datanode. for each block, the client send the block to the 3 datanod

what to do with the remaind space in the last block

Hi all. as we know. the last block of file may be less then 64MB(as default configged), then what will hdfs do with the remained space in the block?? left for other files or just left unused?? any answer or related url is appreciated! thanks xu

Re: about how the hdfs choose datanodes to store the files

Hello xu, On Thu, May 5, 2011 at 12:24 PM, cheng xu wrote: > Hi: >  all! we know that the hdfs divide a large file into several blocks(with > each 64mb, 3 replications default). and  once  the metadata in the namenode > are modified, there goes a thread dataStreamer to transport the blocks to > t

Re: what to do with the remaind space in the last block

Hello xu, On Thu, May 5, 2011 at 1:19 PM, cheng xu wrote: > as we know. the last block of file may be less then 64MB(as default > configged), then what will hdfs do with the remained space in the block?? > left for other files or just left unused?? Since no form of pre-allocation is performed wh

[jira] [Created] (HDFS-1895) Setting up of cluster using ssh - Scripts that help in minimising the cluster setup efforts

Setting up of cluster using ssh - Scripts that help in minimising the cluster setup efforts --- Key: HDFS-1895 URL: https://issues.apache.org/jira/browse/HDFS-1895

Hadoop-Hdfs-trunk - Build # 657 - Still Failing

See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/657/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 788586 lines...] [junit] [junit] 2011-05-0

Re: what to do with the remaind space in the last block

Xu, the datanode only stores the amount of data provided. If the last block only has, e.g., 10 bytes in it, then the datanode will use one low-level block (the minimum space allocation from the underlying file system) to store those 10 bytes. Typically, low-level blocks are 8KB, but that depen

Re: about how the hdfs choose datanodes to store the files

Hello again xu, Apologies over a bad mistake in the earlier post, I believe I had it wrong. The replication is done as you had explained, so do not get confused by my saying that the NN manages it after the writes (I've said this around as well, learnt the right thing thanks to Matthew Foley today

[jira] [Resolved] (HDFS-1892) Update HDFS-1073 branch to deal with OP_INVALID-filled preallocation

[ https://issues.apache.org/jira/browse/HDFS-1892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-1892. --- Resolution: Fixed Fix Version/s: Edit log branch (HDFS-1073) Committed to edit log branch. > U

Hadoop-Hdfs-trunk-Commit - Build # 624 - Still Failing

See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/624/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 2829 lines...] [junit] Running org.apach

[jira] [Created] (HDFS-1896) Additional QA tasks for Edit Log branch

Additional QA tasks for Edit Log branch --- Key: HDFS-1896 URL: https://issues.apache.org/jira/browse/HDFS-1896 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Todd Lipcon Assig

[jira] [Resolved] (HDFS-1799) Refactor log rolling and filename management out of FSEditLog

[ https://issues.apache.org/jira/browse/HDFS-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-1799. --- Resolution: Fixed Hadoop Flags: [Reviewed] > Refactor log rolling and filename management out of

[jira] [Created] (HDFS-1897) Documention refers to removed option dfs.network.script

Documention refers to removed option dfs.network.script --- Key: HDFS-1897 URL: https://issues.apache.org/jira/browse/HDFS-1897 Project: Hadoop HDFS Issue Type: Bug Components: do

[jira] [Created] (HDFS-1898) Tests failing on trunk due to use of NameNode.format

Tests failing on trunk due to use of NameNode.format Key: HDFS-1898 URL: https://issues.apache.org/jira/browse/HDFS-1898 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.0

[jira] [Created] (HDFS-1899) GenericTestUtils.formatNameNode is misplaced

GenericTestUtils.formatNameNode is misplaced Key: HDFS-1899 URL: https://issues.apache.org/jira/browse/HDFS-1899 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.0

Re: about how the hdfs choose datanodes to store the files

Hi Harsh: I really appreciate that! you help me a lot!! ^_^ got it. another question confuses me. assume that a file is divided into several blocks, say a,b,c,d.. and when the file is being writing into the hdfs, are the blocks a,b,c,d. being written sequencely or concurrently?itseems

Re: what to do with the remaind space in the last block

Hi thanks for your reply! it really help alot! thanks again . 2011/5/6 Matthew Foley > Xu, the datanode only stores the amount of data provided. If the last > block only has, e.g., 10 bytes in it, then the datanode will use one > low-level block (the minimum space allocation from the underly

Hadoop-Hdfs-trunk-Commit - Build # 625 - Still Failing

See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/625/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 2827 lines...] [junit] Running org.apach

[jira] [Created] (HDFS-1900) Use the block size key defined by common

Use the block size key defined by common - Key: HDFS-1900 URL: https://issues.apache.org/jira/browse/HDFS-1900 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.21.1 Rep

Hadoop-Hdfs-trunk-Commit - Build # 626 - Still Failing

See https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/626/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 2829 lines...] [junit] Running org.apach