[jira] [Created] (HDFS-8500) Improve refreshNode in DFSAdmin to support hdfs federation

2015-05-30 Thread Zhang Wei (JIRA)
Zhang Wei created HDFS-8500:
---

 Summary: Improve refreshNode in DFSAdmin to support hdfs federation
 Key: HDFS-8500
 URL: https://issues.apache.org/jira/browse/HDFS-8500
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: federation
Affects Versions: 2.7.0
Reporter: Zhang Wei
Priority: Minor


In hdfs federated cluster, we find the dfsadmin command "hdfs dfsadmin 
-refreshNodes"  can only refresh the nameservice (or namenode) which configured 
in fs.defaultFS. The other nameservices configured in hdfs-site.xml can't be 
refreshed unless changing the fs.defaultFS value and run this command again. I 
think we need additional parameters as following to give a convenient way: 
[-a] fresh all namenodes configured in hdfs-site.xml.
[-ns ] specify a nameservice to refresh.
[host:ipc_port] specify a nn to refresh.

please give your opinions. Thanks!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (HDFS-8501) Erasure Coding: Improve memory efficiency of BlockInfoStriped

2015-05-30 Thread Walter Su (JIRA)
Walter Su created HDFS-8501:
---

 Summary: Erasure Coding: Improve memory efficiency of 
BlockInfoStriped
 Key: HDFS-8501
 URL: https://issues.apache.org/jira/browse/HDFS-8501
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Walter Su
Assignee: Walter Su


Erasure Coding: Improve memory efficiency of BlockInfoStriped

Assume we have a BlockInfoStriped:
{noformat}
triplets[] = {s0, s1, s2, s3}
indices[] = {0, 1, 2, 3}
{noformat}

When we run balancer/mover to re-locate replica on s2, firstly it becomes:
{noformat}
triplets[] = {s0, s1, s2, s3, s2}
indices[] = {0, 1, 2, 3, 2}
{noformat}
Then the replica on s1 is removed, finally it becomes:
{noformat}
triplets[] = {s0, s1, null, s3, s2}
indices[] = {0, 1, -1, 3, 2}
{noformat}

The worst case is:
{noformat}
triplets[] = {null, null, null, null, s0, s1, s2, s3}
indices[] = {-1, -1, -1, -1, 0, 1, 2, 3}
{noformat}


We should learn from {{BlockInfoContiguous.removeStorage(..)}}. When a storage 
is removed, we bring the last item front.
With the improvement, the worst case become:
{noformat}
triplets[] = {s0, s1, s2, s3, null}
indices[] = {0, 1, 2, 3, -1}
{noformat}
We have an empty slot.

Notes:
Assume we copy 4 storage first, then delete 4. Even with the improvement, the 
worst case could be:
{noformat}
triplets[] = {s0, s1, s2, s3, null, null, null, null}
indices[] = {0, 1, 2, 3, -1, -1, -1, -1}
{noformat}
But the Balancer strategy won't move same block/blockGroup twice in a row. So 
this case is very rare.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Jenkins build is back to normal : Hadoop-Hdfs-trunk #2141

2015-05-30 Thread Apache Jenkins Server
See 



Jenkins build is back to normal : Hadoop-Hdfs-trunk-Java8 #202

2015-05-30 Thread Apache Jenkins Server
See 



[jira] [Created] (HDFS-8502) Ozone: Storage container data pipeline

2015-05-30 Thread Jitendra Nath Pandey (JIRA)
Jitendra Nath Pandey created HDFS-8502:
--

 Summary: Ozone: Storage container data pipeline
 Key: HDFS-8502
 URL: https://issues.apache.org/jira/browse/HDFS-8502
 Project: Hadoop HDFS
  Issue Type: Sub-task
Reporter: Jitendra Nath Pandey
Assignee: Jitendra Nath Pandey


This jira layout the basic framework of the data pipeline to replicate the 
storage containers while writing. An important design goal is to keep the 
pipeline semantics independent of the storage container implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)