Jenkins build is back to normal : Hadoop-Hdfs-trunk #1860

[Apache Jenkins Server]

See 

[jira] [Resolved] (HDFS-6950) Add Additional unit tests for HDFS-6581

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HDFS-6950.
-
   Resolution: Fixed
Fix Version/s: HDFS-6581
 Hadoop Flags: Reviewed

Committed to the feature branch. Thanks for the contribution [~xyao].

> Add Additional unit tests for HDFS-6581
> ---
>
> Key: HDFS-6950
> URL: https://issues.apache.org/jira/browse/HDFS-6950
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
> Fix For: HDFS-6581
>
> Attachments: HDFS-6950.0.patch, HDFS-6950.1.patch, HDFS-6950.2.patch
>
>
> Create additional unit tests for HDFS-6581 in addition to existing ones in 
> HDFS-6927.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-6991) Notify NN of evicted block before deleting it from RAM disk

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HDFS-6991.
-
Resolution: Duplicate

I'll include the fix in HDFS-6930 since the same code is modified by both 
patches.

> Notify NN of evicted block before deleting it from RAM disk
> ---
>
> Key: HDFS-6991
> URL: https://issues.apache.org/jira/browse/HDFS-6991
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Affects Versions: HDFS-6581
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-6991.01.patch
>
>
> When evicting a block from RAM disk to persistent storage, the DN should 
> notify the NN of the persistent replica before deleting the replica from RAM 
> disk. Else there can be a window of time during which the block is considered 
> 'missing' by the NN.
> Found by [~xyao] via HDFS-6950.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-6992) DN decommission control as a utility for unit testing

[Zhe Zhang (JIRA)]

Zhe Zhang created HDFS-6992:
---

 Summary: DN decommission control as a utility for unit testing
 Key: HDFS-6992
 URL: https://issues.apache.org/jira/browse/HDFS-6992
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor


Currently {{TestDecommission}} contains all test cases as well as control logic 
for decommission-related tests. This is not ideal because the  decommissioning 
process has too many perspectives to fit in one class. As a matter of fact the 
current {{TestDecommission}} class is a bit too long, with ad hoc test cases 
like {{testDecommission}} and {{testDecommission2}}.

It is better to move the decommissioning logic to {{DFSTestUtil}} so it can be 
called in other unit testing classes such as {{TestReplicationPolicy}}. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (HDFS-6991) Notify NN of evicted block before deleting it from RAM disk

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal reopened HDFS-6991:
-

> Notify NN of evicted block before deleting it from RAM disk
> ---
>
> Key: HDFS-6991
> URL: https://issues.apache.org/jira/browse/HDFS-6991
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode, namenode
>Affects Versions: HDFS-6581
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Attachments: HDFS-6991.01.patch
>
>
> When evicting a block from RAM disk to persistent storage, the DN should 
> notify the NN of the persistent replica before deleting the replica from RAM 
> disk. Else there can be a window of time during which the block is considered 
> 'missing' by the NN.
> Found by [~xyao] via HDFS-6950.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (HDFS-6930) Improve replica eviction from RAM disk

[Arpit Agarwal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arpit Agarwal resolved HDFS-6930.
-
   Resolution: Fixed
Fix Version/s: HDFS-6581
 Hadoop Flags: Reviewed

Thanks Colin and Xiaoyu. Committed to the feature branch.

> Improve replica eviction from RAM disk
> --
>
> Key: HDFS-6930
> URL: https://issues.apache.org/jira/browse/HDFS-6930
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: datanode
>Affects Versions: HDFS-6581
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: HDFS-6581
>
> Attachments: HDFS-6930.01.patch, HDFS-6930.02.patch
>
>
> The current replica eviction scheme is inefficient since it performs multiple 
> file operations in the context of block allocation.
> A better implementation would be asynchronous eviction when free space on RAM 
> disk falls below a low watermark to make block allocation faster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSS] Hadoop 2.5.1

[Karthik Kambatla]

Hi folks

Now that all issues with target 2.5.1 are committed, I am planning to cut
an RC for 2.5.1 this Friday. The fixes going into 2.5.1 are -
http://s.apache.org/2Mz

Are there any other "Blocker" issues that we would like to get into 2.5.1?
If there are any, please mark them as "Blocker" and target "2.5.1".

Committers - please avoid committing non-blockers to 2.5.1.

I have put together a new wiki for the release process
. Would
greatly appreciate any eyeballs I can get on improving it. The wiki
reflects our migration to git, the updated create-release script that
avoids manual massaging of tarballs and other minor improvements.

Thanks
Karthik

[jira] [Created] (HDFS-6993) Add a metric for the number of misbehaving DFSClients that try to do no-checksum reads that extend too long

[Colin Patrick McCabe (JIRA)]

Colin Patrick McCabe created HDFS-6993:
--

 Summary: Add a metric for the number of misbehaving DFSClients 
that try to do no-checksum reads that extend too long
 Key: HDFS-6993
 URL: https://issues.apache.org/jira/browse/HDFS-6993
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor


Add a metric on the DataNode for the number of misbehaving DFSClients that try 
to do no-checksum reads that extend too long.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HDFS-6994) libhdfs3 - A native C/C++ HDFS client

[Zhanwei Wang (JIRA)]

Zhanwei Wang created HDFS-6994:
--

 Summary: libhdfs3 - A native C/C++ HDFS client
 Key: HDFS-6994
 URL: https://issues.apache.org/jira/browse/HDFS-6994
 Project: Hadoop HDFS
  Issue Type: Task
  Components: hdfs-client
Reporter: Zhanwei Wang


Hi All

I just got the permission to open source libhdfs3, which is a native C/C++ HDFS 
client based on Hadoop RPC protocol and HDFS Data Transfer Protocol.

libhdfs3 provide the libhdfs style C interface and a C++ interface. Support 
both HADOOP RPC version 8 and 9. Support Namenode HA and Kerberos 
authentication.

libhdfs3 is currently used by HAWQ of Pivotal

I'd like to integrate libhdfs3 into HDFS source code to benefit others.

You can find libhdfs3 code from github
https://github.com/PivotalRD/libhdfs3



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)