[jira] [Created] (HDFS-5441) Wrong use of catalina opts in httpfs.sh

Dridi Boukelmoune created HDFS-5441: --- Summary: Wrong use of catalina opts in httpfs.sh Key: HDFS-5441 URL: https://issues.apache.org/jira/browse/HDFS-5441 Project: Hadoop HDFS Issue Type: B

Re: Decommission DataNode

HI Steve, thanks for your reply. There is the question in hadoop-2.2.0. In hadoop-2.2.0, there are below code in DatanodeManager.handleHeartbeat method : // Check if this datanode should actually be shutdown instead. if (nodeinfo != null &&* nodeinfo.isDisallowed()*) { setDa

Re: Datanode fencing mechanism

Hi Liu Le, You're correct, that's an oversight that was designed but never implemented. It's quite a rare circumstance but we should probably implement the persistent promise as you suggested. Want to have a try at making a patch for trunk? -Todd On Mon, Oct 28, 2013 at 1:57 AM, lei liu wrote:

Re: Replacing the JSP web UIs to HTML 5 applications

So, if I understand correctly, we are using an HTTP based API to access JMX data. The API isn't strictly designed to REST principals but does reflect the management API and query parameters of JMX. While unfortunate that we don't have a better REST based design for it, it may not add enough value

Re: Replacing the JSP web UIs to HTML 5 applications

Just to provide some data point to make the discussion concrete. Here is a part of the dump of the JMX information: curl " http://localhost:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo"; { "beans" : [ { "name" : "Hadoop:service=NameNode,name=NameNodeInfo", "modelerType" : "o

Re: [VOTE] Merge HDFS-4949 to trunk

With 3 +1s, the vote passes. Thanks, all. best, Colin On Fri, Oct 25, 2013 at 4:01 PM, Colin McCabe wrote: > On Fri, Oct 25, 2013 at 10:07 AM, Suresh Srinivas > wrote: >> I posted a comment in the other thread about feature branch merges. >> >> My preference is to make sure the requirements we

Re: Replacing the JSP web UIs to HTML 5 applications

I think it is important that we make provisions for all Ajax calls to be able to go through gateway deployments like Knox with the cluster firewalled off. As I have commented on the Jira, any calls that are currently on the serverside but are moving to the browser will need to either require punchi

Re: Replacing the JSP web UIs to HTML 5 applications

Neither of them will go through JMX. The new Web UI implements hdfs browsing through WebHDFS. The logs are available through the static servlets, which is exactly the same as what we have today. Thanks, Haohui On Mon, Oct 28, 2013 at 6:01 PM, Alejandro Abdelnur wrote: > are you planning to ex

Re: libhdfs portability

On Mon, Oct 28, 2013 at 4:24 PM, Kyle Sletmoe wrote: > I have written a WebHDFSClient and I do not believe that reusing > connections is enough to noticeably speed up transfers in my case. I did > some tests and on average it took roughly 14 minutes to transfer a 3.6 GB > file to an HDFS on my loc

Re: Replacing the JSP web UIs to HTML 5 applications

are you planning to expose things like hdfs browsing and nn/dn logs over jmx? thx Alejandro (phone typing) On Oct 28, 2013, at 17:48, Haohui Mai wrote: > It seems more appealing to me that the UI should JMX directly, because: > > * We're support the JMX in the long term for other management s

Re: Replacing the JSP web UIs to HTML 5 applications

It seems more appealing to me that the UI should JMX directly, because: * We're support the JMX in the long term for other management software. * The information provided by the JMX API will be mostly identical of the JSON API. Today the Web UI covers most of the information provided by JMX. The W

Re: Replacing the JSP web UIs to HTML 5 applications

Isn't using JMX to expose JSON for the web UI misusing JMX? I would think a more appropriate approach would be having /JMX for monitoring integration and a /JSON end point for the UI data. Thanks. On Mon, Oct 28, 2013 at 4:58 PM, Haohui Mai wrote: > Alejandro, > > If I understand correctly, t

Re: Replacing the JSP web UIs to HTML 5 applications

Alejandro, If I understand correctly, that is the exact approach that the new web UI is taking. The new web UI takes the output from JMX and renders them as HTML at the client side. ~Haohui On Mon, Oct 28, 2013 at 4:18 PM, Alejandro Abdelnur wrote: > Haohui, > > If you have NN and DNs producin

Re: libhdfs portability

I have written a WebHDFSClient and I do not believe that reusing connections is enough to noticeably speed up transfers in my case. I did some tests and on average it took roughly 14 minutes to transfer a 3.6 GB file to an HDFS on my local network (I tried the same operation using cURL, with simila

Re: Replacing the JSP web UIs to HTML 5 applications

Haohui, If you have NN and DNs producing JSON instead HTML, then you can build JS based web UIs. Take for example Oozie, Oozie produces JSON, it has a built in JS web ui that consumes JSON and Hue has built an external web UI that also consumes JSON. In the case of Hue UI, Oozie didn't have to cha

Re: libhdfs portability

I believe that the WebHDFS API is your best bet for now. The current implementation of WebHDFSClient does not reuse the HTTP connections, which leads to a large part of the performance penalty. You might want to implement your own version that reuses HTTP connection to see whether it meets your pe

Re: Replacing the JSP web UIs to HTML 5 applications

Hi Alejandro, Can you please elaborate on producing JSON? All information presented in the new Web UIs directly comes from the JMX side. I'm okay with leaving the current JSP right now, since both the old and the new Web UI can happily coexist. When do you think it is a good time to switch the d

Re: Replacing the JSP web UIs to HTML 5 applications

Echo my comments on HDFS-5402: bq. If we're going to remove the old web UI, I think the new web UI has to have the same level of unit testing. We shouldn't go backwards in terms of unit testing. I take a look at TestNamenodeJspHelper / TestDatanodeJspHelper / TestClusterJspHelper. It seems to me

Re: Replacing the JSP web UIs to HTML 5 applications

Producing JSON would be great. Agree with Colin that we should leave for now the current JSP based web ui. thx On Mon, Oct 28, 2013 at 11:16 AM, Colin McCabe wrote: > This is a really interesting project, Haohui. I think it will make > our web UI much nicer. > > I have a few concerns about rem

[jira] [Created] (HDFS-5440) Extract the logic of handling delegation tokens in HftpFileSystem to the TokenAspect class

Haohui Mai created HDFS-5440: Summary: Extract the logic of handling delegation tokens in HftpFileSystem to the TokenAspect class Key: HDFS-5440 URL: https://issues.apache.org/jira/browse/HDFS-5440 Projec

libhdfs portability

Now that Hadoop 2.2.0 is Windows compatible, is there going to be work on creating a portable version of libhdfs for C/C++ interaction with HDFS? I know I can use the WebHDFS REST API, but the data transfer rates are abysmally slow compared to the direct interaction via libhdfs. Regards, -- Kyle S

[jira] [Created] (HDFS-5439) Fix TestPendingReplications

Arpit Agarwal created HDFS-5439: --- Summary: Fix TestPendingReplications Key: HDFS-5439 URL: https://issues.apache.org/jira/browse/HDFS-5439 Project: Hadoop HDFS Issue Type: Sub-task Co

[jira] [Created] (HDFS-5438) Flaws in block report processing can cause data loss

Kihwal Lee created HDFS-5438: Summary: Flaws in block report processing can cause data loss Key: HDFS-5438 URL: https://issues.apache.org/jira/browse/HDFS-5438 Project: Hadoop HDFS Issue Type: Bu

[jira] [Created] (HDFS-5437) Fix TestBlockReport

Arpit Agarwal created HDFS-5437: --- Summary: Fix TestBlockReport Key: HDFS-5437 URL: https://issues.apache.org/jira/browse/HDFS-5437 Project: Hadoop HDFS Issue Type: Sub-task Components

Re: Replacing the JSP web UIs to HTML 5 applications

This is a really interesting project, Haohui. I think it will make our web UI much nicer. I have a few concerns about removing the old web UI, however: * If we're going to remove the old web UI, I think the new web UI has to have the same level of unit testing. We shouldn't go backwards in term

[jira] [Created] (HDFS-5436) Move HsFtpFileSystem and HFtpFileSystem into org.apache.hdfs.web

Haohui Mai created HDFS-5436: Summary: Move HsFtpFileSystem and HFtpFileSystem into org.apache.hdfs.web Key: HDFS-5436 URL: https://issues.apache.org/jira/browse/HDFS-5436 Project: Hadoop HDFS I

[jira] [Resolved] (HDFS-5435) File append fails to initialize storageIDs

[ https://issues.apache.org/jira/browse/HDFS-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-5435. - Resolution: Fixed Fix Version/s: Heterogeneous Storage (HDFS-2832) Release Note: Thanks

[jira] [Created] (HDFS-5435) Fix file append without setting storageIDs

Junping Du created HDFS-5435: Summary: Fix file append without setting storageIDs Key: HDFS-5435 URL: https://issues.apache.org/jira/browse/HDFS-5435 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Created] (HDFS-5434) Write resiliency for replica count 1

Buddy created HDFS-5434: --- Summary: Write resiliency for replica count 1 Key: HDFS-5434 URL: https://issues.apache.org/jira/browse/HDFS-5434 Project: Hadoop HDFS Issue Type: Bug Components: na

Build failed in Jenkins: Hadoop-Hdfs-trunk #1566

See Changes: [bikas] YARN-1022. Unnecessary INFO logs in AMRMClientAsync (haosdent via bikas) -- [...truncated 12964 lines...] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19

Build failed in Jenkins: Hadoop-Hdfs-0.23-Build #774

See -- [...truncated 7678 lines...] [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR]

Hadoop-Hdfs-0.23-Build - Build # 774 - Still Failing

See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/774/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 7871 lines...] [ERROR] location: class com.google.pro

Re: Decommission DataNode

sounds like a question for the cloudera support forums On 28 October 2013 08:59, lei liu wrote: > In CDH3u5, when the DataNode is Decommissioned, the DataNode progress will > be shutdown by NameNode. > > But In CDH4.3.1, when the DataNode is Decommissioned, the DataNode progress > will be not

Decommission DataNode

In CDH3u5, when the DataNode is Decommissioned, the DataNode progress will be shutdown by NameNode. But In CDH4.3.1, when the DataNode is Decommissioned, the DataNode progress will be not shutdown by NameNode. When the datanode is Decommissioned, why the datanode is not automatically shutdown

Datanode fencing mechanism

In https://issues.apache.org/jira/browse/HDFS-1972 jira, there is one below case: Scenario 3: DN restarts during split brain period (this scenario illustrates why I think we need to persistently record the promise about who is active) - block has 2 replicas, user asks to reduce to 1 - NN1 a