Re: [Proposal] Pluggable Namespace

On Oct 3, 2013, at 12:17 PM, Milind Bhandarkar wrote: > Exec Summary: For the last couple of months, we, at Pivotal, along with a > couple of folks in the community have been working on making Namespace > implementation in the namenode pluggable. We have demonstrated that it can > be done without

Re: [Proposal] Pluggable Namespace

Milind, Seems as a proper time to open a Jira. Looks to me nobody is objecting to the general idea of AbstractNamesystem. As usually the details is what finally matters. FSNamesystem does have a formal interface called Namesystem now, but it is somewhat arbitrary and probably rudimentary for your

Re: [Proposal] Pluggable Namespace

Getting back to the technical discussion: based on this proposal email that I sent, I came to know, from Sanjay Radia, of a summer intern project at Hortonworks that implemented namespace using LevelDB. We discussed it on personal email today, and agreed that these two efforts were still orthogo

[jira] [Created] (HDFS-5322) HDFS delegation token not found in cache errors seen on secure HA clusters

Arpit Gupta created HDFS-5322: - Summary: HDFS delegation token not found in cache errors seen on secure HA clusters Key: HDFS-5322 URL: https://issues.apache.org/jira/browse/HDFS-5322 Project: Hadoop HDFS

[jira] [Created] (HDFS-5321) Clean up the HTTP-related configuration in HDFS

Haohui Mai created HDFS-5321: Summary: Clean up the HTTP-related configuration in HDFS Key: HDFS-5321 URL: https://issues.apache.org/jira/browse/HDFS-5321 Project: Hadoop HDFS Issue Type: Sub-tas

[jira] [Resolved] (HDFS-5314) Do not expose CachePool type in AddCachePoolOp

[ https://issues.apache.org/jira/browse/HDFS-5314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe resolved HDFS-5314. Resolution: Fixed Fix Version/s: HDFS-4949 > Do not expose CachePool type in

Re: [VOTE] Release Apache Hadoop 2.2.0

+1 Deployed on secure and nonsecure clusters. Ran some hdfs, MR and yarn tests. Seems good to me. On Mon, Oct 7, 2013 at 12:21 PM, Alejandro Abdelnur wrote: > +1 > > * downloaded source tarball > * verified MD5 > * verified signature > * verified CHANGES.txt files, release # and date > * run 'mv

[jira] [Created] (HDFS-5320) Add datanode caching metrics

Andrew Wang created HDFS-5320: - Summary: Add datanode caching metrics Key: HDFS-5320 URL: https://issues.apache.org/jira/browse/HDFS-5320 Project: Hadoop HDFS Issue Type: Sub-task Compo

[jira] [Created] (HDFS-5319) Links resolving either from active/standby should be same (example clicking on datanodes from Standby)

Siqi Li created HDFS-5319: - Summary: Links resolving either from active/standby should be same (example clicking on datanodes from Standby) Key: HDFS-5319 URL: https://issues.apache.org/jira/browse/HDFS-5319

Re: [Proposal] Pluggable Namespace

On Mon, Oct 7, 2013 at 12:05 PM, Andrew Purtell wrote: > I recognize some of what we recently experienced on a HDFS matter in what > Milind wrote even if this was not the appropriate forum for it. Odd mention > of "conspiracy theories" aside, for people who may come to this thread > later, perhaps

Re: [Proposal] Pluggable Namespace

Folks, If you check the thread, I did respond to Vinod and others on technical aspect of the proposal. My later email was not intended for the hdfs-dev list, but was a personal note to Vinod. Unfortunately, the reply-to meant it went to hdfs-dev, and not to Vinod personally (although my To field

Re: [Proposal] Pluggable Namespace

Milind/Andrew and everyone else, Didn't know an innocuous question regarding a technical issue can blow up so much. I asked a technical question. If you have an answer and you want to respond, please do so. If you don't have an answer, you can reply so. Otherwise please keep this thread for t

Re: [Proposal] Pluggable Namespace

On Oct 6, 2013, at 5:58 PM, Milind Bhandarkar wrote: > Vinod, > > I have received a few emails about concerns that this effort somehow > conflicts with federated namenodes. Most of these emails are from folks > who are directly or remotely associated with Hortonworks. > > Three weeks ago, I sen

Re: [VOTE] Release Apache Hadoop 2.2.0

+1 * downloaded source tarball * verified MD5 * verified signature * verified CHANGES.txt files, release # and date * run 'mvn apache-rat:check' successfully * built distribution * setup speudo cluster * started HDFS/YARN * run some HTTFS tests * run a couple of MR examples * run a few tests using

[jira] [Created] (HDFS-5318) Account for shared storage in NameNode replica counting algorithm

Eric Sirianni created HDFS-5318: --- Summary: Account for shared storage in NameNode replica counting algorithm Key: HDFS-5318 URL: https://issues.apache.org/jira/browse/HDFS-5318 Project: Hadoop HDFS

Re: [VOTE] Release Apache Hadoop 2.2.0

+1 for the release. I have deployed a multinode cluster and extensively tested MR speculative execution, YARN CLI and YARN distributed shell. There were couple of issues I encountered while testing MAPREDUCE-5533, YARN-1168, YARN-1167, YARN-1157, YARN-1131, YARN-1118, YARN-1117 and all of them

Re: [Proposal] Pluggable Namespace

Hi Doug, I recognize some of what we recently experienced on a HDFS matter in what Milind wrote even if this was not the appropriate forum for it. Odd mention of "conspiracy theories" aside, for people who may come to this thread later, perhaps you can recommend an appropriate public Apache commun

[jira] [Created] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI

Suresh Srinivas created HDFS-5317: - Summary: Go back to DFS Home link does not work on datanode webUI Key: HDFS-5317 URL: https://issues.apache.org/jira/browse/HDFS-5317 Project: Hadoop HDFS

[jira] [Created] (HDFS-5316) Namenode ignore the default https port

Suresh Srinivas created HDFS-5316: - Summary: Namenode ignore the default https port Key: HDFS-5316 URL: https://issues.apache.org/jira/browse/HDFS-5316 Project: Hadoop HDFS Issue Type: Sub-ta

Re: [Proposal] Pluggable Namespace

On Sun, Oct 6, 2013 at 5:58 PM, Milind Bhandarkar wrote: > Is Hortonworks concerned that this proposal conflicts with their > development on federated namenode ? This question might better be asked directly of folks at Hortonworks rather than here. Let's please not dwell on such aspects here, ke

Re: [Proposal] Pluggable Namespace

Thanks Bobby. Experimentation with new namespace implementations and parallel development is one of the main intents of starting this project from my end. HDFS has improved a lot, and many of the perceived limitations, such as HA, Performance, Snapshots, (limited) NFS connectivity have been addre

Re: [Proposal] Pluggable Namespace

Chris, CLI is an issue that we had considered, but not in depth. My thinking was that the dfsadmin commands that are relevant only to current FSNamesystem can be separated out later from commands that are applicable to all Namesystem implementations, maybe with a separate nsadmin command group. Ho

Re: [Proposal] Pluggable Namespace

Thank you for sending out these notes, Milind. > Current HDFS design is such that FSNameSystem is baked into even high level > interfaces, this is a major hurdle in cleanly implementing pluggable name > systems. We aim to propose a change in such interfaces into which > FSNameSystem is tightly cou

Re: [Proposal] Pluggable Namespace

Putting all conspiracy theories aside :). Any way we decided to scale the name node is going to have limitations. Federation currently has the problem that we cannot easily move data between different name nodes. It is a static partitioning. It is not a blocker, but it can be annoying. We can f

Re: About debugging HDFS and MapReduce...

I am not sure what you mean by some portion of source code of Hadoop. Because you are running on a fully distributed environment you need to make sure that eclipse is attached to the correct process. If you are trying to debug the data upload pipeline then you need to be sure that eclipse is attac

Re: [VOTE] Release Apache Hadoop 2.2.0

+1 I did some basic testing and it appears to work fine. Daryn On Oct 7, 2013, at 2:00 AM, Arun C Murthy wrote: > Folks, > > I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to > get released - this release fixes a small number of bugs and some > protocol/api issues

Re: [VOTE] Release Apache Hadoop 2.2.0

I ran some basic sanity tests and it seems OK to me. +1 On 10/7/13 2:00 AM, "Arun C Murthy" wrote: >Folks, > >I've created a release candidate (rc0) for hadoop-2.2.0 that I would like >to get released - this release fixes a small number of bugs and some >protocol/api issues which should ensure

[jira] [Created] (HDFS-5315) [HDFS-Snapshot] Return code of SnapshotDiff of two invalid snapshots is 0

Brahma Reddy Battula created HDFS-5315: -- Summary: [HDFS-Snapshot] Return code of SnapshotDiff of two invalid snapshots is 0 Key: HDFS-5315 URL: https://issues.apache.org/jira/browse/HDFS-5315 Pro

Jenkins build is back to normal : Hadoop-Hdfs-trunk #1545

See

Hadoop-Hdfs-0.23-Build - Build # 753 - Still Failing

See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/753/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 7870 lines...] [ERROR] location: class com.google.pro

Build failed in Jenkins: Hadoop-Hdfs-0.23-Build #753

See -- [...truncated 7677 lines...] [ERROR] symbol : class Parser [ERROR] location: package com.google.protobuf [ERROR]

[VOTE] Release Apache Hadoop 2.2.0

Folks, I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to get released - this release fixes a small number of bugs and some protocol/api issues which should ensure they are now stable and will not change in hadoop-2.x. The RC is available at: http://people.apache.org/