Thanks for all the feedback, folks.
I have created a jira: https://issues.apache.org/jira/browse/HDFS-5324.
Let us continue detailed discussions there.
- Milind
---
Milind Bhandarkar
Chief Scientist
Pivotal
+1-650-523-3858 (W)
+1-408-666-8483 (C)
On Mon, Oct 7, 2013 at 9:50 PM, sanjay Radia
On Oct 3, 2013, at 12:17 PM, Milind Bhandarkar wrote:
> Exec Summary: For the last couple of months, we, at Pivotal, along with a
> couple of folks in the community have been working on making Namespace
> implementation in the namenode pluggable. We have demonstrated that it can
> be done without
Milind,
Seems as a proper time to open a Jira.
Looks to me nobody is objecting to the general idea of AbstractNamesystem.
As usually the details is what finally matters.
FSNamesystem does have a formal interface called Namesystem now, but
it is somewhat arbitrary and probably rudimentary for your
Getting back to the technical discussion: based on this proposal email that I
sent, I came to know, from Sanjay Radia, of a summer intern project at
Hortonworks that implemented namespace using LevelDB. We discussed it on
personal email today, and agreed that these two efforts were still orthogo
On Mon, Oct 7, 2013 at 12:05 PM, Andrew Purtell wrote:
> I recognize some of what we recently experienced on a HDFS matter in what
> Milind wrote even if this was not the appropriate forum for it. Odd mention
> of "conspiracy theories" aside, for people who may come to this thread
> later, perhaps
Folks,
If you check the thread, I did respond to Vinod and others on technical
aspect of the proposal.
My later email was not intended for the hdfs-dev list, but was a personal
note to Vinod. Unfortunately, the reply-to meant it went to hdfs-dev, and
not to Vinod personally (although my To field
Milind/Andrew and everyone else,
Didn't know an innocuous question regarding a technical issue can blow up so
much.
I asked a technical question. If you have an answer and you want to respond,
please do so. If you don't have an answer, you can reply so. Otherwise please
keep this thread for t
On Oct 6, 2013, at 5:58 PM, Milind Bhandarkar wrote:
> Vinod,
>
> I have received a few emails about concerns that this effort somehow
> conflicts with federated namenodes. Most of these emails are from folks
> who are directly or remotely associated with Hortonworks.
>
> Three weeks ago, I sen
Hi Doug,
I recognize some of what we recently experienced on a HDFS matter in what
Milind wrote even if this was not the appropriate forum for it. Odd mention
of "conspiracy theories" aside, for people who may come to this thread
later, perhaps you can recommend an appropriate public Apache commun
On Sun, Oct 6, 2013 at 5:58 PM, Milind Bhandarkar
wrote:
> Is Hortonworks concerned that this proposal conflicts with their
> development on federated namenode ?
This question might better be asked directly of folks at Hortonworks
rather than here.
Let's please not dwell on such aspects here, ke
ent of this
> >> development, and would like to quash any conspiracy theories right now,
> >> before they assume a life of their own.
> >>
> >> Thanks,
> >>
> >> Milind
> >>
> >>
> >> -Original Message-
> >
> committers who are primarily focused on HDFS, whose email address I
> had.
> > >> While 2 out of those three responded to me, the third person
> associated
> > >> with Hortonworks, did not.
> > >>
> > >> Is Hortonworks concerned that this
eir
> >> development on federated namenode ? I have explicitly stated that it
> >>does
> >> not, and is orthogonal to federation. But I would like to know if there
> >> are some false assumptions being made about the intent of this
> >> development, and would like to q
anks,
>>
>> Milind
>>
>>
>> -Original Message-
>> From: Vinod Kumar Vavilapalli [mailto:vino...@hortonworks.com]
>> Sent: Sunday, October 06, 2013 12:21 PM
>> To: hdfs-dev@hadoop.apache.org
>> Subject: Re: [Proposal] Pluggable Namespace
>
ginal Message-
> From: Vinod Kumar Vavilapalli [mailto:vino...@hortonworks.com]
> Sent: Sunday, October 06, 2013 12:21 PM
> To: hdfs-dev@hadoop.apache.org
> Subject: Re: [Proposal] Pluggable Namespace
>
> In order to make federation happen, the block pool management was alrea
Vavilapalli [mailto:vino...@hortonworks.com]
Sent: Sunday, October 06, 2013 12:21 PM
To: hdfs-dev@hadoop.apache.org
Subject: Re: [Proposal] Pluggable Namespace
In order to make federation happen, the block pool management was already
separated. Isn't that the same as this effortt?
Thanks,
+
-Original Message-
From: Vinod Kumar Vavilapalli [mailto:vino...@hortonworks.com]
Sent: Sunday, October 06, 2013 12:21 PM
To: hdfs-dev@hadoop.apache.org
Subject: Re: [Proposal] Pluggable Namespace
In order to make federation happen, the block pool management was already
separated. Isn&#
To: hdfs-dev@hadoop.apache.org
> Subject: Re: [Proposal] Pluggable Namespace
>
> Hi Milind,
>
> HDFS federation can solve the NN bottle neck and memory limit problem.
>
> AbstractNameSystem design sounds good. but distributed meta storage using
> HBase should bring performance
:41 PM
To: hdfs-dev@hadoop.apache.org
Subject: Re: [Proposal] Pluggable Namespace
Hi Milind,
HDFS federation can solve the NN bottle neck and memory limit problem.
AbstractNameSystem design sounds good. but distributed meta storage using
HBase should bring performance degration.
On Oct 4, 2013 3
Hi Milind,
HDFS federation can solve the NN bottle neck and memory limit problem.
AbstractNameSystem design sounds good. but distributed meta storage using
HBase should bring performance degration.
On Oct 4, 2013 3:18 AM, "Milind Bhandarkar"
wrote:
> Hi All,
>
> Exec Summary: For the last coupl
separately.
- Milind
-Original Message-
From: Andrew Purtell [mailto:apurt...@apache.org]
Sent: Saturday, October 05, 2013 11:23 AM
To: hdfs-dev@hadoop.apache.org
Subject: Re: [Proposal] Pluggable Namespace
On Thu, Oct 3, 2013 at 12:17 PM, Milind Bhandarkar <
mbhandar...@gopivotal.com>
On Thu, Oct 3, 2013 at 12:17 PM, Milind Bhandarkar <
mbhandar...@gopivotal.com> wrote:
> To get started, we have implemented the same memory-based namespace
> implementation in a remote process, outside of the namenode JVM. In
> addition, work is undergoing to implement the namesystem using HBase.
I think it would be awesome to make this happen. I am +1 for the general
direction. I am by no means an HDFS expert but I would like to see the
patches you have made. If you could file a JIRA that points to the
modifications on github or in patches and any design work you have
explaining it (incl
Hi All,
Exec Summary: For the last couple of months, we, at Pivotal, along with a
couple of folks in the community have been working on making Namespace
implementation in the namenode pluggable. We have demonstrated that it can
be done without major surgery on the namenode, and does not have notic
24 matches
Mail list logo
