Thanks, Sanjay and Suresh.
I've merged the branch into trunk.
Will take care of merging the CHANGES.txt lists, etc, tomorrow, and start
working on the agreed-upon improvements early next week.
Thanks
-Todd
On Wed, Oct 10, 2012 at 9:10 PM, Suresh Srinivas wrote:
> Given the completion of discuss
Given the completion of discussions and the resolution towards next steps I
am +1 for the merge.
On Wed, Oct 10, 2012 at 7:17 PM, sanjay Radia wrote:
>
> On Oct 9, 2012, at 1:44 PM, sanjay Radia wrote:
>
> > I have posted a comment on 3077 listing two areas that i would like to
> fix in the QJM.
On Oct 9, 2012, at 1:44 PM, sanjay Radia wrote:
> I have posted a comment on 3077 listing two areas that i would like to fix
> in the QJM.
> If we agree on that we can merge now and fix those two later.
> Todd has volunteered to fix one and I or suresh can fix the other.
>
>
> sanjay
>
2 j
I have posted a comment on 3077 listing two areas that i would like to fix in
the QJM.
If we agree on that we can merge now and fix those two later.
Todd has volunteered to fix one and I or suresh can fix the other.
sanjay
On Mon, Oct 8, 2012 at 7:21 PM, Suresh Srinivas wrote:
> On Mon, Oct 8, 2012 at 6:20 PM, Todd Lipcon wrote:
>
>> On Mon, Oct 8, 2012 at 6:01 PM, Suresh Srinivas > >wrote:
>>
>> > Todd,
>> >
>> > As I indicated in my comments on the jira, I think some of the design
>> > discussions and further sim
On Mon, Oct 8, 2012 at 8:03 PM, Andrew Purtell wrote:
> Our position on the QJM is we've already "taken delivery" from the feature
> branch and will maintain a private HDFS fork of branch-2 if necessary, i.e.
> we don't have a significant stake in this discussion except at a meta level
> as poten
Sorry, just to be clear "our" and "we" below refer to my employer, nothing
to do with HBase. Please pardon any confusion.
On Tuesday, October 9, 2012, Andrew Purtell wrote:
> Our position on the QJM is we've already "taken delivery" from the feature
> branch and will maintain a private HDFS fork
Our position on the QJM is we've already "taken delivery" from the feature
branch and will maintain a private HDFS fork of branch-2 if necessary, i.e.
we don't have a significant stake in this discussion except at a meta level
as potential contributors.
This is a case study of why feature branch d
On Mon, Oct 8, 2012 at 6:20 PM, Todd Lipcon wrote:
> On Mon, Oct 8, 2012 at 6:01 PM, Suresh Srinivas >wrote:
>
> > Todd,
> >
> > As I indicated in my comments on the jira, I think some of the design
> > discussions and further simplification of design should happen before the
> > merge. See -
>
Sanjay, Suresh,
I don't think we should block the merge on discussion about parts of
the design. It's great that you guys are coming up to speed on the
design and have feedback. We're confident the feedback can be
addressed in follow-on jiras (just as federation and HA have been
iterated on after
On Mon, Oct 8, 2012 at 6:01 PM, Suresh Srinivas wrote:
> Todd,
>
> As I indicated in my comments on the jira, I think some of the design
> discussions and further simplification of design should happen before the
> merge. See -
>
> https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=1
Todd,
As I indicated in my comments on the jira, I think some of the design
discussions and further simplification of design should happen before the
merge. See -
https://issues.apache.org/jira/browse/HDFS-3077?focusedCommentId=13470680&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
Hi Sanjay,
The 7 extra days you requested beyond the original 7-day merge vote have
now elapsed, and we have the requisite three binding +1s to merge.
I'll plan to merge this late tonight unless there are any vetoes in the
meantime.
Of course we can continue to discuss the design and improve the
Todd,
Even though this work was under development over a period of time, during
its development it was not clear when the design was fairly stable to begin a
thorough review. Hence the time of merge is when the real review happens in
such large projects.
I have already indicated on the jira
Hey Sanjay,
While I understand it's substantial and complex code, the code and the
design doc have been available for several months, and the community
has certainly been aware of its development. I also gave a heads up
last week that I would call a merge this week. So I feel like there
has been s
Suresh and I are still reviewing this design and patch.
The 3077 code along with the code pulled from 3092 is fairly substrantial. The
design is also fairly complex and involved.
I would request that we postpone the merge for another week to give folks time
to review this fully.
sanjay
I am in favor of 3077 being part of HDFS.
- It is one of several way of configuring HDFS HA (shared storage is an
option or if we extend the BackupNode a little, then it will be another option
for HA.)
- I believe I can use part of 3092/3077 to move journals to the Datanodes
down th
Sorry for the slowness in response - I've been in bed with a fever the
past couple days and only sporadically on email. I'll respond to a few
different points made in the above thread here:
Konst> We keep growing the HDFS umbrella with competing technologies
Konst> (http/web HDFS as an example) wi
On Thu, Sep 27, 2012 at 2:06 AM, Konstantin Shvachko
wrote:
> The SPOF is in HDFS. This project is about shared storage
> implementation, that could be replaced by NFS or BookKeeper or
> something else.
You cannot equate QJM to a solution that requires an NFS filer. A
filer is just not possible
I am in favor of keeping QJM in HDFS.
QJM is very specific to HDFS and is tightly coupled with HDFS code,
essentially extending the current editlog functionality that writes to
local disk to writing to a separate set of daemons. Clearly there is a need
for this in HDFS. Konstantin, I see your poin
It's true we can package and use HDFS and QJM, if it were separately
distributed, because we supply our internal users with an Apache Hadoop
distribution with our additional polish. Also, Cloudera or Hortonworks or
others could package them together, but then what of the Apache Hadoop
distribution
HA design assumes shared storage. QJM can be used as such, so as other things.
If that was your question.
--Konst
On Thu, Sep 27, 2012 at 2:11 AM, Andrew Purtell wrote:
> I don't follow. For example the QJM and HA NameNode configuration are
> designed together to eliminate the SPOF in HDFS, withi
On Thu, Sep 27, 2012 at 1:50 AM, Andrew Purtell wrote:
> Speaking as an Apache Hadoop user who must do something with the NameNode
> single point of failure this year, I don't subscribe to the view that
> moving that SPOF from the NameNode to a NFS filer is reasonable to ask of
> those not already
I don't follow. For example the QJM and HA NameNode configuration are
designed together to eliminate the SPOF in HDFS, within HDFS. I don't see
how they make sense separately.
On Thursday, September 27, 2012, Konstantin Shvachko wrote:
> The SPOF is in HDFS. This project is about shared storage
>
The SPOF is in HDFS. This project is about shared storage
implementation, that could be replaced by NFS or BookKeeper or
something else.
Suppose failure happened because NFS failed. Where do you go? NetApp.
Same if QJ failed. You go to the creators, which will be somewhat
closer than NetApp, becaus
Speaking as an Apache Hadoop user who must do something with the NameNode
single point of failure this year, I don't subscribe to the view that
moving that SPOF from the NameNode to a NFS filer is reasonable to ask of
those not already set up with NetApp or similar, or those running in a
"cloud" en
On Wed, Sep 26, 2012 at 4:21 PM, Konstantin Shvachko
wrote:
> Don't understand your argument. Else where?
You suggest users should download HDFS and then go to another project
(or subproject) -- i.e. 'elsewhere' -- to get a fundamental, a fix for
the SPOF. IMO, the SPOF-fix belongs in HDFS core.
On Wed, Sep 26, 2012 at 04:17PM, Konstantin Shvachko wrote:
> Hi Todd,
>
> > I had said previously that it's worth
> > discussing if several other people believe the same.
>
> Well let's put it on to general list for discussion then?
> Seems to me an important issue for Hadoop evolution in gener
Don't understand your argument. Else where?
One way or another users will be talking to Todd.
Thanks,
--Konst
On Wed, Sep 26, 2012 at 2:32 PM, Stack wrote:
> On Tue, Sep 25, 2012 at 11:21 PM, Konstantin Shvachko
> wrote:
>> I think this is a great work, Todd.
>> And I think we should not merge
Hi Todd,
> I had said previously that it's worth
> discussing if several other people believe the same.
Well let's put it on to general list for discussion then?
Seems to me an important issue for Hadoop evolution in general.
We keep growing the HDFS umbrella with competing technologies
(http/we
On Tue, Sep 25, 2012 at 11:21 PM, Konstantin Shvachko
wrote:
> I think this is a great work, Todd.
> And I think we should not merge it into trunk or other branches.
> As I suggested earlier on this list I think this should be spinned off
> as a separate project or a subproject.
>
I'd be -1 on th
+1 for the merge.
I've reviewed much of the code as individual patches and tested the whole
system, both in single- and multi-node configurations. I've also tested the
system with security enabled and confirmed that it works as expected. I've done
all of the above testing both with and without
On Wed, Sep 26, 2012 at 10:50 AM, Todd Lipcon wrote:
> On Tue, Sep 25, 2012 at 11:21 PM, Konstantin Shvachko
> wrote:
>> I think this is a great work, Todd.
>> And I think we should not merge it into trunk or other branches.
>> As I suggested earlier on this list I think this should be spinned of
On Tue, Sep 25, 2012 at 11:21 PM, Konstantin Shvachko
wrote:
> I think this is a great work, Todd.
> And I think we should not merge it into trunk or other branches.
> As I suggested earlier on this list I think this should be spinned off
> as a separate project or a subproject.
>
> - The code is
I think this is a great work, Todd.
And I think we should not merge it into trunk or other branches.
As I suggested earlier on this list I think this should be spinned off
as a separate project or a subproject.
- The code is well detached as a self contained package.
- It is a logically stand-alon
+1 Awesome work Todd.
On Tue, Sep 25, 2012 at 4:02 PM, Todd Lipcon wrote:
> Dear fellow HDFS developers,
>
> Per my email thread last week ("Heads up: merge for QJM branch soon"
> at http://markmail.org/message/vkyh5culdsuxdb6t) I would like to
> propose merging the HDFS-3077 branch into trunk.
On Tue, Sep 25, 2012 at 4:02 PM, Todd Lipcon wrote:
> detail in a document attached to HDFS-3077 itself. The code itself has
> been contributed by me, Aaron, and Eli, but I'd be remiss not to also
I should actually amend the above: I had forgotten that some of the
early patches on the branch were
Dear fellow HDFS developers,
Per my email thread last week ("Heads up: merge for QJM branch soon"
at http://markmail.org/message/vkyh5culdsuxdb6t) I would like to
propose merging the HDFS-3077 branch into trunk. The branch has been
active since mid July and has stabilized significantly over the la
38 matches
Mail list logo
