Since the merge vote passed, I have merged the HDFS-347 branch to trunk.
Leaving the JIRA open for now until we also do the merge to branch-2.
Colin, thanks a ton for the monster contribution. This is a long time in
coming.
--
Aaron T. Myers
Software Engineer, Cloudera
On Thu, Apr 11, 2013 at
Thanks Colin.
--Send from my Sony mobile.
On Apr 13, 2013 4:06 AM, "Colin McCabe" wrote:
> Hi Azuryy,
>
> The branch adds new APT documentation which describes the new configuration
> that is needed.
> It's
> in
> ./hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ShortCircuitLocalReads.apt.vm
>
> b
Hi Azuryy,
The branch adds new APT documentation which describes the new configuration
that is needed.
It's
in ./hadoop-hdfs-project/hadoop-hdfs/src/site/apt/ShortCircuitLocalReads.apt.vm
best,
Colin
On Thu, Apr 11, 2013 at 6:37 PM, Azuryy Yu wrote:
> It's good to know HDFS-347 win the votes
It's good to know HDFS-347 win the votes finally.
Does there need some additional configuration to enable these features?
On Fri, Apr 12, 2013 at 2:05 AM, Colin McCabe wrote:
> The merge vote is now closed. With three +1s, it passes.
>
> thanks,
> Colin
>
>
> On Wed, Apr 10, 2013 at 10:00 PM,
The merge vote is now closed. With three +1s, it passes.
thanks,
Colin
On Wed, Apr 10, 2013 at 10:00 PM, Aaron T. Myers wrote:
> I'm +1 as well. I've reviewed much of the code as well and have personally
> seen it running in production at several different sites. I agree with Todd
> that it's
I'm +1 as well. I've reviewed much of the code as well and have personally seen
it running in production at several different sites. I agree with Todd that
it's a substantial improvement in operability.
Best,
Aaron
On Apr 8, 2013, at 1:19 PM, Todd Lipcon wrote:
> +1 for the branch merge. I'v
"hdfs-dev@hadoop.apache.org"
Sent: Sunday, April 7, 2013 8:01 AM
Subject: Re: VOTE: HDFS-347 merge
The patch in HDFS-4661 has addressed the problem I raised. Once the previous
VOTE has be concluded, I will remove my -1. Thanks.
Tsz-Wo
From: T
Let's extend this vote by another 2 days just in case Nicholas doesn't find
time in his schedule today to comment.
He needs to withdraw his -1 before we can proceed.
Colin
On Mon, Apr 8, 2013 at 1:19 PM, Todd Lipcon wrote:
> +1 for the branch merge. I've reviewed all of the code in the branch
+1 for the branch merge. I've reviewed all of the code in the branch, and
we have people now running this code in production scenarios. It is as
functional as the old version and way easier to set up/configure.
-Todd
On Mon, Apr 1, 2013 at 4:32 PM, Colin McCabe wrote:
> Hi all,
>
> I think it's
thers to follow.
>
> Tsz-Wo
>
>
>
>
>
> From: Suresh Srinivas
> To: "hdfs-dev@hadoop.apache.org"
> Sent: Wednesday, March 6, 2013 5:09 AM
> Subject: Re: VOTE: HDFS-347 merge
>
> Thanks Colin. Will check it out as soon as I can.
>
>
> On Tue,
The patch in HDFS-4661 has addressed the problem I raised. Once the previous
VOTE has be concluded, I will remove my -1. Thanks.
Tsz-Wo
From: Tsz Wo Sze
To: "hdfs-dev@hadoop.apache.org"
Sent: Friday, April 5, 2013 12:11 PM
Subject: Re: VOTE
> We usually conclude the last VOTE before starting a new one. Otherwise,
> people may be confused between the VOTEs. (In case you don't know our
> convention. Please check with someone before starting a VOTE. Thanks.)
>
>
> -1
> * The previous VOTE started by Colin has not been concluded.
>
N
On Thu, Apr 4, 2013 at 9:11 PM, Tsz Wo Sze wrote:
> Colin,
>
> We usually conclude the last VOTE before starting a new one. Otherwise,
> people may be confused between the VOTEs. (In case you don't know our
> convention. Please check with someone before starting a VOTE. Thanks.)
>
>
> -1
> *
Colin,
We usually conclude the last VOTE before starting a new one. Otherwise, people
may be confused between the VOTEs. (In case you don't know our convention.
Please check with someone before starting a VOTE. Thanks.)
-1
* The previous VOTE started by Colin has not been concluded.
* The
Sent: Wednesday, April 3, 2013 1:38 AM
Subject: Re: VOTE: HDFS-347 merge
On Mon, Apr 1, 2013 at 6:58 PM, Colin McCabe wrote:
> On Mon, Apr 1, 2013 at 5:04 PM, Suresh Srinivas wrote:
>
>> Colin,
>>
>> For the record, the last email in the previous thread in ended with
+1 for the merge. Some of the minor changes or discussions happening in
some of the related tasks can be addressed post merge to trunk. The pending
discussions/issues should be fixed before this feature gets merged into
branch-2.
On Tue, Apr 2, 2013 at 10:38 AM, Colin McCabe wrote:
> On Mon, Apr
On Mon, Apr 1, 2013 at 6:58 PM, Colin McCabe wrote:
> On Mon, Apr 1, 2013 at 5:04 PM, Suresh Srinivas wrote:
>
>> Colin,
>>
>> For the record, the last email in the previous thread in ended with the
>> following comment from Nicholas:
>> > It is great to hear that you agree to keep HDFS-2246. Pl
On Mon, Apr 1, 2013 at 5:04 PM, Suresh Srinivas wrote:
> Colin,
>
> For the record, the last email in the previous thread in ended with the
> following comment from Nicholas:
> > It is great to hear that you agree to keep HDFS-2246. Please as well
> address my comments posted on HDFS-347 and let
Colin,
For the record, the last email in the previous thread in ended with the
following comment from Nicholas:
> It is great to hear that you agree to keep HDFS-2246. Please as well
address my comments posted on HDFS-347 and let me know once you have posted
a new patch on HDFS-347.
I did not se
Suresh mentioned earlier. I hope you could use our
convention on voting. Otherwise, it is hard for others to follow.
Tsz-Wo
From: Suresh Srinivas
To: "hdfs-dev@hadoop.apache.org"
Sent: Wednesday, March 6, 2013 5:09 AM
Subject: Re: VOTE: HDFS
Thanks Colin. Will check it out as soon as I can.
On Tue, Mar 5, 2013 at 12:24 PM, Colin McCabe wrote:
> On Tue, Feb 26, 2013 at 5:09 PM, Suresh Srinivas
> wrote:
> >>
> >> Suresh, if you're willing to "support and maintain" HDFS-2246, do you
> >> have cycles to propose a patch to the HDFS-347
On Tue, Feb 26, 2013 at 5:09 PM, Suresh Srinivas wrote:
>>
>> Suresh, if you're willing to "support and maintain" HDFS-2246, do you
>> have cycles to propose a patch to the HDFS-347 branch reintegrating
>> HDFS-2246 with the simplifications you outlined? In your review, did
>> you find anything el
+1
It sounds like restoring 2246 to the 347 patch is the only path
forward (ie there will be zero compromise on a proposal that removes
2246 in any form) in which case this seems like a good way to
implement that (similar to what Sanjay suggested earlier). We can have
a separate jira for removing
Suresh offered to write a patch restoring HDFS-2246, so unless his
timeline is unacceptable, I think we're done.
On Wed, Feb 27, 2013 at 11:45 AM, sanjay Radia wrote:
> It is not being held back of for the windows port. It is being held back
> because 2246 should not be removed as part of 347; a
Here is a compromise proposal, which hopefully will satisfy both sides:
We keep the old block reader and have a configuration option that enables it.
So in addition to dfs.client.use.legacy.blockreader, which we already
have, we would have dfs.client.use.legacy.blockreader.local.
Does that make s
On Wed, Feb 27, 2013 at 11:45 AM, sanjay Radia wrote:
>
> On Feb 26, 2013, at 1:51 PM, Eli Collins wrote:
>
>> it doesn't seem right to hold up 347 up for Windows support given that
>> Windows support has not been merged to trunk yet, is not in any Apache
>> release, etc. Personally I don't like e
On Feb 26, 2013, at 1:51 PM, Eli Collins wrote:
> it doesn't seem right to hold up 347 up for Windows support given that
> Windows support has not been merged to trunk yet, is not in any Apache
> release, etc. Personally I don't like establishing the precedent here
> that we can hold up a merge d
On Feb 20, 2013, at 5:12 PM, Aaron T. Myers wrote:
> On Wed, Feb 20, 2013 at 4:29 PM, Chris Douglas wrote:
>
>> Given that HDFS-347 is a strictly better approach, once committed,
>> there will be ample motivation to add support for other OSes and
>> remove HDFS-2246 entirely. Nobody is confused
>
> Suresh, if you're willing to "support and maintain" HDFS-2246, do you
> have cycles to propose a patch to the HDFS-347 branch reintegrating
> HDFS-2246 with the simplifications you outlined? In your review, did
> you find anything else you'd like to address prior to the merge, or is
> this the
Eli, you've sent the same email a half dozen times in the last ~24
hours. You might try a different tactic.
Suresh, if you're willing to "support and maintain" HDFS-2246, do you
have cycles to propose a patch to the HDFS-347 branch reintegrating
HDFS-2246 with the simplifications you outlined? In
Hi Bikas,
I completely agree with you in principle -- short circuit reads end up
ceding control of the data path from the DataNode to the user applications.
This has a few disadvantages which you've mentioned, and have been brought
up in the JIRA as well: particularly QoS, metrics, the flexibility
On Tue, Feb 26, 2013 at 11:35 AM, Suresh Srinivas
wrote:
>>
>>
>> I assume you mean in trunk? Given that ATM's proposal is to only
>> remove HDFS-2246 from branch-2 once (a) we're confident in HDFS-347
>> and (b) adds Windows support, and we won't be releasing from trunk any
>> time soon - from
Hi,
In my opinion, this feature of short circuit reads (HDFS-347 or HDFS-2246)
is not a desirable feature for HDFS. We should be working towards removing
this feature instead of enhancing it and making it popular.
Maybe short-circuit reads were something that HBase needed for performance
at a poi
>
>
> I assume you mean in trunk? Given that ATM's proposal is to only
> remove HDFS-2246 from branch-2 once (a) we're confident in HDFS-347
> and (b) adds Windows support, and we won't be releasing from trunk any
> time soon - from a user perspective - HDFS-2246 will only be replaced
> with HDFS
On Tue, Feb 26, 2013 at 9:33 AM, Suresh Srinivas wrote:
>>
>>
>> There's no reason to maintain multiple implementations of the same
>> feature, that's why per the 2246 jira it was proposed as a "good short
>> term solution till HDFS-347 is completed". Why is ATM's compromise
>> unacceptable?
>>
>
>
> There's no reason to maintain multiple implementations of the same
> feature, that's why per the 2246 jira it was proposed as a "good short
> term solution till HDFS-347 is completed". Why is ATM's compromise
> unacceptable?
>
We have already discussed this.
Here is the recap:
HDFS-347 do
On Mon, Feb 25, 2013 at 4:09 PM, Suresh Srinivas wrote:
> ATM's suggestion of removing HDFS-2246 in trunk, but not branch-2, is
>> a rational compromise: it allows some period for others to adapt, but
>> not an indefinite one. It's not clear what you're proposing, if
>> anything.
>>
>
>
>
> I am n
ATM's suggestion of removing HDFS-2246 in trunk, but not branch-2, is
> a rational compromise: it allows some period for others to adapt, but
> not an indefinite one. It's not clear what you're proposing, if
> anything.
>
I am not sure why a release that supports both these is such a bad idea.
A
t;> Tsz-Wo
>>>
>>>
>>>
>>>
>>>
>>> From: Eli Collins
>>> To: "hdfs-dev@hadoop.apache.org" ; Tsz Wo Sze
>>>
>>> Sent: Monday, February 25, 2013 10:24 AM
>>> Sub
gt;
>>
>>
>>
>>
>> From: Eli Collins
>> To: "hdfs-dev@hadoop.apache.org" ; Tsz Wo Sze
>>
>> Sent: Monday, February 25, 2013 10:24 AM
>> Subject: Re: VOTE: HDFS-347 merge
>>
>> On Sat, Feb
e.org" ; Tsz Wo Sze
>
> Sent: Monday, February 25, 2013 10:24 AM
> Subject: Re: VOTE: HDFS-347 merge
>
> On Sat, Feb 23, 2013 at 4:23 PM, Tsz Wo Sze wrote:
>> I still do not see a valid reason to remove HDFS-2246 immediately. Some
>> users may have insecure clu
Tsz Wo Sze
Sent: Monday, February 25, 2013 10:24 AM
Subject: Re: VOTE: HDFS-347 merge
On Sat, Feb 23, 2013 at 4:23 PM, Tsz Wo Sze wrote:
> I still do not see a valid reason to remove HDFS-2246 immediately. Some
> users may have insecure clusters and they don't want to change their
>
.
Colin
>
>
> Tsz-Wo
>
>
>
>
> From: Aaron T. Myers
> To: "hdfs-dev@hadoop.apache.org" ; Tsz Wo Sze
>
> Sent: Friday, February 22, 2013 6:40 PM
> Subject: Re: VOTE: HDFS-347 merge
>
> On Fri, Feb 22, 2013 at 6:
On Sat, Feb 23, 2013 at 4:23 PM, Tsz Wo Sze wrote:
> I still do not see a valid reason to remove HDFS-2246 immediately. Some
> users may have insecure clusters and they don't want to change their
> configuration.
Because it doesn't make sense to support multiple mechanisms for the
same thing.
sz-Wo
From: Aaron T. Myers
To: "hdfs-dev@hadoop.apache.org" ; Tsz Wo Sze
Sent: Friday, February 22, 2013 6:40 PM
Subject: Re: VOTE: HDFS-347 merge
On Fri, Feb 22, 2013 at 6:32 PM, Tsz Wo Sze wrote:
> Another
> substantive concern is that HD
On Fri, Feb 22, 2013 at 6:32 PM, Tsz Wo Sze wrote:
> Another
> substantive concern is that HDFS-347 is not as well tested as
> HDFS-2246. So, we should keep HDFS-2246 around for sometime and remove
> it later. Is this the usual practice?
>
I'm proposing we do just that - keep HDFS-2246 around
st.
Tsz-Wo
From: Eli Collins
To: "hdfs-dev@hadoop.apache.org"
Sent: Friday, February 22, 2013 1:55 PM
Subject: Re: VOTE: HDFS-347 merge
On Thu, Feb 21, 2013 at 1:24 PM, Chris Douglas wrote:
> On Wed, Feb 20, 2013 at 5:12 PM, Aaron T. Myers wrote
On Thu, Feb 21, 2013 at 1:24 PM, Chris Douglas wrote:
> On Wed, Feb 20, 2013 at 5:12 PM, Aaron T. Myers wrote:
>> Given that the only substantive concerns with HDFS-347 seem to be about
>> Windows support for local reads, for now we only merge this branch to
>> trunk. Support for doing HDFS-2246
On Thu, Feb 21, 2013 at 1:24 PM, Chris Douglas wrote:
> On Wed, Feb 20, 2013 at 5:12 PM, Aaron T. Myers wrote:
>> Given that the only substantive concerns with HDFS-347 seem to be about
>> Windows support for local reads, for now we only merge this branch to
>> trunk. Support for doing HDFS-2246
On Wed, Feb 20, 2013 at 5:12 PM, Aaron T. Myers wrote:
> Given that the only substantive concerns with HDFS-347 seem to be about
> Windows support for local reads, for now we only merge this branch to trunk.
Another substantive concern is that HDFS-347 is not as well tested as
HDFS-2246. So, w
On Wed, Feb 20, 2013 at 5:12 PM, Aaron T. Myers wrote:
> Given that the only substantive concerns with HDFS-347 seem to be about
> Windows support for local reads, for now we only merge this branch to
> trunk. Support for doing HDFS-2246 style local reads will be removed from
> trunk, but retained
On Wed, Feb 20, 2013 at 5:12 PM, Aaron T. Myers wrote:
> On Wed, Feb 20, 2013 at 4:29 PM, Chris Douglas
> wrote:
>
> > Given that HDFS-347 is a strictly better approach, once committed,
> > there will be ample motivation to add support for other OSes and
> > remove HDFS-2246 entirely. Nobody is
> Only once someone adds support for doing HDFS-347 style local reads which
work on Windows will we consider merging HDFS-347 to branch-2.
There's no chance of having both HDFS-347 and HDFS-2246 style local reads
coexisting in branch-2?
It would be nice if HDFS-347 was in branch-2 sooner rather t
On Wed, Feb 20, 2013 at 4:29 PM, Chris Douglas wrote:
> Given that HDFS-347 is a strictly better approach, once committed,
> there will be ample motivation to add support for other OSes and
> remove HDFS-2246 entirely. Nobody is confused about this. There's
> ample precedent for retaining obscure
On Wed, Feb 20, 2013 at 4:29 PM, Chris Douglas wrote:
> The throughput on this thread is too high.
>
> Nicholas/Suresh: do either of you disagree with the general approach
> in HDFS-347? Holding an inevitable branch open causes a lot of pain
> (Suresh, you endured much of that personally with sec
The throughput on this thread is too high.
Nicholas/Suresh: do either of you disagree with the general approach
in HDFS-347? Holding an inevitable branch open causes a lot of pain
(Suresh, you endured much of that personally with security, which had
a lot of follow-on work). While it makes sense t
>
> The patches even going back as far as last September have all removed
> the old code path. I sort of assumed that, if you are taking time to
> review the patches, you would have noticed this... additionally,
> Colin's comments on the JIRA said as much... eg:
>
Todd, we have different ways of r
On Wed, Feb 20, 2013 at 4:04 PM, Suresh Srinivas wrote:
>
> HDFS-347 does not clearly state old short circuit will be removed any where
> in the jira or design. If this was made clear in the jira, this discussion
> would
> have happened much earlier than now.
>
> You seem to be taking the comments
>
> >
> > I have to disagree. No where in the jira or the design it is explicitly
> > stated that
> > the old short circuit functionality is being removed. My assumption has
> been
> > that it will not be removed.
>
> I've tried this avenue in the past on other insecurities which were
> fixed. Sorr
On Wed, Feb 20, 2013 at 3:31 PM, Suresh Srinivas wrote:
>> Given that this is an optimization, and we have a ton of optimizations
>> which don't yet run on Windows, I don't think that should be
>> considered. Additionally, the Windows support has not yet been merged,
>> nor is it in any release, s
On Wed, Feb 20, 2013 at 3:13 PM, Todd Lipcon wrote:
> On Wed, Feb 20, 2013 at 3:08 PM, Tsz Wo Sze wrote:
> > The reason to keep it around is that the HDFS-347 only support Unix but
> not
> > other OS.
>
> Given that this is an optimization, and we have a ton of optimizations
> which don't yet ru
>
>
> This was just an error with the "consolidate merge patch". Like I said
> in the previous email, these patches are just for Jenkins QA to run
> on, and I assume that any HDFS committer is able to look at the branch
> itself to understand the changes in it. It's easy to accidentally end
> up wi
ay, February 20, 2013 3:06 PM
>
> Subject: Re: VOTE: HDFS-347 merge
>
> On Wed, Feb 20, 2013 at 3:01 PM, Tsz Wo Sze wrote:
>> Also, the patch seems to have removed the existing short-circuit read
>> feature (HDFS-2246). It is an incompatible change. I think the patch is
The reason to keep it around is that the HDFS-347 only support Unix but not
other OS.
Tsz-Wo
From: Todd Lipcon
To: hdfs-dev@hadoop.apache.org; Tsz Wo Sze
Sent: Wednesday, February 20, 2013 3:06 PM
Subject: Re: VOTE: HDFS-347 merge
On Wed, Feb 20, 2013
On Wed, Feb 20, 2013 at 3:06 PM, Todd Lipcon wrote:
> https://issues.apache.org/jira/browse/HDFS-4476 where I pointed out
sorry, meant to link to:
https://issues.apache.org/jira/browse/HDFS-2246?focusedCommentId=13102013&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#commen
On Wed, Feb 20, 2013 at 3:01 PM, Tsz Wo Sze wrote:
> Also, the patch seems to have removed the existing short-circuit read feature
> (HDFS-2246). It is an incompatible change. I think the patch is farther
> away from being ready and I would keep my -1.
The existing short circuit feature is in
review.
>>
>> -Todd
>>
>> On Wed, Feb 20, 2013 at 11:56 AM, Tsz Wo Sze wrote:
>> > -1
>> > The patch seems not ready yet. I have posted some comments/suggestions
>> on the JIRA. Colin also has agreed that there are some bugs to be fixed.
>> So
e.org"
Sent: Wednesday, February 20, 2013 11:56 AM
Subject: Re: VOTE: HDFS-347 merge
-1
The patch seems not ready yet. I have posted some comments/suggestions on the
JIRA. Colin also has agreed that there are some bugs to be fixed. Sorry.
Tsz-Wo
-1
> > The patch seems not ready yet. I have posted some comments/suggestions
> on the JIRA. Colin also has agreed that there are some bugs to be fixed.
> Sorry.
> >
> > Tsz-Wo
> >
> >
> >
> >
> > ________
> >
JIRA. Colin also has agreed that there are some bugs to be fixed.
>> Sorry.
>>
>> Tsz-Wo
>>
>>
>>
>>
>>
>> From: Todd Lipcon
>> To: hdfs-dev@hadoop.apache.org
>> Sent: Tuesday, February 19, 2013 4:
From: Todd Lipcon
To: hdfs-dev@hadoop.apache.org; Tsz Wo Sze
Sent: Wednesday, February 20, 2013 12:16 PM
Subject: Re: VOTE: HDFS-347 merge
Hi Nicholas,
I looked at your comments on the JIRA, and they all seem like trivial
things that could be addressed post-merge, and none of them would
affect
Subject: Re: VOTE: HDFS-347 merge
+1 (binding)
I code-reviewed almost all of the code in this branch, and also spent some
time benchmarking and testing under various workloads. We've also done
significant testing on clusters here at Cloudera, both secure and insecure,
and verified integration
ache.org
> Sent: Tuesday, February 19, 2013 4:11 PM
> Subject: Re: VOTE: HDFS-347 merge
>
> +1 (binding)
>
> I code-reviewed almost all of the code in this branch, and also spent some
> time benchmarking and testing under various workloads. We've also done
> significan
+1 (non binding)
This is great...
On Tue, Feb 19, 2013 at 7:11 PM, Todd Lipcon wrote:
> +1 (binding)
>
> I code-reviewed almost all of the code in this branch, and also spent some
> time benchmarking and testing under various workloads. We've also done
> significant testing on clusters here at
+1 (binding)
I code-reviewed almost all of the code in this branch, and also spent some
time benchmarking and testing under various workloads. We've also done
significant testing on clusters here at Cloudera, both secure and insecure,
and verified integration with a number of other ecosystem compo
Hi Colin,
The latest HDFS-347 patch was posted on Feb 16. Because of the long weekends,
I still do not have a chance to check it. Will do it in this week.
Nicholas
From: Colin McCabe
To: hdfs-dev@hadoop.apache.org
Sent: Sunday, February 17, 2013 1:48 PM
+1 (non binding)
If this gets in, a backport to branch-2 would be most appreciated.
On Sun, Feb 17, 2013 at 1:48 PM, Colin McCabe wrote:
> Hi all,
>
> I would like to merge the HDFS-347 branch back to trunk. It's been
> under intensive review and testing for several months. The branch
> adds
+1
On Sun, Feb 17, 2013 at 1:48 PM, Colin McCabe wrote:
> Hi all,
>
> I would like to merge the HDFS-347 branch back to trunk. It's been
> under intensive review and testing for several months. The branch
> adds a lot of new unit tests, and passes Jenkins as of 2/15 [1]
>
> We have tested HDFS
78 matches
Mail list logo
