Re: [DISCUSS] Supporting Hadoop-1 and experimental features

[Alan Gates]

Xuefu Zhang <mailto:xzh...@cloudera.com>
May 15, 2015 at 22:29
Thanks for the explanation, Alan!

While I have understood more on the proposal, I actually see more 
problems than the confusion of two lines of releases. Essentially, 
this proposal forces a user to make a hard choice between a stabler, 
legacy-aware release line and an adventurous, pioneering release line. 
And once the choice is made, there is no easy way back or forward.

Here is my interpretation. Let's say we have two main branches as 
proposed. I develop a new feature which I think useful for both 
branches. So, I commit it to both branches. My feature requires 
additional schema support, so I provide upgrade scripts for both 
branches. The scripts are different because the two branches have 
already diverged in schema.

Now the two branches evolve in a diverging fashion like this. This is 
all good as long as a user stays in his line. The moment the user 
considers a switch, mostly likely, from branch-1 to branch-2, he is 
stuck. Why? Because there is no upgrade path from a release in 
branch-1 to a release in branch-2!

If we want to provide an upgrade path, then there will be MxN paths, 
where M and N are the number of releases in the two branches, 
respectively. This is going to be next to a nightmare, not only for 
users, but also for us.
MxN would indeed be bad, but there is no reason to approach it that 
way.  It's highly unlikely that users will want to migrate from 2.x -> 
1.y.  And for a given 1.x release, we can assume that users will want to 
be able to migrate to the current head of branch for 2.y.  So this means 
we would need two upgrade scripts from each 1.x release.  This is extra 
effort but it is not that bad.

Also, the proposal will require two sets of things that Hive provides: 
double documentation, double feature tracking, double build/test 
infrastructures, etc.
Our documentation already handles the fact that certain features are 
only supported in certain releases.  Our test and build infrastructure 
can already be made to work on multiple branches.  I'm not sure what you 
mean by double feature tracking.

This approach can also potentially cause the problem we saw in hadoop 
releases, where 0.23 release was greater than 1.0 release.
I'm sorry, I don't follow what you're saying here.  You mean the numbers 
are just bigger (like 23 > 1)?  We already have that problem, this 
doesn't make it worse.

To me, the problem we are trying to solve is deprecating old things 
such hadoop-1, Hive CLI, etc. This a valid problem to be solved. As I 
see, however, we approached the problem in less favorable ways.
That is only one of the two problems.  The other is to provide a 
mechanism for experimental features.

First, it seemed we wanted to deprecate something just for the sake of 
deprecation, and it's not based on the rationale that supports the 
desire. Dev might write code that accidentally break hadoop-1 build. 
However, this is more a build infrastructure problem rather than the 
burden of supporting hadoop-1. If our build could catch it at 
precommit test, then I would think the accident can be well avoided. 
Most of the times, fixing the build is trivial. And we have already 
addressed the build infrastructure problem.

Secondly, if we do have a strong reason to deprecate something, we 
should have a deprecation plan rather than declaring on the spot that 
the current release is the last one supporting X. I think Microsoft 
did a better job in terms production deprecation. For instance, they 
announced long before the last day desupporting Windows XP. In my 
opinion, we should have a similar vision, giving users, distributions 
enough time to adjust rather than shocking them with breaking news.

In summary, I do see the need of deprecation in Hive, but I am afraid 
the way we take, including the proposal here, isn't going to nicely 
solve the problem. On the contrary, I foresee a spectrum of confusion, 
frustration, and burden for the user as well as for developers.

Thanks,
Xuefu


Xuefu Zhang <mailto:xzh...@cloudera.com>
May 15, 2015 at 17:31
Just make sure that I understand the proposal correctly: we are going to
have two main branches, one for hadoop-1 and one for hadoop-2. New features
are only merged to branch-2. That essentially says we stop development for
hadoop-1, right? Are we also making two lines of releases: ene for branch-1
and one for branch-2? Won't that be confusing and also burdensome if we
release say 1.3, 2.0, 2.1, 1.4...

Please note that we will have hadoop 3 soon. What's the story there?

Thanks,
Xuefu



On Fri, May 15, 2015 at 4:43 PM, Vaibhav Gumashta<vgumas...@hortonworks.com
wrote:

  +1 on the new branch. I think it’ll help in faster dev time for these
important changes.

  —Vaibhav

   From: Alan Gates<alanfga...@gmail.com>
Reply-To: "dev@hive.apache.org"<dev@hive.apache.org>
Date: Friday, May 15, 2015 at 4:11 PM
To: "dev@hive.apache.org"<dev@hive.apache.org>
Subject: Re: [DISCUSS] Supporting Hadoop-1 and experimental features

  Anyone else have feedback on this?  If not I'll start a vote next week.

Alan.

    Gopal Vijayaraghavan<gop...@apache.org>
May 14, 2015 at 10:44
   Hi,

+1 on the idea.

Having a stable release branch with ongoing fixes where we do not drop
major features would be good all around.

It lets us accelerate the pace of development, drop major features or
rewrite them entirely without dragging everyone else kicking&  screaming
into that release.

Cheers,
Gopal



    Sergey Shelukhin<ser...@hortonworks.com>
May 11, 2015 at 19:17
   That sounds like a good idea.
Some features could be back ported to branch-1 if viable, but at least new
stuff would not be burdened by Hadoop 1/MR code paths.
Probably also a good place to enable vectorization and other perf features
by default while we make alpha releases.

+1


    Alan Gates<alanfga...@gmail.com>
May 11, 2015 at 15:38
   There is a lot of forward-looking work going on in various branches of
Hive:  LLAP, the HBase metastore, and the work to drop the CLI.  It would
be good to have a way to release this code to users so that they can
experiment with it.  Releasing it will also provide feedback to developers.

At the same time there are discussions on whether to keep supporting
Hadoop-1.  The burden of supporting older, less used functionality such as
Hadoop-1 is becoming ever harder as many new features are added.

I propose that the best way to deal with this would be to make a
branch-1.  We could continue to make new feature releases off of this
branch (1.3, 1.4, etc.).  This branch would not drop old functionality.
This provides stability and continuity for users and developers.

We could then merge these new features branches (LLAP, HBase metastore,
CLI drop) into the trunk, as well as turn on by default newer features such
as the vectorization and ACID.  We could also drop older, less used
features such as support for Hadoop-1 and MapReduce.  It will be a while
before we are ready to make stable, production ready releases of this
code.  But we could start making alpha quality releases soon.  We would
call these releases 2.x, to stress the non-backward compatible changes such
as dropping Hadoop-1.  This will give users a chance to play with the new
code and developers a chance to get feedback.

Thoughts?



Vaibhav Gumashta <mailto:vgumas...@hortonworks.com>
May 15, 2015 at 16:43
+1 on the new branch. I think it’ll help in faster dev time for these 
important changes.

—Vaibhav

From: Alan Gates <alanfga...@gmail.com <mailto:alanfga...@gmail.com>>
Reply-To: "dev@hive.apache.org <mailto:dev@hive.apache.org>" 
<dev@hive.apache.org <mailto:dev@hive.apache.org>>
Date: Friday, May 15, 2015 at 4:11 PM
To: "dev@hive.apache.org <mailto:dev@hive.apache.org>" 
<dev@hive.apache.org <mailto:dev@hive.apache.org>>
Subject: Re: [DISCUSS] Supporting Hadoop-1 and experimental features

Anyone else have feedback on this?  If not I'll start a vote next week.

Alan.

Alan Gates <mailto:alanfga...@gmail.com>
May 15, 2015 at 16:11
Anyone else have feedback on this?  If not I'll start a vote next week.

Alan.

Gopal Vijayaraghavan <mailto:gop...@apache.org>
May 14, 2015 at 10:44
Hi,

+1 on the idea.

Having a stable release branch with ongoing fixes where we do not drop
major features would be good all around.

It lets us accelerate the pace of development, drop major features or
rewrite them entirely without dragging everyone else kicking & screaming
into that release.

Cheers,
Gopal