On Tue, May 10, 2016, at 11:34, Hitesh Shah wrote:
> There seems to be some incorrect assumptions on why the application had
> an issue. For rolling upgrade deployments, the application bundles the
> client-side jars that it was compiled against and uses them in its
> classpath and expects to be able to communicate with upgraded servers.
> Given that hadoop-common is a monolithic jar, it ends up being used on
> both client-side and server-side. The problem in this case was caused by
> the fact that the ResourceManager was generating the credentials file
> with a format understood only by hadoop-common from 3.x. For an
> application compiled against 2.x and has *only* hadoop-common from 2.x on
> its classpath, trying to read this file fails.
>
> This is not about whether internal implementations can change for
> non-public APIs. The file format for the Credential file in this scenario
> is *not* internal implementation especially when you can have different
> versions of the library trying to read the file. If an older client is
> talking to a newer versioned server, the general backward compat
> assumption is that the client should receive a response that it can parse
> and understand. In this scenario, the credentials file provided to the
> YARN app by the RM should have been written out with the older version or
> at the very least been readable by the older hadoop-common.jar.
>
> In any case, does anyone have any specific concerns with changing
> LimitedPrivate({"MapReduce"}) to Public?
I think Allen's concern is valid. These APIs were annotated as
LimitedPrivate so that we could change them in the future. If they
become public, we lose that flexibility.
>
> And sure, if we are saying that Hadoop-3.x requires all apps built
> against it to go through a full re-compile as well as downtime as
> existing apps may no longer work out of the box, lets call it out very
> explicitly in the Release notes.
I think it's reasonable to require all applications to be re-compiled
for Hadoop 3.x. One reason is because we make incompatible changes to
Hadoop's dependencies in major releases. This dependency change also
means that "existing apps may no longer work out of the box," as you
stated, since they may need to be updated to use different versions of
our dependencies.
For the same reasons, it is reasonable to require downtime when
upgrading from 2.x to 3.x (or generally, from any major release to any
other major release). For example, I believe the HDFS rolling upgrade
document explicitly calls out the fact that the mechanism is targetted
at going to between minor releases, not major ones.
If we need to document this better, then let's file a JIRA to do that.
best,
Colin
>
> — Hitesh
>
> > On May 10, 2016, at 9:24 AM, Allen Wittenauer <allenwittena...@yahoo.com>
> > wrote:
> >
> >
> >> On May 10, 2016, at 8:37 AM, Hitesh Shah <hit...@apache.org> wrote:
> >>
> >> There have been various discussions on various JIRAs where upstream
> >> projects such as YARN apps ( Tez, Slider, etc ) are called out for using
> >> the above so-called Private APIs. A lot of YARN applications that have
> >> been built out have picked up various bits and pieces of implementation
> >> from MapReduce and DistributedShell to get things to work.
> >>
> >> A recent example is a backward incompatible change introduced ( where the
> >> API is not even directly invoked ) in the Credentials class related to the
> >> ability to read tokens/credentials from a file.
> >
> > Let’s be careful here. It should be noted that the problem happened
> > primarily because the application jar appears to have included some hadoop
> > jars in them. So the API invocation isn’t the problem: it’s the fact
> > that the implementation under the hood changed. If the application jar
> > didn’t bundle hadoop jars —especially given that were already on the
> > classpath--this problem should never have happened.
> >
> >> This functionality is required by pretty much everyone as YARN provides
> >> the credentials to the app by writing the credentials/tokens to a local
> >> file which is read in when UserGroupInformation.getCurrentUser() is
> >> invoked.
> >
> > What you’re effectively arguing is that implementations should never
> > change for public (and in this case LimitedPrivate) APIs. I don’t think
> > that’s reasonable. Hadoop is filled with changes in major branches where
> > the implementations have changed but the internals have been reworked to
> > perform the work in a slightly different manner.
> >
> >> This change breaks rolling upgrades for yarn applications from 2.x to 3.x
> >> (whether we end up supporting rolling upgrades across 2.x to 3.x is a
> >> separate discussion )
> >
> >
> > At least today, according to the document attached to YARN-666 (lol),
> > rolling upgrades are only supported within the same major version.
> >
> >>
> >> I would like to change our compatibility docs to state that any API that
> >> is marked as LimitedPrivate{Mapreduce} implies LimitedPrivate{YARN
> >> Applications}.
> >>
> >> Comments/concerns?
> >
> >
> > a) That isn’t good enough. No one reads the compatibility guidelines
> > as it is given how many times someone says “X” isn’t covered when it
> > clearly is.
> >
> > b) LimitedPrivate{“YARN Applications”} makes zero sense. At that point
> > it’s Public and the source should be changed to reflect that. Especially
> > given those flags impacts things like how the javadocs are generated.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
