On Sep 28, 2009, at 3:15 AM, Steve Loughran wrote:
Dhruba Borthakur wrote:
> It is really nice to have wire-compatibility between clients and
servers
> running different versions of hadoop. The reason we would like
this is
> because we can allow the same client (Hive, etc) submit jobs to two
> different clusters running different versions of hadoop. But I am
not stuck
> up on the name of the release that supports wire-compatibility, it
can be
> either 1.0 or something later than that.
> API compatibility +1
> Data compatibility +1
> Job Q compatibility -1Wire compatibility +0
That's stability of the job submission network protocol you are
looking
for there.
* We need a job submission API that is designed to work over long-
haul
links and versions
* It does not have to be the same as anything used in-cluster
* It does not actually need to run in the JobTracker. An independent
service bridging the stable long-haul API to an unstable datacentre
protocol does work, though authentication and user-rights are a
troublespot
I think you are misinterpreting what Job Q compatibility means.
It is about jobs already in the queue surviving an upgrade across a
release.
See my initial proposal on Jan 16th:
https://issues.apache.org/jira/browse/HADOOP-5071?focusedCommentId=12664691&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
#action_12664691
Doug argued that it is nice to have but not required for 1.0 - can be
added later.
sanjay
Similarly, it would be good for a stable long-haul HDFS protocol, such
as FTP or webdav. Again, no need to build into the namenode .
see http://www.slideshare.net/steve_l/long-haul-hadoop
and commentary under http://wiki.apache.org/hadoop/BristolHadoopWorkshop
