Jenkins is running a full test run on trunk using protoc 2.5.0. https://builds.apache.org/job/Hadoop-trunk/480
And it seems go be going just fine. If everything looks OK, I'm planing to backport HADOOP-9845 to the 2.1.0-beta branch midday PST tomorrow. This will normalize all builds failures do the protoc mismatch. Thanks. Alejandro On Mon, Aug 12, 2013 at 5:53 PM, Alejandro Abdelnur <t...@cloudera.com>wrote: > shooting to get it i n for 2.1.0. > > at moment is in trunk till the nightly finishes. then we'll decide > > in the mean time, you can have multiple versions installed in diff dirs > and set the right one in the path > > thx > > Alejandro > (phone typing) > > On Aug 12, 2013, at 17:47, Konstantin Shvachko <shv.had...@gmail.com> > wrote: > > > Ok. After installing protobuf 2.5.0 I can compile trunk. > > But now I cannot compile Hadoop-2 branches. None of them. > > So if I switch between branches I need to reinstall protobuf? > > > > Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL > > versions? > > I did not get definite impression there is. > > If not it could be a pretty big disruption. > > > > Thanks, > > --Konst > > > > > > > > On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <t...@cloudera.com > >wrote: > > > >> I've just committed HADOOP-9845 to trunk (only trunk at the moment). > >> > >> To build trunk now you need protoc 2.5.0 (the build will fail with a > >> warning if you don't have it). > >> > >> We'd propagate this to the 2 branches once the precommit build is back > to > >> normal and see things are OK. > >> > >> Thanks. > >> > >> > >> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <t...@cloudera.com > >>> wrote: > >> > >>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk > use > >>> protoc 2.5.0. > >>> > >>> thx > >>> > >>> > >>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan < > >>> gkesa...@hortonworks.com> wrote: > >>> > >>>> I can take care of re-installing 2.4 and installing 2.5 in a different > >>>> location. This would fix 2.0 branch builds as well. > >>>> Thoughts? > >>>> > >>>> -Giri > >>>> > >>>> > >>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur < > t...@cloudera.com > >>>>> wrote: > >>>> > >>>>> Giri, > >>>>> > >>>>> first of all, thanks for installing protoc 2.5.0. > >>>>> > >>>>> I didn't know we were installing them as the only version and not > >>>> driven by > >>>>> env/path settings. > >>>>> > >>>>> Now we have a bit of a problem, precommit builds are broken because > of > >>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1). > >>>>> > >>>>> We have to options: > >>>>> > >>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out > >> any > >>>>> follow up issues. > >>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and > >>>> 2.5.0 > >>>>> coexisting > >>>>> > >>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues > >> an > >>>>> then merge it to the other branches. > >>>>> > >>>>> We need to sort this out quickly as precommits are not working. > >>>>> > >>>>> I'll wait till 3PM today for objections to option #1, if none I'll > >>>> commit > >>>>> it to trunk. > >>>>> > >>>>> Thanks. > >>>>> > >>>>> Alejandro > >>>>> > >>>>> > >>>>> > >>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan < > >>>>> gkesa...@hortonworks.com> wrote: > >>>>> > >>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the > >> default > >>>>> path. > >>>>>> If we still need 2.4 I may have to install it. Let me know > >>>>>> > >>>>>> -Giri > >>>>>> > >>>>>> > >>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur < > >>>> t...@cloudera.com > >>>>>>> wrote: > >>>>>> > >>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both > >> so > >>>> we > >>>>>> can > >>>>>>> use and env to set it in the jobs? > >>>>>>> > >>>>>>> thx > >>>>>>> > >>>>>>> Alejandro > >>>>>>> (phone typing) > >>>>>>> > >>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan < > >>>> gkesa...@hortonworks.com > >>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0 > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -Giri > >>>>>>>> > >>>>>>>> > >>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan < > >>>>>>>> gkesa...@hortonworks.com> wrote: > >>>>>>>> > >>>>>>>>> Alejandro, > >>>>>>>>> > >>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9. > >>>>>>>>> > >>>>>>>>> -Giri > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur < > >>>>> t...@cloudera.com > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> pinging again, I need help from somebody with sudo access to > >> the > >>>>>> hadoop > >>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of > >>>>> hours > >>>>>> to > >>>>>>>>>> set > >>>>>>>>>> up myself. > >>>>>>>>>> > >>>>>>>>>> Please!!! > >>>>>>>>>> > >>>>>>>>>> thx > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur < > >>>>>> t...@cloudera.com > >>>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache > >>>>> hadoop > >>>>>>>>>>> jenkins boxes. > >>>>>>>>>>> > >>>>>>>>>>> Who can help with this? I assume somebody at Y!, right? > >>>>>>>>>>> > >>>>>>>>>>> Thx > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark < > >>>> ecl...@apache.org> > >>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need > >> to > >>>>> have > >>>>>>> the > >>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are > >>>>> running. > >>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and > >>>> 2.5.x > >>>>> on > >>>>>>>>>>>> the same class path. > >>>>>>>>>>>> > >>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new > >>>> classes > >>>>> we > >>>>>>>>>>>> could use. With that said HBase is getting pretty close to > >> a > >>>>>> rather > >>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this > >> in > >>>>>> sooner > >>>>>>>>>>>> rather than later would be great. If we could get this into > >>>>> 2.1.0 > >>>>>> it > >>>>>>>>>>>> would be great as that would allow us to have a pretty easy > >>>> story > >>>>>> to > >>>>>>>>>>>> users with regards to protobuf version. > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee < > >>>> kih...@yahoo-inc.com > >>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention > >>>> Avro. > >>>>> See > >>>>>>>>>>>> HADOOP-9672. > >>>>>>>>>>>>> The version we are using has memory leak and inefficiency > >>>>> issues. > >>>>>>>>>> We've > >>>>>>>>>>>> seen users running into it. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Kihwal > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> ________________________________ > >>>>>>>>>>>>> From: Tsuyoshi OZAWA <ozawa.tsuyo...@gmail.com> > >>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" < > >>>>> common-dev@hadoop.apache.org> > >>>>>>>>>>>>> Cc: "hdfs-...@hadoop.apache.org" < > >> hdfs-...@hadoop.apache.org > >>>>> ; > >>>>> " > >>>>>>>>>>>> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; " > >>>>>>>>>>>> mapreduce-...@hadoop.apache.org" < > >>>>> mapreduce-...@hadoop.apache.org> > >>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM > >>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 > >> release, > >>>>>>>>>>>> HADOOP-9845 > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Hi, > >>>>>>>>>>>>> > >>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in > >>>> HADOOP-9346. > >>>>>>>>>>>>> For more detail, please see the JIRA ticket: > >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346 > >>>>>>>>>>>>> > >>>>>>>>>>>>> - Tsuyoshi > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur < > >>>>>>>>>> t...@cloudera.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 > >> release. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant > >>>>>> benefits > >>>>>>>>>> to > >>>>>>>>>>>>>> justify the upgrade. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make > >> things > >>>>>> easier > >>>>>>>>>> for > >>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and > >> adopting > >>>>>> Hadoop > >>>>>>>>>> 2. > >>>>>>>>>>>> If > >>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to > >>>>>> support 2 > >>>>>>>>>>>>>> different versions and they my get in nasty waters due to > >>>>>> classpath > >>>>>>>>>>>> issues. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of > >>>>>> 2.1.0-beta > >>>>>>>>>>>> branch > >>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC > >>>>> layer > >>>>>>>>>>>> YARN-885). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Now, to do this it will require a few things: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box > >>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers > >>>>> should > >>>>>>>>>>>> install > >>>>>>>>>>>>>> locally protobuf 2.5.0 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> -- > >>>>>>>>>>>>>> Alejandro > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Alejandro > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Alejandro > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Alejandro > >>> > >>> > >>> > >>> -- > >>> Alejandro > >> > >> > >> > >> -- > >> Alejandro > >> > -- Alejandro
