I've filed https://issues.apache.org/jira/browse/HADOOP-9872 addressing the following:
--------- - handles protoc version correctly independently of the exit code - if HADOOP_PROTOC_PATH env var is defined, it uses it as the protoc executable * if HADOOP_PROTOC_PATH is not defined, it picks protoc from the PATH - documentation updated to reflect 2.5.0 is required - enforces the version of protoc and protobuf JAR are the same - Added to VersionInfo the protoc version used (sooner or later this will be useful for in a troubleshooting situation). Luke Lu <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=vicaya> suggested to make the version check for protoc lax (i.e. 2.5.*). While working on the patch I've thought about that. But that would introduce a potential mismatch between protoc and protobuff JAR. Still If you want to use different version of protoc/protobuff from the one defined in the POM, you can use the -Dprotobuf.version=#### to specify your alternate version. But I would recommend not to do this, because if you publish the artifacts to a Maven repo, the fact you used -Dprotobuf.version=#### will be lost and the version defined in the POM properties will be used (IMO Maven should use the effective POM on deploy, but they don't). --------- It would be great if a few people test the patch locally. Once this is committed to trunk I'll bacport HADOOP-9845 & HADOOP-9872 to all the 2 branches. Thx. On Tue, Aug 13, 2013 at 1:09 PM, Alejandro Abdelnur <t...@cloudera.com>wrote: > > There is no indication that protoc 2.5.0 is breaking anything. > > Hadoop-trunk builds have been failing way before 1/2 way with: > > ------- > > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on > project hadoop-yarn-client: ExecutionException; nested exception is > java.util.concurrent.ExecutionException: java.lang.RuntimeException: The > forked VM terminated without saying properly goodbye. VM crash or System.exit > called ? -> [Help 1] > org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute > goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test > (default-test) on project hadoop-yarn-client: ExecutionException; nested > exception is java.util.concurrent.ExecutionException: > java.lang.RuntimeException: The forked VM terminated without saying properly > goodbye. VM crash or System.exit called ? > > ------- > > > The Hadoop-trunk #480 build failed with a JVM abort in a testcase towards > the end of mapreduce tests. > > Until then there were no failures at all. > > I've increased heap size and tried a second run and the failure was > earlier. > > I've looked a Hadoop-trunk builds prior to the HADOOP-9845 and it has been > failing the same way in all the kept builds. > > We need to fix Hadoop-trunk builds independently of this. > > Any objection to commit HADOOP-9845 to branch-2 and the 2.1.0-beta > branches to get all the other jenkins jobs working? > > I'll wait till tomorrow morning before proceeding. > > Thx > > > > > On Mon, Aug 12, 2013 at 8:35 PM, Alejandro Abdelnur <t...@cloudera.com>wrote: > >> Jenkins is running a full test run on trunk using protoc 2.5.0. >> >> https://builds.apache.org/job/Hadoop-trunk/480 >> >> And it seems go be going just fine. >> >> If everything looks OK, I'm planing to backport HADOOP-9845 to the >> 2.1.0-beta branch midday PST tomorrow. This will normalize all builds >> failures do the protoc mismatch. >> >> Thanks. >> >> Alejandro >> >> >> On Mon, Aug 12, 2013 at 5:53 PM, Alejandro Abdelnur <t...@cloudera.com>wrote: >> >>> shooting to get it i n for 2.1.0. >>> >>> at moment is in trunk till the nightly finishes. then we'll decide >>> >>> in the mean time, you can have multiple versions installed in diff dirs >>> and set the right one in the path >>> >>> thx >>> >>> Alejandro >>> (phone typing) >>> >>> On Aug 12, 2013, at 17:47, Konstantin Shvachko <shv.had...@gmail.com> >>> wrote: >>> >>> > Ok. After installing protobuf 2.5.0 I can compile trunk. >>> > But now I cannot compile Hadoop-2 branches. None of them. >>> > So if I switch between branches I need to reinstall protobuf? >>> > >>> > Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL >>> > versions? >>> > I did not get definite impression there is. >>> > If not it could be a pretty big disruption. >>> > >>> > Thanks, >>> > --Konst >>> > >>> > >>> > >>> > On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <t...@cloudera.com >>> >wrote: >>> > >>> >> I've just committed HADOOP-9845 to trunk (only trunk at the moment). >>> >> >>> >> To build trunk now you need protoc 2.5.0 (the build will fail with a >>> >> warning if you don't have it). >>> >> >>> >> We'd propagate this to the 2 branches once the precommit build is >>> back to >>> >> normal and see things are OK. >>> >> >>> >> Thanks. >>> >> >>> >> >>> >> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur < >>> t...@cloudera.com >>> >>> wrote: >>> >> >>> >>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make >>> trunk use >>> >>> protoc 2.5.0. >>> >>> >>> >>> thx >>> >>> >>> >>> >>> >>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan < >>> >>> gkesa...@hortonworks.com> wrote: >>> >>> >>> >>>> I can take care of re-installing 2.4 and installing 2.5 in a >>> different >>> >>>> location. This would fix 2.0 branch builds as well. >>> >>>> Thoughts? >>> >>>> >>> >>>> -Giri >>> >>>> >>> >>>> >>> >>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur < >>> t...@cloudera.com >>> >>>>> wrote: >>> >>>> >>> >>>>> Giri, >>> >>>>> >>> >>>>> first of all, thanks for installing protoc 2.5.0. >>> >>>>> >>> >>>>> I didn't know we were installing them as the only version and not >>> >>>> driven by >>> >>>>> env/path settings. >>> >>>>> >>> >>>>> Now we have a bit of a problem, precommit builds are broken >>> because of >>> >>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1). >>> >>>>> >>> >>>>> We have to options: >>> >>>>> >>> >>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron >>> out >>> >> any >>> >>>>> follow up issues. >>> >>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 >>> and >>> >>>> 2.5.0 >>> >>>>> coexisting >>> >>>>> >>> >>>>> My take would be to commit HADOOP-9845 in trunk, iron out any >>> issues >>> >> an >>> >>>>> then merge it to the other branches. >>> >>>>> >>> >>>>> We need to sort this out quickly as precommits are not working. >>> >>>>> >>> >>>>> I'll wait till 3PM today for objections to option #1, if none I'll >>> >>>> commit >>> >>>>> it to trunk. >>> >>>>> >>> >>>>> Thanks. >>> >>>>> >>> >>>>> Alejandro >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan < >>> >>>>> gkesa...@hortonworks.com> wrote: >>> >>>>> >>> >>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the >>> >> default >>> >>>>> path. >>> >>>>>> If we still need 2.4 I may have to install it. Let me know >>> >>>>>> >>> >>>>>> -Giri >>> >>>>>> >>> >>>>>> >>> >>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur < >>> >>>> t...@cloudera.com >>> >>>>>>> wrote: >>> >>>>>> >>> >>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both >>> >> so >>> >>>> we >>> >>>>>> can >>> >>>>>>> use and env to set it in the jobs? >>> >>>>>>> >>> >>>>>>> thx >>> >>>>>>> >>> >>>>>>> Alejandro >>> >>>>>>> (phone typing) >>> >>>>>>> >>> >>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan < >>> >>>> gkesa...@hortonworks.com >>> >>>>>> >>> >>>>>>> wrote: >>> >>>>>>> >>> >>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0 >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> -Giri >>> >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan < >>> >>>>>>>> gkesa...@hortonworks.com> wrote: >>> >>>>>>>> >>> >>>>>>>>> Alejandro, >>> >>>>>>>>> >>> >>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9. >>> >>>>>>>>> >>> >>>>>>>>> -Giri >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur < >>> >>>>> t...@cloudera.com >>> >>>>>>>> wrote: >>> >>>>>>>>> >>> >>>>>>>>>> pinging again, I need help from somebody with sudo access to >>> >> the >>> >>>>>> hadoop >>> >>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of >>> >>>>> hours >>> >>>>>> to >>> >>>>>>>>>> set >>> >>>>>>>>>> up myself. >>> >>>>>>>>>> >>> >>>>>>>>>> Please!!! >>> >>>>>>>>>> >>> >>>>>>>>>> thx >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur < >>> >>>>>> t...@cloudera.com >>> >>>>>>>>>>> wrote: >>> >>>>>>>>>> >>> >>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache >>> >>>>> hadoop >>> >>>>>>>>>>> jenkins boxes. >>> >>>>>>>>>>> >>> >>>>>>>>>>> Who can help with this? I assume somebody at Y!, right? >>> >>>>>>>>>>> >>> >>>>>>>>>>> Thx >>> >>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark < >>> >>>> ecl...@apache.org> >>> >>>>>>>>>> wrote: >>> >>>>>>>>>>> >>> >>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need >>> >> to >>> >>>>> have >>> >>>>>>> the >>> >>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are >>> >>>>> running. >>> >>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and >>> >>>> 2.5.x >>> >>>>> on >>> >>>>>>>>>>>> the same class path. >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new >>> >>>> classes >>> >>>>> we >>> >>>>>>>>>>>> could use. With that said HBase is getting pretty close to >>> >> a >>> >>>>>> rather >>> >>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this >>> >> in >>> >>>>>> sooner >>> >>>>>>>>>>>> rather than later would be great. If we could get this into >>> >>>>> 2.1.0 >>> >>>>>> it >>> >>>>>>>>>>>> would be great as that would allow us to have a pretty easy >>> >>>> story >>> >>>>>> to >>> >>>>>>>>>>>> users with regards to protobuf version. >>> >>>>>>>>>>>> >>> >>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee < >>> >>>> kih...@yahoo-inc.com >>> >>>>>> >>> >>>>>>>>>> wrote: >>> >>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention >>> >>>> Avro. >>> >>>>> See >>> >>>>>>>>>>>> HADOOP-9672. >>> >>>>>>>>>>>>> The version we are using has memory leak and inefficiency >>> >>>>> issues. >>> >>>>>>>>>> We've >>> >>>>>>>>>>>> seen users running into it. >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> Kihwal >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> ________________________________ >>> >>>>>>>>>>>>> From: Tsuyoshi OZAWA <ozawa.tsuyo...@gmail.com> >>> >>>>>>>>>>>>> To: "common-dev@hadoop.apache.org" < >>> >>>>> common-dev@hadoop.apache.org> >>> >>>>>>>>>>>>> Cc: "hdfs-...@hadoop.apache.org" < >>> >> hdfs-...@hadoop.apache.org >>> >>>>> ; >>> >>>>> " >>> >>>>>>>>>>>> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; " >>> >>>>>>>>>>>> mapreduce-...@hadoop.apache.org" < >>> >>>>> mapreduce-...@hadoop.apache.org> >>> >>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM >>> >>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 >>> >> release, >>> >>>>>>>>>>>> HADOOP-9845 >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> Hi, >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in >>> >>>> HADOOP-9346. >>> >>>>>>>>>>>>> For more detail, please see the JIRA ticket: >>> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346 >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> - Tsuyoshi >>> >>>>>>>>>>>>> >>> >>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur < >>> >>>>>>>>>> t...@cloudera.com> >>> >>>>>>>>>>>> wrote: >>> >>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 >>> >> release. >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant >>> >>>>>> benefits >>> >>>>>>>>>> to >>> >>>>>>>>>>>>>> justify the upgrade. >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make >>> >> things >>> >>>>>> easier >>> >>>>>>>>>> for >>> >>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and >>> >> adopting >>> >>>>>> Hadoop >>> >>>>>>>>>> 2. >>> >>>>>>>>>>>> If >>> >>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to >>> >>>>>> support 2 >>> >>>>>>>>>>>>>> different versions and they my get in nasty waters due to >>> >>>>>> classpath >>> >>>>>>>>>>>> issues. >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of >>> >>>>>> 2.1.0-beta >>> >>>>>>>>>>>> branch >>> >>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC >>> >>>>> layer >>> >>>>>>>>>>>> YARN-885). >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Now, to do this it will require a few things: >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box >>> >>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers >>> >>>>> should >>> >>>>>>>>>>>> install >>> >>>>>>>>>>>>>> locally protobuf 2.5.0 >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> Thanks. >>> >>>>>>>>>>>>>> >>> >>>>>>>>>>>>>> -- >>> >>>>>>>>>>>>>> Alejandro >>> >>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>>> >>> >>>>>>>>>>> -- >>> >>>>>>>>>>> Alejandro >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> >>> >>>>>>>>>> -- >>> >>>>>>>>>> Alejandro >>> >>>>> >>> >>>>> >>> >>>>> >>> >>>>> -- >>> >>>>> Alejandro >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> Alejandro >>> >> >>> >> >>> >> >>> >> -- >>> >> Alejandro >>> >> >>> >> >> >> >> -- >> Alejandro >> > > > > -- > Alejandro > -- Alejandro
