There is no indication that protoc 2.5.0 is breaking anything. Hadoop-trunk builds have been failing way before 1/2 way with:
------- [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-yarn-client: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ? -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.12.3:test (default-test) on project hadoop-yarn-client: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ? ------- The Hadoop-trunk #480 build failed with a JVM abort in a testcase towards the end of mapreduce tests. Until then there were no failures at all. I've increased heap size and tried a second run and the failure was earlier. I've looked a Hadoop-trunk builds prior to the HADOOP-9845 and it has been failing the same way in all the kept builds. We need to fix Hadoop-trunk builds independently of this. Any objection to commit HADOOP-9845 to branch-2 and the 2.1.0-beta branches to get all the other jenkins jobs working? I'll wait till tomorrow morning before proceeding. Thx On Mon, Aug 12, 2013 at 8:35 PM, Alejandro Abdelnur <t...@cloudera.com>wrote: > Jenkins is running a full test run on trunk using protoc 2.5.0. > > https://builds.apache.org/job/Hadoop-trunk/480 > > And it seems go be going just fine. > > If everything looks OK, I'm planing to backport HADOOP-9845 to the > 2.1.0-beta branch midday PST tomorrow. This will normalize all builds > failures do the protoc mismatch. > > Thanks. > > Alejandro > > > On Mon, Aug 12, 2013 at 5:53 PM, Alejandro Abdelnur <t...@cloudera.com>wrote: > >> shooting to get it i n for 2.1.0. >> >> at moment is in trunk till the nightly finishes. then we'll decide >> >> in the mean time, you can have multiple versions installed in diff dirs >> and set the right one in the path >> >> thx >> >> Alejandro >> (phone typing) >> >> On Aug 12, 2013, at 17:47, Konstantin Shvachko <shv.had...@gmail.com> >> wrote: >> >> > Ok. After installing protobuf 2.5.0 I can compile trunk. >> > But now I cannot compile Hadoop-2 branches. None of them. >> > So if I switch between branches I need to reinstall protobuf? >> > >> > Is there a consensus about going towards protobuf 2.5.0 upgrade in ALL >> > versions? >> > I did not get definite impression there is. >> > If not it could be a pretty big disruption. >> > >> > Thanks, >> > --Konst >> > >> > >> > >> > On Mon, Aug 12, 2013 at 3:19 PM, Alejandro Abdelnur <t...@cloudera.com >> >wrote: >> > >> >> I've just committed HADOOP-9845 to trunk (only trunk at the moment). >> >> >> >> To build trunk now you need protoc 2.5.0 (the build will fail with a >> >> warning if you don't have it). >> >> >> >> We'd propagate this to the 2 branches once the precommit build is back >> to >> >> normal and see things are OK. >> >> >> >> Thanks. >> >> >> >> >> >> On Mon, Aug 12, 2013 at 2:57 PM, Alejandro Abdelnur <t...@cloudera.com >> >>> wrote: >> >> >> >>> About to commit HADOOP-9845 to trunk, in 5 mins. This will make trunk >> use >> >>> protoc 2.5.0. >> >>> >> >>> thx >> >>> >> >>> >> >>> On Mon, Aug 12, 2013 at 11:47 AM, Giridharan Kesavan < >> >>> gkesa...@hortonworks.com> wrote: >> >>> >> >>>> I can take care of re-installing 2.4 and installing 2.5 in a >> different >> >>>> location. This would fix 2.0 branch builds as well. >> >>>> Thoughts? >> >>>> >> >>>> -Giri >> >>>> >> >>>> >> >>>> On Mon, Aug 12, 2013 at 11:37 AM, Alejandro Abdelnur < >> t...@cloudera.com >> >>>>> wrote: >> >>>> >> >>>>> Giri, >> >>>>> >> >>>>> first of all, thanks for installing protoc 2.5.0. >> >>>>> >> >>>>> I didn't know we were installing them as the only version and not >> >>>> driven by >> >>>>> env/path settings. >> >>>>> >> >>>>> Now we have a bit of a problem, precommit builds are broken because >> of >> >>>>> mismatch of protoc (2.5.0) and protobuf JAR( 2.4.1). >> >>>>> >> >>>>> We have to options: >> >>>>> >> >>>>> 1* commit HADOOP-9845 that will bring protobuf to 2.5.0 and iron out >> >> any >> >>>>> follow up issues. >> >>>>> 2* reinstall protoc 2.4.1 in the jenkins machines and have 2.4.1 and >> >>>> 2.5.0 >> >>>>> coexisting >> >>>>> >> >>>>> My take would be to commit HADOOP-9845 in trunk, iron out any issues >> >> an >> >>>>> then merge it to the other branches. >> >>>>> >> >>>>> We need to sort this out quickly as precommits are not working. >> >>>>> >> >>>>> I'll wait till 3PM today for objections to option #1, if none I'll >> >>>> commit >> >>>>> it to trunk. >> >>>>> >> >>>>> Thanks. >> >>>>> >> >>>>> Alejandro >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Mon, Aug 12, 2013 at 11:30 AM, Giridharan Kesavan < >> >>>>> gkesa...@hortonworks.com> wrote: >> >>>>> >> >>>>>> Like I said protoc is upgraded from 2.4 to 2.5. 2.5 is in the >> >> default >> >>>>> path. >> >>>>>> If we still need 2.4 I may have to install it. Let me know >> >>>>>> >> >>>>>> -Giri >> >>>>>> >> >>>>>> >> >>>>>> On Sat, Aug 10, 2013 at 7:01 AM, Alejandro Abdelnur < >> >>>> t...@cloudera.com >> >>>>>>> wrote: >> >>>>>> >> >>>>>>> thanks giri, how do we set 2.4 or 2.5., what is the path to both >> >> so >> >>>> we >> >>>>>> can >> >>>>>>> use and env to set it in the jobs? >> >>>>>>> >> >>>>>>> thx >> >>>>>>> >> >>>>>>> Alejandro >> >>>>>>> (phone typing) >> >>>>>>> >> >>>>>>> On Aug 9, 2013, at 23:10, Giridharan Kesavan < >> >>>> gkesa...@hortonworks.com >> >>>>>> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>>> build slaves hadoop1-hadoop9 now has libprotoc 2.5.0 >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> -Giri >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Fri, Aug 9, 2013 at 10:56 PM, Giridharan Kesavan < >> >>>>>>>> gkesa...@hortonworks.com> wrote: >> >>>>>>>> >> >>>>>>>>> Alejandro, >> >>>>>>>>> >> >>>>>>>>> I'm upgrading protobuf on slaves hadoop1-hadoop9. >> >>>>>>>>> >> >>>>>>>>> -Giri >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> On Fri, Aug 9, 2013 at 1:15 PM, Alejandro Abdelnur < >> >>>>> t...@cloudera.com >> >>>>>>>> wrote: >> >>>>>>>>> >> >>>>>>>>>> pinging again, I need help from somebody with sudo access to >> >> the >> >>>>>> hadoop >> >>>>>>>>>> jenkins boxes to do this or to get sudo access for a couple of >> >>>>> hours >> >>>>>> to >> >>>>>>>>>> set >> >>>>>>>>>> up myself. >> >>>>>>>>>> >> >>>>>>>>>> Please!!! >> >>>>>>>>>> >> >>>>>>>>>> thx >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Thu, Aug 8, 2013 at 2:29 PM, Alejandro Abdelnur < >> >>>>>> t...@cloudera.com >> >>>>>>>>>>> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> To move forward with this we need protoc 2.5.0 in the apache >> >>>>> hadoop >> >>>>>>>>>>> jenkins boxes. >> >>>>>>>>>>> >> >>>>>>>>>>> Who can help with this? I assume somebody at Y!, right? >> >>>>>>>>>>> >> >>>>>>>>>>> Thx >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> On Thu, Aug 8, 2013 at 2:24 PM, Elliott Clark < >> >>>> ecl...@apache.org> >> >>>>>>>>>> wrote: >> >>>>>>>>>>> >> >>>>>>>>>>>> In HBase land we've pretty well discovered that we'll need >> >> to >> >>>>> have >> >>>>>>> the >> >>>>>>>>>>>> same version of protobuf that the HDFS/Yarn/MR servers are >> >>>>> running. >> >>>>>>>>>>>> That is to say there are issues with ever having 2.4.x and >> >>>> 2.5.x >> >>>>> on >> >>>>>>>>>>>> the same class path. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Upgrading to 2.5.x would be great, as it brings some new >> >>>> classes >> >>>>> we >> >>>>>>>>>>>> could use. With that said HBase is getting pretty close to >> >> a >> >>>>>> rather >> >>>>>>>>>>>> large release (0.96.0 aka The Singularity) so getting this >> >> in >> >>>>>> sooner >> >>>>>>>>>>>> rather than later would be great. If we could get this into >> >>>>> 2.1.0 >> >>>>>> it >> >>>>>>>>>>>> would be great as that would allow us to have a pretty easy >> >>>> story >> >>>>>> to >> >>>>>>>>>>>> users with regards to protobuf version. >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Thu, Aug 8, 2013 at 8:18 AM, Kihwal Lee < >> >>>> kih...@yahoo-inc.com >> >>>>>> >> >>>>>>>>>> wrote: >> >>>>>>>>>>>>> Sorry to hijack the thread but, I also wanted to mention >> >>>> Avro. >> >>>>> See >> >>>>>>>>>>>> HADOOP-9672. >> >>>>>>>>>>>>> The version we are using has memory leak and inefficiency >> >>>>> issues. >> >>>>>>>>>> We've >> >>>>>>>>>>>> seen users running into it. >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Kihwal >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> ________________________________ >> >>>>>>>>>>>>> From: Tsuyoshi OZAWA <ozawa.tsuyo...@gmail.com> >> >>>>>>>>>>>>> To: "common-...@hadoop.apache.org" < >> >>>>> common-...@hadoop.apache.org> >> >>>>>>>>>>>>> Cc: "hdfs-dev@hadoop.apache.org" < >> >> hdfs-dev@hadoop.apache.org >> >>>>> ; >> >>>>> " >> >>>>>>>>>>>> yarn-...@hadoop.apache.org" <yarn-...@hadoop.apache.org>; " >> >>>>>>>>>>>> mapreduce-...@hadoop.apache.org" < >> >>>>> mapreduce-...@hadoop.apache.org> >> >>>>>>>>>>>>> Sent: Thursday, August 8, 2013 1:59 AM >> >>>>>>>>>>>>> Subject: Re: Upgrade to protobuf 2.5.0 for the 2.1.0 >> >> release, >> >>>>>>>>>>>> HADOOP-9845 >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> Hi, >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> About Hadoop, Harsh is dealing with this problem in >> >>>> HADOOP-9346. >> >>>>>>>>>>>>> For more detail, please see the JIRA ticket: >> >>>>>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-9346 >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> - Tsuyoshi >> >>>>>>>>>>>>> >> >>>>>>>>>>>>> On Thu, Aug 8, 2013 at 1:49 AM, Alejandro Abdelnur < >> >>>>>>>>>> t...@cloudera.com> >> >>>>>>>>>>>> wrote: >> >>>>>>>>>>>>>> I' like to upgrade to protobuf 2.5.0 for the 2.1.0 >> >> release. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> As mentioned in HADOOP-9845, Protobuf 2.5 has significant >> >>>>>> benefits >> >>>>>>>>>> to >> >>>>>>>>>>>>>> justify the upgrade. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Doing the upgrade now, with the first beta, will make >> >> things >> >>>>>> easier >> >>>>>>>>>> for >> >>>>>>>>>>>>>> downstream projects (like HBase) using protobuf and >> >> adopting >> >>>>>> Hadoop >> >>>>>>>>>> 2. >> >>>>>>>>>>>> If >> >>>>>>>>>>>>>> we do the upgrade later, downstream projects will have to >> >>>>>> support 2 >> >>>>>>>>>>>>>> different versions and they my get in nasty waters due to >> >>>>>> classpath >> >>>>>>>>>>>> issues. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> I've locally tested the patch in a pseudo deployment of >> >>>>>> 2.1.0-beta >> >>>>>>>>>>>> branch >> >>>>>>>>>>>>>> and it works fine (something is broken in trunk in the RPC >> >>>>> layer >> >>>>>>>>>>>> YARN-885). >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Now, to do this it will require a few things: >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> * Make sure protobuf 2.5.0 is available in the jenkins box >> >>>>>>>>>>>>>> * A follow up email to dev@ aliases indicating developers >> >>>>> should >> >>>>>>>>>>>> install >> >>>>>>>>>>>>>> locally protobuf 2.5.0 >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> Thanks. >> >>>>>>>>>>>>>> >> >>>>>>>>>>>>>> -- >> >>>>>>>>>>>>>> Alejandro >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> -- >> >>>>>>>>>>> Alejandro >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> -- >> >>>>>>>>>> Alejandro >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Alejandro >> >>> >> >>> >> >>> >> >>> -- >> >>> Alejandro >> >> >> >> >> >> >> >> -- >> >> Alejandro >> >> >> > > > > -- > Alejandro > -- Alejandro
