Hi Zoltan, As you said, migrating this job to the new ci-hadoop instance looks good as Hadoop also shares the same armN slaves. I am able to login the new ci-hadoop instance with Apache LDAP credentials, but i am not able to see the job creation option. Should I request access or the process for creation of a job is different than jenkin?. Please guide me to create the new job in the ci-hadoop instance. I will migrate this job after connecting the armN slaves to the new system.
Thanks Chinna On Wed, Jun 17, 2020 at 11:57 AM Zhenyu Zheng <zhengzhenyul...@gmail.com> wrote: > Hi Zoltan, > > Thanks alot for the information, so looks like one possible solution is as > you suggest, move the current ARM2 and ARM3 (those two were donate to > builds.apache.org by us) to the new ci-hadoop cluster and set up the jobs > just as what has been done in current jenkins. > > I will also ask our team member works on other projects to find out what > the status of other projects is. > > BR, > > On Tue, Jun 16, 2020 at 6:41 PM Zoltan Haindrich <k...@rxd.hu> wrote: > >> Hey, >> >> There is an effort by the Apache Infra to change the way Jenkins stuff is >> organized; a couple months ago Gavin wrote an email about it: >> >> http://mail-archives.apache.org/mod_mbox/tez-dev/202004.mbox/%3ccan0gg1dodepzatjz9bofe-2ver7qg7h0hmvyjmsldgjr8_r...@mail.gmail.com%3E >> The resources for running these jobs are coming from the H0~H21 slaves >> which will be migrated to the new jenkins master eventually. >> >> >> So please >> >> suggest a way which direction we can move and can you share some >> details >> >> about the new ci-hadoop instance. >> >> Since Hadoop testing is also happening on ARM - I think the best would be >> to also migrate the armN slaves and the Hive arm nightly over to the new >> ci-hadoop instance. >> >> On 6/16/20 8:40 AM, Zhenyu Zheng wrote: >> > Thanks for the info, I wonder if where does the resource of ci-hadoop >> and >> > hive-test-kube come from? Do they include ARM resources? >> >> Interesting question; the resources for Hive testing are donated by >> Cloudera. >> About the ARM workers I think Chinna could provide more details. >> ...I've no idea don't know who sponsors the Hxx slaves >> >> > Can you provide some more information about how the new hive-test-kube >> is >> > running? >> It's basically a Jenkins instance which is using kubernetes pods to run >> things. >> The whole thing is running on a GKE cluster. >> While I was working on it I collected stuff needed for it in this repo: >> https://github.com/kgyrtkirk/hive-test-kube/ >> it should be possible to start a new deployment using that stuff >> >> cheers, >> Zoltan >> >> > >> > BR, >> > Kevin Zheng >> > >> > On Tue, Jun 16, 2020 at 12:41 PM Chinna Rao Lalam < >> > lalamchinnara...@gmail.com> wrote: >> > >> >> Hi Zoltan, >> >> >> >> Thanks for the update. >> >> >> >> Current https://builds.apache.org/job/Hive-linux-ARM-trunk/ job is >> >> targeting to run hive tests daily on "arm" slaves, it is using 2 arm >> >> slaves. >> >> To find any potential issues with "arm" and fix the issues. So please >> >> suggest a way which direction we can move and can you share some >> details >> >> about the new ci-hadoop instance. >> >> >> >> Thanks, >> >> Chinna >> >> >> >> On Mon, Jun 15, 2020 at 3:56 PM Zoltan Haindrich <k...@rxd.hu> wrote: >> >> >> >>> Hey all, >> >>> >> >>> In an ticket (INFRA-20416) Gavin asked me if we are completely off >> >>> builds.apache.org - when I went over the jobs I've saw that >> >>> https://builds.apache.org/job/Hive-linux-ARM-trunk/ is running there >> >>> once a day. >> >>> >> >>> Since builds.apache.org will be shut down in sometime in the future >> - we >> >>> should move this job to the new ci-hadoop instance or to >> hive-test-kube. >> >>> The key feature of the job is that it runs the test on the "armX" >> slaves; >> >>> which are statically configured on b.a.o. >> >>> Not sure which way to go - but we will have to move in some direction. >> >>> >> >>> cheers, >> >>> Zoltan >> >>> >> >>> >> >>> On 3/13/20 7:22 AM, Zhenyu Zheng wrote: >> >>>> Hi Chinna, >> >>>> >> >>>> Thanks alot for the reply, I uploaded a patch and also a github PR >> for >> >>>> https://issues.apache.org/jira/browse/HIVE-21939 . >> >>>> In the patch, I bumped the protobuf used in standalone-metadata to >> 2.6.1 >> >>>> and added a new profile, this profile will identify >> >>>> the hardware architecture and if it is Aarch64, it will override the >> >>>> protobuf group.id and package to com.github.os72 which >> >>>> includes ARM support. For X86 platform, Hive will still download the >> >>>> protobuf packages from org.google repo. I think with >> >>>> this method, we can keep the influence to existing x86 users to the >> >>>> minimum. I hope this could be a acceptable short-term >> >>>> solution. >> >>>> >> >>>> I've manually tested on my machine and the github PR travis CI test >> has >> >>>> already passed, so the build process is OK, so let's >> >>>> wait for the full test result from builds.apache.org. >> >>>> >> >>>> BR, >> >>>> >> >>>> Zhenyu >> >>>> >> >>>> On Thu, Mar 12, 2020 at 9:23 PM Chinna Rao Lalam < >> >>> lalamchinnara...@gmail.com> >> >>>> wrote: >> >>>> >> >>>>> Hi Zhenyu, >> >>>>> >> >>>>> Until HBase dependency resolved, without effecting the existing code >> >>> on X86 >> >>>>> i suggest create a separate profile with "os72" repo. >> >>>>> >> >>>>> Down the line we should have common version for both X86 and ARM. >> >>>>> >> >>>>> Hope It Helps, >> >>>>> Chinna >> >>>>> >> >>>>> On Wed, Mar 11, 2020 at 8:39 AM Zhenyu Zheng < >> >>> zhengzhenyul...@gmail.com> >> >>>>> wrote: >> >>>>> >> >>>>>> Hi Chinna, David and others might interested, >> >>>>>> >> >>>>>> Thanks for bring this up, we are currently working on improving >> >>> enabling >> >>>>>> big-data software on the ARM platform, >> >>>>>> we have already done fixes and providing CIs to some of the >> well-know >> >>>>>> projects like: >> >>>>>> 1. Hadoop: >> >>>>>> >> >>>>>> >> >>>>> >> >>> >> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ >> >>>>>> 2. Spark: https://amplab.cs.berkeley.edu/jenkins/label/spark-arm/ >> >>>>>> 3. HBase: >> >>>>>> >> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Nightly-ARM/ >> >>>>>> >> >>>>>> And we are now working on projects including Hive, Kudu, etc. >> >>>>>> >> >>>>>> Regarding to the protobuf upgrades in Hive, except upgrading to 3.x >> >>> and >> >>>>>> break dependency for HBase, there can >> >>>>>> be some possible short-term plan(or walk-arounds), doing thes can >> make >> >>>>> Hive >> >>>>>> work on ARM without break any >> >>>>>> dependencies, and then we can interact with Hbase project to see >> how >> >>> can >> >>>>> we >> >>>>>> both upgrade to 3.x(since this >> >>>>>> make take some time). >> >>>>>> >> >>>>>> Those possible solutions can be: >> >>>>>> 1. Using pre-patched protobuf 2.5.0 with ARM support >> >>>>>> from org.openlabtesting repo, some projects(HBase did >> >>>>>> this: https://github.com/apache/hbase/pull/959, and we will add a >> >>>>> profile >> >>>>>> for this, it will detact the host arch and >> >>>>>> will only use this pre-patched protobuf when the host arch is ARM >> so >> >>> that >> >>>>>> nothing will be affected for existing x86 >> >>>>>> users; Spark and Hadoop used some other package from this repo as >> they >> >>>>> have >> >>>>>> already upgraded protobuf). >> >>>>>> >> >>>>>> 2. Using pre-patched protobuf 2.6.1-build3 from os72's repo: >> >>>>>> >> >>> >> https://mvnrepository.com/artifact/com.github.os72/protoc/2.6.1-build3 >> >>>>>> Hive already used ``protoc-jar-maven-plugin`` from this repo: >> >>>>>> >> >>>>>> >> >>>>> >> >>> >> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/pom.xml#L484 >> >>>>>> >> >>>>>> >> >>>>>> >> >>>>> >> >>> >> https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/pom.xml#L472 >> >>>>>> so it is a minor update. >> >>>>>> >> >>>>>> I've tested both ways, worked OK on the ARM machine. >> >>>>>> >> >>>>>> So my ideas will be that we make this a two-step-work, first step >> we >> >>> use >> >>>>>> one of the short-term plan to enable >> >>>>>> Hive on ARM, and 2nd Step, we look for the long run to discuss with >> >>> HBase >> >>>>>> to find out a fesiable solution. >> >>>>>> >> >>>>>> BR, >> >>>>>> >> >>>>>> Zhenyu Zheng >> >>>>>> >> >>>>>> On Tue, Mar 10, 2020 at 9:29 PM Chinna Rao Lalam < >> >>>>>> lalamchinnara...@gmail.com> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> Thanks David for the quick reply. >> >>>>>>> Yes currently HBase protobuf version mismatch is the issue. >> >>>>>>> >> >>>>>>> We are investigating any common version of protobuf is there to >> >>> support >> >>>>>> for >> >>>>>>> HBase an HIVE. >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> On Tue, Mar 10, 2020 at 6:08 PM David Lavati >> >>>>>> <dlav...@cloudera.com.invalid >> >>>>>>>> >> >>>>>>> wrote: >> >>>>>>> >> >>>>>>>> Hi, >> >>>>>>>> >> >>>>>>>> I recently investigated this in >> >>>>>>>> https://issues.apache.org/jira/browse/HIVE-20359 and it didn't >> look >> >>>>>>>> feasible, as protobuf has to be in sync across the whole stack, >> but >> >>>>> the >> >>>>>>>> external protobuf version for hbase is on 2.5.0 for hbase-2.x, >> see >> >>> my >> >>>>>>>> comment on the jira and >> >>>>>>>> https://github.com/apache/hbase/blob/branch-2.2/pom.xml#L1414. >> >>>>>>>> >> >>>>>>>> Kind Regards, >> >>>>>>>> David >> >>>>>>>> >> >>>>>>>> On Tue, Mar 10, 2020 at 1:11 PM Chinna Rao Lalam < >> >>>>>>>> lalamchinnara...@gmail.com> >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> Hi All, >> >>>>>>>>> >> >>>>>>>>> We are working on HIVE building and running on ARM >> architecture. As >> >>>>>>> part >> >>>>>>>> of >> >>>>>>>>> this i want to add one periodic HIVE JENKINS JOB on ARM >> machines. >> >>>>>> Which >> >>>>>>>>> helps to identity gaps for ARM, If any issues identified we will >> >>>>> keep >> >>>>>>> on >> >>>>>>>>> working to fix those issues. >> >>>>>>>>> >> >>>>>>>>> Currently to run HIVE on aarch64, we need to upgrade to >> >>>>>> protobuf-3.7.1. >> >>>>>>>>> HIVE-21939 <https://issues.apache.org/jira/browse/HIVE-21939> >> >>>>>> upgraded >> >>>>>>>>> protobuf-2.5.0 to protobuf-3.7.1, It have mainly 2 failures. We >> are >> >>>>>>>> working >> >>>>>>>>> on this. >> >>>>>>>>> >> >>>>>>>>> Thanks, >> >>>>>>>>> Chinna Rao Lalam >> >>>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> -- >> >>>>>>> Hope It Helps, >> >>>>>>> Chinna >> >>>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Hope It Helps, >> >>>>> Chinna >> >>>>> >> >>>> >> >>> >> >> >> >> >> >> -- >> >> Hope It Helps, >> >> Chinna >> >> >> > >> > -- Hope It Helps, Chinna
