Hi All, Some updates for ARM CI, our team has succesfully donated ARM resources and setup an ARM CI for Apache Spark: https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-maven-arm/ it will set to periodic job and then PR trigger when we think it is stable enough.
I really hope we can do the same for Hadoop. BR, On Fri, Sep 6, 2019 at 6:11 AM Vinayakumar B <vinayakum...@apache.org> wrote: > Thanks @Anu > > I understand the concern. I took it in different manner. > > Anyway, since protobuf upgrade looks huge, and need everyone's eyes on > changes as early as possible, its better to do it trunk itself. > > I was able to come to successfull attempt of upgrading protobuf as per > suggestion of stack in Jira > https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253 > . > > Have created the PR. Please review. Changes looks huge, because all > references of "com.google.protobuf" relocated to > "o.a.h.shaded.com.google.protobuf". > Otherwise changes are reasonable. > > This change is with still keeping the current 2.5.0 dependency, for > downstream builds. So essentially nothing should be changed for downstreams. > > -Vinay > > > On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer <aengin...@cloudera.com> > wrote: > >> Yes, I think that is what Sunil and I are trying to suggest; the complex >> dependencies like Protobuf, if you do it in the trunk you have a better >> change of getting it done. Otherwise, at merge point random downstream >> applications which you have never heard of will object, and Hadoop >> compatibility rules are very clear so you cannot fix it. >> >> With that said, even doing this in the trunk is complex; It might be >> good for you to host a meeting and get some feedback. I have openly said it >> is a great idea like "belling the cat", but the effort is in getting the >> community to agree and align. Solve that, most of your technical problems >> will be easier to solve. >> >> If you go into a branch, it might be that the community might forget >> about your work; and when you come in to merge you will see issues which >> you did not think about. >> >> So, Here is what would be great if you can make this happen; for ARM >> work, get a list of dependencies that needed to be upgraded; see if you can >> get the community aligned with this goal; since ARM might not be in the >> target for many users. You need to convince users that even without ARM, >> this is a great idea. >> >> If you like we can get together during one of the HDFS meetups hosted by >> Wei-chiu. >> >> Thanks >> Anu >> >> >> >> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B <vinayakum...@apache.org> >> wrote: >> >>> Hi all, >>> >>> Thanks for the response. >>> As I see, protobuf upgrade is long pending and most awaited one. >>> >>> @Sunil >>> Protobuf upgrade looks to be a non-trivial task. >>> Thanks @Duo Zhang for the suggestion of >>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the >>> problem >>> of dependency on build environment. >>> However more problem lies in upgrade protobuf without breaking the >>> downstream builds. >>> Reason is many downstream projects directly refers server specific jars >>> and >>> they expect protobuf-2.5.0 jar to get added to classpath by transitive >>> dependency. >>> >>> There are some historical discussions and suggestions on >>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf >>> upgrade. >>> Recommended path for solution is, try to upgrade protobuf using shading >>> of >>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in >>> dependencies for downstreams. >>> I am trying out ideas on this and if it gets completed within time, may >>> be >>> we can target trunk itself for the protobuf upgrade. >>> >>> separate branch idea is suggested for the overall ARM support including >>> protobuf upgrade and other problems mentioned in the discussion above. >>> >>> I dont expect separate branch to have a huge changes, apart from bug >>> fixes, >>> since there are no separate features specific to ARM is being planned. >>> So timely rebase of separate branch would reduce the overhead on branch >>> review/merge task. >>> >>> Still, if the solution to protobuf upgrade winds up early, without any >>> side >>> effects, I am more than happy to land it in trunk itself. >>> >>> Thanks, >>> Vinay >>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan <sun...@apache.org> wrote: >>> >>> > Thanks Vinay for starting the thread. >>> > >>> > I agree to Anu's view point related to protobuf. And with the >>> suggestion >>> > pointed out by Duo Zhang, if we can make use >>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to >>> 3.0.0 >>> > of protobuf will also be more easier. >>> > >>> > However i think its better to do this effort in trunk itself. >>> > In offline talks, few members were interested to start 3.3.0 release. >>> And >>> > given that happens soon, I feel its better >>> > we do this task in trunk itself as branch diverge is very much >>> possible. >>> > And to bring to call a merge on such a big branch will be even more >>> tough >>> > task. >>> > >>> > my 2 cents. >>> > >>> > Thanks >>> > Sunil >>> > >>> > On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <palomino...@gmail.com> >>> > wrote: >>> > >>> >> Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to >>> >> generate >>> >> the protobuf code. It will download the protoc binary from the maven >>> >> central so we do not need to install protoc on the build machine any >>> more. >>> >> >>> >> Zhenyu Zheng <zhengzhenyul...@gmail.com> 于2019年9月4日周三 下午5:27写道： >>> >> >>> >> > BTW, I also noticed that the Hadoop-trunk-Commit job has been >>> failling >>> >> for >>> >> > over 2 month related to the Protobuf problem . >>> >> > According to the latest successful build log: >>> >> > >>> >> >>> https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull >>> >> > the >>> >> > os version was ubuntu 14.04 and for the jobs that are failling now >>> such >>> >> > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console >>> , >>> >> > the os version is 18.04. I'm not very familiar with the version >>> changing >>> >> > for the jobs but I did a little search, according to: >>> >> > >>> >> > >>> >> >>> https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names >>> >> > & >>> >> > >>> >> > >>> >> >>> https://packages.ubuntu.com/search?suite=default§ion=all&arch=any&keywords=libprotoc-dev&searchon=names >>> >> > it both said that the version of libprotc-dev and protobuf-compiler >>> >> > available for ubuntu 18.04 is 3.0.0 >>> >> > >>> >> > >>> >> > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ayush...@gmail.com> >>> wrote: >>> >> > >>> >> >> Thanx Vinay for the initiative, Makes sense to add support for >>> >> different >>> >> >> architectures. >>> >> >> >>> >> >> +1, for the branch idea. >>> >> >> Good Luck!!! >>> >> >> >>> >> >> -Ayush >>> >> >> >>> >> >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <palomino...@gmail.com >>> > >>> >> >> wrote: >>> >> >> > >>> >> >> > For HBase, we purged all the protobuf related things from the >>> public >>> >> >> API, >>> >> >> > and then upgraded to a shaded and relocated version of protobuf. >>> We >>> >> have >>> >> >> > created a repo for this: >>> >> >> > >>> >> >> > https://github.com/apache/hbase-thirdparty >>> >> >> > >>> >> >> > But since the hadoop dependencies still pull in the protobuf 2.5 >>> >> jars, >>> >> >> our >>> >> >> > coprocessors are still on protobuf 2.5. Recently we have opened a >>> >> >> discuss >>> >> >> > on how to deal with the upgrading of coprocessor. Glad to see >>> that >>> >> the >>> >> >> > hadoop community is also willing to solve the problem. >>> >> >> > >>> >> >> > Anu Engineer <aengin...@cloudera.com.invalid> 于2019年9月3日周二 >>> 上午1:23写道： >>> >> >> > >>> >> >> >> +1, for the branch idea. Just FYI, Your biggest problem is >>> proving >>> >> that >>> >> >> >> Hadoop and the downstream projects work correctly after you >>> upgrade >>> >> >> core >>> >> >> >> components like Protobuf. >>> >> >> >> So while branching and working on a branch is easy, merging back >>> >> after >>> >> >> you >>> >> >> >> upgrade some of these core components is insanely hard. You >>> might >>> >> want >>> >> >> to >>> >> >> >> make sure that community buys into upgrading these components >>> in the >>> >> >> trunk. >>> >> >> >> That way we will get testing and downstream components will >>> notice >>> >> when >>> >> >> >> things break. >>> >> >> >> >>> >> >> >> That said, I have lobbied for the upgrade of Protobuf for a >>> really >>> >> long >>> >> >> >> time; I have argued that 2.5 is out of support and we cannot >>> stay on >>> >> >> that >>> >> >> >> branch forever; or we need to take ownership of the Protobuf 2.5 >>> >> code >>> >> >> base. >>> >> >> >> It has been rightly pointed to me that while all the arguments I >>> >> make >>> >> >> is >>> >> >> >> correct; it is a very complicated task to upgrade Protobuf, and >>> the >>> >> >> worst >>> >> >> >> part is we will not even know what breaks until downstream >>> projects >>> >> >> pick up >>> >> >> >> these changes and work against us. >>> >> >> >> >>> >> >> >> If we work off the Hadoop version 3 — and assume that we have >>> >> >> "shading" in >>> >> >> >> place for all deployments; it might be possible to get there; >>> still >>> >> a >>> >> >> >> daunting task. >>> >> >> >> >>> >> >> >> So best of luck with the branch approach — But please remember, >>> >> Merging >>> >> >> >> back will be hard, Just my 2 cents. >>> >> >> >> >>> >> >> >> — Anu >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng < >>> >> zhengzhenyul...@gmail.com >>> >> >> > >>> >> >> >> wrote: >>> >> >> >> >>> >> >> >>> Hi, >>> >> >> >>> >>> >> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. >>> A >>> >> >> separate >>> >> >> >>> branch with it's own ARM CI seems a really good idea. >>> >> >> >>> By doing this we won't break any of the undergoing development >>> in >>> >> >> trunk >>> >> >> >> and >>> >> >> >>> a CI can be a very good way to show what are the >>> >> >> >>> current problems and what have been fixed, it will also >>> provide a >>> >> very >>> >> >> >> good >>> >> >> >>> view for contributors that are intrested to working on >>> >> >> >>> this. We can finally merge back the branch to trunk until the >>> >> >> community >>> >> >> >>> thinks it is good enough and stable enough. We can donate >>> >> >> >>> ARM machines to the existing CI system for the job. >>> >> >> >>> >>> >> >> >>> I wonder if this approch possible? >>> >> >> >>> >>> >> >> >>> BR, >>> >> >> >>> >>> >> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu < >>> >> liusheng2...@gmail.com> >>> >> >> >>> wrote: >>> >> >> >>> >>> >> >> >>>> Hi, >>> >> >> >>>> >>> >> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab" >>> >> community >>> >> >> >>>> mentioned by Vinay. I am working on building and >>> >> >> >>>> testing Hadoop components on aarch64 server these days, >>> besides >>> >> the >>> >> >> >>> missing >>> >> >> >>>> dependices of ARM platform issues #1 #2 #3 >>> >> >> >>>> mentioned by Vinay, other similar issue has also be found, >>> such as >>> >> >> the >>> >> >> >>>> "PhantomJS" dependent package also missing for aarch64. >>> >> >> >>>> >>> >> >> >>>> To promote the ARM support for Hadoop, we have discussed and >>> >> hoped to >>> >> >> >> add >>> >> >> >>>> an ARM specific CI to Hadoop repo. we are not >>> >> >> >>>> sure about if there is any potential effect or confilict on >>> the >>> >> trunk >>> >> >> >>>> branch, so maybe creating a ARM specific branch for doing >>> these >>> >> stuff >>> >> >> >>>> is a better choice, what do you think? >>> >> >> >>>> >>> >> >> >>>> Hope to hear thoughts from you :) >>> >> >> >>>> >>> >> >> >>>> BR, >>> >> >> >>>> Liu sheng >>> >> >> >>>> >>> >> >> >>>> Vinayakumar B <vinayakum...@apache.org> 于2019年8月27日周二 >>> 上午5:34写道： >>> >> >> >>>> >>> >> >> >>>>> Hi Folks, >>> >> >> >>>>> >>> >> >> >>>>> ARM is becoming famous lately in its processing capability >>> and >>> >> has >>> >> >> >> got >>> >> >> >>>> the >>> >> >> >>>>> potential to run Bigdata workloads. >>> >> >> >>>>> Many users have been moving to ARM machines due to its low >>> cost. >>> >> >> >>>>> >>> >> >> >>>>> In the past there were attempts to compile Hadoop on ARM >>> >> (Rasberry >>> >> >> >> PI) >>> >> >> >>>> for >>> >> >> >>>>> experimental purposes. Today ARM architecture is taking some >>> of >>> >> the >>> >> >> >>>>> serverside processing as well. So there will be/is a real >>> need of >>> >> >> >>> Hadoop >>> >> >> >>>> to >>> >> >> >>>>> support ARM architecture as well. >>> >> >> >>>>> >>> >> >> >>>>> There are bunch of users who are trying out building Hadoop >>> on >>> >> ARM, >>> >> >> >>>> trying >>> >> >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some >>> >> >> >>>>> >>> >> >> >>>>> As of today, Hadoop does not compile on ARM due to below >>> issues, >>> >> >> >> found >>> >> >> >>>> from >>> >> >> >>>>> testing done in openlab in [2]. >>> >> >> >>>>> >>> >> >> >>>>> 1. Protobuf : >>> >> >> >>>>> ------------------- >>> >> >> >>>>> Hadoop project (also some downstream projects) stuck to >>> >> protobuf >>> >> >> >>>> 2.5.0 >>> >> >> >>>>> version, due to backward compatibility reasons. >>> Protobuf-2.5.0 is >>> >> >> not >>> >> >> >>>> being >>> >> >> >>>>> maintained in the community. While protobuf 3.x is being >>> actively >>> >> >> >>> adopted >>> >> >> >>>>> widely, still protobuf 3.x provides wire compatibility for >>> proto2 >>> >> >> >>>> messages. >>> >> >> >>>>> Due to some compilation issues in the generated java code, >>> which >>> >> can >>> >> >> >>>> induce >>> >> >> >>>>> problems in downstream. Due to this reason protobuf upgrade >>> from >>> >> >> >> 2.5.0 >>> >> >> >>>> was >>> >> >> >>>>> not taken up. >>> >> >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to >>> avoid >>> >> >> >>> classpath >>> >> >> >>>>> problem in downstream projects. >>> >> >> >>>>> There are patches available to fix compilation in Hadoop. >>> But >>> >> >> >> need >>> >> >> >>> to >>> >> >> >>>>> find a way to upgrade protobuf to latest version and still >>> >> maintain >>> >> >> >> the >>> >> >> >>>>> downstream's classpath using shading feature of Hadoop build. >>> >> >> >>>>> >>> >> >> >>>>> There is a Jira for protobuf upgrade[3] created even >>> before >>> >> >> >> shade >>> >> >> >>>>> support was added to Hadoop. Now need to revisit the Jira and >>> >> >> >> continue >>> >> >> >>>>> explore possibilities. >>> >> >> >>>>> >>> >> >> >>>>> 2. leveldbjni: >>> >> >> >>>>> --------------- >>> >> >> >>>>> Current leveldbjni used in YARN doesnot support ARM >>> >> architecture, >>> >> >> >>>> need >>> >> >> >>>>> to check whether any of the future versions support ARM and >>> can >>> >> >> >> hadoop >>> >> >> >>>>> upgrade to that version. >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1' >>> >> >> >>>>> ------------------------- >>> >> >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM >>> executable by >>> >> >> >>> default >>> >> >> >>>> in >>> >> >> >>>>> the maven repository. Workaround is to build it locally and >>> keep >>> >> in >>> >> >> >>> local >>> >> >> >>>>> maven repository. >>> >> >> >>>>> Need to check whether any future versions of >>> >> 'protoc-gen-grpc-java' >>> >> >> >> is >>> >> >> >>>>> having ARM executable and whether hadoop-yarn-csi can >>> upgrade it? >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> Once the compilation issues are solved, then there might be >>> many >>> >> >> >> native >>> >> >> >>>>> code related issues due to different architectures. >>> >> >> >>>>> So to explore everything, need to join hands together and >>> >> proceed. >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> Let us discuss and check, whether any body else out there who >>> >> also >>> >> >> >> need >>> >> >> >>>> the >>> >> >> >>>>> support of Hadoop on ARM architectures and ready to lend >>> their >>> >> hands >>> >> >> >>> and >>> >> >> >>>>> time in this work. >>> >> >> >>>>> >>> >> >> >>>>> >>> >> >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358 >>> >> >> >>>>> [2] >>> >> >> >> >>> >> >> >>> >> >>> https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887 >>> >> >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363 >>> >> >> >>>>> >>> >> >> >>>>> -Vinay >>> >> >> >> >>> >> >> >>> >> > >>> >> >>> > >>> >>
