I took the ut tests on my arm instance before and reported an issue in https://issues.apache.org/jira/browse/SPARK-27721, and seems there was no leveldbjni native package for aarch64 in leveldbjni-all.jar(or 1.8) https://mvnrepository.com/artifact/org.fusesource.leveldbjni/leveldbjni-all/1.8 , we can find https://github.com/fusesource/leveldbjni/pull/82 this pr added the aarch64 support and merged on 2 Nov 2017, but the latest release of the repo is on 17 Oct 2013, unfortunately it didn't include the aarch64 supporting.
I will running the test on the job mentioned above, and will try to fix the issue above, or if anyone have any idea of it, welcome reply me, thank you. On Wed, Jun 26, 2019 at 8:11 PM Sean Owen <sro...@gmail.com> wrote: > Can you begin by testing yourself? I think the first step is to make > sure the build and tests work on ARM. If you find problems you can > isolate them and try to fix them, or at least report them. It's only > worth getting CI in place when we think builds will work. > > On Tue, Jun 25, 2019 at 9:26 PM Tianhua huang <huangtianhua...@gmail.com> > wrote: > > > > Thanks Shane :) > > > > This sounds good, and yes I agree that it's best to keep the test/build > infrastructure in one place. If you can't find the ARM resource we are > willing to support the ARM instance :) Our goal is to make more open > source software to be more compatible for aarch64 platform, so let's to do > it. I will be happy if I can give some help for the goal. > > > > Waiting for you good news :) > > > > On Wed, Jun 26, 2019 at 9:47 AM shane knapp <skn...@berkeley.edu> wrote: > >> > >> ...or via VM as you mentioned earlier. :) > >> > >> shane (who will file a JIRA tomorrow) > >> > >> On Tue, Jun 25, 2019 at 6:44 PM shane knapp <skn...@berkeley.edu> > wrote: > >>> > >>> i'd much prefer that we keep the test/build infrastructure in one > place. > >>> > >>> we don't have ARM hardware, but there's a slim possibility i can scare > something up in our older research stock... > >>> > >>> another option would be to run the build in a arm-based docker > container, which (according to the intarwebs) is possible. > >>> > >>> shane > >>> > >>> On Tue, Jun 25, 2019 at 6:35 PM Tianhua huang < > huangtianhua...@gmail.com> wrote: > >>>> > >>>> I forked apache/spark project and propose a job( > https://github.com/theopenlab/spark/pull/1) for spark building in OpenLab > ARM instance, this is the first step to build spark on ARM, I can enable a > periodic job for arm building for apache/spark master if you guys like. > Later I will run tests for spark. I also willing to be the maintainer of > the arm ci of spark. > >>>> > >>>> Thanks for you attention. > >>>> > >>>> On Thu, Jun 20, 2019 at 10:17 AM Tianhua huang < > huangtianhua...@gmail.com> wrote: > >>>>> > >>>>> Thanks Sean. > >>>>> > >>>>> I am very happy to hear that the community will put effort to fix > the ARM-related issues. I'd be happy to help if you like. And could you > give the trace link of this issue, then I can check it is fixed or not, > thank you. > >>>>> As far as I know the old versions of spark support ARM, and now the > new versions don't, this just shows that we need a CI to check whether the > spark support ARM and whether some modification break it. > >>>>> I will add a demo job in OpenLab to build spark on ARM and do a > simple UT test. Later I will give the job link. > >>>>> > >>>>> Let me know what you think. > >>>>> > >>>>> Thank you all! > >>>>> > >>>>> > >>>>> On Wed, Jun 19, 2019 at 8:47 PM Sean Owen <sro...@gmail.com> wrote: > >>>>>> > >>>>>> I'd begin by reporting and fixing ARM-related issues in the build. > If > >>>>>> they're small, of course we should do them. If it requires > significant > >>>>>> modifications, we can discuss how much Spark can support ARM. I > don't > >>>>>> think it's yet necessary for the Spark project to run these CI > builds > >>>>>> until that point, but it's always welcome if people are testing that > >>>>>> separately. > >>>>>> > >>>>>> On Wed, Jun 19, 2019 at 7:41 AM Holden Karau <hol...@pigscanfly.ca> > wrote: > >>>>>> > > >>>>>> > Moving to dev@ for increased visibility among the developers. > >>>>>> > > >>>>>> > On Wed, Jun 19, 2019 at 1:24 AM Tianhua huang < > huangtianhua...@gmail.com> wrote: > >>>>>> >> > >>>>>> >> Thanks for your reply. > >>>>>> >> > >>>>>> >> As I said before, I met some problem of build or test for spark > on aarch64 server, so it will be better to have the ARM CI to make sure the > spark is compatible for AArch64 platforms. > >>>>>> >> > >>>>>> >> I’m from OpenLab team(https://openlabtesting.org/ ,a community > to do open source project testing. And we can support some Arm virtual > machines to AMPLab Jenkins, and also we have a developer team that willing > to work on this, we willing to maintain build CI jobs and address the CI > issues. What do you think? > >>>>>> >> > >>>>>> >> > >>>>>> >> Thanks for your attention. > >>>>>> >> > >>>>>> >> > >>>>>> >> On Wed, Jun 19, 2019 at 6:39 AM shane knapp <skn...@berkeley.edu> > wrote: > >>>>>> >>> > >>>>>> >>> yeah, we don't have any aarch64 systems for testing... this > has been asked before but is currently pretty low on our priority list as > we don't have the hardware. > >>>>>> >>> > >>>>>> >>> sorry, > >>>>>> >>> > >>>>>> >>> shane > >>>>>> >>> > >>>>>> >>> On Mon, Jun 10, 2019 at 7:08 PM Tianhua huang < > huangtianhua...@gmail.com> wrote: > >>>>>> >>>> > >>>>>> >>>> Hi, sorry to disturb you. > >>>>>> >>>> The CI testing for apache spark is supported by AMPLab > Jenkins, and I find there are some computers(most of them are Linux (amd64) > arch) for the CI development, but seems there is no Aarch64 computer for > spark CI testing. Recently, I build and run test for spark(master and > branch-2.4) on my arm server, and unfortunately there are some problems, > for example, ut test is failed due to a LEVELDBJNI native package, the > details for java test see http://paste.openstack.org/show/752063/ and > python test see http://paste.openstack.org/show/752709/ > >>>>>> >>>> So I have a question about the ARM CI testing for spark, is > there any plan to support it? Thank you very much and I will wait for your > reply! > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> > >>>>>> >>> -- > >>>>>> >>> Shane Knapp > >>>>>> >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>>>>> >>> https://rise.cs.berkeley.edu > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > -- > >>>>>> > Twitter: https://twitter.com/holdenkarau > >>>>>> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 > >>>>>> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > >>> > >>> > >>> > >>> -- > >>> Shane Knapp > >>> UC Berkeley EECS Research / RISELab Staff Technical Lead > >>> https://rise.cs.berkeley.edu > >> > >> > >> > >> -- > >> Shane Knapp > >> UC Berkeley EECS Research / RISELab Staff Technical Lead > >> https://rise.cs.berkeley.edu >
