I think an ARM release would be cool. To actually support that properly, we would need something like an ARM profile for the CI builds (at least in the nightly tests), otherwise ARM support would probably be broken frequently. Maybe that could be a way to start? Create a Travis CI ARM build (if possible) and see what tests pass and which parts of the system would need to be adjusted?
On Thu, Jul 11, 2019 at 9:24 AM Xiyuan Wang <wangxiyuan1...@gmail.com> wrote: > Hi yun: > I didn't try to build rocksdb with vagrant, but just `make -j8 > rocksdbjava` directly in an ARM machine. We hit some issues as well. My > colleague has created an issue in rocksdb[1]. Rocksdb doesn't contains ARM > .so file in his offical jar package. If you have the same request, let's > work together there. > > Thanks. > > [1]: https://github.com/facebook/rocksdb/issues/5559 > > Yun Tang <myas...@live.com> 于2019年7月11日周四 下午12:01写道： > > > Hi Xiyuan > > > > Have you ever tried to release RocksDB on ARM like official doc[1] > > suggests? From our experience, cross-building for ARM did not work fine > > with Vagrant and we have to build rocksDB's binary file on ARM > separately. > > > > As frocksdb [2] might not always maintained in Flink, I think we'd better > > support to release RocksDB-java with ARM officially. > > > > > > [1] https://github.com/facebook/rocksdb/blob/master/java/RELEASE.md > > [2] https://github.com/dataArtisans/frocksdb > > > > Best > > Yun Tang > > > > > > ________________________________ > > From: Xiyuan Wang <wangxiyuan1...@gmail.com> > > Sent: Tuesday, July 9, 2019 10:52 > > To: dev@flink.apache.org > > Subject: Re: [DISCUSS] ARM support for Flink > > > > Thanks for your help. I built the frocksdb locally on ARM and all the > > related tests are passed now. Except some tests which can be fixed > easily, > > it seems that both building and testing are ran well on ARM. > > > > Basing on my test, Is it possible to support Flink on ARM officailly? > Seem > > the worklist is not too long. And I can help with the CI testing part. > > > > Need Flink team's idea. > > > > Thanks. > > > > Dian Fu <dian0511...@gmail.com> 于2019年7月8日周一 上午10:23写道： > > > > > Hi Xiyuan, > > > > > > Thanks for bring the discussion. > > > > > > WRT the exception, it's because the native bundled in the rocksdb jar > > file > > > isn't compiled with cross platform support. You can refer [1] for how > to > > > build rocksdb which has ARM platform. > > > > > > WRT ARM support, the rocksdb currently used in Flink is hosted in the > > > Ververica git [2], so it won't be difficult to make it support ARM. > > > However, I guess this git exists just for temporary [3], not because we > > > want to add much feature in rocksdb. > > > > > > [1] https://github.com/facebook/rocksdb/issues/678 < > > > https://github.com/facebook/rocksdb/issues/678> > > > [2] https://github.com/dataArtisans/frocksdb < > > > https://github.com/dataArtisans/frocksdb> > > > [3] https://issues.apache.org/jira/browse/FLINK-10471 < > > > https://issues.apache.org/jira/browse/FLINK-10471> > > > > > > Regards, > > > Dian > > > > > > > 在 2019年7月8日，上午9:17，Xiyuan Wang <wangxiyuan1...@gmail.com> 写道： > > > > > > > > Hi Flink: > > > > Recently we meet a problem that we have to test and run Flink on ARM > > > > arch. While after searching Flink community, I didn’t find an > official > > > ARM > > > > release version. > > > > > > > > Since Flink is made by Java and Scala language which can be ran > > > > cross-platform usually, I think Flink can be built and ran on ARM > > > directly > > > > as well. Then with my local test, Flink was built and deployed > success > > as > > > > expected. But some tests were failed due to ARM arch. For example: > > > > > > > > 1. MemoryArchitectureTest.testArchitectureNotUnknown:34 Values should > > be > > > > different. Actual: UNKNOWN > > > > 2. [ERROR] > > > > > > > > > > testIterator(org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest) > > > > Time elapsed: 0.234 s <<< ERROR! > > > > java.io.IOException: Could not load the native RocksDB library > > > > at > > > > > > > > > > org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest.testIteratorHelper(RocksDBRocksStateKeysIteratorTest.java:90) > > > > at > > > > > > > > > > org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest.testIterator(RocksDBRocksStateKeysIteratorTest.java:63) > > > > Caused by: java.lang.UnsatisfiedLinkError: > > > > > > > > > > /tmp/rocksdb-lib-81ca7930b92af2cca143a050c0338d34/librocksdbjni-linux64.so: > > > > > > > > > > /tmp/rocksdb-lib-81ca7930b92af2cca143a050c0338d34/librocksdbjni-linux64.so: > > > > cannot open shared object file: No such file or directory (Possible > > > cause: > > > > can't load AMD 64-bit .so on a AARCH64-bit platform) > > > > … > > > > > > > > Since the test isn’t passed totally, we are not sure if Flink 100% > > > > support ARM or not. Is it possible for Flink to support ARM release > > > > officially? I guess it may be not a very huge work basing on Java. I > > > notice > > > > that Flink now uses trivis-ci which is X86 only for build & test > check. > > > Is > > > > it possible to add an ARM arch CI as well? It can be non-voting > first. > > > Then > > > > we can keep monitoring and fixing ARM related error. One day it’s > > stable > > > > enough, we can remove the non-voting tag and create Flink ARM > release. > > > > > > > > There is an open source CI community called OpenLab[1] which can > > provide > > > > CI function and ARM resource to Flink by free. I’m one of the OpenLab > > > > members. If Flink commun think ARM support is fine, I can keep > helping > > > > Flink to build and maintain the ARM CI job. There is an POC for > Flink > > > ARM > > > > build job made by me on OpenLab system[2] and a live demo which built > > and > > > > run on an ARM VM[3]. You can take a look first. > > > > > > > > Eager to get everybody’s feedback. Any question is welcome. > > > > > > > > Thanks. > > > > > > > > [1]: https://openlabtesting.org/ > > > > [2]: https://github.com/theopenlab/flink/pull/1 > > > > [3]: http://114.115.168.52:8081/#/overview > > > > > > > > >
