We (as in the Flink PMC) cannot add apps to the GitHub repo; please
check first with INFRA whether these CI systems are allowed.
On 30/07/2019 03:44, Xiyuan Wang wrote:
Hi Stephan,
Maybe I misled you in the previous email. We don't need to migrate CI
completely, travis-ci is still there working for X86 arch. What we need to
do is to add another CI tool for ARM arch.
There are some ways to do it. As I wrote on
https://issues.apache.org/jira/browse/FLINK-13199 to @Chesnay:
1. Add OpenLab CI system for ARM arch test.OpenLab is very similar with
travis-ci. What Flilnk need to do is adding the openlab github app to the
repo, then add the job define files inner Flink repo, Here is a POC by me:
https://github.com/theopenlab/flink/pull/1
2. OpenLab will donate ARM resouces to Apache Infra team as well. Then
Flink can use the Apache offical Jenkins system for Flink ARM test in the
future. https://builds.apache.org/
3. Use Drony CI which support ARM arch as well. https://drone.io/
Since I'm from OpenLab community, if Flink choose OpenLab CI, My OpenLab
colleague and I can keep helping and maintaining the ARM CI job. If choose
the 2nd way, the CI maintainance work may be handled by apache-infra team I
guess. If choose the 3rd Drony CI, what we can help is very limited.
AFAIK, Drony use container for CI test, which may not satisfy some
requiremnts. And OpenLab use VM for test.
Need Flink core team's decision and reply.
Thanks.
Stephan Ewen <se...@apache.org> 于2019年7月29日周一 下午6:05写道：
I don't think it is feasible for Flink to migrate CI completely.
Is there a way to add ARM tests on an external CI in addition?
@Chesnay what do you think?
On Fri, Jul 12, 2019 at 4:45 AM Xiyuan Wang <wangxiyuan1...@gmail.com>
wrote:
Hi Stephan
yeah, we should add an ARM CI first. But Travis CI doesn't support ARM
arch itself. OpenLab community support it. As I mentioned before, OpenLab
is an opensource CI system like travis-ci.[1], it uses opensource CI
project `zuul`[2] for its deployment. Now some opensource project has
intergreted with it already. For example, `contained` project from
CNCF community[3]. And I have a POC for Flink ARM build and test using
OpenLab. Now the build is passed[4], and I'm working on debugging with
the
`test` part[5]. Is it fine for Flink to using?
[1]: https://openlabtesting.org
[2]: https://zuul-ci.org/docs/zuul/
[3]: https://status.openlabtesting.org/projects
[4]:
https://status.openlabtesting.org/build/2aa33f1a87854679b70f36bd6f75a890
[5]: https://github.com/theopenlab/flink/pull/1
Stephan Ewen <se...@apache.org> 于2019年7月11日周四 下午9:56写道：
I think an ARM release would be cool.
To actually support that properly, we would need something like an ARM
profile for the CI builds (at least in the nightly tests), otherwise
ARM
support would probably be broken frequently.
Maybe that could be a way to start? Create a Travis CI ARM build (if
possible) and see what tests pass and which parts of the system would
need
to be adjusted?
On Thu, Jul 11, 2019 at 9:24 AM Xiyuan Wang <wangxiyuan1...@gmail.com>
wrote:
Hi yun:
I didn't try to build rocksdb with vagrant, but just `make -j8
rocksdbjava` directly in an ARM machine. We hit some issues as well.
My
colleague has created an issue in rocksdb[1]. Rocksdb doesn't
contains
ARM
.so file in his offical jar package. If you have the same request,
let's
work together there.
Thanks.
[1]: https://github.com/facebook/rocksdb/issues/5559
Yun Tang <myas...@live.com> 于2019年7月11日周四 下午12:01写道：
Hi Xiyuan
Have you ever tried to release RocksDB on ARM like official doc[1]
suggests? From our experience, cross-building for ARM did not work
fine
with Vagrant and we have to build rocksDB's binary file on ARM
separately.
As frocksdb [2] might not always maintained in Flink, I think we'd
better
support to release RocksDB-java with ARM officially.
[1]
https://github.com/facebook/rocksdb/blob/master/java/RELEASE.md
[2] https://github.com/dataArtisans/frocksdb
Best
Yun Tang
________________________________
From: Xiyuan Wang <wangxiyuan1...@gmail.com>
Sent: Tuesday, July 9, 2019 10:52
To: dev@flink.apache.org
Subject: Re: [DISCUSS] ARM support for Flink
Thanks for your help. I built the frocksdb locally on ARM and all
the
related tests are passed now. Except some tests which can be fixed
easily,
it seems that both building and testing are ran well on ARM.
Basing on my test, Is it possible to support Flink on ARM
officailly?
Seem
the worklist is not too long. And I can help with the CI testing
part.
Need Flink team's idea.
Thanks.
Dian Fu <dian0511...@gmail.com> 于2019年7月8日周一 上午10:23写道：
Hi Xiyuan,
Thanks for bring the discussion.
WRT the exception, it's because the native bundled in the rocksdb
jar
file
isn't compiled with cross platform support. You can refer [1] for
how
to
build rocksdb which has ARM platform.
WRT ARM support, the rocksdb currently used in Flink is hosted in
the
Ververica git [2], so it won't be difficult to make it support
ARM.
However, I guess this git exists just for temporary [3], not
because
we
want to add much feature in rocksdb.
[1] https://github.com/facebook/rocksdb/issues/678 <
https://github.com/facebook/rocksdb/issues/678>
[2] https://github.com/dataArtisans/frocksdb <
https://github.com/dataArtisans/frocksdb>
[3] https://issues.apache.org/jira/browse/FLINK-10471 <
https://issues.apache.org/jira/browse/FLINK-10471>
Regards,
Dian
在 2019年7月8日，上午9:17，Xiyuan Wang <wangxiyuan1...@gmail.com> 写道：
Hi Flink:
Recently we meet a problem that we have to test and run Flink
on
ARM
arch. While after searching Flink community, I didn’t find an
official
ARM
release version.
Since Flink is made by Java and Scala language which can be ran
cross-platform usually, I think Flink can be built and ran on
ARM
directly
as well. Then with my local test, Flink was built and deployed
success
as
expected. But some tests were failed due to ARM arch. For
example:
1. MemoryArchitectureTest.testArchitectureNotUnknown:34 Values
should
be
different. Actual: UNKNOWN
2. [ERROR]
testIterator(org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest)
Time elapsed: 0.234 s <<< ERROR!
java.io.IOException: Could not load the native RocksDB library
at
org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest.testIteratorHelper(RocksDBRocksStateKeysIteratorTest.java:90)
at
org.apache.flink.contrib.streaming.state.RocksDBRocksStateKeysIteratorTest.testIterator(RocksDBRocksStateKeysIteratorTest.java:63)
Caused by: java.lang.UnsatisfiedLinkError:
/tmp/rocksdb-lib-81ca7930b92af2cca143a050c0338d34/librocksdbjni-linux64.so:
/tmp/rocksdb-lib-81ca7930b92af2cca143a050c0338d34/librocksdbjni-linux64.so:
cannot open shared object file: No such file or directory
(Possible
cause:
can't load AMD 64-bit .so on a AARCH64-bit platform)
…
Since the test isn’t passed totally, we are not sure if Flink
100%
support ARM or not. Is it possible for Flink to support ARM
release
officially? I guess it may be not a very huge work basing on
Java.
I
notice
that Flink now uses trivis-ci which is X86 only for build &
test
check.
Is
it possible to add an ARM arch CI as well? It can be non-voting
first.
Then
we can keep monitoring and fixing ARM related error. One day
it’s
stable
enough, we can remove the non-voting tag and create Flink ARM
release.
There is an open source CI community called OpenLab[1] which
can
provide
CI function and ARM resource to Flink by free. I’m one of the
OpenLab
members. If Flink commun think ARM support is fine, I can keep
helping
Flink to build and maintain the ARM CI job. There is an POC
for
Flink
ARM
build job made by me on OpenLab system[2] and a live demo which
built
and
run on an ARM VM[3]. You can take a look first.
Eager to get everybody’s feedback. Any question is welcome.
Thanks.
[1]: https://openlabtesting.org/
[2]: https://github.com/theopenlab/flink/pull/1
[3]: http://114.115.168.52:8081/#/overview
