[
https://issues.apache.org/jira/browse/FLINK-24932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17569889#comment-17569889
]
Yun Tang commented on FLINK-24932:
----------------------------------
I have published my personal jar package
[https://mavenlibs.com/maven/dependency/io.github.myasuka/frocksdbjni#6.20.3-ververica-1.1]
.
It has newly built mac-related binaries with other binaries from
{{{}com.ververica.frocksdb:6.20.3-ververica-1.0{}}}. In other words, the
behavior and performance should stay the same as before on linux platform.
I triggered the CI:
[https://myasuka.visualstudio.com/flink/_build/results?buildId=425&view=results],
and it works fine.
I also trigger a state-benchmark and get the results below:
|RocksDB state access|6.20.3-1.0|6.20.3-1.1|performance diff|
|org.apache.flink.state.benchmark.ListStateBenchmark.listAdd|609.648787|599.759513|1.62%|
|org.apache.flink.state.benchmark.ListStateBenchmark.listAddAll|402.725298|413.361382|-2.64%|
|org.apache.flink.state.benchmark.ListStateBenchmark.listAppend|584.213647|578.372979|1.00%|
|org.apache.flink.state.benchmark.ListStateBenchmark.listGet|165.589927|166.683339|-0.66%|
|org.apache.flink.state.benchmark.ListStateBenchmark.listGetAndIterate|169.162846|168.137545|0.61%|
|org.apache.flink.state.benchmark.ListStateBenchmark.listUpdate|619.011581|616.667451|0.38%|
|org.apache.flink.state.benchmark.MapStateBenchmark.mapAdd|530.982488|521.362366|1.81%|
|org.apache.flink.state.benchmark.MapStateBenchmark.mapContains|66.026026|66.09366|-0.10%|
|org.apache.flink.state.benchmark.MapStateBenchmark.mapEntries|450.608818|455.196487|-1.02%|
|org.apache.flink.state.benchmark.MapStateBenchmark.mapGet|65.990952|66.614229|-0.94%|
|org.apache.flink.state.benchmark.MapStateBenchmark.mapIsEmpty|58.177144|57.683105|0.85%|
|org.apache.flink.state.benchmark.MapStateBenchmark.mapIterator|452.456979|452.730404|-0.06%|
|org.apache.flink.state.benchmark.MapStateBenchmark.mapKeys|462.294596|457.377305|1.06%|
|org.apache.flink.state.benchmark.MapStateBenchmark.mapPutAll|160.517354|160.890812|-0.23%|
|org.apache.flink.state.benchmark.MapStateBenchmark.mapRemove|524.405322|521.075357|0.64%|
|org.apache.flink.state.benchmark.MapStateBenchmark.mapUpdate|519.14977|528.62154|-1.82%|
|org.apache.flink.state.benchmark.MapStateBenchmark.mapValues|452.350788|451.491916|0.19%|
|org.apache.flink.state.benchmark.ValueStateBenchmark.valueAdd|520.534334|522.024747|-0.29%|
|org.apache.flink.state.benchmark.ValueStateBenchmark.valueGet|856.137319|843.489822|1.48%|
|org.apache.flink.state.benchmark.ValueStateBenchmark.valueUpdate|522.79689|524.357156|-0.30%|
As you can see, the performance behaves the same as expected.
I also use two macbook laptops (one is Intel CPU and another is M1) to run all
tests under flink-statebackend-rocksdb modules, they both behave well.
The rest part of work is that I will update the frocksdb release-notes and make
[https://github.com/ververica/frocksdb/pull/50] merged. Then let the jar
package published under ”com.ververica" group.
> Frocksdb cannot run on Apple M1
> -------------------------------
>
> Key: FLINK-24932
> URL: https://issues.apache.org/jira/browse/FLINK-24932
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Reporter: Yun Tang
> Assignee: Yun Tang
> Priority: Minor
> Fix For: 1.16.0
>
>
> After we bump up RocksDB version to 6.20.3, we support to run RocksDB on
> linux arm cluster. However, according to the feedback from Robert, Apple M1
> machines cannot run FRocksDB yet:
> {code:java}
> java.lang.Exception: Exception while creating StreamOperatorStateContext.
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:255)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:268)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:109)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreGates(StreamTask.java:711)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:687)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at
> org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:766)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
> ~[flink-runtime-1.14.0.jar:1.14.0]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for StreamFlatMap_c21234bcbf1e8eb4c61f1927190efebd_(1/1) from
> any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> Caused by: java.io.IOException: Could not load the native RocksDB library
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:882)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> Caused by: java.lang.UnsatisfiedLinkError:
> /private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib:
>
> dlopen(/private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib,
> 0x0001): tried:
> '/private/var/folders/js/yfk_y2450q7559kygttykwk00000gn/T/rocksdb-lib-5783c058ce68d31d371327abc9b51cac/librocksdbjni-osx.jnilib'
> (mach-o file, but is an incompatible architecture (have 'x86_64', need
> 'arm64e')), '/usr/lib/librocksdbjni-osx.jnilib' (no such file)
> at java.lang.ClassLoader$NativeLibrary.load(Native Method)
> ~[?:1.8.0_312]
> at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1950)
> ~[?:1.8.0_312]
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1832)
> ~[?:1.8.0_312]
> at java.lang.Runtime.load0(Runtime.java:811) ~[?:1.8.0_312]
> at java.lang.System.load(System.java:1088) ~[?:1.8.0_312]
> at
> org.rocksdb.NativeLibraryLoader.loadLibraryFromJar(NativeLibraryLoader.java:79)
> ~[frocksdbjni-6.20.3-ververica-1.0.jar:?]
> at
> org.rocksdb.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:57)
> ~[frocksdbjni-6.20.3-ververica-1.0.jar:?]
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.ensureRocksDBIsLoaded(EmbeddedRocksDBStateBackend.java:856)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.EmbeddedRocksDBStateBackend.createKeyedStateBackend(EmbeddedRocksDBStateBackend.java:402)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:345)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:87)
> ~[flink-statebackend-rocksdb_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:329)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:346)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:164)
> ~[flink-streaming-java_2.11-1.14.0.jar:1.14.0]
> ... 11 more
> {code}
> This issue is tracked by RocksDB community:
> [rocksdb/issues/7720|https://github.com/facebook/rocksdb/issues/7720]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
