Using Flink in a NI does not make much sense. However, it's definitely worthwhile to enable Flink applications to use Flink as a library and bundle that in a NI.
Btw reflection is not the root problem. You can use reflection just fine in a NI. The issue with NI is (next to Unsafe) that NI has a close-world assumption. That means we would only run Flink without any UDFs (in the sense of real user code, not UDFs provided by Flink). However, in theory, it should be possible to compile Flink against a specific application and build a NI when everything that is needed for a particular Flink application is known to the Graal NI compiler. In practise, Flink would need that compiler to figure out, which classes/methods are actually needed and which not, since some Flink magic would break the close world assumption. Finally, Table API with code generation will probably not work with Graal for the time being at all. Not sure if there is any effort to support that as well on Graal side. On Tue, Jun 30, 2020 at 9:51 AM Roman Grebennikov <[email protected]> wrote: > Hi, > > the error it the original message means that inside a Kryo serializer, > there is something that uses sun.misc.Unsafe to compute field offset in raw > memory to access class fields. The problem is that memory layout in JVM and > in NI is different and most probably will result in a segfault if you leave > it as is. > > If you really want to go further, there is a @RecomputeFieldValue(kind = > Kind.FieldOffset) annotation in the graalvm to update the offset for NI. > > But I guess the overall perspective of running Flink as a NI is not that > good for the following reasons: > * GraalVM NI requires that the whole application is completely static and > there is no dynamic classloading happening in runtime. NI still can handle > some sort of dynamic classloading, but only if you define all possible > combinations of loaded classes on compile time. Flink itself uses dynamic > classloading when you submit a job there as a fat jar file. So you will > need to statically compile a bundle of your app fat jar AND taskmanager > code at once into a single binary. > * Flink also relies quite a lot on dynamic reflection in the serialization > code, so you also have to build static reflection configuration at > compile-time (by probably running your job+taskmanager with tracing agent > on JVM) > * As GraalVM NI has no way of doing JIT compilation (as everything must be > compiled statically), you should expect a lower overall job performance. > > GraalVM and Flink can be quite a nice code gymnastics exercise, but what > is your final business goal? > > Roman Grebennikov | [email protected] > > > On Thu, Jun 25, 2020, at 11:48, [email protected] wrote: > > Whats up guys, > > > > I'm trying to run an Apache Flink Application with the GraalVM Native > Image but I get the following error: (check attached file) > > > > I suppose this happens, because Flink uses a lot of low-level-code and is > highly optimized. > > > > When I googled the combination of GraalVM Native Image and Apache Flink I > get no results. > > > > Did anyone ever succeeded in making it work and how? > > > > Best regards, > > > > Ivo > > > *Attachments:* > > - Flink GraalVM Native Image Error > > > -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
