[
https://issues.apache.org/jira/browse/FLINK-19176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17194219#comment-17194219
]
Igal Shilman commented on FLINK-19176:
--------------------------------------
Hi [~galenwarren], welcome to project!
Indeed your analysis is correct, currently we support a selected few
serializers that are built in, and really the only one that is officially
supported and recommend for production use is Protobuf. It is also the only one
that works across remote functions (Python, Rust, Golang). I'd highly advice
to stick with that when working with embedded functions.
I've took a quick peek into ScalaPB and it seems like it generates classes that
do not implement the Protobuf Message interface, so I'm assuming that our
Protobuf serializer will not be able to serializer classes generated with the
ScalaPB.
Unfortunately adding ScalaPB payload serializer to the project directly (even
with runtime scope) will be very hard to get a consensus for, since maintaining
different Scala version is problematic and we have experienced that in Flink in
many different instances.
I think that the long term solution should be a Scala native remote SDK, and I
would be perfectly happy to help kicking off this effort if you are interested
:)
Meanwhile, if you would like to use ScalaPB for the embedded functions, then we
would need to support pluggable MessagePayloadSerializer, that would be picked
up in runtime from the class-path. If you are interesting on working on that, I
can try to outline the steps needed to do that.
> Support ScalaPB as a message payload serializer in Stateful Functions
> ---------------------------------------------------------------------
>
> Key: FLINK-19176
> URL: https://issues.apache.org/jira/browse/FLINK-19176
> Project: Flink
> Issue Type: Improvement
> Components: Stateful Functions
> Affects Versions: 2.0.0
> Reporter: Galen Warren
> Priority: Major
> Fix For: statefun-2.1.0
>
>
> Currently, Stateful Functions supports four options for serialization of
> message payloads:
> * Protobuf (based on code generated for Java)
> * Kryo
> * Multilanguage
> * Raw
> This proposal is to add a fifth option to this list, to support serialization
> of message payloads based on [ScalaPB |[https://scalapb.github.io/docs/]].
> This would allow Scala developers to use ScalaPB-generated classes as message
> types in Stateful Functions directly as message types, without having to
> convert to/from code generated for Java and/or raw byte[] messages.
> This would be a simple implementation, as there is already a
> [MessagePayloadSerializer
> |[https://github.com/apache/flink-statefun/blob/8ffe619a94917d676cf1054c8a0e60de544663db/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/message/MessagePayloadSerializer.java]]
> interface that is implemented for each of the existing serialization
> methods; supporting ScalaPB would require a new class implementing
> MessagePayloadSerializer in terms of ScalaPB-generated code, and a means to
> select it, by adding a new value to the [MessageFactoryType
> |[https://github.com/apache/flink-statefun/blob/8ffe619a94917d676cf1054c8a0e60de544663db/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/message/MessageFactoryType.java]]enumeration.
> Plus testing :)
> I've done this already locally, the implementation is very similar to the
> existing [MessagePayloadSerializerPb
> |[https://github.com/apache/flink-statefun/blob/8ffe619a94917d676cf1054c8a0e60de544663db/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/message/MessagePayloadSerializerPb.java]]implementation.
> It uses a reference to ScalaPB in "provided" scope.
> Would you be interested in a pull request to add this? Primary benefit is to
> make it easy to use Stateful Functions in a Scala environment. Thanks.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
