[
https://issues.apache.org/jira/browse/FLINK-19176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17195564#comment-17195564
]
Galen Warren commented on FLINK-19176:
--------------------------------------
Thanks [~igal].
Yep, that's almost exactly how I approached it! I thought of MessageFactoryKey
as needing to containing two pieces of information – the existing
MessageFactoryType (with one new value added to the enumeration,
USES_CUSTOM_PAYLOADS) and a string field (customPayloadSerializerName) that
contains the name of the custom payload serializer class, which would be used
only with USES_CUSTOM_PAYLOADS.
However, one question. Several classes that currently have MessageFactoryKey as
a field are serializable, and so changing the type of that field would break
serialization backward compatibility if nothing else were done.
MessageTypeSerializer is an example of such a class. I can see a few options
for how to proceed:
* Just replace MessageFactoryType with MessageFactoryKey anyway, bump the
serialization uid and accept the incompatibility
* Replace MessageFactoryType with MessageFactoryKey, bump the serialization
uid, and implement custom Java serialization so that the new versions of the
classes can deserialize serialized data from the old version of the classes
with the different field
* Use MessageFactoryKey for parameters everywhere but, in serializable classes
that currently contain MessageFactoryKey, leave that MessageFactoryType as a
field and also add customPayloadSerializerName as a field. This should allow
both the new and old versions of the class to read the same serialized data and
not require a bump of the version uid.
Does that sound right, and if so, any recommendations on how to proceed? Thanks.
> Support ScalaPB as a message payload serializer in Stateful Functions
> ---------------------------------------------------------------------
>
> Key: FLINK-19176
> URL: https://issues.apache.org/jira/browse/FLINK-19176
> Project: Flink
> Issue Type: Improvement
> Components: Stateful Functions
> Affects Versions: 2.0.0
> Reporter: Galen Warren
> Priority: Major
> Fix For: statefun-2.1.0
>
>
> Currently, Stateful Functions supports four options for serialization of
> message payloads:
> * Protobuf (based on code generated for Java)
> * Kryo
> * Multilanguage
> * Raw
> This proposal is to add a fifth option to this list, to support serialization
> of message payloads based on [ScalaPB |[https://scalapb.github.io/docs/]].
> This would allow Scala developers to use ScalaPB-generated classes as message
> types in Stateful Functions directly as message types, without having to
> convert to/from code generated for Java and/or raw byte[] messages.
> This would be a simple implementation, as there is already a
> [MessagePayloadSerializer
> |[https://github.com/apache/flink-statefun/blob/8ffe619a94917d676cf1054c8a0e60de544663db/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/message/MessagePayloadSerializer.java]]
> interface that is implemented for each of the existing serialization
> methods; supporting ScalaPB would require a new class implementing
> MessagePayloadSerializer in terms of ScalaPB-generated code, and a means to
> select it, by adding a new value to the [MessageFactoryType
> |[https://github.com/apache/flink-statefun/blob/8ffe619a94917d676cf1054c8a0e60de544663db/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/message/MessageFactoryType.java]]enumeration.
> Plus testing :)
> I've done this already locally, the implementation is very similar to the
> existing [MessagePayloadSerializerPb
> |[https://github.com/apache/flink-statefun/blob/8ffe619a94917d676cf1054c8a0e60de544663db/statefun-flink/statefun-flink-core/src/main/java/org/apache/flink/statefun/flink/core/message/MessagePayloadSerializerPb.java]]implementation.
> It uses a reference to ScalaPB in "provided" scope.
> Would you be interested in a pull request to add this? Primary benefit is to
> make it easy to use Stateful Functions in a Scala environment. Thanks.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
